API reference

This module takes care of integrating and optimizing over the ability/difficulty space. It includes TrackedResponses, which can store cumulative results during a test.

struct LikelihoodAbilityEstimator <: ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

The ability likelihood distribution.

LikelihoodAbilityEstimator()

struct MeanAbilityEstimator{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, IntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator} <: ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

Point ability estimate given by the mean (EAP) of dist_est, computed using integrator.

MeanAbilityEstimator(bits...)

Bag-of-config-bits constructor: uses any given DistributionAbilityEstimator and AbilityIntegrator found in bits, or builds default ones from the rest of bits.

struct ModeAbilityEstimator{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, OptimizerT<:ComputerAdaptiveTesting.Aggregators.AbilityOptimizer} <: ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

Point ability estimate given by the mode of dist_est (e.g. MLE for a LikelihoodAbilityEstimator or MAP for a PosteriorAbilityEstimator), found using optim.

ModeAbilityEstimator(bits...)

Bag-of-config-bits constructor: uses any given DistributionAbilityEstimator and AbilityOptimizer found in bits, or builds default ones from the rest of bits.

struct PosteriorAbilityEstimator{PriorT<:Distributions.Distribution} <: ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

Ability posterior distribution: the response likelihood times a prior distribution over ability (a standard normal by default).

PosteriorAbilityEstimator(; ncomp=0)

Constructs with a standard normal prior (ncomp=0) or a ncomp-dimensional standard multivariate normal prior.

In case an item bank is enumerable (e.g. GriddedItemBank), then this method integrates over the ability likihood given a set of responses with a given coefficient function using a Riemann sum (aka the rectangle rule).

struct TrackedResponses{BareResponsesT<:ComputerAdaptiveTesting.Responses.BareResponses, ItemBankT<:FittedItemBanks.AbstractItemBank, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker}

Responses to items in item_bank (as BareResponses), together with an ability_tracker that maintains an incrementally-updated ability estimate (or distribution) as responses are added. This is the object threaded through a CAT run and passed to next item rules, termination conditions and ability estimators.

This method returns a tracked point estimate if it is has the given ability estimator, otherwise it computes it using the given ability estimator.

Types and functions for recording examinee responses and evaluating the resulting ability likelihood function.

struct AbilityLikelihood{ItemBankT<:FittedItemBanks.AbstractItemBank, BareResponsesT<:ComputerAdaptiveTesting.Responses.BareResponses}

The likelihood of ability θ given responses to items in item_bank, i.e. θ -> prod(P(response | θ) for response in responses). Callable as a function of θ; also has function_xs/function_ys methods for item banks that support evaluation at a fixed grid of xs.

struct BareResponses{ResponseTypeT<:FittedItemBanks.ResponseType, ConcreteResponseTypeT, IndicesVecT<:AbstractVector{Int64}, ValuesVecT<:AbstractArray{ConcreteResponseTypeT, 1}}

A bare (untracked) sequence of responses, stored as parallel vectors of item indices and response values, sharing a common rt (response type). See also TrackedResponses, which additionally tracks the item bank and ability estimate.

struct Response{ResponseTypeT<:FittedItemBanks.ResponseType, ConcreteResponseTypeT}

A single response of the given ResponseType to the item at index.

add_response!(
    responses::ComputerAdaptiveTesting.Responses.BareResponses,
    response::ComputerAdaptiveTesting.Responses.Response
) -> ComputerAdaptiveTesting.Responses.BareResponses

Append response to responses in-place.

function_xs(
    ability_lh::ComputerAdaptiveTesting.Responses.AbilityLikelihood{FittedItemBanks.DichotomousPointsItemBank}
) -> Any

The grid of ability values (xs) at which ability_lh's item bank tabulates response probabilities.

function_ys(
    ability_lh::ComputerAdaptiveTesting.Responses.AbilityLikelihood{FittedItemBanks.DichotomousPointsItemBank}
) -> Any

The likelihood of ability_lh's responses evaluated at each point in function_xs, i.e. the product over responses of the tabulated response probability at each grid point.

pop_response!(
    responses::ComputerAdaptiveTesting.Responses.BareResponses
) -> ComputerAdaptiveTesting.Responses.BareResponses

Remove and discard the last response from responses in-place.

Runs a configured CAT (CatRules) in a loop (CatLoop/run_cat), obtaining responses interactively or from simulated data, and optionally recording the run (see recorder.jl).

struct CatLoop
CatLoop(; rules=..., get_response=..., new_response_callback=...)

• rules::Any: An object which implements the CAT engine. Implementations exist for:
  • CatRules
  • Stateful.StatefulCat

• get_response::Any: The function (index, label) -> Int8` which obtains the testee's response for a given question, e.g. by prompting or simulation from data.

• new_response_callback::Any: A callback called each time there is a new responses. If provided, it is passed (responses::TrackedResponses, terminating).

• init_callback::Any: A callback called each time a CAT is run If provided, it is passed (loop::CatLoop, responses::TrackedResponses).

Configuration for a simulatable CAT.

RecordedCatLoop(;
    rules::CatRules,
    item_bank::AbstractItemBank = nothing,
    responses::Union{Nothing, Vector{ResponseType}} = nothing,
    dims::Union{Nothing, Tuple{Int, Int}} = nothing,
    expected_responses::Int = 0,
    get_response::Function = nothing,
    new_response_callback::Function = nothing,
    new_response_callbacks::Vector{Function} = Any[]
    requests...
)

This RecordedCatLoop is a simplified construction of a [CatRules](@ref)-based [CatLoop](@ref) and [CatRecorder](@ref).

It can be constructed with just some cat rules, an item_bank, and a response memory responses, as well as usually one or more requests for the [CatRecorder](@ref). In this casedimsare provided by theitembank, andexpectedresponsesis set to the length ofresponsesas well as used to provide responses usinggetresponses, otherwise the respective arguments must be provided. The argumentsgetresponse,newresponsecallback, andnewresponsecallbacksare passed to the underlyingCatLoop`.

The resulting RecordedCatLoop can be run directly with run_cat.

auto_responder(
    responses
) -> ComputerAdaptiveTesting.Sim.var"#auto_responder##0#auto_responder##1"

This function constructs a next item function which automatically responds according to responses.

consume!(dict, key) do value
    ...
end

Execute the callback with the value at key in dict if it exists, and remove that key from the dictionary.

prompt_response(index_, label) -> Int8

This response callback simply prompts the user for the response using the console

record!(
    recorder::CatRecorder,
    tracked_responses
) -> Union{Bool, Vector{Bool}}

run_cat(
    loop::ComputerAdaptiveTesting.Sim.RecordedCatLoop,
    item_bank::FittedItemBanks.AbstractItemBank;
    ib_labels
) -> Tuple{Union{ComputerAdaptiveTesting.Responses.BareResponses{FittedItemBanks.BooleanResponse, Bool, Vector{Int64}, Vector{Bool}}, ComputerAdaptiveTesting.Responses.BareResponses{FittedItemBanks.MultinomialResponse, Int64, Vector{Int64}, Vector{Int64}}}, Any}

Run a given RecordedCatLoop by delegating the call to the wrapped CatLoop.

In case item_bank is not provided, the item bank provided during the construction of RecordedCatLoop is used.

run_cat(cat_config::CatLoop, item_bank::AbstractItemBank; ib_labels=nothing)

Run a given CatLoop cat_config on the given item_bank. If ib_labels is not given, default labels of the form <<item #$index>> are passed to the callback.

struct FixedLength <: ComputerAdaptiveTesting.TerminationConditions.TerminationCondition

• num_items::Int64

abstract type TerminationCondition <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

This module implements the next item selection rules, which form the main part of CAT.

Bibliography

[1] Linden, W. J., & Pashley, P. J. (2009). Item selection and ability estimation in adaptive testing. In Elements of adaptive testing (pp. 3-30). Springer, New York, NY.

struct AbilityVariance{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, IntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator} <: ComputerAdaptiveTesting.NextItemRules.StateCriterion

• dist_est::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

• integrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

• skip_zero::Bool

This StateCriterion returns the variance of the ability estimate given a set of responses.

abstract type CriterionBase <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract base type all criteria should inherit from

In equation 10 of [1] we see that we can compute information using 2nd derivatives of log likelihood or 1st derivative squared. For single categories, we need to an extra term which disappears when we calculate the total see [2]. For this reason RawEmpiricalInformationPointwiseItemCategoryCriterion computes without this factor, while EmpiricalInformationPointwiseItemCategoryCriterion computes with it.

So in general, only use the former with TotalItemInformation

[1] ``Information Functions of the Generalized Partial Credit Model'' Eiji Muraki https://doi.org/10.1177/014662169301700403

[2] https://mark.reid.name/blog/fisher-information-and-log-likelihood.html

struct ExhaustiveSearch <: ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

struct ExpectationBasedItemCriterion{ResponseExpectationT<:ComputerAdaptiveTesting.NextItemRules.ResponseExpectation, CriterionT<:Union{ComputerAdaptiveTesting.NextItemRules.ItemCategoryCriterion, ComputerAdaptiveTesting.NextItemRules.ItemCriterion, ComputerAdaptiveTesting.NextItemRules.StateCriterion}} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion

• response_expectation::ComputerAdaptiveTesting.NextItemRules.ResponseExpectation

• criterion::Union{ComputerAdaptiveTesting.NextItemRules.ItemCategoryCriterion, ComputerAdaptiveTesting.NextItemRules.ItemCriterion, ComputerAdaptiveTesting.NextItemRules.StateCriterion}

This ItemCriterion wraps a a ResponseExpectation and a StateCriterion or ItemCriterion to look at the criterion's expected value for a particular item 1-ply ahead.

struct FixedRuleSequencer{RulesT} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

• breaks::Tuple{Int64}

• rules::Any

struct GreedyForcedContentBalancer{InnerRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

• targets::Vector{Float64}

• groups::Vector{Int64}

• inner_rule::ComputerAdaptiveTesting.NextItemRules.NextItemRule

This content balancing procedure takes target proportions for each group of items. At each step the group with the lowest ratio of seen items to target is selected.

http://dx.doi.org/10.1207/s15324818ame0403_4

abstract type ItemCategoryCriterion <: ComputerAdaptiveTesting.NextItemRules.SubItemCriterionBase

abstract type ItemCriterion <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

struct ItemCriterionRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

• strategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

• criterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion

ItemCriterionRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.

ItemCriterionRule(bits...; ability_estimator=nothing

Implicit constructor for ItemCriterionRule. Will default to ExhaustiveSearch when no NextItemStrategy is given.

struct MemoryNextItemRule{MemoryT} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

• item_idxs::Any

abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract base type for all item selection rules. All descendants of this type are expected to implement the interface (::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int.

In practice, all adaptive rules in this package use ItemCriterionRule.

NextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemCriterionRule the default instance.

abstract type NextItemStrategy <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract type for next item strategies, tightly coupled with ItemCriterionRule. All descendants of this type are expected to implement the interface (rule::ItemCriterionRule{::NextItemStrategy, ::ItemCriterion})(responses::TrackedResponses, items) where {ItemCriterionT <: }(strategy::NextItemStrategy)(; parallel=true)::NextItemStrategy`

This calculates the pointwise information criterion for an item response model.

struct PointResponseExpectation{PointAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ResponseExpectation

• ability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

This ResponseExpectation gets expected outcomes based on a point ability estimates.

abstract type PointwiseItemCategoryCriterion <: ComputerAdaptiveTesting.NextItemRules.SubItemCriterionBase

abstract type PointwiseItemCriterion <: ComputerAdaptiveTesting.NextItemRules.SubItemCriterionBase

struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

• rng::Random.AbstractRNG: Default: Xoshiro()

This is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.

struct RandomesqueStrategy <: ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

• rng::Random.AbstractRNG

• k::Int64

http://dx.doi.org/10.1207/s15324818ame0204_6

See EmpiricalInformationPointwiseItemCategoryCriterion for more details.

abstract type StateCriterion <: ComputerAdaptiveTesting.NextItemRules.CriterionBase

struct UrryItemCriterion{AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion

• ability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

This item criterion just picks the item with the raw difficulty closest to the current ability estimate.

struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

• next_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.

• termination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.

• ability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.

• ability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()

Configuration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.

This may be more a more convenient layer to integrate than CatLoop if you want to write your own CAT loop rather than using hooks.

CatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)

Explicit constructor for CatRules.

CatRules(bits...)

Implicit constructor for CatRules.

This module defines the interface for a stateful CAT as well as an implementation in terms of CatRules. The interface is meant to enable polymorphic use of different CAT implementations.

abstract type StatefulCat

Abstract supertype for implementation of the stateful CAT interface.

struct StatefulCatRules{TrackedResponsesT<:ComputerAdaptiveTesting.Aggregators.TrackedResponses} <: ComputerAdaptiveTesting.Stateful.StatefulCat

This is a the StatefulCat implementation in terms of CatRules. It is also the de-facto standard for the behavior of the interface.

julia add_response!(config::StatefulCat, index::IndexT, response::ResponseT)`

The exact response type ResponseT depends on the item bank. It should be chosen to interoperate with any equivalent item bank according to the implementation in ComputerAdaptiveTesting.jl.

get_ability(config::StatefulCat) -> AbilityT

Return the current ability estimate according to the CAT. The type of the ability estimate AbilityT depends on the CAT implementation but should attempt to interoperate with ComputerAdaptiveTesting.jl.

get_responses(config::StatefulCat) -> Tuple{AbstractVector{IndexT}, AbstractVector{ResponseT}}

Returns a tuple of the indices and responses of the items that have been added to the CAT with add_response! so far.

julia item_bank_size(config::StatefulCat)`

Return number of items in the current item bank.

item_criteria(config::StatefulCat) -> AbstractVector{CriteriaT}

Returns a vector of criteria values for each item in the item bank.

The criteria can vary, but should attempt to interoperate with ComputerAdaptiveTesting.jl.

julia item_response_functions(config::StatefulCat, index::IndexT, ability::AbilityT) -> AbstractVector{Float}`

Return the vector of probability of different responses to item at index for someone with a certain ability according to the IRT model backing the CAT.

likelihood(config::StatefulCat, ability::AbilityT) -> Float64

TODO

next_item(config::StatefulCat) -> IndexT

Returns the index of the best next item according to the CAT.

Ideally IndexT will be an integer and the return type a 1-based index, however it should at least be the same type as accepted by add_response!.

ranked_items(config::StatefulCat) -> AbstractVector{IndexT}

Return a vector of indices of the sorted from best to worst item according to the CAT.

reset!(config::StatefulCat)

Reset the CAT to its initial state, removing all responses.

rollback!(config::StatefulCat)

Rollback the last response added with add_response!.

Some CAT implementations may not support this operation in which case they will throw an error.

set_item_bank!(config::StatefulCat, item_bank::AbstractItemBank)

Set the current item bank of the CAT. This will also reset the CAT to its initial state, removing all responses.

Some CAT implementations may not support this operation in which case they will throw an error.

Run a given CatLoop with a MaterializedDecisionTree

CatComparisonConfig(;
    rules::NamedTuple{Symbol, StatefulCat},
    strategy::CatComparisonExecutionStrategy,
    phases::Union{NamedTuple{Symbol, Callable}, Tuple{Symbol}},
    skips::Set{Tuple{Symbol, Symbol}},
    callback::Callable
) -> CatComparisonConfig

CatComparisonConfig sets up a evaluation-oriented comparison between different CAT systems.

Specify the comparison by listing: CAT systems in rules, a NamedTuple which gives identifiers to implementations of the StatefulCat interface; the strategy to use, an implementation of CatComparisonExecutionStrategy; the phases to run listed as either as a NamedTuple with names of phases and corresponding callbacks or nothing a Tuple of phases to run; and a callback which will be used as a fallback in cases where no callback is provided.

The exact phases depend on the strategy used. See their individual documentation for more.

This function compares a given CAT configuration with a CAT using a random next item selection rule.

struct DichotomousPointsWithLogsItemBank{DomainT} <: FittedItemBanks.PointsItemBank

A DichotomousPointsItemBank (inner_bank) with its tabulated response probabilities precomputed in log-space (log_ys), so that likelihoods over many responses can be accumulated by summation instead of repeated multiplication.

struct LogItemBank{ItemBankT<:FittedItemBanks.AbstractItemBank} <: FittedItemBanks.AbstractItemBank

Wraps inner, an item bank whose response function FittedItemBanks.log_resp gives log-probabilities, exposing it as an ordinary AbstractItemBank whose resp/resp_vec return ULogarithmic values. Used where response probabilities are so small that computing in log-space avoids underflow.

abstract type CatConfigBase