API reference

This module takes care of integrating and optimizing over the ability/difficulty space. It includes TrackedResponses, which can store cumulative results during a test.

In case an item bank is enumerable (e.g. GriddedItemBank), then this method integrates over the ability likihood given a set of responses with a given coefficient function using a Riemann sum (aka the rectangle rule).

This method returns a tracked point estimate if it is has the given ability estimator, otherwise it computes it using the given ability estimator.

struct CatLoop
CatLoop(; rules=..., get_response=..., new_response_callback=...)

• rules::Any: An object which implements the CAT engine. Implementations exist for:
  • CatRules
  • Stateful.StatefulCat

• get_response::Any: The function (index, label) -> Int8` which obtains the testee's response for a given question, e.g. by prompting or simulation from data.

• new_response_callback::Any: A callback called each time there is a new responses. If provided, it is passed (responses::TrackedResponses, terminating).

Configuration for a simulatable CAT.

auto_responder(
    responses
) -> ComputerAdaptiveTesting.Sim.var"#11#12"

This function constructs a next item function which automatically responds according to responses.

prompt_response(index_, label) -> Int8

This response callback simply prompts the user for the response using the console

record!(
    recorder::CatRecorder,
    tracked_responses
) -> Vector{Bool}

run_cat(cat_config::CatLoop, item_bank::AbstractItemBank; ib_labels=nothing)

Run a given CatLoop cat_config on the given item_bank. If ib_labels is not given, default labels of the form <<item #$index>> are passed to the callback.

struct FixedLength <: ComputerAdaptiveTesting.TerminationConditions.TerminationCondition

• num_items::Int64

abstract type TerminationCondition <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

This module implements the next item selection rules, which form the main part of CAT.

Bibliography

[1] Linden, W. J., & Pashley, P. J. (2009). Item selection and ability estimation in adaptive testing. In Elements of adaptive testing (pp. 3-30). Springer, New York, NY.

struct AbilityVariance{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, IntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator} <: ComputerAdaptiveTesting.NextItemRules.StateCriterion

• dist_est::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

• integrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

• skip_zero::Bool

This StateCriterion returns the variance of the ability estimate given a set of responses.

abstract type CriterionBase <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract base type all criteria should inherit from

In equation 10 of [1] we see that we can compute information using 2nd derivatives of log likelihood or 1st derivative squared. For single categories, we need to an extra term which disappears when we calculate the total see [2]. For this reason RawEmpiricalInformationPointwiseItemCategoryCriterion computes without this factor, while EmpiricalInformationPointwiseItemCategoryCriterion computes with it.

So in general, only use the former with TotalItemInformation

[1] ``Information Functions of the Generalized Partial Credit Model'' Eiji Muraki https://doi.org/10.1177/014662169301700403

[2] https://mark.reid.name/blog/fisher-information-and-log-likelihood.html

struct ExhaustiveSearch <: ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

struct ExpectationBasedItemCriterion{ResponseExpectationT<:ComputerAdaptiveTesting.NextItemRules.ResponseExpectation, CriterionT<:Union{ComputerAdaptiveTesting.NextItemRules.ItemCategoryCriterion, ComputerAdaptiveTesting.NextItemRules.ItemCriterion, ComputerAdaptiveTesting.NextItemRules.StateCriterion}} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion

• response_expectation::ComputerAdaptiveTesting.NextItemRules.ResponseExpectation

• criterion::Union{ComputerAdaptiveTesting.NextItemRules.ItemCategoryCriterion, ComputerAdaptiveTesting.NextItemRules.ItemCriterion, ComputerAdaptiveTesting.NextItemRules.StateCriterion}

This ItemCriterion wraps a a ResponseExpectation and a StateCriterion or ItemCriterion to look at the criterion's expected value for a particular item 1-ply ahead.

struct FixedRuleSequencer{RulesT} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

• breaks::Tuple{Int64}

• rules::Any

struct GreedyForcedContentBalancer{InnerRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

• targets::Vector{Float64}

• groups::Vector{Int64}

• inner_rule::ComputerAdaptiveTesting.NextItemRules.NextItemRule

This content balancing procedure takes target proportions for each group of items. At each step the group with the lowest ratio of seen items to target is selected.

http://dx.doi.org/10.1207/s15324818ame0403_4

abstract type ItemCategoryCriterion <: ComputerAdaptiveTesting.NextItemRules.SubItemCriterionBase

abstract type ItemCriterion <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

struct ItemCriterionRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

• strategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

• criterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion

ItemCriterionRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.

ItemCriterionRule(bits...; ability_estimator=nothing

Implicit constructor for ItemCriterionRule. Will default to ExhaustiveSearch when no NextItemStrategy is given.

struct MemoryNextItemRule{MemoryT} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

• item_idxs::Any

abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract base type for all item selection rules. All descendants of this type are expected to implement the interface (::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int.

In practice, all adaptive rules in this package use ItemCriterionRule.

NextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemCriterionRule the default instance.

abstract type NextItemStrategy <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract type for next item strategies, tightly coupled with ItemCriterionRule. All descendants of this type are expected to implement the interface (rule::ItemCriterionRule{::NextItemStrategy, ::ItemCriterion})(responses::TrackedResponses, items) where {ItemCriterionT <: }(strategy::NextItemStrategy)(; parallel=true)::NextItemStrategy`

This calculates the pointwise information criterion for an item response model.

struct PointResponseExpectation{PointAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ResponseExpectation

• ability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

This ResponseExpectation gets expected outcomes based on a point ability estimates.

abstract type PointwiseItemCategoryCriterion <: ComputerAdaptiveTesting.NextItemRules.SubItemCriterionBase

abstract type PointwiseItemCriterion <: ComputerAdaptiveTesting.NextItemRules.SubItemCriterionBase

struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

• rng::Random.AbstractRNG: Default: Xoshiro()

This is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.

struct RandomesqueStrategy <: ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

• rng::Random.AbstractRNG

• k::Int64

http://dx.doi.org/10.1207/s15324818ame0204_6

See EmpiricalInformationPointwiseItemCategoryCriterion for more details.

abstract type StateCriterion <: ComputerAdaptiveTesting.NextItemRules.CriterionBase

struct UrryItemCriterion{AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion

• ability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

This item criterion just picks the item with the raw difficulty closest to the current ability estimate.

struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

• next_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.

• termination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.

• ability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.

• ability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()

Configuration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.

This may be more a more convenient layer to integrate than CatLoop if you want to write your own CAT loop rather than using hooks.

CatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)

Explicit constructor for CatRules.

CatRules(bits...)

Implicit constructor for CatRules.