GitHub

Creating a CAT

This guide gives a brief overview of how to create a CAT using the configuration structs in ComputerAdaptiveTesting.jl.

API design

The configuration of a CAT is built up as a tree of configuration structs. These structs are all subtypes of CatConfigBase.

The constructors for the configuration structs in this package tend to have smart defaults. In general most constructors have two forms. The first is an explicit keyword constructor form, where all arguments are given:

ConfigurationObject(
  field1=value1,
  field1=value2,
)

The second is an implicit form, where arguments are given in any order. If possible, they will be used together will appropriately guessed defaults to construct the configuration:

ConfigurationObject(value2, value1)

The implicit form is particularly useful for quick prototyping. Implicit form constructors are also available for some abstract types. In this case, they will return a concrete type that is a reasonable default for the abstract type.

After using the implicit form, you can print the object to see what values have been filled in. This may be useful in case you want to modify some of the defaults or switch to the explicit form.

Item banks

Item banks are the source of items for the test. The basic definitions are provided by FittedItemBanks.jl and can be fit to data using RIrtWrappers.jl. See the documentation pages of those packages for more information.

CatRules

This is the main type for configuring a CAT. It contains the item bank, the next item selection rule, and the stopping rule. CatRules has explicit and implicit constructors.

ComputerAdaptiveTesting.CatConfig.CatRules — Type
struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase
  • next_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.
  • termination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.
  • ability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.
  • ability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()

Configuration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.

This may be more a more convenient layer to integrate than CatLoopConfig if you want to write your own CAT loop rather than using hooks.

CatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)

Explicit constructor for CatRules.

CatRules(bits...)

Implicit constructor for CatRules.

source

Next item selection with NextItemRule

The next item selection rule is the most important part of the CAT. Each rule extends the NextItemRule abstract type.

ComputerAdaptiveTesting.NextItemRules.NextItemRule — Type
abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract base type for all item selection rules. All descendants of this type are expected to implement the interface (::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int.

In practice, all adaptive rules in this package use ItemStrategyNextItemRule.

NextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemStrategyNextItemRule the default instance.

source

A sort of null hypothesis next item selection rule is RandomNextItemRule, which

ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule — Type
struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule
  • rng::Random.AbstractRNG: Default: Xoshiro()

This is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.

source

Other rules are created by combining a ItemCriterion – which somehow rates items according to how good they are – with a NextItemStrategy using an ItemStrategyNextItemRule, which acts as an adapter. The default NextItemStrategy (and currently only) is ExhaustiveSearch. When using the implicit constructors, ItemCriterion can therefore be used directly without wrapping in any place an NextItemRule is expected.

ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule — Type
struct ItemStrategyNextItemRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

  • strategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

  • criterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion

ItemStrategyNextItemRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.

ItemStrategyNextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for ItemStrategyNextItemRule. Will default to ExhaustiveSearch when no NextItemStrategy is given.

source
ComputerAdaptiveTesting.NextItemRules.NextItemStrategy — Type
abstract type NextItemStrategy <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract type for next item strategies, tightly coupled with ItemStrategyNextItemRule. All descendants of this type are expected to implement the interface (rule::ItemStrategyNextItemRule{::NextItemStrategy, ::ItemCriterion})(responses::TrackedResponses, items) where {ItemCriterionT <: }(strategy::NextItemStrategy)(; parallel=true)::NextItemStrategy`

source

Evaluating item and state merit with ItemCriterion and StateCriterion

The ItemCriterion abstract type is used to rate items according to how good they are as a candidate for the next item. A typical example is InformationItemCriterion, which using the current ability estimate $\theta$ and the item response function irf to calculate each item's information $\frac{irf_θ'^2}{irf_θ * (1 - irf_θ)}$.

Within this, you can use ExpectationBasedItemCriterion as an adapter. It takes a ResponseExpectation: either PointResponseExpectation or DistributionResponseExpectation and a a StateCriterion, which evaluates how good a particular state is in terms getting a good estimate of the test takers ability. They look one ply ahead to get the expected value of the $StateCriterion$ after selecting the given item. The AbilityVarianceStateCriterion looks at the variance of the ability $\theta$ estimate at that state.

Stopping rules with TerminationCondition

Currently the only TerminationCondition is FixedItemsTerminationCondition, which ends the test after a fixed number of items.