GitHub

API reference

Density integration policies

PosteriorAbilityEstimator(prior, LogSpace()) and LikelihoodAbilityEstimator(LogSpace()) select log-density integration when constructing an integrator from a numerical backend. Config bits may be given in either order; omitting the space defaults to LinSpace(). SafeLikelihoodAbilityEstimator(LogSpace(); ncomp=2) applies the same policy to both its likelihood and prior fallback branches.

MeanAbilityEstimator(dist, grid) selects LogGridIntegrator for a log-space estimator and a supported equal-weight grid. With a supported continuous backend, also supply a maximizing optimizer: MeanAbilityEstimator(dist, backend, optimizer). An explicitly constructed AbilityIntegrator overrides the default policy. Both pdf and logpdf remain available in either space; density modes continue to maximize logpdf. Normalized moments remain ordinary numbers.

ComputerAdaptiveTesting.Aggregators.calculation_space — Function
calculation_space(_)

Default integration policy for a distribution estimator. Custom estimators default to LinSpace() and may specialize this method. Both pdf and logpdf retain their usual meaning regardless of this policy.

source

Stateful.logdensity(cat, θ) evaluates the configured estimator's native, unnormalized log density. It includes the prior when the CAT uses a posterior estimator, matching the density returned by the existing Stateful.likelihood. It is available with either integration policy and does not first compute a potentially underflowed probability. Custom estimators need native logpdf.

The creation guide shows propagation through CatRules, optional tracking, and explicit overrides.

Likelihood-weighted criteria

NextItemRules.LikelihoodWeightedItemCriterion and NextItemRules.LikelihoodWeightedItemCategoryCriterion integrate their pointwise criterion against an unnormalized ability density. Their ability integrator applies the response likelihood and, when using a posterior estimator, the prior once. The pointwise criterion supplies only its value at the ability being integrated.

Log-density interface

Responses.AbilityLogLikelihood wraps an AbilityLikelihood, or can be constructed directly from an item bank and BareResponses, or from Aggregators.TrackedResponses. Calling it sums the item bank's native FittedItemBanks.log_resp values. It does not first multiply probabilities.

Distributions.logpdf(est, tracked_responses) returns a callable log density; Distributions.logpdf(est, tracked_responses, θ) evaluates it at θ. Likelihood, posterior and guarded distribution estimators support this interface. Posterior log densities add Distributions.logpdf(prior, θ) directly. Like the existing pdf interface, these densities are unnormalized.

julia> using ComputerAdaptiveTesting.Responses, ComputerAdaptiveTesting.Aggregators,
           FittedItemBanks, Distributions

julia> bank = ItemBank2PL([0.0, 0.0], [1.0, 1.0]);

julia> history = BareResponses(ResponseType(bank), [1, 2], [false, true]);

julia> tracked = TrackedResponses(history, bank);

julia> AbilityLogLikelihood(tracked)(0.0) ≈ log(0.25)
true

julia> est = PosteriorAbilityEstimator(Normal());

julia> logpdf(est, tracked)(0.0) ≈ logpdf(est, tracked, 0.0) ≈ log(0.25) + logpdf(Normal(), 0.0)
true

For tabulated dichotomous banks, Responses.function_log_ys(likelihood) returns log likelihoods on Responses.function_xs(likelihood). Equivalently, Responses.function_ys(AbilityLogLikelihood(likelihood)) returns these log values. An empty response history returns zeros; impossible responses produce -Inf. Use FittedItemBanks.DichotomousPointsWithLogsItemBank to reuse the item log cache across calls. Constructing logs from a table cannot recover probabilities that were already rounded to zero or one.

Aggregators.ModeAbilityEstimator with Aggregators.FunctionOptimizer maximizes this log density directly for MLE/MAP. This also applies to the explicit optimizer(IntegralCoeffs.one, distribution_estimator, tracked_responses) call. General coefficient-weighted objectives, including signed objectives, and the two-argument optimizer(coefficient, density_function) interface retain their probability-space product semantics. Custom AbilityOptimizers control their own objective evaluation.

DerivedMeasures.LaplaceApproxEstimator evaluates log-density curvature directly at the mode. Its result remains (mode, negative_second_derivative): the second value is precision, not standard deviation. Custom distribution estimators used for log-density optimization implement the two-argument logpdf method; there is no automatic fallback through log(pdf(...)).

Fixed-grid log normalization and tracking

Aggregators.LogGridIntegrator opts into stable fixed-grid inference. Wrap a PsychometricsBazaarBase.Integrators.FixedGridIntegrator, its preallocated form, or an IterativeFixedGridIntegrator. These are equal-weight sums, without a grid-spacing factor; normalized expectations are unaffected by that common factor. Adaptive quadrature and unequal quadrature weights are not supported by this wrapper.

The wrapper evaluates the estimator's native logpdf on the grid, subtracts the largest log density, exponentiates, and normalizes the resulting weights. Means, variances, covariance matrices and response predictions reuse that normalization. Signed moments are calculated with ordinary numbers. It returns ordinary values for expectations, but logarithmic numbers for raw integrals and normdenom, preserving their unnormalized scale. Float64(mass) may underflow; log(mass) preserves the log mass. An explicit denominator supplied to expectation is still an unnormalized denominator, not a log denominator.

Use LogGridAbilityTracker to cache the grid log densities, normalized weights, and normalization scale across calculations. Both new constructors accept config bits in any order. Attach the tracker to TrackedResponses, or let CatRules collect it from the integrator embedded in the ability estimator.

julia> using ComputerAdaptiveTesting.Aggregators, ComputerAdaptiveTesting.Responses,
           FittedItemBanks, Distributions, PsychometricsBazaarBase.Integrators

julia> bank = ItemBank2PL([0.0], [1.0]);

julia> dist = PosteriorAbilityEstimator(Normal());

julia> grid = FixedGridIntegrator(collect(-6.0:0.05:6.0));

julia> tracker = LogGridAbilityTracker(dist, grid);

julia> integral = LogGridIntegrator(tracker);

julia> history = BareResponses(ResponseType(bank), fill(1, 2000),
                              repeat([false, true], 1000));

julia> tracked = TrackedResponses(history, bank, tracker); track!(tracked);

julia> abs(MeanAbilityEstimator(dist, integral)(tracked)) < 1e-10
true

julia> isfinite(variance(integral, dist, tracked))
true

julia> sum(response_expectation(dist, integral, tracked, 1)) ≈ 1
true

LogGridIntegrator(grid) works without a tracker, recomputing the weights on each calculation. Before a tracker's first track!, it also computes temporary weights. Adding, popping or clearing tracked responses refreshes the tracker. Integrating a different history (including a speculative response), bank or estimator computes temporary weights without modifying the live cache. The cache includes a snapshot of response indices and values, so bare-history edits cannot reuse stale weights. Treat grid coordinates, item parameters and prior parameters as fixed; explicitly refresh with track! after changing them. Parallel reads are supported; concurrent mutation of the tracker is not.

Predictions evaluate every response category directly, including nominal categories, avoiding cancellation from 1 - P(true). Tabulated dichotomous banks use their log-likelihood arrays and require an exactly matching grid. No probability-space fallback is supplied for custom estimators lacking logpdf. An empty grid, a grid with zero density everywhere, or NaN/+Inf log densities raise errors. A grid with insufficient resolution can still miss the posterior peak: stable normalization does not correct quadrature error.

Continuous log-density integration

Aggregators.LogFunctionIntegrator(backend, optimizer) opts into stable continuous integration. Both arguments are config bits and may be supplied in either order. It reuses PsychometricsBazaarBase's quadrature and maximization implementations; it does not change the existing FunctionIntegrator default. The estimator must implement native Distributions.logpdf.

julia> using ComputerAdaptiveTesting.Aggregators, ComputerAdaptiveTesting.Responses,
           FittedItemBanks, Distributions, PsychometricsBazaarBase.Integrators

julia> using PsychometricsBazaarBase.Optimizers: NativeOneDimOptimOptimizer

julia> bank = ItemBank2PL([0.0], [1.0]);

julia> tracked = TrackedResponses(BareResponses(ResponseType(bank)), bank);

julia> backend = QuadGKIntegrator(; lo=-8.0, hi=8.0, rtol=1e-8);

julia> optimizer = NativeOneDimOptimOptimizer(; lo=-8.0, hi=8.0);

julia> integral = LogFunctionIntegrator(backend, optimizer);

julia> dist = PosteriorAbilityEstimator(Normal(-0.7, 1.0));

julia> abs(MeanAbilityEstimator(dist, integral)(tracked) + 0.7) < 1e-8
true

julia> abs(variance(integral, dist, tracked) - 1) < 1e-8
true

Supported backends are QuadGKIntegrator, FixedGKIntegrator, MultiDimFixedGKIntegrator, HCubatureIntegrator and CubatureIntegrator. Their domain, tolerances and scalar/vector/matrix output restrictions still apply. For example, use HCubature for vector and matrix moments. The optimizer must maximize its input function and search a region containing a representative density peak inside the integration domain. It may use a finite search interval when the quadrature domain is infinite.

Each expectation prepares one log density and chooses a reference value c using the optimizer. Both numerator and denominator integrate against exp(log_density(x) - c) using ordinary arithmetic, then divide before restoring any absolute scale. Signed moments stay ordinary numbers. A separate calculation prepares a fresh reference, so response-history edits and speculative histories cannot reuse stale density state. Predictions integrate every response category directly rather than subtracting a probability from one.

Raw integration returns the backend's result type with its value and error multiplied by exp(c) using logarithmic numbers. normdenom remains the absolute, unnormalized mass. Inspect it with log(mass); converting it to Float64 can underflow or overflow. An explicit denominator supplied to expectation is an absolute mass, not a log mass or a rescaled mass.

Use IntPassthrough() (from PsychometricsBazaarBase.Integrators) with expectation to inspect its ordinary-valued result and error via intval and interr, or IntMeasurement() for a normalized measurement. When both integrals provide errors, the ratio estimate includes both numerator and normalization error. It treats an explicitly supplied scalar denominator as fixed. These estimates depend on the backend's error estimates; they are not certified bounds. Backends without error estimates retain BareIntegrationResult semantics.

A nonfinite reference, invalid log density, overflow after rescaling, or zero or nonfinite normalization mass raises DomainError. Normalization also rejects a mass error estimate as large as the mass itself. Coefficients must be defined at the reference point; zero-density integration points skip their evaluation. Choosing a log scale does not ensure the optimizer or quadrature finds every peak, resolve an inadequate integration range, or make an improper density integrable. Adjust the domain, optimizer or quadrature when needed.

ComputerAdaptiveTesting.DerivedMeasures.LaplaceApproxEstimator — Type
struct LaplaceApproxEstimator{DistEstT<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, OptimizerT<:ComputerAdaptiveTesting.Aggregators.AbilityOptimizer} <: ComputerAdaptiveTesting.DerivedMeasures.PointAndSpreadEstimator

Estimate a scalar ability density's mode and its log-density curvature. Returns (mode, -d²logpdf/dθ²) at the mode. The second value is curvature (precision), not variance or standard deviation. Evaluates the log density directly, including when the probability-space density underflows.

Construct from a ModeAbilityEstimator, or from a distribution ability estimator and an ability optimizer.

source
ComputerAdaptiveTesting.Aggregators — Module

This module takes care of integrating and optimizing over the ability/difficulty space. It includes TrackedResponses, which can store cumulative results during a test.

source
ComputerAdaptiveTesting.Aggregators.LikelihoodAbilityEstimator — Type
struct LikelihoodAbilityEstimator{SpaceT<:ComputerAdaptiveTesting.Aggregators.CalculationSpace} <: ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

The ability likelihood distribution. Optional LinSpace() (default) or LogSpace() config bits select the default integration policy.

LikelihoodAbilityEstimator(bits...)
source
ComputerAdaptiveTesting.Aggregators.LinSpace — Type
struct LinSpace <: ComputerAdaptiveTesting.Aggregators.CalculationSpace

Construct ordinary density integrators (the default). This does not change the meaning of pdf/logpdf, or log-objective optimization of density modes.

source
ComputerAdaptiveTesting.Aggregators.LogFunctionIntegrator — Type
struct LogFunctionIntegrator{I<:Union{PsychometricsBazaarBase.Integrators.FixedGKIntegrator, PsychometricsBazaarBase.Integrators.CubatureIntegrator, PsychometricsBazaarBase.Integrators.HCubatureIntegrator, PsychometricsBazaarBase.Integrators.MultiDimFixedGKIntegrator, PsychometricsBazaarBase.Integrators.QuadGKIntegrator}, O<:PsychometricsBazaarBase.Optimizers.Optimizer} <: ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

Continuous integration of native log ability densities. Construct with LogFunctionIntegrator(backend, optimizer) (config bits in either order), where backend is a QuadGKIntegrator, FixedGKIntegrator, MultiDimFixedGKIntegrator, HCubatureIntegrator or CubatureIntegrator, and optimizer is a PsychometricsBazaarBase Optimizer that maximizes a function. The backend retains its integration domain, tolerances and output-shape limits. Choose an optimizer search domain containing a representative density peak inside the integration domain.

For each calculation, maximize the estimator's logpdf to choose a finite reference log density c, then integrate f(x) * exp(logpdf(x) - c). An expectation uses the same reference for its numerator and denominator and cancels the scale before conversion. Raw integrals and normdenom retain their absolute scale as logarithmic numbers, including backend error estimates. An explicit expectation denominator is an absolute mass, not its logarithm.

IntValue() returns ordinary normalized moments. IntPassthrough() also retains the quadrature error estimate; for ratios this uses (numerator_error + abs(value) * mass_error) / (mass - mass_error) when both integrals supply errors. This is conditional on the backend error estimates, not a rigorous guarantee. A backend without errors yields a bare result.

Nonfinite reference densities, invalid log-density evaluations, overflow of the rescaled density, and zero/nonfinite normalization mass raise DomainError. Scaling cannot repair missed peaks, an unsuitable integration domain or a divergent integral. No density or response-history state is cached. Coefficients must be defined at the reference point; zero-density quadrature points skip coefficient evaluation, including for vector and matrix moments.

source
ComputerAdaptiveTesting.Aggregators.LogGridAbilityTracker — Type
mutable struct LogGridAbilityTracker{EstimatorT<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, IntegratorT<:Union{PsychometricsBazaarBase.Integrators.FixedGridIntegrator, PsychometricsBazaarBase.Integrators.IterativeFixedGridIntegrator, PsychometricsBazaarBase.Integrators.PreallocatedFixedGridIntegrator}} <: ComputerAdaptiveTesting.Aggregators.AbilityTracker

Cache log densities, normalized weights and their normalization scale on an equal-weight fixed grid. Construct with LogGridAbilityTracker(estimator, grid) (config bits may be supplied in either order), then use LogGridIntegrator(tracker) for means, variances, covariance and predictions.

track!(responses, tracker) evaluates logpdf(estimator, responses) on the grid. tracker.cache.density contains log_values, normalized weights, log_scale and scaled_sum; the unnormalized grid mass is exp(log_scale) * scaled_sum. An all-zero density raises DomainError rather than manufacturing a posterior.

The cache records a copy of the response history. Integrations for different histories (including speculation), banks or estimators use temporary weights without modifying this cache. An uninitialized tracker also works this way. Treat the grid, bank parameters and estimator as fixed; call track! after changing their contents. Concurrent integrations may read a tracker, but must not run concurrently with track! on that same tracker.

source
ComputerAdaptiveTesting.Aggregators.LogGridIntegrator — Type
struct LogGridIntegrator{IntegratorT<:Union{PsychometricsBazaarBase.Integrators.FixedGridIntegrator, PsychometricsBazaarBase.Integrators.IterativeFixedGridIntegrator, PsychometricsBazaarBase.Integrators.PreallocatedFixedGridIntegrator}, TrackerT<:Union{Nothing, ComputerAdaptiveTesting.Aggregators.LogGridAbilityTracker}} <: ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

Integrate ability densities using normalized log weights on an equal-weight fixed grid. Construct with LogGridIntegrator(grid) or LogGridIntegrator(tracker::LogGridAbilityTracker). Accepts FixedGridIntegrator, its preallocated form, and IterativeFixedGridIntegrator. Like those integrators, raw integrals are sums without a grid-spacing factor.

Expectations return ordinary scalars, vectors or matrices. Raw integration and normdenom preserve the absolute scale using logarithmic numbers: use log to inspect tiny masses, and explicitly convert to floating point only if desired. An explicit denominator passed to expectation retains its usual meaning.

Requires the estimator's native logpdf interface. Tabulated dichotomous banks are supported when their grid exactly matches the integration grid. The grid must be nonempty and have nonzero density somewhere. NaN and +Inf log densities are rejected. No adaptive-quadrature error estimate is provided.

source
ComputerAdaptiveTesting.Aggregators.LogSpace — Type
struct LogSpace <: ComputerAdaptiveTesting.Aggregators.CalculationSpace

Construct integrators using native log densities and stable normalization. Equal-weight grids use LogGridIntegrator; supported continuous backends use LogFunctionIntegrator and require an optimizer for choosing the log scale. Estimates and normalized moments remain ordinary numbers.

source
ComputerAdaptiveTesting.Aggregators.MeanAbilityEstimator — Type
struct MeanAbilityEstimator{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, IntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator} <: ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

Point ability estimate given by the mean (EAP) of dist_est, computed using integrator.

MeanAbilityEstimator(bits...)

Bag-of-config-bits constructor: uses any given DistributionAbilityEstimator and AbilityIntegrator found in bits, or adapts a numerical backend using the distribution's calculation_space. Explicit ability integrators override this default. Log-space continuous integration also needs a maximizing optimizer config bit; grid tracking can be requested with GriddedAbilityTracker.

source
ComputerAdaptiveTesting.Aggregators.ModeAbilityEstimator — Type
struct ModeAbilityEstimator{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, OptimizerT<:ComputerAdaptiveTesting.Aggregators.AbilityOptimizer} <: ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

Point ability estimate given by the mode of dist_est (e.g. MLE for a LikelihoodAbilityEstimator or MAP for a PosteriorAbilityEstimator), found using optim.

With FunctionOptimizer, maximizes logpdf(dist_est, tracked_responses) directly to avoid likelihood underflow. Custom distribution estimators must implement the two-argument logpdf interface; custom AbilityOptimizers control their own objective evaluation.

ModeAbilityEstimator(bits...)

Bag-of-config-bits constructor: uses any given DistributionAbilityEstimator and AbilityOptimizer found in bits, or builds default ones from the rest of bits.

source
ComputerAdaptiveTesting.Aggregators.PosteriorAbilityEstimator — Type
struct PosteriorAbilityEstimator{PriorT<:Distributions.Distribution, SpaceT<:ComputerAdaptiveTesting.Aggregators.CalculationSpace} <: ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

Ability posterior distribution: the response likelihood times a prior distribution over ability (a standard normal by default).

PosteriorAbilityEstimator(bits...; ncomp=0)

Accepts a prior distribution and a LinSpace() (default) or LogSpace() policy in either order. Without a prior, constructs with a standard normal (ncomp=0) or a ncomp-dimensional standard multivariate normal prior. The policy affects automatic integrator construction, not the meanings of pdf and logpdf.

source
ComputerAdaptiveTesting.Aggregators.TrackedResponses — Type
struct TrackedResponses{BareResponsesT<:ComputerAdaptiveTesting.Responses.BareResponses, ItemBankT<:FittedItemBanks.AbstractItemBank, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker}

Responses to items in item_bank (as BareResponses), together with an ability_tracker that maintains an incrementally-updated ability estimate (or distribution) as responses are added. This is the object threaded through a CAT run and passed to next item rules, termination conditions and ability estimators.

source
Distributions.logpdf — Method
logpdf(
    ability_est::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator,
    tracked_responses::ComputerAdaptiveTesting.Aggregators.TrackedResponses,
    x
) -> Any

Evaluate the unnormalized log density of a distribution ability estimator at x. The two-argument form, logpdf(est, tracked_responses), returns a callable log-density function instead.

For LikelihoodAbilityEstimator, this is the sum of item log probabilities; for PosteriorAbilityEstimator, it additionally includes logpdf(prior, x). No normalizing constant is subtracted, matching the existing pdf convention. GuardedAbilityEstimator selects the same branch as pdf when the callable is constructed. Custom distribution estimators implement the two-argument form; there is deliberately no fallback through log(pdf(...)).

source
ComputerAdaptiveTesting.Responses.AbilityLikelihood — Type
struct AbilityLikelihood{ItemBankT<:FittedItemBanks.AbstractItemBank, BareResponsesT<:ComputerAdaptiveTesting.Responses.BareResponses}

The likelihood of ability θ given responses to items in item_bank, i.e. θ -> prod(P(response | θ) for response in responses). Callable as a function of θ; also has function_xs/function_ys methods for item banks that support evaluation at a fixed grid of xs. See AbilityLogLikelihood for evaluation without multiplying probabilities.

source
ComputerAdaptiveTesting.Responses.AbilityLogLikelihood — Type
struct AbilityLogLikelihood{LikelihoodT<:ComputerAdaptiveTesting.Responses.AbilityLikelihood}

The log likelihood of ability given a response history. Construct with AbilityLogLikelihood(item_bank, responses), AbilityLogLikelihood(likelihood), or AbilityLogLikelihood(tracked_responses).

Callable as log_likelihood(θ): sums FittedItemBanks.log_resp for the observed responses, rather than taking the logarithm of a probability product. An empty history has log likelihood zero; an impossible response has log likelihood -Inf. Pointwise evaluation requires the item bank to implement log_resp.

For supported tabulated banks, function_xs returns the grid and function_ys returns log likelihoods. The wrapped response history is shared, not copied, so later changes to it affect subsequent evaluations.

source
ComputerAdaptiveTesting.Responses.BareResponses — Type
struct BareResponses{ResponseTypeT<:FittedItemBanks.ResponseType, ConcreteResponseTypeT, IndicesVecT<:AbstractVector{Int64}, ValuesVecT<:AbstractArray{ConcreteResponseTypeT, 1}}

A bare (untracked) sequence of responses, stored as parallel vectors of item indices and response values, sharing a common rt (response type). See also TrackedResponses, which additionally tracks the item bank and ability estimate.

source
ComputerAdaptiveTesting.Responses.add_response! — Method
add_response!(
    responses::ComputerAdaptiveTesting.Responses.BareResponses,
    response::ComputerAdaptiveTesting.Responses.Response
) -> ComputerAdaptiveTesting.Responses.BareResponses

Append response to responses in-place.

source
ComputerAdaptiveTesting.Responses.function_log_ys — Method
function_log_ys(
    ability_lh::ComputerAdaptiveTesting.Responses.AbilityLikelihood{<:FittedItemBanks.DichotomousPointsWithLogsItemBank}
) -> Any

Return the response log likelihood at each point in function_xs, without exponentiating the accumulated log probabilities. Supports DichotomousPointsItemBank and DichotomousPointsWithLogsItemBank.

For repeated evaluations, use FittedItemBanks.DichotomousPointsWithLogsItemBank to reuse its item log-probability cache. An ordinary points bank constructs that cache on each call. These logs retain the precision of the tabulated probabilities; they cannot recover probabilities already rounded to zero or one. An empty history returns zeros and impossible responses yield -Inf.

source
ComputerAdaptiveTesting.Responses.function_xs — Method
function_xs(
    ability_lh::ComputerAdaptiveTesting.Responses.AbilityLikelihood{<:Union{FittedItemBanks.DichotomousPointsItemBank, FittedItemBanks.DichotomousPointsWithLogsItemBank}}
) -> Any

The grid of ability values (xs) at which ability_lh's item bank tabulates response probabilities.

source
ComputerAdaptiveTesting.Responses.function_ys — Method
function_ys(
    ability_lh::ComputerAdaptiveTesting.Responses.AbilityLikelihood{<:Union{FittedItemBanks.DichotomousPointsItemBank, FittedItemBanks.DichotomousPointsWithLogsItemBank}}
) -> Any

The likelihood of ability_lh's responses evaluated at each point in function_xs, i.e. the product over responses of the tabulated response probability at each grid point.

For an AbilityLogLikelihood, returns log likelihoods instead.

source
ComputerAdaptiveTesting.Sim.CatLoop — Type
struct CatLoop
CatLoop(; rules=..., get_response=..., new_response_callback=...)
  • get_response::Any: The function (index, label) -> Int8` which obtains the testee's response for a given question, e.g. by prompting or simulation from data.
  • new_response_callback::Any: A callback called each time there is a new responses. If provided, it is passed (responses::TrackedResponses, terminating).
  • init_callback::Any: A callback called each time a CAT is run If provided, it is passed (loop::CatLoop, responses::TrackedResponses).

Configuration for a simulatable CAT.

source
ComputerAdaptiveTesting.Sim.RecordedCatLoop — Method
RecordedCatLoop(;
    rules::CatRules,
    item_bank::AbstractItemBank = nothing,
    responses::Union{Nothing, Vector{ResponseType}} = nothing,
    dims::Union{Nothing, Tuple{Int, Int}} = nothing,
    expected_responses::Int = 0,
    get_response::Function = nothing,
    new_response_callback::Function = nothing,
    new_response_callbacks::Vector{Function} = Any[]
    requests...
)

This RecordedCatLoop is a simplified construction of a CatRules-based CatLoop and CatRecorder.

It can be constructed with just some cat rules, an item_bank, and a response memory responses, as well as usually one or more requests for the CatRecorder. In this case dims are provided by the item_bank, and expected_responses is set to the length of responses as well as used to provide responses using get_responses, otherwise the respective arguments must be provided. The arguments get_response, new_response_callback, and new_response_callbacks are passed to the underlying CatLoop.

The resulting RecordedCatLoop can be run directly with run_cat.

source
ComputerAdaptiveTesting.Sim.auto_responder — Method
auto_responder(
    responses
) -> ComputerAdaptiveTesting.Sim.var"#auto_responder##0#auto_responder##1"

This function constructs a next item function which automatically responds according to responses.

source
ComputerAdaptiveTesting.Sim.run_cat — Method
run_cat(
    loop::ComputerAdaptiveTesting.Sim.RecordedCatLoop,
    item_bank::FittedItemBanks.AbstractItemBank;
    ib_labels
) -> Tuple{Union{ComputerAdaptiveTesting.Responses.BareResponses{FittedItemBanks.BooleanResponse, Bool, Vector{Int64}, Vector{Bool}}, ComputerAdaptiveTesting.Responses.BareResponses{FittedItemBanks.MultinomialResponse, Int64, Vector{Int64}, Vector{Int64}}}, Any}

Run a given RecordedCatLoop by delegating the call to the wrapped CatLoop.

In case item_bank is not provided, the item bank provided during the construction of RecordedCatLoop is used.

source
ComputerAdaptiveTesting.Sim.run_cat — Method
run_cat(cat_config::CatLoop, item_bank::AbstractItemBank; ib_labels=nothing)

Run a given CatLoop cat_config on the given item_bank. If ib_labels is not given, default labels of the form <<item #$index>> are passed to the callback.

source
ComputerAdaptiveTesting.TerminationConditions.LengthBoundedTermination — Type
struct LengthBoundedTermination{InnerT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition} <: ComputerAdaptiveTesting.TerminationConditions.TerminationCondition

  • min_length::Int64

  • max_length::Int64

  • termination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition

Wraps another termination condition so that the test always administers at least min_length items and never more than max_length items.

source
ComputerAdaptiveTesting.TerminationConditions.StateCriterionThresholdTermination — Type
struct StateCriterionThresholdTermination{InnerT<:ComputerAdaptiveTesting.NextItemRules.StateCriterion} <: ComputerAdaptiveTesting.TerminationConditions.TerminationCondition

  • threshold::Float64

  • criterion::ComputerAdaptiveTesting.NextItemRules.StateCriterion

Terminates the test once a StateCriterion reaches threshold. When the criterion is one which should be minimised, the test terminates once it drops to or below the threshold, otherwise once it reaches or exceeds it.

source
ComputerAdaptiveTesting.NextItemRules — Module

This module implements the next item selection rules, which form the main part of CAT.

Bibliography

[1] Linden, W. J., & Pashley, P. J. (2009). Item selection and ability estimation in adaptive testing. In Elements of adaptive testing (pp. 3-30). Springer, New York, NY.

source
ComputerAdaptiveTesting.NextItemRules.AbilityVariance — Type
struct AbilityVariance{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, IntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator} <: ComputerAdaptiveTesting.NextItemRules.StateCriterion

  • dist_est::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

  • integrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

  • skip_zero::Bool

This StateCriterion returns the variance of the ability estimate given a set of responses.

source
ComputerAdaptiveTesting.NextItemRules.EmpiricalInformationPointwiseItemCategoryCriterion — Type

In equation 10 of [1] we see that we can compute information using 2nd derivatives of log likelihood or 1st derivative squared. For single categories, we need to an extra term which disappears when we calculate the total see [2]. For this reason RawEmpiricalInformationPointwiseItemCategoryCriterion computes without this factor, while EmpiricalInformationPointwiseItemCategoryCriterion computes with it.

So in general, only use the former with TotalItemInformation

[1] ``Information Functions of the Generalized Partial Credit Model'' Eiji Muraki https://doi.org/10.1177/014662169301700403

[2] https://mark.reid.name/blog/fisher-information-and-log-likelihood.html

source
ComputerAdaptiveTesting.NextItemRules.ExpectationBasedItemCriterion — Type
struct ExpectationBasedItemCriterion{ResponseExpectationT<:ComputerAdaptiveTesting.NextItemRules.ResponseExpectation, CriterionT<:Union{ComputerAdaptiveTesting.NextItemRules.ItemCategoryCriterion, ComputerAdaptiveTesting.NextItemRules.ItemCriterion, ComputerAdaptiveTesting.NextItemRules.StateCriterion}} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion

  • response_expectation::ComputerAdaptiveTesting.NextItemRules.ResponseExpectation

  • criterion::Union{ComputerAdaptiveTesting.NextItemRules.ItemCategoryCriterion, ComputerAdaptiveTesting.NextItemRules.ItemCriterion, ComputerAdaptiveTesting.NextItemRules.StateCriterion}

This ItemCriterion wraps a a ResponseExpectation and a StateCriterion or ItemCriterion to look at the criterion's expected value for a particular item 1-ply ahead.

source
ComputerAdaptiveTesting.NextItemRules.GreedyForcedContentBalancer — Type
struct GreedyForcedContentBalancer{InnerRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

  • targets::Vector{Float64}

  • groups::Vector{Int64}

  • inner_rule::ComputerAdaptiveTesting.NextItemRules.NextItemRule

This content balancing procedure takes target proportions for each group of items. At each step the group with the lowest ratio of seen items to target is selected.

http://dx.doi.org/10.1207/s15324818ame0403_4

source
ComputerAdaptiveTesting.NextItemRules.ItemCriterionRule — Type
struct ItemCriterionRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

  • strategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

  • criterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion

ItemCriterionRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.

ItemCriterionRule(bits...; ability_estimator=nothing

Implicit constructor for ItemCriterionRule. Will default to ExhaustiveSearch when no NextItemStrategy is given.

source
ComputerAdaptiveTesting.NextItemRules.LikelihoodWeightedItemCategoryCriterion — Type
struct LikelihoodWeightedItemCategoryCriterion{PointwiseItemCategoryCriterionT<:ComputerAdaptiveTesting.NextItemRules.PointwiseItemCategoryCriterion, AbilityIntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ItemCategoryCriterion

  • criterion::ComputerAdaptiveTesting.NextItemRules.PointwiseItemCategoryCriterion

  • integrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

  • estimator::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

Integrate a pointwise item-category criterion against the unnormalized ability density. The integrator applies the response likelihood and, for a posterior estimator, its prior exactly once.

source
ComputerAdaptiveTesting.NextItemRules.LikelihoodWeightedItemCriterion — Type
struct LikelihoodWeightedItemCriterion{PointwiseItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.PointwiseItemCriterion, AbilityIntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion

  • criterion::ComputerAdaptiveTesting.NextItemRules.PointwiseItemCriterion

  • integrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

  • estimator::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

Integrate a pointwise item criterion against the unnormalized ability density. The integrator applies the response likelihood and, for a posterior estimator, its prior exactly once.

source
ComputerAdaptiveTesting.NextItemRules.NextItemRule — Type
abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract base type for all item selection rules. All descendants of this type are expected to implement the interface (::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int.

In practice, all adaptive rules in this package use ItemCriterionRule.

NextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemCriterionRule the default instance.

source
ComputerAdaptiveTesting.NextItemRules.NextItemStrategy — Type
abstract type NextItemStrategy <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract type for next item strategies, tightly coupled with ItemCriterionRule. All descendants of this type are expected to implement the interface (rule::ItemCriterionRule{::NextItemStrategy, ::ItemCriterion})(responses::TrackedResponses, items) where {ItemCriterionT <: }(strategy::NextItemStrategy)(; parallel=true)::NextItemStrategy`

source
ComputerAdaptiveTesting.NextItemRules.PointResponseExpectation — Type
struct PointResponseExpectation{PointAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ResponseExpectation
  • ability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

This ResponseExpectation gets expected outcomes based on a point ability estimates.

source
ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule — Type
struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule
  • rng::Random.AbstractRNG: Default: Xoshiro()

This is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.

source
ComputerAdaptiveTesting.NextItemRules.UrryItemCriterion — Type
struct UrryItemCriterion{AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion
  • ability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

This item criterion just picks the item with the raw difficulty closest to the current ability estimate.

source
ComputerAdaptiveTesting.Rules.CatRules — Type
struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase
  • next_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.
  • termination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.
  • ability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.
  • ability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()

Configuration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.

This may be more a more convenient layer to integrate than CatLoop if you want to write your own CAT loop rather than using hooks.

CatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)

Explicit constructor for CatRules.

CatRules(bits...)

Implicit constructor for CatRules. Supply a point estimator instance, or a point estimator type (such as MeanAbilityEstimator) together with its distribution and numerical backend. Constructors for item/state criteria inherit the resolved estimator and compatible integration adapter. Explicitly supplied components retain their own configuration. Shared trackers from estimation, selection and stopping are registered once.

source
ComputerAdaptiveTesting.Stateful — Module

This module defines the interface for a stateful CAT as well as an implementation in terms of CatRules. The interface is meant to enable polymorphic use of different CAT implementations.

source
ComputerAdaptiveTesting.Stateful.StatefulCatRules — Type
struct StatefulCatRules{TrackedResponsesT<:ComputerAdaptiveTesting.Aggregators.TrackedResponses} <: ComputerAdaptiveTesting.Stateful.StatefulCat

This is a the StatefulCat implementation in terms of CatRules. It is also the de-facto standard for the behavior of the interface.

source
ComputerAdaptiveTesting.Stateful.add_response! — Function

julia add_response!(config::StatefulCat, index::IndexT, response::ResponseT)`

The exact response type ResponseT depends on the item bank. It should be chosen to interoperate with any equivalent item bank according to the implementation in ComputerAdaptiveTesting.jl.

source
ComputerAdaptiveTesting.Stateful.get_ability — Function
get_ability(config::StatefulCat) -> AbilityT

Return the current ability estimate according to the CAT. The type of the ability estimate AbilityT depends on the CAT implementation but should attempt to interoperate with ComputerAdaptiveTesting.jl.

source
ComputerAdaptiveTesting.Stateful.item_criteria — Function
item_criteria(config::StatefulCat) -> AbstractVector{CriteriaT}

Returns a vector of criteria values for each item in the item bank.

The criteria can vary, but should attempt to interoperate with ComputerAdaptiveTesting.jl.

source
ComputerAdaptiveTesting.Stateful.item_response_functions — Function

julia item_response_functions(config::StatefulCat, index::IndexT, ability::AbilityT) -> AbstractVector{Float}`

Return the vector of probability of different responses to item at index for someone with a certain ability according to the IRT model backing the CAT.

source
ComputerAdaptiveTesting.Stateful.likelihood — Function
likelihood(config::StatefulCat, ability::AbilityT) -> Float64

Evaluate the configured ability distribution's unnormalized density. For a posterior estimator this includes the prior. See logdensity for native log-density evaluation without first forming this potentially tiny value.

source
ComputerAdaptiveTesting.Stateful.logdensity — Function

Evaluate the configured ability distribution's unnormalized log density at ability. Includes the prior for posterior estimators, and follows the selected branch for guarded estimators. Available independently of the integration-space policy; the distribution estimator must implement native logpdf.

source
ComputerAdaptiveTesting.Stateful.next_item — Function
next_item(config::StatefulCat) -> IndexT

Returns the index of the best next item according to the CAT.

Ideally IndexT will be an integer and the return type a 1-based index, however it should at least be the same type as accepted by add_response!.

source
ComputerAdaptiveTesting.Stateful.set_item_bank! — Function
set_item_bank!(config::StatefulCat, item_bank::AbstractItemBank)

Set the current item bank of the CAT. This will also reset the CAT to its initial state, removing all responses.

Some CAT implementations may not support this operation in which case they will throw an error.

source
ComputerAdaptiveTesting.Comparison.CatComparisonConfig — Method
CatComparisonConfig(;
    systems::Dict{String, Union{AbstractComparisonSystem, Callable}},
    strategy::CatComparisonExecutionStrategy,
    phases::Union{NamedTuple{Symbol, Callable}, Tuple{Symbol}},
    skips::Set{Tuple{Symbol, Symbol}},
    callback::Callable
) -> CatComparisonConfig

CatComparisonConfig sets up a evaluation-oriented comparison between different CAT systems.

Specify the comparison by listing: CAT systems in systems, a NamedTuple which gives identifiers to implementations of the StatefulCat interface; the strategy to use, an implementation of CatComparisonExecutionStrategy; the phases to run listed as either as a NamedTuple with names of phases and corresponding callbacks or nothing a Tuple of phases to run; and a callback which will be used as a fallback in cases where no callback is provided.

The exact phases depend on the strategy used. See their individual documentation for more.

source