Item Response Theory and Rasch Measurement Theory

About me

• Lic. psychologist (Uppsala) & PhD behavior analysis (Oslo)
• Scientist at RISE Research Institutes of Sweden
  • ORCID: https://orcid.org/0000-0003-1669-592X
• Background:
  • CEO in private care
  • 10 years consulting; OBM, leadership/groups/organizations, sustainability, etc.
  • Swedish pilot trial of the PAX Good Behavior Game (universal prevention in elementary school)
  • Prevention and measurement

Latent constructs

We want to measure something that is not directly accessible by using one or more proxy indicators.

• in psychology - tests and questionnaires
  • almost always ordered categorical data (ordinal)
• tests of ability/IQ/etc, often consisting of subtests or multiple items
  • type of data: correct/incorrect response; response time
• questionnaires to measure depression, wellbeing, anxiety, loneliness, etc
  • type of data: yes/no, Likert scales, visual analogue scales, etc

Latent variable

Based on observed indicators (response data collected from participants), the latent variable is assigned a value for each respondent/participant. The value of the latent variable is the measurement. The indicators themselves are not measurements, they are indicators of a latent variable.

HS.model <- ' visual  =~ x1 + x2 + x3 
              textual =~ x4 + x5 + x6
              speed   =~ x7 + x8 + x9 '

fit <- cfa(HS.model, data = HolzingerSwineford1939)

lavaanPlot(model = fit)

Types of indicator data

• Dichotomous data - two categories (yes/no; correct/incorrect)
• Polytomous data - more than two ordered categories (Likert, etc)
• Interval data - i.e response time
• Count data - i.e frequency of behavior during a time period

We’ll primarily look at the first two types, as they are most commonly used in psychology and psychometrics.

“Sum score and alpha”

• This is a huge problem in published research.
• Many papers include “measures” that consist of questionnaires created ad hoc for a study, report Cronbach’s alpha, and use a sum score based on putting numbers on response categories.
• Most psychological “measures” are actually used only once (Elson et al. 2023)

We need a reasonable psychometric analysis to justify the use of a sum score! And even then the “sum score” is a debatable metric in itself since it is ordinal data, but often gets treated like interval data in statistical models.

Ability testing

• As a first and simple example, we’ll look at data from a test where a participant can either score correct (1) or incorrect (0). More items solved correctly = higher ability.
• We will use the dichotomous Rasch model.
• A key assumption of the Rasch model (shared with all IRT models) is that items will have a systematic ordering of difficulty that is similar across participants.
• This is the structure of the dataset (items = columns):

Guttman pattern

The assumed basic structure of the data is that there is a systematic pattern across items and participants of an increased probability of correct responses as the latent ability increases. The Rasch model can be described as a probabilistic Guttman scale (Andrich 1985).

RIheatmap(rasch_data)

This figure shows items and persons sorted based on the number of correct responses (colored blue). You can see the gradual shift from lower left to upper right that shows the Guttman pattern.

Item difficulty/location

rasch <- RM(rasch_data)
plotICC(rasch, item.subset = 4, ask = FALSE, xlim = c(-5,6))

This is a key figure in understanding IRT and the concept of item difficulty/location. The x axis is the latent ability (aka latent variable/dimension/continuum), and the y axis is the probability of a correct response.

Item difficulty/location

The point on the x-axis where the y-axis = 0.5 is the item difficulty (aka item location or item threshold). This is the threshold when the probability of a correct response is equal to the probability of an incorrect response. A correct response on this item indicates a higher ability than the item difficulty, and an incorrect response indicates a lower ability than the item difficulty.

A note on terminology

• By tradition in Rasch/IRT terminology, the term “difficulty” is used to describe the item location on the latent dimension/variable/continuum and “ability” is used to describe the person location on the latent dimension/variable/continuum.

• Ability and difficulty are intuitive when describing ability tests, but may be confusing when looking at other types of latent constructs, such as depression or well-being.

• A more generic term is the “location” of items and persons on the latent variable. We will move towards using that more consistently in this lecture, but there will be some variation in terminology (sorry!).

  • Hopefully, this inconsistency on my part will make it easier to read other materials that use different terms for the same concepts.

Item difficulty pt 2

plotjointICC(rasch, xlim = c(-5,5), main = "ICC for 9 dichotomous items")
abline(h = 0.5, lty = "dashed")

This figure illustrates how the items are ordered (“item hierarchy”) according to the locations on the latent dimension/variable where they provide the most information - the item threshold - which is the location on the x axis where probability = 0.5 (on the y axis) for each dichotomous item.

Item difficulty sorted

plotPImap(rasch, sorted = TRUE)

A note on scaling

• IRT/Rasch uses the logit scale, which is an interval scale, for both items and persons. This means that the distance between two points on the scale is the same, regardless of where on the scale you are. Great for statistical analysis!

• However, the values on the logit scale have no inherent meaning or external reference point. This is why we need to look at the item difficulty and person ability in relation to each other. Do not conflate the zero point on the logit scale with something like 0 on a Z-score scale.

• You can also not interpret a person location as good or bad, or high or low, without looking at it in the context of more person locations. A person with a location of 0 is not necessarily “average”, “normal”, or “healthy” (or anything else). It is just a location on the latent variable.

Targeting

Targeting pt 2

Summing up so far

We have looked at the Rasch model for dichotomous data, also sometimes (not quite correctly) referred to as the IRT 1PL model. The key parameter is the item location/difficulty. 1PL stands for “one parameter logistic” model, and the parameter is the item location.

I generally use the term “location” for both items and persons as it is more generic. But for didactic purposes, when speaking of ability tests, it is probably easier to think about the specific term “difficulty” for items and how it related to the persons latent “ability”.

In IRT terminology, “person location” is frequently referred to as “theta”, often using this symbol: \(\theta\)

Other IRT models

The 2PL and 3PL models are also commonly used. The 2PL model adds a second parameter, item discrimination, which describes how well the item separates between persons with high and low ability.

mirt(rasch_data, 1, itemtype = "Rasch", verbose = FALSE) %>% plot(type="trace", facet_items = FALSE)
mirt(rasch_data, 1, itemtype = "2PL", verbose = FALSE) %>% plot(type="trace", facet_items = FALSE)

3PL model

The 3PL model adds a third parameter, which makes the figure look like this.

mirt(rasch_data, 1, itemtype = "3PL", verbose = FALSE) %>% plot(type="trace", facet_items = FALSE)

Can you guess what the third parameter is?

ICC for polytomous models

This is the same type of figure as before, and it now shows probabilities for all response categories for one item. How do you interpret it?

Guesstimating theta

Recall \(\theta\) (person location)?

• Let’s say a person responds “Often” to this item, in which range is it most likely that their theta is?

Guesstimating theta

This is an item from the Perceived Stress Scale (PSS).

Estimating theta

In context

Is this person’s theta a high or low value? How does it relate to the overall sample?

Adding another item

q8n: “found that you could not cope with all the things that you had to do?” - “Sometimes”

One more item…

q2n: “felt that you were unable to control important things in your life?” - “Seldom”

See how that error bar gets smaller and smaller as we add more items? That’s because we’re getting more and more information about the respondents theta with each item.

PSS-7 targeting

Here are all 7 items from the PSS negative items (Rozental, Forsström, and Johansson 2023). Discuss 2 & 2 how you interpret this figure.

RItargeting(pss7, xlim = c(-4,4))

PSS-7 item hierarchy

Why is this figure of interest?

RIitemHierarchy2(pss7) + theme_rise()

Reliability

Test information - reflects item properties, not sample/person properties.

• The same goes for test-retest as a reliability test, it should be used to assess stability in item properties over time, not sample properties.

Unidimensionality

While multidimensional constructs are possible, it is outside the scope of this lecture. Most often, even unidimensional measures are not well constructed. The most common problem (in my experience) with dimensionality is residual correlations. We’ll look at four ways to assess unidimensionality in a Rasch model:

• Conditional item fit statistics
• Residual correlation matrix
• Plot of factor loadings on the first residual contrast
• PCA of residuals

PSS-14 example

We’ll use the same published dataset and paper analyzing the PSS-14 scale as before (Rozental, Forsström, and Johansson 2023) as an example for dimensionality analysis.

We use multiple methods to analyze dimensionality since there is no single method that allows one to assess unidimensionality properly. This is also true for factor analysis.

Does everyone know what residuals are?

PCA of residuals

The highest eigenvalue should be below 2.0 (closer to 1.5)

Loadings on 1st contrast

We clearly have two clusters of items. Can you spot the pattern?

Residual correlation matrix

Another residual corr matrix

Unidimensionality - Item fit

Ordered response categories

A higher person location on the latent variable should entail an increased probability of a higher response (category) for all items and vice versa. This is sometimes referred to as ‘monotonicity’.

We can check this by looking at the item characteristic curves (ICC). So far, we have only seen ICCs with ordered response categories. We will look at an example with disordered response categories.

An important note on types of data

• Ordinal data - ordered response categories with unknown distance between categories.

This is by far the most common type of data in psychology and social sciences. But it is extremely common to pretend that ordinal data is interval data, and use it as such in everything from simple calculations such as mean/SD to more complex statistical models. This is a problem (Liddell and Kruschke 2018) and there are methods for analyzing ordinal data properly (i.e Bürkner and Vuorre 2019).

Adding more response categories does not make them anything else than ordinal, neither does removing the labels. Visual Analogue Scales have no strong case for being interval level either. All three methods usually add to problems with disordered response categories and invariance issues.

Example: Flourishing Scale

We’ll use an open dataset (Didino et al. 2019) for the Flourishing Scale (FS) (Diener et al. 2010). It has 8 items:

FS response categories

All items share the same set of 7 response categories:

Disordered categories

What we look for in this figure is response categories that at no point on the x-axis has the highest probability. Visually, this means that the problematic probability curve does not pass “above” other categories. Which ones can you identify in this example?

Addressing disordered response categories

There are several ways to address disordered response categories in the analysis phase. The most common is to merge adjacent categories and rerun the ICC analysis to check results. However, disordered thresholds can (at least partially) be a sign of other problems, such as misfitting items, multidimensionality, or local dependence.

Long term, it is important to investigate the cause and revise the questionnaire. Usually, the cause of disordered categories is related to having too many response categories, having bad or no labels on categories (only endpoints labeled is bad practice). Also, item formulation needs to work well with the response categories.

Invariance

• In essence, invariance is about the stability of item locations across subgroups. It can be explained as an interaction effect between item location and a demographic variable.
• Invariance is important to establish to ensure that we can compare measurements between groups of interest.
• For instance, for two persons with same theta, belonging to different groups (i.e gender), each item should have about the same response threshold location.
• (We can also look at other item properties, such as item fit, and how it varies across subgroups, but that is outside the scope of this lecture)

Invariance pt 2

• We can use global tests of invariance, that assess the measurement model as a whole (all items together) and compare groups
  • Likelihood ratio test (LRT)
• And, usually more interesting, tests that examine each item separately
  • Differential Item Functioning (DIF)
• Several types of tests are available for both
  • some tests can only compare two groups, some can deal with interactions between two types of groups (education level+sex) and continuous variables (age in years)

Invariance pt 3

A preprint reviewing invariance tests in published research (D’Urso et al. 2022) concluded that:

(1) 4% of the 918 scales underwent MI testing across groups or time and that these tests were generally poorly reported, (2) none of the reported MI tests could be successfully reproduced, and (3) of 161 newly performed MI tests, a mere 46 (29%) reached sufficient MI (scalar invariance), and MI often failed completely (89; 55%). Thus, MI tests were rarely done and poorly reported in psychological studies, and the frequent violations of MI indicate that reported group differences cannot be solely attributed to group differences in the latent constructs.

DIF comments

• Important and complex topic
• The practical effects of DIF found are affected both by the total number of items and the magnitude of the DIF
• Large sample sizes will indicate statistically significant DIF, but DIF magnitude is the key metric
• How to deal with problematic DIF is outside the scope of this lecture
• For more examples and details on DIF, see: (Andrich and Hagquist 2012, 2015; Hagquist 2019; Hagquist and Andrich 2017)

Reliability

• Affects statistical power to detect changes/differences in the latent trait. Reliability is a function of the number of items/item thresholds and the item locations. Having more items with more thresholds (ordered response categories), and the more dispersed the item locations are (instead of overlapping), the higher the reliability.

• It is important to understand the reliability of the measure itself, and then how it applies to the population of interest. Since reliability is not constant across the latent variable continuum, targeting becomes a factor in determining reliability in a practical use case.

• In IRT/Rasch, the Person Separation Index (PSI) is sometimes used, as it is comparable to the ubiquitous Cronbach’s alpha in having a 0-1 range. However, the PSI represents the sample SEM, not the test (see ?RItif).

  • Presenting the sample reliability is relevant for subsequent statistical analyses, just remember that it is not necessarily representative of the reliability of the test itself.

Standard error of measurement

Recall this figure? The horizontal lines (SEM) around the diamond shapes (estimated theta values) become shorter for each item we add.

TIF and SEM

The Test Information Function (Samejima 1969, 1994) is basically the inverse of the standard error of measurement.

Classical vs modern test theory

• CTT does not include analysis of item location/difficulty, which is a key aspect of IRT
• CTT assumes that reliability is constant across the continuum (and equal for all respondents)
• CTT has no analysis of monotonicity/ordering of response categories
• IRT enables the analysis of items and persons on the same (interval level) scale
• IRT produces model fit metrics for both items and persons
• IRT enables estimation of latent variable scores on an interval scale
• Only Rasch models can justify ordinal sum scores as a sufficient metric to represent measurement on the latent construct scale
  • Also, Rasch is a special case… (see Andrich 2004; Kyngdon 2008).