Several metrics are reported, RMU, PSI, and EAP. It is recommended to also
use the function RIrelRep() to determine conditional reliability.
Details
RMU, Relative Measurement Uncertainty:
This function uses the TAM library to estimate the Rasch model using
Marginal Maximum Likelihood and then generates plausible values
(PVs; Mislevy, 1991). The function uses borrowed code, see ?RMUreliability.
The PVs are then used with the RMU method described by Bignardi et al. (2025) to estimate a mean and confidence interval. The mean is generally similar to the expected a posteriori (EAP) reliability point estimate (Adams, 2005). The confidence interval uses the highest continuous density interval (HDCI) based on the distribution of correlations.
Default setting is to generate 1000 PVs. More are recommended for stable estimates/CIs. How many more has not been systematically evaluated, but 4000 might be a good starting point. For smaller samples, more PVs is not very demanding computationally, but be wary of the time it takes to create thousands of PVs for each respondent in large samples.
PSI, Person Separation Index:
Estimated using functions in the eRm package, see ?eRm::SepRel.
EAP, Expected a posteriori:
Estimated using functions in the TAM package, see ?TAM::EAPrel.
References
Bignardi, G., Kievit, R., & Bürkner, P. C. (2025). A general method for estimating reliability using Bayesian Measurement Uncertainty. PsyArXiv. doi:10.31234/osf.io/h54k8
Mislevy, R. J. (1991). Randomization-Based Inference about Latent Variables from Complex Samples. Psychometrika, 56(2), 177–196. doi:10.1007/BF02294457
Adams, R. J. (2005). Reliability as a measurement design effect. Studies in Educational Evaluation, 31(2), 162–172. doi:10.1016/j.stueduc.2005.05.008
Examples
if (FALSE) { # \dontrun{
# comparison of a fully Bayesian Rasch model and PVs
df <- eRm::raschdat1[,1:20] %>%
rownames_to_column("id") %>%
pivot_longer(!id, names_to = "item")
library(brms)
brms_model <- brm(
value ~ 1 + (1 | item) + (1 | id),
data = df,
chains = 4,
cores = 4,
family = "bernoulli"
)
posterior_draws <- brms_model %>%
as_draws_df() %>%
dplyr::select(starts_with("r_id")) %>%
t()
RMUreliability(posterior_draws)
RIreliability(eRm::raschdat1[,1:20], draws = 4000)
} # }