Trifactor Models for Multiple-Ratings Data

Abstract In this study we extend and assess the trifactor model for multiple-ratings data in which two different raters give independent scores for the same responses (e.g., in the GRE essay or to subset of PISA constructed-responses). The trifactor model was extended to incorporate a cross-classified data structure (e.g., items and raters) instead of a strictly hierarchical structure. we present a set of simulations to reflect the incompleteness and imbalance in real-world assessments. The effects of the rate of missingness in the data and of ignoring differences among raters are investigated using two sets of simulations. The use of the trifactor model is also illustrated with empirical data analysis using a well-known international large-scale assessment.

• Publication year

  2019

• Venue

  Multivariate Behavioral Research

• Publication date

  2019-03-28

• Fields of study

  Mathematics, Medicine, Psychology

• Identifiers
  DOI 10.1080/00273171.2018.1530091PMID 30919664
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Many-facet Rasch measurement

  2019cited by this paper
• R: A language and environment for statistical computing.

  2014cited by this paper
• A Third-Order Item Response Theory Model for Modeling the Effects of Domains and Subdomains in Large-Scale Educational Assessment Surveys

  2014influential reference
• A trifactor model for integrating ratings across multiple informants.

  2013influential reference
• Application of Latent Trait Models to Identifying Substantively Interesting Raters

  2012cited by this paper
• Rater Cognition: Implications for Validity.

  2012cited by this paper
• Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences

  2012cited by this paper
• Introduction to the special section.

  2012cited by this paper
• Introduction to the Special Section: More Than Measurement Error: Discovering Meaning Behind Informant Discrepancies in Clinical Assessments of Children and Adolescents

  2011cited by this paper
• Generalized full-information item bifactor analysis.

  2011cited by this paper
• A Hierarchical Rater Model for Constructed Responses, with a Signal Detection Rater Model

  2011cited by this paper
• Commentary: Definitely More Than Measurement Error: But How Should We Understand and Deal With Informant Discrepancies?

  2011cited by this paper
• Introduction to Many-Facet Rasch Measurement: Analyzing and Evaluating Rater-Mediated Assessments

  2011cited by this paper
• Introduction to the Special Section: More Than Measurement Error: Discovering Meaning Behind Informant Discrepancies in Clinical Assessments of Children and Adolescents

  2011cited by this paper
• Simsum: Analyses of Simulation Studies Including Monte Carlo Error

  2010cited by this paper
• A Markov Chain Monte Carlo Approach to Confirmatory Item Factor Analysis

  2010cited by this paper
• Studies of a Latent Class Signal Detection Model for Constructed Response Scoring II: Incomplete and Hierarchical Designs. Research Report. ETS RR-10-08.

  2010influential reference
• Formal Relations and an Empirical Comparison among the Bi‐Factor, the Testlet, and a Second‐Order Multidimensional IRT Model

  2010cited by this paper
• BUGS Code for Item Response Theory

  2010cited by this paper
• An Examination of Rater Drift within a Generalizability Theory Framework.

  2009cited by this paper
• On the Assessment of Monte Carlo Error in Simulation-Based Statistical Analyses

  2009cited by this paper
• Simultaneous Estimation of Overall and Domain Abilities: A Higher-Order IRT Model Approach

  2009cited by this paper
• STUDIES OF A LATENT-CLASS SIGNAL-DETECTION MODEL FOR CONSTRUCTED-RESPONSE SCORING

  2008cited by this paper
• Covariates of the Rating Process in Hierarchical Models for Multiple Ratings of Test Items

  2007cited by this paper
• Generalizability in Item Response Modeling

  2007cited by this paper
• A Comparison of Alternative Models for Testlets

  2006cited by this paper
• Analysing multitrait-multimethod data with structural equation models for ordinal variables applying the WLSMV estimator: what sample size is needed for valid results?

  2006cited by this paper
• Markov Chain Monte Carlo

  2006cited by this paper
• Appropriate assessment of neighborhood effects on individual health: integrating random and fixed effects in multilevel logistic regression.

  2005cited by this paper
• Detecting and measuring rater effects using many-facet Rasch measurement: Part II.

  2004cited by this paper
• Generalized multilevel structural equation modeling

  2004cited by this paper
• A comparison of generalizability theory and many-facet Rasch measurement in an analysis of college sophomore writing

  2004cited by this paper
• Detecting and measuring rater effects using many-facet Rasch measurement: part I.

  2003cited by this paper
• The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment Data

  2002influential reference
• A General Bayesian Model for Testlets: Theory and Applications

  2002cited by this paper
• Bayesian measures of model complexity and fit

  2002cited by this paper
• Construction of measures from many-facet data.

  2002cited by this paper
• The Information in Multiple Ratings

  2002cited by this paper
• Real-time feedback on rater drift in constructed-response items: An example from the golden state examination

  2001cited by this paper
• An IRT Model for Multiple Raters

  2001cited by this paper
• The Rater Bundle Model

  2001influential reference
• WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility

  2000cited by this paper
• Interpreting Parameters in the Logistic Regression Model with Random Effects

  2000cited by this paper
• Ordinal Data Modeling

  2000cited by this paper
• The Linear Logistic Test Model

  1995cited by this paper
• Sampling Variability of Performance Assessments.

  1993cited by this paper
• Multitrait-Multimethod Analyses: Inferring Each Trait-Method Combination with Multiple Indicators.

  1993cited by this paper
• Inference from Iterative Simulation Using Multiple Sequences

  1992influential reference
• Generalizability Theory

  1992cited by this paper
• Full-information item bi-factor analysis

  1992cited by this paper
• Items bundles

  1988cited by this paper
• Item Clusters and Computerized Adaptive Testing: A Case for Testlets

  1987cited by this paper
• The design and analysis of clinical experiments

  1987cited by this paper
• Rating the ratings: Assessing the psychometric quality of rating data

  1980cited by this paper
• INFERENCE AND MISSING DATA

  1975cited by this paper
• The linear logistic test model as an instrument in educational research

  1973cited by this paper
• مراجعة كتاب الإطار المرجعي الأوروبي المشترك للغات: دراسة، تدريس، تقييم / The Common European Framework of Reference for Languages: Learning, Teaching, Assessment

  1970influential reference
• Convergent and discriminant validation by the multitrait-multimethod matrix.

  1959cited by this paper

• Bayesian Nonparametric Models for Multiple Raters: A General Statistical Framework

  2024cites this paper
• Reducing Attenuation Bias in Regression Analyses Involving Rating Scale Data via Psychometric Modeling

  2024cites this paper
• Item response theory model highlighting rating scale of a rubric and rater–rubric interaction in objective structured clinical examination

  2024cites this paper
• Linking essay-writing tests using many-facet models and neural automated essay scoring

  2024cites this paper
• Discrepant Results in Mental Health Research

  2024cites this paper
• Improving the Precision of Classroom Observation Scores Using a Multi-Rater and Multi-Timepoint Item Response Theory Model

  2024cites this paper
• Integration of Prediction Scores From Various Automated Essay Scoring Models Using Item Response Theory

  2023influential citation
• Using Linkage Sets to Improve Connectedness in Rater Response Model Estimation

  2023cites this paper
• An Item Response Theory Approach to Enhance Peer Assessment Effectiveness in Massive Open Online Courses

  2022cites this paper
• The Operations Triad Model and Youth Mental Health Assessments: Catalyzing a Paradigm Shift in Measurement Validation

  2022cites this paper
• A Bayesian many-facet Rasch model with Markov modeling for rater severity drift

  2022cites this paper
• Bridging the Grade Gap: Reducing Assessment Bias in a Multi-Grader Class

  2022cites this paper
• A Multidimensional Item Response Theory Model for Rubric-Based Writing Assessment

  2021cites this paper
• Examining the Performance of the Trifactor Model for Multiple Raters

  2021cites this paper
• A multidimensional generalized many-facet Rasch model for rubric-based performance assessment

  2021cites this paper
• Measurement of aggressive behavior in early childhood: A critical analysis using five informants.

  2021cites this paper
• A Multi-Rater Latent Growth Curve Model

  2021cites this paper
• A Dyadic IRT Model

  2019influential citation
• Investigating Rater Effects in International Large-Scale Assessments

  2019cites this paper