Developments in Psychometric Population Models for Technology-Based Large-Scale Assessments: An Overview of Challenges and Opportunities

International large-scale assessments (ILSAs) transitioned from paper-based assessments to computer-based assessments (CBAs) facilitating the use of new item types and more effective data collection tools. This allows implementation of more complex test designs and to collect process and response time (RT) data. These new data types can be used to improve data quality and the accuracy of test scores obtained through latent regression (population) models. However, the move to a CBA also poses challenges for comparability and trend measurement, one of the major goals in ISLAs. We provide an overview of current methods used in ILSAs to examine and assure the comparability of data across different assessment modes and methods that improve the accuracy of test scores by making use of new data types provided by a CBA.

• Publication year

  2019

• Venue

  Journal of educational and behavioral statistics

• Publication date

  2019-10-23

• Fields of study

  Computer Science, Psychology

• Identifiers
  DOI 10.3102/1076998619881789
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Understanding and detecting data fabrication in large-scale assessments

  2018cited by this paper
• Modeling Omitted and Not-Reached Items in IRT Models

  2017cited by this paper
• AND FUTURE DIRECTIONS

  2017cited by this paper
• Automated Assessment of Patients’ Self-Narratives for Posttraumatic Stress Disorder Screening Using Natural Language Processing and Text Mining

  2017cited by this paper
• Joint Modeling of Ability and Differential Speed Using Responses and Response Times

  2016cited by this paper
• A New Procedure for Detection of Students’ Rapid Guessing Responses Using Response Time

  2016cited by this paper
• Sometimes More Is Better, and Sometimes Less Is Better: Task Complexity Moderates the Response Time Accuracy Correlation

  2016cited by this paper
• Analyzing Process Data from Problem-Solving Items with N-Grams: Insights from a Computer-Based Large-Scale Assessment

  2016influential reference
• Using Response Time Data to Inform the Coding of Omitted Responses

  2016cited by this paper
• Markov Response Models 1 RUNNING HEAD : Markov Response Models Hidden Markov IRT Models for Responses and Response Times

  2016cited by this paper
• A Bivariate Generalized Linear Item Response Theory Modeling Framework to the Analysis of Responses and Response Times

  2015cited by this paper
• Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving

  2015cited by this paper
• Identifying Feature Sequences from Process Data in Problem-Solving Items with N -Grams

  2015cited by this paper
• More is not Always Better: The Relation between Item Response and Item Response Time in Raven's Matrices

  2015cited by this paper
• Nonignorable data in IRT models: Polytomous responses and response propensity models with covariates

  2015cited by this paper
• An Alternative Way to Model Population Ability Distributions in Large-Scale Educational Surveys

  2015cited by this paper
• Statistical Learning with Sparsity: The Lasso and Generalizations

  2015cited by this paper
• The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment.

  2014cited by this paper
• Predicting self-monitoring skills using textual posts on Facebook

  2014cited by this paper
• Using response time to investigate students' test-taking behaviors in a NAEP computer-based study

  2014cited by this paper
• Analytics in International Large-Scale Assessments: Item Response Theory and Population Models

  2013cited by this paper
• Imputing Proficiency Data under Planned Missingness in Population Models

  2013cited by this paper
• Handbook of International Large-Scale Assessment : Background, Technical Issues, and Methods of Data Analysis

  2013cited by this paper
• Faster on easy items, more accurate on difficult ones: Cognitive ability and performance on a task of varying difficulty

  2013cited by this paper
• Screening for posttraumatic stress disorder using verbal features in self narratives: a text mining approach.

  2012cited by this paper
• Can fast and slow intelligence be differentiated

  2012cited by this paper
• Bayesian tests of measurement invariance.

  2012cited by this paper
• A review of recent response-time analyses in educational testing

  2011cited by this paper
• Investigation of model fit and score scale comparability in international assessments

  2011cited by this paper
• Measuring Growth in a Longitudinal Large-Scale Assessment with a General Latent Variable Model

  2011cited by this paper
• A brief survey on sequence classification

  2010cited by this paper
• A Mixture Rasch Model With Item Response Time Components

  2010cited by this paper
• IRT Parameter Estimation With Response Times as Collateral Information

  2010cited by this paper
• Testing Measurement Invariance Using Item Response Theory in Longitudinal Data: An Introduction

  2010cited by this paper
• 双対平坦空間におけるLeast Angle Regressionと情報量規準

  2009cited by this paper
• Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing

  2008cited by this paper
• A general diagnostic model applied to language testing data.

  2008cited by this paper
• Testlet Response Theory and Its Applications

  2007cited by this paper
• Forward stagewise regression and the monotone lasso

  2007cited by this paper
• A hierarchical framework for modeling speed and accuracy on test items

  2007cited by this paper
• Detecting Differential Speededness in Multistage Testing

  2007cited by this paper
• A Hierarchical Framework for Modeling Speed and Accuracy on Test Items

  2007cited by this paper
• An Application of Item Response Time: The Effort‐Moderated IRT Model

  2006cited by this paper
• Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests

  2005cited by this paper
• A GENERAL DIAGNOSTIC MODEL APPLIED TO LANGUAGE TESTING DATA

  2005cited by this paper
• Sequence Data Mining

  2005cited by this paper
• Applied Linear Regression

  2005cited by this paper
• A UNIFIED APPROACH TO IRT SCALE LINKING AND SCALE TRANSFORMATIONS

  2004cited by this paper
• An Extensive Empirical Study of Feature Selection Metrics for Text Classification

  2003cited by this paper
• A hierarchical bayesian statistical framework for response time distributions

  2003cited by this paper
• Categorical Data Analysis

  2003cited by this paper
• Model-Based Clustering, Discriminant Analysis, and Density Estimation

  2002cited by this paper
• The role of secondary covariates when estimating latent trait population distributions

  2002influential reference
• Foundations of Statistical Natural Language Processing

  2001cited by this paper
• Random Forests

  2001cited by this paper
• A method based on the chi-square test for document classification

  2001cited by this paper
• Text Classification from Labeled and Unlabeled Documents using EM

  2000cited by this paper
• An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization

  2000cited by this paper
• Feature Generation for Sequence Categorization

  1998cited by this paper
• Regression Shrinkage and Selection via the Lasso

  1996cited by this paper
• Additive and multiplicative models for gamma distributed random variables, and their application as psychometric models for response times

  1993cited by this paper
• Measurement invariance, factor analysis and factorial invariance

  1993influential reference
• The Use of Collateral Information in Proficiency Estimation for the Trial State Assessment.

  1992cited by this paper
• Organization for Economic Cooperation and Development

  1992cited by this paper
• Randomization-based inference about latent variables from complex samples

  1991cited by this paper
• ON THE RELIABILITY OF TESTLET‐BASED TESTS

  1991cited by this paper
• Model selection and Akaike's Information Criterion (AIC): The general theory and its analytical extensions

  1987cited by this paper
• Estimating the Dimension of a Model

  1978cited by this paper
• A new look at the statistical model identification

  1974cited by this paper

• Assessing financial literacy among the young

  2025cites this paper
• Exploring the Influence of Response Time Allocation on Item Revisiting: Implications for Test‐Taking Strategies in Cognitive Diagnostic Assessments

  2025cites this paper
• A composite intelligence scoring framework for identifying high-potential individuals using multi-metric predictive models

  2025cites this paper
• Comparing Weighted RMSD, Weighted MD, Infit, and Outfit Item Fit Statistics Under Uniform Differential Item Functioning

  2025cites this paper
• Irt-latent regression with many predictors: limits and solutions

  2025cites this paper
• Principal component analysis on the covariance matrix for data reduction in large-scale assessments

  2025cites this paper
• Paper-based and digital reading in 14 countries: exploring cross-country variation in mode effects

  2025influential citation
• Predictive insights into U.S. students’ mathematics performance on PISA 2022 using ensemble tree-based machine learning models

  2025cites this paper
• The use of process data in large-scale assessments: a literature review

  2024cites this paper
• Addressing Threats to Validity in Supervised Machine Learning: A Framework and Best Practices for Education Researchers

  2024cites this paper
• Generally Applicable Variance Estimation Methods for Common-Population Linking

  2024cites this paper
• Examining Gender Differences in TIMSS 2019 Using a Multiple‐Group Hierarchical Speed‐Accuracy‐Revisits Model

  2024cites this paper
• Remote Testing of Reading Comprehension in 8-Year-Old Children: Mode and Setting Effects

  2023cites this paper
• Using Process Data to Improve Classification Accuracy of Cognitive Diagnosis Model

  2023cites this paper
• What Can We Learn from a Semiparametric Factor Analysis of Item Responses and Response Time? An Illustration with the PISA 2015 Data

  2023cites this paper
• 20 years of interactive tasks in large-scale assessments: Process data as a way towards sustainable change?

  2023cites this paper
• Test mode effect: Sources, detection, and applications

  2023cites this paper
• A Latent Hidden Markov Model for Process Data

  2023cites this paper
• Incorporating Test‐Taking Engagement into Multistage Adaptive Testing Design for Large‐Scale Assessments

  2023cites this paper
• Collaborative problem-solving design in large-scale assessments: Shedding lights in sequential conversation-based measurement

  2023cites this paper
• Generating group-level scores under response accuracy-time conditional dependence

  2022cites this paper
• Latent Space Model for Process Data

  2022cites this paper
• Some thoughts on analytical choices in the scaling model for test scores in international large-scale assessment studies

  2022cites this paper
• Exploring the Multiverse of Analytical Decisions in Scaling Educational Large-Scale Assessment Data: A Specification Curve Analysis for PISA 2018 Mathematics Data

  2022cites this paper
• Leveraging process data to assess adults' problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks

  2021cites this paper
• Combining Clickstream Analyses and Graph-Modeled Data Clustering for Identifying Common Response Processes

  2021cites this paper
• On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

  2021cites this paper
• Robust and Nonrobust Linking of Two Groups for the Rasch Model with Balanced and Unbalanced Random DIF: A Comparative Simulation Study and the Simultaneous Assessment of Standard Errors and Linking Errors with Resampling Techniques

  2021cites this paper
• Identifying Mixture Components From Large-Scale Keystroke Log Data

  2021cites this paper
• Plausible Values: Principles of Item Response Theory and Multiple Imputations

  2020cites this paper
• Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects

  2020cites this paper
• Technology and Assessment

  2020cites this paper
• Old questions, new challenges and possible solutions

  2020cites this paper
• Rapid guessing rates across administration mode and test setting

  2020cites this paper
• TIMSS 2019 Scaling Methodology: Item Response Theory, Population Models, and Linking Across Modes

  2020cites this paper
• H New Procedures for PISA 2018 Population Modelling

  2020cites this paper
• Changing times, changing needs: enhancing the utility of international large-scale assessments

  2020cites this paper
• The Detection of Cheating on E-Exams in Higher Education—The Performance of Several Old and Some New Indicators

  2020cites this paper
• Evaluating item response theory linking and model fit for data from PISA 2000–2012

  2019cites this paper
• The Uses of Process Data in PISA

  year unknowncites this paper