Challenges in Survey Research

While being an important and often used research method, survey research has been less often discussed on a methodological level in empirical software engineering than other types of research. This chapter compiles a set of important and challenging issues in survey research based on experiences with several large-scale international surveys. The chapter covers theory building, sampling, invitation and follow-up, statistical as well as qualitative analysis of survey data and the usage of psychometrics in software engineering surveys.

• Publication year

  2019

• Venue

  Contemporary Empirical Methods in Software Engineering

• Publication date

  2019-08-16

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1007/978-3-030-32489-6_4arXiv 1908.05899
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Happiness and the productivity of software engineers

  2019cited by this paper
• Scientists rise up against statistical significance

  2019cited by this paper
• CERSE - Catalog for empirical research in software engineering: A Systematic mapping study

  2019cited by this paper
• Empirical Software Engineering: From Discipline to Interdiscipline

  2018cited by this paper
• Status Quo in Requirements Engineering

  2018influential reference
• Standards of validity and the validity of standards in behavioral software engineering research: the perspective of psychological test theory

  2018influential reference
• Taxonomies in software engineering: A Systematic mapping study and a revised taxonomy development method

  2017cited by this paper
• Survey Research in Software Engineering: Problems and Mitigation Strategies

  2017cited by this paper
• What happens when software developers are (un)happy

  2017cited by this paper
• On the Unhappiness of Software Developers

  2017cited by this paper
• Lessons Learnt in Conducting Survey Research

  2017cited by this paper
• Supporting Defect Causal Analysis in Practice with Cross-Company Data on Causes of Requirements Engineering Problems

  2017cited by this paper
• On Evidence-Based Risk Management in Requirements Engineering

  2017cited by this paper
• Grounded Theory in Software Engineering Research: A Critical Review and Guidelines

  2016cited by this paper
• Worse Than Spam: Issues In Sampling Software Developers

  2016cited by this paper
• Statistics

  2016cited by this paper
• Naming the pain in requirements engineering

  2016cited by this paper
• Naming the Pain in Requirements Engineering: Comparing Practices in Brazil and Germany

  2015cited by this paper
• Understanding the affect of developers: theoretical background and guidelines for psychoempirical software engineering

  2015cited by this paper
• A systematic literature review on agile requirements engineering practices and challenges

  2015cited by this paper
• Forty years of research on personality in software engineering: A mapping study

  2015cited by this paper
• Theory-oriented software engineering

  2015cited by this paper
• Behavioral software engineering: A definition and systematic literature review

  2015cited by this paper
• Investigating probabilistic sampling approaches for large-scale surveys in software engineering

  2015cited by this paper
• A Handbook of Test Construction (Psychology Revivals) : Introduction to Psychometric Design

  2015cited by this paper
• Constructing Grounded Theory

  2014cited by this paper
• Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0

  2014cited by this paper
• The standards for educational and psychological testing.

  2013cited by this paper
• Naming the pain in requirements engineering: design of a global family of surveys and first results from Germany

  2013cited by this paper
• Experimentation in Software Engineering

  2012cited by this paper
• Evidence-Based Guidelines to Defect Causal Analysis

  2012cited by this paper
• Case Study Research in Software Engineering - Guidelines and Examples

  2012cited by this paper
• Statistical Inference: The Big Picture.

  2011cited by this paper
• An Introduction to Psychological Assessment and Psychometrics

  2010cited by this paper
• Guidelines for conducting and reporting case study research in software engineering

  2009cited by this paper
• Personal Opinion Surveys

  2008cited by this paper
• Towards individualized software engineering: empirical studies should collect psychometrics

  2008cited by this paper
• A Critical Assessment of Null Hypothesis Significance Testing in Quantitative Communication Research

  2008cited by this paper
• Excerpts from : The Discovery of Grounded Theory : Strategies for Qualitative Research

  2008cited by this paper
• Building Theories in Software Engineering

  2008cited by this paper
• A Systematic Review of Theory Use in Software Engineering Experiments

  2007cited by this paper
• The Nature of Theory in Information Systems

  2006cited by this paper
• Personality and the Fate of Organizations

  2006cited by this paper
• Designing an Effective Survey

  2005cited by this paper
• Practical Experiences in the Design and Conduct of Surveys in Empirical Software Engineering

  2003cited by this paper
• Principles of survey research part 2: designing a survey

  2002cited by this paper
• Principles of survey research part 4: questionnaire evaluation

  2002cited by this paper
• Principles of survey research: part 3: constructing a survey instrument

  2002cited by this paper
• Principles of survey research: part 5: populations and samples

  2002cited by this paper
• Principles of survey research: part 1: turning lemons into lemonade

  2001cited by this paper
• Requirements Elicitation Using a Combination of Prototypes and Scenarios

  2001cited by this paper
• Standards for Educational and Psychological Testing

  1999cited by this paper
• An assessment of survey research in POM: from constructs to theory

  1998cited by this paper
• Survey research methods

  1998cited by this paper
• Viewpoints for requirements elicitation: a practical approach

  1998cited by this paper
• Bootstrap confidence intervals

  1996cited by this paper
• John wiley & sons.

  1994cited by this paper
• Survey Research Methodology in Management Information Systems: An Assessment

  1993cited by this paper
• Measuring the MBTI. . .And Coming Up Short.

  1993cited by this paper
• Basics of Qualitative Research

  1992influential reference
• Basics of Grounded Theory Analysis: Emergence Vs. Forcing

  1992influential reference
• Modern Psychometrics: The Science of Psychological Assessment

  1989cited by this paper
• Sampling techniques.

  1971influential reference
• Statistics: An Introductory Analysis

  1965cited by this paper
• Convergent and discriminant validation by the multitrait-multimethod matrix.

  1959cited by this paper
• POLITECNICO DI TORINO Repository ISTITUZIONALE Naming the Pain in Requirements Engineering : Contemporary Problems , Causes , and effects in Practice /

  year unknowncited by this paper
• Naming the pain in requirements engineering

  year unknowncited by this paper

• Understanding npm Developers' Practices, Challenges, and Recommendations for Secure Package Development

  2026cites this paper
• Adoption of Large Language Models in Scrum Management: Insights from Brazilian Practitioners

  2026cites this paper
• Practitioner Views on Mobile App Accessibility: Practices and Challenges

  2026cites this paper
• How vulnerability explanations help software practitioners confirm and fix code vulnerabilities

  2026cites this paper
• M&SCheck: Towards a Checklist to Support Software Engineering Newcomers to the Modeling and Simulation Area

  2025cites this paper
• Understanding Computer Science Students' Career Fair Experiences: Goals, Preparation, and Outcomes

  2025cites this paper
• Why do women pursue a PhD in Computer Science?

  2025influential citation
• Industrial Practices of Requirements Engineering for ML-Enabled Systems in Brazil: An Extended Analysis

  2025cites this paper
• An Empirical Study of Generative AI Adoption in Software Engineering

  2025cites this paper
• An Investigation on How AI-Generated Responses Affect SoftwareEngineering Surveys

  2025influential citation
• Analysis of User-Generated Content to Unfold Privacy Concerns with CBDCs

  2025cites this paper
• A study on reported under-documented non-functional requirements as an indicator of technical debt

  2025cites this paper
• Da Robótica à Inclusão Digital: Fortalecendo a Presença Feminina na Computação com Pesquisa e Extensão

  2025cites this paper
• Synthetic Personas: Enhancing Demographic Response Simulation through Large Language Models and Genetic Algorithms

  2025cites this paper
• Exploring Code Comprehension in Scientific Programming: Preliminary Insights from Research Scientists

  2025cites this paper
• Requirements Engineering for ML-Enabled Systems: Status Quo and Problems

  2024cites this paper
• Comparing Design Thinking Techniques in the Context of IoT Requirements Elicitation: An Empirical Study

  2024influential citation
• Do Developers Use Static Application Security Testing (SAST) Tools Straight Out of the Box? A large-scale Empirical Study

  2024cites this paper
• The Impact of Generative AI-Powered Code Generation Tools on Software Engineer Hiring: Recruiters' Experiences, Perceptions, and Strategies

  2024cites this paper
• A Developer-Centric Study Exploring Mobile Application Security Practices and Challenges

  2024cites this paper
• On the Rationale and Use of Assertion Messages in Test Code: Insights from Software Practitioners

  2024cites this paper
• Teaching Survey Research in Software Engineering

  2024influential citation
• ML-Enabled Systems Model Deployment and Monitoring: Status Quo and Problems

  2024cites this paper
• Industrial Practices of Requirements Engineering for ML-Enabled Systems in Brazil

  2024cites this paper
• Naming the Pain in Machine Learning-Enabled Systems Engineering

  2024influential citation
• Automated Security Findings Management: A Case Study in Industrial DevOps

  2024cites this paper
• Could Gameplay Data of Games with a Purpose Assist in Training Hazardous Situations in the Industry?

  2024cites this paper
• An Experience Report on the Use of Active Learning in Empirical Software Engineering Education: Understanding the Pros and Cons from the Student's Perspective

  2024cites this paper
• Different Debt: An Addition to the Technical Debt Dataset and a Demonstration Using Developer Personality

  2024cites this paper
• A Conceptual Framework for Studying Factors that Influence Digital Technology Adoption Among Manufacturing Sector: Leadership Effectiveness as a Mediating Variable

  2023cites this paper
• How do Developers Really Feel About Bug Fixing? Directions for Automatic Program Repair

  2023cites this paper
• Software integration model: An assessment tool for global software development vendors

  2023cites this paper
• The Type to Take Out a Loan? A Study of Developer Personality and Technical Debt

  2023influential citation
• Developers’ Perception of GitHub Actions: A Survey Analysis

  2023cites this paper
• Post-pandemic Resilience of Hybrid Software Teams

  2023cites this paper
• Empirical research in software architecture - Perceptions of the community

  2023cites this paper
• Current Risk Situation Training in Industry, and Games as a Strategy for Playful, Engaging and Motivating Training

  2023cites this paper
• CharM - Evaluating a model for characterizing service-based architectures

  2023cites this paper
• Status Quo and Problems of Requirements Engineering for Machine Learning: Results from an International Survey

  2023cites this paper
• On Building an Instrument to Assess the Organizational Climate of Agile Software Development Teams

  2023cites this paper
• Impostor Phenomenon in Software Engineers

  2023influential citation
• Adaptação de Papéis Ágeis em Projetos Distribuídos em Larga Escala: Percepção de Profissionais da Indústria

  2023influential citation
• Analysing the Factors Influencing Digital Technology Adoption in Manufacturing Sectors: Leadership Effectiveness as a Mediator

  2022cites this paper
• Design Thinking and Non-Functional Requirements Elicitation: A Survey

  2022cites this paper
• Take a deep breath: Benefits of neuroplasticity practices for software developers and computer workers in a family of experiments

  2022cites this paper
• Explorando a especificação de cenários de teste de segurança no desenvolvimento orientado por comportamento (Behavior Driven Development)

  2022cites this paper
• Empirical Formal Methods: Guidelines for Performing Empirical Studies on Formal Methods

  2022cites this paper
• Lessons Learned to Improve the UX Practices in Agile Projects Involving Data Science and Process Automation

  2022cites this paper
• Agent Based Learning Environment for Survey Research

  2022cites this paper
• Using an Instant Visual and Text Based Feedback Tool to Teach Path Finding Algorithms: A Concept

  2021cites this paper
• Pattern Discovery and Validation Using Scientific Research Methods

  2021cites this paper
• Enterprise-Driven Open Source Software: A Case Study on Security Automation

  2021cites this paper
• "We do not appreciate being experimented on": Developer and Researcher Views on the Ethics of Experiments on Open-Source Projects

  2021cites this paper
• A Survey Research on Feedback Practices in Agile Software Development Teams

  2021cites this paper
• Lyzeli: a tool for identifying the clues in survey research data

  2021cites this paper
• Cataloging Dependency Injection Anti-patterns in Software Systems

  2021cites this paper
• Integration of Security Standards in DevOps Pipelines: An Industry Case Study

  2021cites this paper
• Risk Management Practices in Information Security: Exploring the Status Quo in the DACH Region

  2020cites this paper
• Factors leading to retention and organizational turnover in software engineering : a conceptual model

  2020cites this paper
• Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines

  2020cites this paper
• Behavioral Software Engineering: Methodological Introduction to Psychometrics

  2020cites this paper
• When to Update Systematic Literature Reviews in Software Engineering

  2020cites this paper
• Awareness of Secure Coding Guidelines in the Industry - A first data analysis

  2020cites this paper
• Sifu - a cybersecurity awareness platform with challenge assessment and intelligent coach

  2020cites this paper
• ACM SIGSOFT Empirical Standards

  2020cites this paper
• A methodology for psycho-biological assessment of stress in software engineering

  2020cites this paper
• Gender Differences in Personality Traits of Software Engineers

  2020cites this paper
• Having Fun Doing Research on Agile Methods

  2019cites this paper
• On Psychometric Instruments in Software Engineering Research: An Ongoing Study

  2018cites this paper
• Repositório ISCTE-IUL

  year unknowncites this paper