Abstract: Survey Research
While being an important and often used research method, survey research has been less often discussed on a methodological level in empirical software engineering than other types of research. This chapter compiles a set of important and challenging issues in survey research based on experiences with several large-scale international surveys.
The chapter covers theory building, sampling, invitation, and follow-up, statistical as well as qualitative analysis of survey data and the usage of psychometric in software engineering surveys.
Introduction
Empirical software engineering started with a strong focus on controlled experiments. It widened only later to case studies and similar research methods. Both methodologies have been discussed extensively for software engineering (Wohlin et al. 2012, Runeson et al. 2012). While survey research has been used to capture a broader sample for mostly cross-sectional studies, methodological issues have rarely been discussed.
The aim of this chapter is to complement existing more general literature on survey research and questionnaire design as well as the existing software engineering specific literature. Therefore, this is not a tutorial to survey research, but it provides a compact description of important issues and lessons learned that any empirical software engineering research can make use of in their next surveys. To not only discuss pure methodology and theory, but we also provide concrete examples of our experiences with the methodologies based on two lines of survey research:
First, the project Naming the Pain in requirements engineering (NaPiRE) has the goal to provide an empirical basis for requirements engineering research by capturing the state of the practice and current problems and challenges with requirements engineering. We have already made three rounds of surveys in this project over seven years and over ten countries. In these, we developed a theory as the basis for the questionnaire, several variations on questions for similar concepts and also experimented with different methodological options (Méndez Fernández & Wagner 2015, Wagner et al. 2019, Méndez Fernández et al. 2017). We will discuss these variations and experiences in the following.
We complement the NaPiRE experiences with a study that aimed to assess the happiness of software developers and targeted GitHub developers with a psychometrically validated test (Graziotin & Fagerholm 2019, Graziotin et al. 2018, 2017).
This example, described in Section 7.3 is different in the target population and how the questionnaire was created. Hence, it gives us even more possibilities to discuss. The chapter is organized so that we discuss different areas that we consider interesting and challenging. We start with a discussion on how survey research can be integrated with theory building, then explain what we need to consider when using psychometric tests in our questionnaires and why we need to consider psychometric properties. We then discuss the limited possibilities in evaluating the sample of a survey study including a short discussion of ethics. We continue with the closely related issue of how and whom to invite to a survey and how to manage follow-ups. The last two sections discuss issues in quantitative statistical and qualitative analysis of the data from a survey.
Conclusion
Survey research is becoming a more and more elementary tool in empirical software engineering as it allows us to capture cross-sectional snapshots of current states of practice, i.e. they allow us to describe and explain contemporary phenomena in practice (e.g. opinions, beliefs, or experiences). Survey research is indeed a powerful method and its wide adoption in the software engineering community is also steered, we believe, by the prejudice of that, it is easy to employ while there exist, in fact, many non-trivial pitfalls that render survey research cumbersome.
In response to this problem, the community has started to contribute hands-on experiences and lessons learned contributions, such as by Torchiano et al. (2017).
In this chapter, we have complemented existing literature on challenges in survey research by discussing more advanced topics. Those topics range from how to use survey research to build and evaluate scientific theories oversampling and subject invitation strategies to data analysis topics considering both quantitative and
30 Stefan Wagner et al. qualitative data, and we complemented it with specialized use cases such as using surveys for psychometric studies. To this end, we drew from our experiences in running a globally distributed and bi-yearly replicated family of large-scale surveys in requirements engineering. While we are certainly aware that our own journey in learning from our own mistakes and slips is not done yet. We hope that by reporting
And discussing these lessons we learned over the past years, we already support other members of our research community in further improving their own survey projects.
Acknowledgments:
We are grateful to all collaborating researchers in the NaPiRE initiative.
About KSRA
The Kavian Scientific Research Association (KSRA) is a non-profit research organization to provide research / educational services in December 2013. The members of the community had formed a virtual group on the Viber social network. The core of the Kavian Scientific Association was formed with these members as founders. These individuals, led by Professor Siavosh Kaviani, decided to launch a scientific / research association with an emphasis on education.
KSRA research association, as a non-profit research firm, is committed to providing research services in the field of knowledge. The main beneficiaries of this association are public or private knowledge-based companies, students, researchers, researchers, professors, universities, and industrial and semi-industrial centers around the world.
Our main services Based on Education for all Spectrum people in the world. We want to make an integration between researches and educations. We believe education is the main right of Human beings. So our services should be concentrated on inclusive education.
The KSRA team partners with local under-served communities around the world to improve the access to and quality of knowledge based on education, amplify and augment learning programs where they exist, and create new opportunities for e-learning where traditional education systems are lacking or non-existent.
FULL Paper PDF file:
Challenges in Survey ResearchStefan Wagner
University of Stuttgart, Stuttgart, Germany, e-mail: [email protected]
Daniel Mendez
Technical University of Munich, Munich, Germany
Blekinge Institute of Technology, Karlskrona, Sweden
fortiss GmbH, Munich, Germany
ORCID: 0000-0003-0619-6027 e-mail: [email protected]
Michael Felderer
University of Innsbruck, Innsbruck, Austria
Blekinge Institute of Technology, Karlskrona, Sweden
e-mail: [email protected]
Daniel Graziotin
University of Stuttgart, Stuttgart, Germany, e-mail: [email protected]
Marcos Kalinowski
Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro, Brazil, e-mail: [email protected]
https://arxiv.org/pdf/1908.05899.pdf
Professor Siavosh Kaviani was born in 1961 in Tehran. He had a professorship. He holds a Ph.D. in Software Engineering from the QL University of Software Development Methodology and an honorary Ph.D. from the University of Chelsea.
