What does reproducibility mean for qualitative research?
Written by Shalini Kurapati and Marta Teperek
Sebastian Karcher of Qualitative Data Repository was at TU Delft on Jan 28 to share his thoughts and expertise on this topic in a two-part seminar organised by the 4TU.Centre for Research Data. Given that we are based at the Delft University of Technology, where qualitative research is not mainstream, we were positively surprised to see a lot of interest in the seminars and that the talk was attended by around 60 participants. The majority of researchers were from the faculty of Technology Policy and Management followed by Industrial Design Engineering. The second part was a hands-on workshop on qualitative data management techniques, which was attended by 20 participants.
Limits of Reproducibility
During the opening talk “Limits of Reproducibility: Strategies for Transparent Qualitative Research”, Sebastian raised the issue of terminology problems surrounding reproducibility discussions. To clarify, he presented lucid definitions of the various terms that are often used interchangeably.
- Reproducibility: Using same data and methods to produce same results
- Replicability: Using same methods, different data/sample and arriving at same results
- Transparency: Providing all necessary information to evaluate a study
Sebastian argued that reproducibility and replicability are based on a positivist view of science (e.g. testing a hypothesis in a lab), whereas qualitative research may follow interpretive research methods where the study conditions are quite impossible to recreate. Therefore, Sebastian suggested that the rigour and quality of qualitative research should not be judged based on reproducibility or replicability, but by the transparency of the research process.
He distinguished three types of transparency most relevant in qualitative research:
- Production transparency: Information on how data are collected or generated, such as research questions, sampling, subject recruitment; this information should be recorded following best practices in the research field.
- Analytic transparency: Documentation process of qualitative data preparation and analysis leading to the conclusions of a study.
- Data access: Considerations with regards to data sharing and access conditions.
However, Sebastian stressed that the idea of using transparency as a way to evaluate the rigour of qualitative research is only the beginning of a debate and not a solution for all aspects of qualitative research evaluation. In addition, following the path of transparency is not always straightforward. For example, data preparation, data anonymisation and providing access to data can be costly (both in terms of time and resources). In addition, there might be ethical and legal challenges with respect to privacy regulations and additional copyright implications of textual data. Nonetheless, Sebastian was confident that thanks to improved technology, existence of instruments like data management plans and raising awareness of best practices, these problems can be overcome.
Managing Qualitative Data for Sharing and Transparency
In the second part of the seminar, Sebastian focussed on practical issues related to de-identification of personal data, access controls for data and informed consent forms.
De-identification of qualitative data is highly dependent on research questions and context. For example, knowledge of a specific language might influence the level of difficulty in de-identification. In addition, local circumstances need to be understood in order to know what gives away the data subject’s identity. Using a practical exercise based on a politicians’ profile, Sebastian asked the participants to de-identify personal data, while also thinking about minimising the loss of richness of the information provided. The final results were truly interesting since different groups de-identified the same text in a myriad of ways, and they was no model answer. An ensuing discussion allowed participants to reflect on the considerations they made while de-identification.
An ensuing exercise was to understand the guidelines on writing clear and informative text in consent forms. Getting consent right is important to make sure that data collection is efficient, compliant to regulations, and that there are considerations of a potential reuse of research data. It is also important for researchers to negotiate with ethics committees on the text of the informed consent and sometimes receive advice form them to avoid risk averse decisions by the committees.
Before retiring to questions from audience, Sebastian spoke on the importance of awareness of access controls for sharing qualitative data which often contains personal information. Therefore, despite the fact that research data contains information which could potentially disclose individuals, their identity can be still protected through access controls. Sebastian mentioned some examples of access control to research data which can be offered by research repositories:
- Conditional online access – data can be accessed online, but only by registered users, who in addition might be required to fulfil some obligations, e.g. have ethics approval or be affiliated with research institutions.
- Depositor-approved access – data can be accessed online by registered users, who are in addition approved by the depositor. Sebastian advised caution with relying on this approach to data access, given that these routes might not be sustainable long-term
- Offline access – datasets can be accessed by registered users at a secure location, on a computer with no network access
- Embargoed data – datasets can be only accessed after a certain date/after certain amount of time past/certain even happen – conditions need to be specified and justified
Summary
The take-home message of the workshop was that there is no one size fits all solution when it comes to responsible working with qualitative research data – contextual information is often key. That said, in all cases, transparent documentation of all the datasets and processes are very important, as well as getting appropriate consent and ensuring adequate access controls to data.
Materials
Sebastian’s talk, as well as all the workshop materials are publicly available:
Talk on Limits of Reproducibility: Strategies for Transparent Qualitative Research
Workshop on Managing Qualitative Data for Sharing and Transparency
Open Working 