Authors (in alphabetical order): Maria Cruz, Shalini Kurapati, Yasemin Türkyilmaz-van der Velden
With contribution from workshop participants (in alphabetical order): Patrick Aerts (Netherlands eScience Center + DANS), Kees den Heijer (TU Delft), Jelle de Plaa (SRON), Jordi Domingo (KNMI), Martin Donnelly (University of Edinburgh), Raman Ganguly (University of Vienna), Rolf Hut (TU Delft), Karsten Kryger Hansen (Aalborg University), Carlos Martinez (Netherlands eScience center), Joakim Philipson (Stockholm University), Wessel Sloof (University Medical Center Groningen), Martijn Staats (Wageningen University & Research), Michael Svendsen (Royal Danish Library), Jan van der Ploeg (University Medical Center Groningen), Ronald van Haren (Netherlands eScience Center), Egbert Westerhof (DIFFER).
How to cite: A citable version of this report is available since July 06, 2018 through the Open Science Framework. DOI: 10.31219/osf.io/z48cm.
On 24 May 2018, Maria Cruz, Shalini Kurapati, and Yasemin Türkyilmaz-van der Velden led a workshop titled “Software Reproducibility: How to put it into practice?”, as part of the event Towards cultural change in data management – data stewardship in practice held at TU Delft, the Netherlands. There were 17 workshop participants, including researchers, data stewards, and research software engineers. Here we describe the rationale of the workshop, what happened on the day, key discussions and insights, and suggested next steps.
Rationale for the workshop
There is no denying about a reproducibility crisis in science. In some fields, over half of published studies fail reproducibility tests. A survey of 1576 scientists conducted by Nature in 2016 revealed that over 90% of the respondents agreed that there was some level of crisis and over 70% said they had tried and failed to reproduce another group’s experiments. Given the ubiquitousness of software in many areas of contemporary scientific research, it could be argued that there can’t be reproducibility in science without reproducible software.
In a recent Comment in Water Resources Research, in response to “Most computational hydrology is not reproducible, so is it really science?”, Hut, Van de Giesen and Drost (2017) argue that documenting and archiving code and data is not enough to guarantee the reproducibility of computational results. They suggest the use of software containers and open interfaces, and that researchers work more closely with research software engineers (RSEs) to learn best practices in software design. This advice is presented in the context of hydrology, but it could be applied more generally.
Inspired by the article and its advice, the workshop aimed to explore the various topics of software reproducibility— how some of the advice could be put in practice, and what role could institutions, data stewards, and research software engineers play in this regard.
What happened on the day
The workshop session lasted one hour. It started with the moderators introducing themselves, followed by a short survey of the audience using Mentimeter, led by Yasemin Türkyilmaz-van der Velden. Maria Cruz then gave a presentation setting the scene, providing information on reproducibility, and summarising the paper and the suggestions by Hut, Van de Giesen and Drost (2017). One of the authors of the paper, Rolf Hut, attended the session and also said a few words about his paper and his ideas. Shalini Kurapati then moderated the main activity described below.
Using Mentimeter, we asked a few questions to the audience to get familiar with their background and their experiences with research software. As seen in the responses below, there was an almost perfectly balanced audience formed by researchers, research software engineers, data stewards, and people in other research support positions.
There was also a very good balance in terms of the participants’ research backgrounds, which ranged from various disciplines in the physical sciences and medical research to intellectual history and information science. Almost all participants had experience with research software.
The majority (65%) of the participants agreed that there is a reproducibility crisis in science. The reproducibility crisis was a hot topic during the main event (Towards cultural change in data management – data stewardship in practice) and had been already discussed comprehensively earlier in the programme, by the keynote speaker Danny Kingsley. Therefore, a potential bias in the responses of the participants cannot be excluded. Regardless, it was interesting to see that there is an increasing awareness of this important issue.
Before moving to the presentation about software reproducibility, we asked the participants what came to their mind about this topic. The answers, which ranged from sustainability, preservation, and integrity to GitHub, Zenodo, containers, and Docker, clearly show that the audience was already very familiar with the topic of software reproducibility.
Prior to the workshop, we hoped to have a group of participants with diverse backgrounds and interests. Fortunately, that turned out to be the case, and we could form groups with the ideal representation from all stakeholders of interest. We divided the participants into 4 groups, each containing at least a data steward/research support staff, a research software engineer, and a researcher. The groups were invited to answer the following questions within a collaborative google document:
- What do you think about the advice of Hut, Van de Giesen and Drost, i.e., use containers (e.g. Docker), use open interfaces, and closely collaborate with Research Software Engineers to improve software reproducibility?
- Any additional advice to Hut et al., to improve software reproducibility?
- How can researchers, RSEs and data stewards work together towards implementing the above advice?
The groups were allotted 20 minutes to discuss answers to the questions and record them in the google document. The workshop moderators were able to actively monitor the google document to steer the groups towards timely conclusion of their activity. After the activity was concluded, a representative from each group pitched their activity summary and their key findings for a minute. The contents of the google document and the pitches, which were recorded live in the workshop slides, provide us the insights on the challenges and corresponding solutions for software sustainability and reproducibility, that are reported in the next section.
Key discussion points and insights on the advice by Hut, van de Giesen & Drost
Lack of funding for Research Software Engineers
Lack of (sustainable) funding for hiring RSEs is one of the obstacles to putting the advice of Hut, Van de Giesen and Drost into practice. Larger projects typically already have RSEs on board, but for smaller projects this is not always possible. It is difficult to recruit and hire RSEs across disciplines. However, the Netherlands eScience Center is a good example of a way to centrally fund research software development and to pool developer expertise across disciplines.
Open source software is not always an option
Because of scientific competition, commercial and IP interests, it is not always an option to make research software available as open source software. Dockers (containers) are also not an option for commercial software.
High-level documentation is very important. A good README file does part of the job, but documentation and a user manual are also important. Any information (e.g. equations, model) behind the software also needs to be shared.
Lack of support for software validation is also a problem. As an addition to the advice by Hut, van de Giesen and Drost, one of the groups suggested that support should also be provided for software validation (in-house code review). In cases where professional software support is limited, it would already be helpful if researchers would review each others’ code, just like they would do with papers. If the goal is to make code understandable to other researchers, then their feedback will be paramount. Organizing code reviews in a research group could improve the quality of the code significantly with only a small time investment.
The role of data stewards, RSEs and researchers
Data stewards – the link between researchers and RSEs?
Two groups saw the role of data stewards as brokers between researchers and RSEs. It was acknowledged that researchers and RSEs should interact more to improve research codes (e.g. review of codes). Data stewards could be the link between the two. Data stewards could monitor possible synergy between projects and link researchers with specialist RSE expertise. One group felt that data stewards should provide the toolbox, with principles (e.g FAIR principles) and guidance, and RSEs should help implement those principles, because they have the knowledge to do so.
Could RSEs do more to promote best practices?
Two groups thought that RSEs could take a more proactive role in providing training for researchers, promoting best practices, and generally propagating their knowledge. Without assigning roles, one of the groups felt that implementing the advice of Hut, Van de Giesen and Drost required programming courses, support staff to help out researchers at departmental level, and the breakdown of problems into smaller problems that could be solved with up-to-date techniques based on expert knowledge. Could RSEs also help with this?
Opportunities and barriers, and the role of institutions
Integrated teams working across university faculties, departments, and institutes, with a single point of contact, could provide a way for researchers, data stewards, and RSEs to work together. Fear of stepping into others’ “working areas” and different working cultures may create barriers, as well as the potential lack of scientific/research expertise from RSEs and software developers.
Sustainable funding is a challenge, so is the lack of recognition for developing research software in the current academic rewards system. There also needs to be a persuasive driver beyond just doing the right thing. This can come from funders, publishers and possibly institutions. Any driver will be most persuasive when it comes from the research community itself.
Universities and institutes should promote good practice for software engineering as part of open science.
The short-term goal of this workshop was to start a conversation on the topic of software reproducibility between researchers, research support staff (data stewards and others with a similar role), and research software engineers, and to make the results of this discussion public via this forum.
The immediate next step is to bring the results of this interaction to the attention of the community Working towards Sustainable Software for Science: Practice and Experiences (WSSSPE), which includes researchers and research software engineers, but lacks a strong connection with data stewards and research data support. We plan to submit a paper for the 9th International Workshop on Sustainable Software for Science: Practice and Experiences, to be held in Amsterdam on 29 October 2018.
The time available for the workshop was limited and not all the issues were discussed or discussed in enough depth. For example, it would be interesting to discuss in more detail what training and resources researchers and research supports need to help software reproducibility become more of a reality and what role could data stewards and research software engineers play in this regard.
Institutions could certainly do more in terms of funding and rewards for software development, and promoting best practices. How to make this happen in a global and concerted manner?
In the long-term we will continue to engage with the necessary stakeholders to keep the discussion alive and to define operational solutions towards improving software reproducibility and sustainability.
- Workshop slides
- Water Resources Research paper by Hut, van de Giesen & Drost, in response to “Most computational hydrology is not reproducible, so is it really science?”
- Participants’ contribution through collaborative google document
The TU Delft Library website now hosts a webpage with information on the process the Library has initiated to get a better understanding of the ideas and needs regarding open source software at TU Delft.
One aspect of this initiative was a series of sandbox sessions, “open to all who want to contribute to the dialogue to bring together a community with the purpose of engaging on open source software.”
The presentations from the sandbox are now available on the new Open Source Software webpage.
The 4TU.Centre for Research Data announces its report on research data management within the 4TU Research Centres.
Over the last few months, the 4TU.Centre for Research Data had the chance to make contact and to speak with several of the Scientific Directors of the 4TU Research Centres about research data management. The report published today highlights the findings from these contacts and conversations.
A citable version of the report is available on OSF Preprints (DOI: 10.17605/OSF.IO/SGFTW).
1. Research data management is not addressed at a strategic level by the 4TU Research Centres, but left to individual research groups or to individual researchers connected to the Centres.
2. Within the 4TU Research Centres, there is a broad range of attitudes towards data and a broad range of data types and characteristics, including large datasets; commercially sensitive datasets; privacy and ethical concerns regarding data; software and its sustainability.
3. Software sustainability is an important and much discussed topic, but there are currently no standards or systematic way of looking after software.
4. Research on human subjects and datasets including personally identifiable information or sensitive personal information are more prominent than might be expected in engineering and the technical sciences. Lack of transparency and reproducibility of scientific results can be an issue in these areas because the underlying datasets are often not available.
An Opportunity to Collaborate
Research data management is increasingly viewed as an important part of high-quality research. International and national funding bodies now mandate institutions and researchers to make data available. Data sharing is predicated on good research data management and has the potential to make scientific research more transparent, open, and efficient. In view of these principles and developments, the 4TU.Centre for Research Data wishes to maintain and deepen its links with the 4TU Research Centres and to support the Centres in various aspects of research data management.
The framework is entitled “Impact for a better society” and “openness” is listed as one of the four major guiding principles. The principle of openness was apparent already during the consultation phase of the framework: “more than 600 internal and external stakeholders have been actively participating” in the process.
The purpose of the strategic framework is “to serve as a high-level compass that will guide decision-making bodies at all levels within our university in the years ahead”. But what does the framework really mean for Open Science? In this blog post, I highlighted the key quotations from the strategic framework which are likely to have the highest impact on future Open Science developments at TU Delft.
Impact for a better society
First, Open Science fits neatly with the overall title of the framework “Impact for a better society”. The framework states in the preface that “societal impact and academic excellence can be mutually reinforcing”. And this is indeed the case. Open Science means that research results can be accessed and re-used by everyone in the society, including the members of the public. TU Delft also wishes to increase its societal engagement by “promoting public participation in scientific research (‘citizen science’). Which is all deeply in line with the principles of Open Science.
Open Access publishing
Within Open Access publishing, TU Delft wishes to first develop a stronger awareness among its researchers. Second, the strategic framework also emphasises the need for a sustainable transition to Open Access publishing and it thus includes the commitment to “reducing costs for Open Access publishing by negotiating journal subscriptions with publishers.” At the same time, TU Delft will explore “new ways to present and disseminate knowledge”, which will not necessarily rely on publishing via the traditional scientific journals. Finally, researchers are encouraged “to serve on relevant Editorial Boards”, suggesting that TU Delft researchers take an active part in shaping publishers’ policies.
The importance of good data management and sharing is also stipulated in the strategic framework. TU Delft wishes to stimulate the sharing of research data, and it realises that in order to achieve this, researchers need to be provided “with the necessary support, for example by appointing data stewards and data engineers within all faculties who advise researchers in managing their data.”
In addition, TU Delft will implement a “policy for research data, and enable researchers to control their own research data in accordance with this policy.” And, quite importantly, the strategic framework states that TU Delft wants to “involve researchers in contributing to TU Delft’s policy for research data management.”
Finally, the strategic framework recognises the importance of the new EU General Data Protection Regulation, and will “set up an integrity policy that protects scientific data and personal data in line with the EU directives.”
Software is an integral part of research and is necessary for research reproducibility. It is therefore not surprising that the commitment to open source software has been stated in several locations in the strategic framework. First, TU Delft will develop “best practices for working with open source software, for example in relation to copyright and archiving of source code” and “facilitate a central place of support for researchers who want to use open source software.” Furthermore, TU Delft stresses the importance of communities in raising awareness and reinforcing good practice. It will, therefore, create “an open source software community with active ambassadors.”
Rewards for Open Science
The Strategic Framework is aiming at recognising the engagement with Open Science by changing the ways in which researchers are evaluated. TU Delft wants to include a more explicit recognition of “engagement with Open Science and Open Education” in yearly R&O evaluation cycles. To facilitate this, TU Delft supports “(inter) national initiatives aimed at finding alternative indicators that positively value open access publications” and is “collaborating with (inter)national leaders in the field of non-traditional metrics.”
Supporting researchers in their transition to Open Science
Importantly, TU Delft recognises that researchers need to be professionally supported in order to ensure that the objectives of the strategic framework can be successfully met. Therefore, it aims to “improve the quality of [its] professional services” and wants to provide researchers with a clear, ‘one-stop-shop’ contacts for requests which should be “simple and effective”, “digital where possible, and personalised where needed”.
TU Delft also plans to appropriately recognise and reward those supporting researchers in their transition to Open Science. TU Delft will “take the lead in national initiatives aimed at extending the job classification for support staff with positions that support recent developments, such as data stewards that advise researchers in managing their (open) research data”.
Strategy for Open Education was also widely mentioned in the framework. The one-page summary outlines TU Delft’s commitment to “promote and facilitate Open Education”, which is then followed by a declaration: “we wholeheartedly support Open Education and want to make Open Educational Resources part of our educational policy”. To achieve this, TU Delft will support lecturers and students in the use of open education resources and will encourage “lecturers to publish their educational material under an open license”
Importantly, TU Delft also wishes to appropriately reward those engaged in Open Education activities. It wishes to strengthen a culture “in which education and teaching receive more appreciation and recognition” and “will refine [its] HR policy so that it will offer further scope for professional development and career opportunities within education”. In addition, as part of its educational policy, TU Delft wants to make “open education part of the basic teaching qualification programme and the evaluation criteria of courses.”
Last, the framework also states that TU Delft has the ambition to replace “commercial textbooks by open resources in all BSc programmes as much as possible.”
How important is the strategic framework?
So how important is the framework? Will the statements be really implemented?
To answer these questions I will conclude with the final quotation from the framework: “this framework is more than a formal requirement; it is our moral responsibility”.
As part of the TU Delft Open Science initiative, the Library together with ICT are looking at the issues related to open source software created by researchers at TU Delft – sustainability, career recognition, training, archiving, licensing and copyright. As part of the process that will help deliver a strategic framework for Open Science at TU Delft, we have been interviewing researchers about their ideas and needs regarding open source software. The interview with Hugo Ledoux, an associate-professor in the 3D geoinformation research group, part of the Department of Urbanism of the Faculty of Architecture & the Built Environment, has been published in the last edition of the Library Online Magazine.
On Thursday 26 October, TU Deft Library together with ICT hosted the third, but possibly not the last, of a series of Innovation Sandbox Sessions on Open Source Software at TU Delft. The topic of the session was training and support.
There were three very interesting presentations and lots of engagement from the audience.
- Carlos Martinez Ortiz – Netherlands eScience Center. He talked about open source software and software sustainability.
- Julian Kooij – TU Delft, Assistant Professor “Visual Sensing and Learning” in the Intelligent Vehicles group, part of the Biomechanical Engineering department, 3mE faculty. Julian talked about the use of Gitlab (and Robot Operating System, ROS) in the Prius Demonstrator vehicle.
- Rob van Laarhoven – TU Delft, Manager of the Data Management department, ICT. Rob talked about the same Gitlab project and broader ambition of setting up a TU Delft-wide Gitlab.
How to keep research software alive?
This discussion caught my attention. Software decays over time because it depends on other code or technology (operating systems, browsers, etc.) that change over time. To keep up with these changes, software needs to be maintained and updated, but this takes time, skill and resources.
10 Ways to keep your successful scientific software alive
This is the title of a blogpost from Vincent van Hees, an eScience Research Engineer at The Netherlands eScience Center. One of his recommendations is to build a community of developers. Building a large community may only be possible for generic research software, but it may still be worth the effort for more domain-specific pieces of software if that leads to a reduction of the maintenance work load.
There is no magic recipe for how to build a community. The Netherlands Research Software Engineer community is a recent initiative “to bring together the community of research software engineers from Dutch universities, knowledge institutes, companies and other relevant organizations to share knowledge, to organize meetings and raise awareness for the scientific recognition of research software.”
ICT and the Library are once again jointly running a session on Software as Research Output at TU Delft.
The session will take place in the Blue Room, in the TU Delft Library, and it will run from 12:15 to approximately 13:30 on Thursday 26 October. A light lunch is included.
ICT conducted research on the issues related to open source software at TU Delft – sustainability, career recognition, training, archiving, open source vs closed source, licencing and copyright.
The idea is to continue the dialogue, focusing on the topics and actions that are considered to be the most important and areas that could be collaboratively worked on further.
In this session, the third and final of a series of Innovation Sandbox Sessions on Open Source Software at TU Delft, we’ll be exploring: Software as output of the researcher – training and support for researchers.
The session is open, so feel free to share the invitation with your colleagues. If you want to come please email Julie Beardsell (J.A.Beardsell *AT* tudelft.nl), so we know how many to cater for.