ABSTRACT Scientific communication relies on evidence that cannot be entirely included in publications, but the rise of computational science has added a new layer of inaccessibility.

Although it is now accepted that data should be made available on request, the current regulations regarding the availability of software are inconsistent. We argue that, with some

exceptions, anything less than the release of source programs is intolerable for results that depend on computation. The vagaries of hardware, software and natural language will always

ensure that exact reproducibility remains uncertain, but withholding code increases the chances that efforts to reproduce results will fail. Access through your institution Buy or subscribe

calculated during checkout ADDITIONAL ACCESS OPTIONS: * Log in * Learn about institutional subscriptions * Read our FAQs * Contact customer support SIMILAR CONTENT BEING VIEWED BY OTHERS

ENCORE: A PRACTICAL IMPLEMENTATION TO IMPROVE REPRODUCIBILITY AND TRANSPARENCY OF COMPUTATIONAL RESEARCH Article Open access 16 September 2024 TOWARD PRACTICAL TRANSPARENT VERIFIABLE AND

LONG-TERM REPRODUCIBLE RESEARCH USING GUIX Article Open access 04 October 2022 JULIA FOR BIOLOGISTS Article 06 April 2023 REFERENCES * Schwab, M., Karrenbach, M. & Claerbout, J. Making

scientific computations reproducible. _Comput. Sci. Eng._ 2, 61–67 (2000) Article  Google Scholar  * Barnes, N. Publish your computer code: it is good enough. _Nature_ 467, 753 (2010)

Article  CAS  Google Scholar  * McCafferty, D. Should code be released? _Commun. ACM_ 53, 16–17 (2010) Article  Google Scholar  * Merali, Z. …Error. _Nature_ 467, 775–777 (2010) Article  CAS

  Google Scholar  * Hanson, B., Sugden, A. & Alberts, B. Making data maximally available. _Science_ 331, 649 (2011) Article  CAS  ADS  Google Scholar  * Peng, R. D. Reproducible research

and biostatistics. _Biostatistics_ 10, 405–408 (2009)THIS WORK PROVIDES A SUCCINCT AND CONVINCING ARGUMENT FOR REPRODUCIBILITY— _BIOSTATISTICS_ IS AT THE FOREFRONT OF ENSURING THAT CODE AND

DATA ARE PROVIDED FOR OTHER RESEARCHERS. Article  Google Scholar  * Devil in the details. _Nature_ 470, 305–306 (2011) * Pedersen, T. Empiricism is not a matter of faith. _Comput.

Linguist._ 34, 465–470 (2008) Article  Google Scholar  * Donoho, D. L. An invitation to reproducible computational research. _Biostatistics_ 11, 385–388 (2010) Article  Google Scholar  *

Raymond, E. S. _The Cathedral and the Bazaar_ (O’Reilly, 2001) Google Scholar  * He, Y. & Ding, C. H. Q. Using accurate arithmetics to improve numerical reproducibility and stability in

parallel applications. _J. Supercomput._ 18, 259–277 (2001) Article  Google Scholar  * Hatton, L. et al. The seismic kernel system—a large scale exercise in Fortran 77 portability. _Softw.

Pract. Exper._ 18, 301–329 (1988) Article  ADS  Google Scholar  * Hatton, L. The T experiments: errors in scientific software. _IEEE Comput. Sci. Eng._ 4, 27–38 (1997) Article  Google

Scholar  * Hey, T. Tansley, S. Tolle, K., eds. _The Fourth Paradigm: Data-Intensive Scientific Discovery_ (Microsoft Press, 2009) * Gervasi, V. & Zowghi, D. On the role of ambiguity in

RE. In _Requirements Engineering: Foundation for Software Quality_ (eds Wieringa, R. & Persson, A ) 248–254 (Springer, 2010) Book  Google Scholar  * van Deemter, K. _Not Exactly: In

Praise of Vagueness_ (Oxford University Press, 2010) Google Scholar  * Yang, H., Willis, A., De Roeck, A. & Nuseibeh, B. Automatic detection of nocuous coordination ambiguities in

natural language requirements. In _Proc. IEEE/ACM Intl Conf. on Automated Software Engineering_ 53–62 (ACM, 2010) Book  Google Scholar  * de Bruijn, F. & Dekkers, H. L. Ambiguity in

natural language software requirements: a case study. In _Requirements Engineering: Foundation for Software Quality_ (eds Wieringa, R, & Persson, A., ) 233–247 (Springer, 2010) Book 

Google Scholar  * Pfleeger, S. L. & Hatton, L. Investigating the influence of formal methods. _IEEE Computer_ 30, 33–43 (1997) Article  Google Scholar  * van der Meulen, M. J. P. &

Revilla, M. A. The effectiveness of software diversity in a large population of programs. _IEEE Trans. Softw. Eng._ 34, 753–764 (2008) Article  Google Scholar  * Boehm, B., Rombach, H. D.,

Zelkowitz, M. V., eds. _Foundations of Empirical Software Engineering: the Legacy of Victor R. Basili_ (Springer, 2005) * Monniaux, D. The pitfalls of verifying floating-point computations.

_ACM Trans._ _ Programming Languages Systems_ 30 (3). 1–41 (2008) * Thomas, S. J., Hacker, J. P., Desagne, M. & Stull, R. B. An ensemble analysis of forecast errors related to floating

point performance. _Weather Forecast._ 17, 898–906 (2002) Article  ADS  Google Scholar  * Revol, N. Standardized interval arithmetic and interval arithmetic used in libraries. _Proc._ _ 3rd

Intl Congr._ _ Mathematical Software_ 337–341. (2010) * Rump, S. M., Ogita, T. & Oishi, S. Accurate floating point summation. Part 1: Faithful rounding. _SIAM J. Sci. Comput._ 31,

189–224 (2009) Article  Google Scholar  * Rump, S. M. Accurate and reliable computing in floating-point arithmetic. _Proc._ _ 3rd Intl Congr._ _ Mathematical Software_ 105–108. (2010) *

Badin, M., Bic, L. & Dillencourt, M. &. Nicolau, A. Improving accuracy for matrix multiplications on GPUs. _Sci. Prog._ 19, 3–11 (2011) Google Scholar  * Pan, V. Y., Murphy, B.,

Qian, G. & Rosholt, R. E. A new error-free floating-point summation algorithm. _Comput. Math. Appl._ 57, 560–564 (2009) Article  MathSciNet  Google Scholar  * Boldo, S. & Muller,

J.-M. Exact and approximated error of the FMA. _IEEE Trans. Comput._ 60, 157–164 (2011) Article  MathSciNet  Google Scholar  * Kahan, W. Desperately needed remedies for the undebuggability

of large floating-point computations in science and engineering. _IFIP/SIAM/NIST Working Conf._ _ Uncertainty Quantification in Scientific Computing_ (Springer, in the press). * Mesirov, J.

P. Accessible reproducible research. _Science_ 327, 415–416 (2010) Article  CAS  Google Scholar  * Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M. & Stodden, V. Reproducible

research in computational harmonic analysis. _Comput. Sci. Eng._ 11, 8–18 (2009)DONOHO AND FELLOW RESEARCHERS HAVE BEEN AT THE FOREFRONT OF REPRODUCIBILITY FOR MANY YEARS; THIS ARTICLE

REVIEWS THEIR WORK, INCLUDING FACILITIES FOR CODE PRESENTATION. Article  Google Scholar  * Yool, A., Popova, E. E. & Anderson, T. R. Medusa-1.0: a new intermediate complexity plankton

ecosystem model for the global domain. _Geosci. Model Develop._ 4, 381–417 (2011)AN EXAMPLE OF AN ARTICLE FROM A JOURNAL THAT ASKS FOR CODE RELEASE FOR MODEL DESCRIPTION PAPERS AND

ENCOURAGES CODE RELEASE FOR OTHER CATEGORIES OF PAPER. Article  ADS  Google Scholar  * Easterbrook, S. M. & Johns, T. C. Engineering the software for understanding climate change.

_Comput. Sci. Eng._ 11, 65–74 (2009) Article  Google Scholar  * Brohan, P., Kennedy, J. J., Harris, I., Tett, S. F. B. & Jones, P. D. Uncertainty estimates in regional and global

observed temperature changes: a new dataset from 1850. _J. Geophys. Res._ 111, D12106 (2006) Article  ADS  Google Scholar  * Adams, E. N. Optimizing preventive service of software products.

_IBM J. Res. Develop._ 28, 2–14 (1984) Article  Google Scholar  * Hatton, L. & Roberts, A. How accurate is scientific software? _IEEE Trans. Softw. Eng._ 20, 785–797 (1994) Article 

Google Scholar  Download references ACKNOWLEDGEMENTS We thank D. Hales of the Department of Design at the Open University and P. Piwak of the Department of Computing for pointing out some

reproducibility work outside computing. J.G.-C. is grateful to I. Goz for pointing out the calculation errors in the CRUTEM data from the Meteorological Office. AUTHOR INFORMATION AUTHORS

AND AFFILIATIONS * Department of Computing Open University, Walton Hall, Milton Keynes MK7 6AA, UK, Darrel C. Ince * School of Computing and Information Systems, Kingston University,

Kingston KT1 2EE, UK, Leslie Hatton * 83 Victoria Street, London SW1H 0HW, UK, John Graham-Cumming Authors * Darrel C. Ince View author publications You can also search for this author

inPubMed Google Scholar * Leslie Hatton View author publications You can also search for this author inPubMed Google Scholar * John Graham-Cumming View author publications You can also

search for this author inPubMed Google Scholar CONTRIBUTIONS D.C.I., L.H. and J.G.-C. contributed to all aspects of this article. CORRESPONDING AUTHOR Correspondence to Darrel C. Ince.

ETHICS DECLARATIONS COMPETING INTERESTS The authors declare no competing financial interests. RIGHTS AND PERMISSIONS Reprints and permissions ABOUT THIS ARTICLE CITE THIS ARTICLE Ince, D.,

Hatton, L. & Graham-Cumming, J. The case for open computer programs. _Nature_ 482, 485–488 (2012). https://doi.org/10.1038/nature10836 Download citation * Received: 09 May 2011 *

Accepted: 05 January 2012 * Published: 22 February 2012 * Issue Date: 23 February 2012 * DOI: https://doi.org/10.1038/nature10836 SHARE THIS ARTICLE Anyone you share the following link with

will be able to read this content: Get shareable link Sorry, a shareable link is not currently available for this article. Copy to clipboard Provided by the Springer Nature SharedIt

content-sharing initiative