rescience / rescience-article Goto Github PK

Hi all,

I've uploaded a very first and incomplete draft but I won't have much time to work further on it until 2 to 3 weeks. If anyone feel to work on it...

Introduction

• The replication crisis today
  → Medicine, Biomedical, Psychology, Political Sciences, etc.
• Computational science is no exception
  → Software is still a second-class citizen in Science
  → Missing/unavailable code, not compilable, not replicable, etc.
• Pre-publication solutions (for new work)
  → Good practices, notebooks, active formats, virtual containers etc.
• Post-publication solutions (for old work)
  → Mostly no solution but things are starting to change

Motivation

• Use-cases (see #1)
  → J. Stachelek
  → N. Rougier
  → B. Girard
• ReRun, Repeatable, Replicable, Repoducible, Reusable or Remixable ?
  → ReRun (variation on experiment and set-up)
  → Repeatable (same experiment, same set-up, same lab)
  → Replicable (same experiment, same set-up, independent lab)
  → Reproducible (variations on experiments, on setup, independent labs)
  → Reusable (different experiment)
  → Remixable ()
• Reproducibility criterion
  → Quantitative
  → Qualitative
  → Other ?

Editorial process

(see http://rescience.github.io/write/ and http://rescience.github.io/read/)

• The editorial board
• Submission
  → Code
  → Article
  → Data
• Review
  → Public
  → Interactive
• Edition (criterion for accept & reject)
• Publication (Github / Zenodo)

Conclusion

• The added value is the article, not the code
  → Original article + ReScience article should be sufficient for future replication
• What about failed replication ?
  → See http://rescience.github.io/faq/
• Expanding the model (the CoScience journal)
  → Instead of post-reproduction, publish articles including independent replication

Unsorted references

• Writing Software Specifications K. Hinsen, Computing in Science and Engineering, 2015.
• A long journey into reproducible computational neuroscience M. Topalidou, A. Leblois, T. Boraud and N.P. Rougier, Frontiers in Computational Neuroscience, 2015.
• Computational science: shifting the focus from tools to models K. Hinsen, F1000Research, 2014.
• Best Practices for Scientific Computing Greg Wilson , D. A. Aruliah, C. Titus Brown, Neil P. Chue Hong, Matt Davis, Richard T. Guy, Steven H. D. Haddock, Kathryn D. Huff, Ian M. Mitchell, Mark D. Plumbley, Ben Waugh, Ethan P. White, Paul Wilson, PLOS Biology, 2014.
• Learning from the Past: Approaches for Reproducibility in Computational Neuroscience S.M. Crook, A.P. Davison, H.E. Plesser, 20 Years of Computational Neuroscience, 2013
• Four aspects to make science open “by design” and not as an after-thought Y.O. Halchenko, M.Hanke, GigaScience, 2015.

Replicability vs. reproducibility

• Replicability vs. reproducibility — or is it the other way around? Mark Liberman (2015)
• Open code for open science? Steve M. Easterbrook, Nature Geoscience 7, 779–781 (2014)

Similar initiative

• Push journal - Research & applied theory in writing with sources
• IPOL Journal - Image Processing On Line
• PSRI - Political Science Replication Initiative

• Associate editors could bring useful testimony on the practice in their field regarding reproducibility problem and the extent of the problem. How are replication considered ? valuable ? Is there any place where such reproduction/replication can be published, etc.
• Active reviewers could explain the whole review process and how different it is from regular review, how different is the public interaction with the author, how hard it is to review code, etc.
• Authors could explain what is the initial motivation for replicating results of a paper. Did they already replicated some results before ReScience and what did they do with such implementation? Was the review useful at all ? Did they get new interaction following the publication? etc.

Wish list (mine):

• Open access
• Open reviewing
• Low cost
• read by computational scientists from all fields

I don't think any journal satisfies all criteria.

Candidates (please suggest more!):

• F1000Research

  Open access, open review, and a good topic section (publishing, education, communication), but also a serious price tag (500 or 1000 US$ depending on length)

• Computing in Science & Engineering

  Focused on computational science, no page charges, but no open access either
  (not even for money), nor open reviewing.
  There are two channels: peer-reviewed articles, and department contributions that are evaluated
  by the department editors. Disclaimer: I am department editor for "Scientific Programming".

• PeerJ Computer Science

  In spite of the name there are many articles on computational science as well. Open access
  but not open reviewing. High page charges (695 US$).

The idea is to converge on definition of these words according to our respective scientific domains. At this point it is not even sure that all words are relevant to all domains and this may also depend on the kind of software we consider (see discussion in #4)

Rerunable:

Repeatable:

Replicable:

Reproducible:

Reusable:

Remixable:

Reimplementable: