PLoS ONE plos plosone PLOS ONE 1932-6203 Public Library of Science San Francisco, CA USA 10.1371/journal.pone.0289171 PONE-D-23-11089 Research Article Research and analysis methodsResearch facilitiesInformation centersArchives Computer and information sciencesSoftware engineeringComputer software Engineering and technologySoftware engineeringComputer software Computer and information sciencesSoftware engineeringSource code Engineering and technologySoftware engineeringSource code Research and analysis methodsResearch assessmentReproducibility Computer and information sciencesData management Biology and life sciencesMolecular biologyMolecular biology techniquesSequencing techniquesRNA sequencing Research and analysis methodsMolecular biology techniquesSequencing techniquesRNA sequencing Biology and life sciencesMolecular biologyMolecular biology techniquesMolecular biology assays and analysis techniquesNucleic acid analysisRNA analysis Research and analysis methodsMolecular biology techniquesMolecular biology assays and analysis techniquesNucleic acid analysisRNA analysis Biology and life sciencesOrganismsEukaryotaAnimalsVertebratesAmniotesMammalsWildebeest Biology and life sciencesZoologyAnimalsVertebratesAmniotesMammalsWildebeest Scikick: A sidekick for workflow clarity and reproducibility during extensive data analysis Workflow clarity and reproducibility with Scikick Carlucci Matthew Software Writing – original draft Writing – review & editing 1 2 Bareikis Tadas Software Writing – review & editing 1 2 Koncevičius Karolis Software Supervision Writing – review & editing 2 Gibas Povilas Software Writing – review & editing 1 2 Kriščiūnas Algimantas Software Writing – review & editing 2 Petronis Art Supervision Writing – review & editing 1 2 https://orcid.org/0000-0002-7651-1312 Oh Gabriel Supervision Writing – original draft Writing – review & editing 1 2 3 * The Krembil Family Epigenetics Laboratory, The Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada Institute of Biotechnology, Life Sciences Center, Vilnius University, Vilnius, Lithuania Stanford University School of Medicine, Stanford, California, United States of America Bernasconi Anna Editor Politecnico di Milano, ITALY

The authors have declared that no competing interests exist.

 * E-mail: gabriel.oh@stanford.edu 27 7 2023 2023 18 7 e0289171 12 4 2023 13 7 2023 2023 Carlucci et al This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Reproducibility is crucial for scientific progress, yet a clear research data analysis workflow is challenging to implement and maintain. As a result, a record of computational steps performed on the data to arrive at the key research findings is often missing. We developed Scikick, a tool that eases the configuration, execution, and presentation of scientific computational analyses. Scikick allows for workflow configurations with notebooks as the units of execution, defines a standard structure for the project, automatically tracks the defined interdependencies between the data analysis steps, and implements methods to compile all research results into a cohesive final report. Utilities provided by Scikick help turn the complicated management of transparent data analysis workflows into a standardized and feasible practice. Scikick version 0.2.1 code and documentation is available as supplementary material. The Scikick software is available on GitHub (https://github.com/matthewcarlucci/scikick) and is distributed with PyPi (https://pypi.org/project/scikick/) under a GPL-3 license.

 http://dx.doi.org/10.13039/501100004895 European Social Fund No 09.3.3-LMT-K-712-17-0008 Petronis Art This project was supported by the European Social Fund, ec.europa.eu/esf (project No 09.3.3-LMT-K-712-17-0008) under grant agreement with the Research Council of Lithuania (LMTLT; lmt.lt) awarded to A.P. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data Availability All relevant data are within the paper and its Supporting information files. 1. Introduction

Research reproducibility, in the many forms it takes [1–3], is essential to the scientific method. Multiple insights can often be gained from a single large dataset, however, the breadth of such investigations has placed a heavy burden on researchers who aim to practice full computational transparency. It is essential that analytical procedures are clearly documented throughout an investigation, including details of their intent, background rationale, implementation, and analysis outputs [4, 5]. In its absence, investigative decisions, assumptions, and results can lose their context, and in turn, lower the quality of research communication [6].

Computational notebook formats (e.g., Jupyter Notebooks [7] and Rmarkdown [8]) and their associated development environments have paved a path for streamlined generation and sharing of computational results. Notebooks enable investigators to compile the analytical context (i.e., text), implementation (code), and results (figures) within a single document. This results in a report that reflects the entire process of analysis and serves as a transparent lens into how computations unfolded.

In order to develop larger projects, there is a demand to use multiple notebooks within the same analysis. To this end, in addition to best practice guidelines [9] and improvements to the notebook format [10], tools have been designed for some specific project types; generating reading materials on computational topics (e.g., bookdown [11] and Jupyter Book [12]) or developing software packages fully within notebooks (e.g., nbdev [13]). However, these solutions do not emphasize the ordered and interdependent execution of notebooks common to computational research projects. As such, reproducibility is compromised when projects are not configured to execute notebooks in the correct order, and transparency is compromised when projects do not clearly document this execution order.

The clarity of notebook outputs gives it an advantage to tools whose main purpose is to configure ordered computations (e.g., GNU Make [14], Snakemake [15], Nextflow [16], etc.). To benefit from both toolsets, researchers often use them in tandem to reproducibly configure the execution of a notebook collection. Further, researchers can produce graphical representations of this configuration to transparently represent the execution to the reader. However, assembling and maintaining these configurations throughout evolving projects is cumbersome. Therefore, many rapidly developing projects cannot dedicate the resources necessary for this level of transparency and reproducibility (Fig 1a).

10.1371/journal.pone.0289171.g001 Scikick workflow development use-case, practices, and features.

a) An illustration of the problem Scikick aims to address. Left) A schematic of a rendered computational notebook with contextual descriptions accompanying code and results demonstrating the clarity of the notebook format. Centre) A minimal “notebook collection” where execution order of notebooks is undocumented and not configured, compromising both transparency and reproducibility. Right) A graphical representation of a workflow management configuration which supplements the notebook collection to execute the notebooks in the specified order. b) The illustration shows the main Scikick features used to manage a collection of notebooks throughout a project. An unstructured collection of notebooks are initially executed by Scikick to generate a structured report. New content inside the workflow, including modifications to a notebook, upstream modifications or the addition of new notebooks, are all detected by Scikick. Pending updates are computed by Scikick to generate an up-to-date report which can be easily navigated. Users repeatedly apply the modification, state management, and execution features throughout a project’s development.

To simplify and improve transparency during these projects, we developed Scikick, a minimalistic command-line utility for maintaining computational notebook workflows. Scikick integrates the notebook format, workflow tools, and other elements necessary to streamline data-intensive research [17]. Our tool is designed to promote the maintenance of computational workflow archives that are easily inspectable by a diverse scientific readership at all project stages, conceptually extending the benefits of computational notebooks to computational workflows.

 2. Results 2.1 Feature overview

The features of Scikick are designed to maintain structure and re-executability while performing an investigation which spans multiple computational notebooks that may depend on one-another. Notebooks are arranged into a workflow graph using Scikick’s command-line interface, implemented with Python ver. 3.6+, to update the configuration file and inspect the workflow state (Fig 1b). Notebooks are executed through Scikick’s Snakemake workflow configuration and are compiled into a static report containing all data analysis outputs. Scikick features are broadly separated into notebook, workflow, and repository management functionalities that seamlessly work together.

2.1.1 Command-line interface overview

The main commands for accessing this functionality are: “init”, “add”, “run”, and “status”. The “sk init” command checks for software requirements and creates an empty configuration file within the project directory. The “sk add” command is used to add notebooks and define the workflow graph. “sk run” calls on Snakemake to execute notebooks in the specified order. Then, as notebooks are modified, the “sk status” command displays which notebooks require (re)execution. Further usage information can be found within the Scikick documentation and command-line help outputs.

 2.2 Notebook execution and metadata capture

To promote easy access for recording results in a notebook format, Scikick supports a plurality of file and notebook formats that are automatically executed using appropriate methods (e.g., R scripts are converted to Rmarkdown) to capture code, console, and graphical outputs in a markdown document which would otherwise go unrecorded. All notebook outputs are compiled with Rmarkdown into a cohesive final report where the report navigation, which is automatically configured by Scikick, allows readers to easily identify the notebook source file. Scikick also logs useful technical information related to each notebook into the resulting markdown files, such as software versions, user defined functions, execution time, and versioning history of the notebook. Usage of these features increases the amount of information available to readers revisiting old computational archives.

 2.3 Concurrent management of automation and reporting

As a project matures and more analyses are added, it is reasonable to expect that long notebooks will be broken up into many smaller notebooks [9] in order to focus on specific topics, reduce resource usage (e.g., memory and runtime), and simplify computing environment namespaces. Creation of notebooks which depend on one-another during this stage puts new strains on automating the workflow while also ensuring the report remains transparent. Therefore, Scikick provides simple features to define dependence between notebooks and to manage each notebook’s state. After authoring a new notebook and placing it in the workflow graph, the status of all notebooks can be verified by the user. Any pending notebook executions will run in an appropriate order to overwrite older outputs such that results represent a procedural execution of each notebook’s code. Finally, embedding notebook results and the order of notebook executions into a final report in a human-readable format allows for the report archive to be fully understood by readers (e.g., Fig 2).

10.1371/journal.pone.0289171.g002 Report and project map generated by Scikick for the single-cell RNA sequencing analysis demonstration.

A series of notebooks shown in the file tree under the “notebooks” directory were added to the project’s “scikick.yml” YAML configuration file using the command-line interface. The notebooks were executed to generate the displayed HTML archive (orange border) which resides in the “report” directory. A workflow graph (“Project Map”) was automatically generated by Scikick using the graphviz.dot format [18]. The graph illustrates the order of notebook executions, outlines the organization of the project’s source code, and allows for report navigation such that a reader can trace the provenance of results among the notebooks. Each grouping (grey box) represents a series of notebooks that were used to analyze an individual transcriptomic dataset with a final group containing two notebooks used to perform a combined analysis of the datasets. The red node represents the current notebook results page being viewed in the report. A table of contents is provided on the left for each page in the report.

 2.4 Demonstrations and guides

A series of vignettes in the Scikick documentation (S1 File; https://github.com/matthewcarlucci/scikick) includes a quick-start guide, a comprehensive user manual, details on methods and design, as well as a demonstration of Scikick’s capabilities through an analysis of single-cell RNA sequencing (scRNA-seq) datasets [19]. Here, we demonstrated how the implementation of a complex workflow involving data inspection, quality control, normalization, dimensionality reduction, and data exploration, is managed through the usage of Scikick. This allowed the pursuit of nonlinear avenues of investigation across multiple datasets and project stages and automatically embedded a navigable graph of the workflow within the archive (Fig 2).

 2.5 Project management with robust versioned archives

Advanced practices such as “Continuous Analysis” [20] provide schemes for producing verifiable end-to-end runs of computational analyses by ensuring that code can execute from start to finish within a specified software environment. Despite having verified a machine-readable workflow in this way, maintaining an archive that is also human-readable is a difficult, but necessary, complementary task [9]. Crucial to this practice, Scikick projects remain reader-ready throughout their development with the use of features such as a simplistic configuration and a navigable project graph (Fig 2). Additionally, Scikick projects can be stored entirely as plain text and sets a clear structure for which files are tracked with version control, further simplifying the adoption of Continuous Analysis on version control platforms (e.g., GitHub, GitLab, BitBucket, etc.). Lastly, demanding knowledge needed to use advanced computing is reduced for Scikick projects; minimal additional configuration can achieve distributed notebook executions over a cluster’s resources in a prespecified container environment (e.g., Docker [21], Singularity [22], etc.). Since all Scikick projects use the same execution and reporting methods, a single configuration of continuous analysis on a given set of computational infrastructure can be applied to all projects. We implemented such a configuration on common academic infrastructure which monitors source code for changes and re-runs scientific analysis automatically (Fig 3). Altogether, Scikick methods are sufficiently flexible to implement as part of a variety of computing infrastructure and environments.

10.1371/journal.pone.0289171.g003 Developing Scikick projects to maturation with continuous analysis.

a) This schematic illustration represents a configuration for an automated re-execution of a Scikick project. Local development of notebook collections with Scikick proceeds as described in Fig 1b. Upon pushing a version of the project’s source code to a remote server (GitLab), the project is executed utilizing resources of a SLURM compute cluster where each notebook is submitted in parallel as cluster jobs. To provide fixed software environments for the remote machines, Singularity containers are defined in the Scikick configuration file. The final reproducible report is stored alongside the code that produced it as a “verified report”, which is known to have executed from start to finish within the specified software environment. b) A sample of the ‘Continuous Integration’ section of GitLab shows versions of the analysis that are executed as they are “pushed” to the GitLab server, as configured in (a). Some versions encountered errors which were rapidly addressed which may have been detected much later if re-execution was not performed. Successful analysis verification runs store a final Scikick report on the server, alongside the source code that produced it, where results may be inspected at any time in the future, and further, may be precisely reproduced with usage of the specified container.

 3. Conclusion

Computational notebooks have provided transparency to research reporting, and yet require supporting tooling for the analysis of large-scale datasets with many investigative branches. Projects lack access to streamlined build tools to support computational notebooks, and therefore lose the reproducibility and transparency needed for auditing analyses. Scikick provides the ability to maintain clarity of data analysis and research workflow development all the way through to verified analysis archives. Scikick can be used to execute polyglot projects, and underlying tooling updates (e.g., Quarto [23]) may improve language agnosticism in the future. Well-structured and annotated investigative projects, supported by the use of tools like Scikick, will help improve transparency to allow for more rigorous review and thereby improve reliability of scientific works.

 Supporting information Scikick 0.2.1 source code and documentation.

(ZIP)

The authors thank all of those involved in providing comments throughout the development of Scikick which provided valuable feedback for the design and usage of the tool.

 References Goodman SN, Fanelli D, Ioannidis JPA. What does research reproducibility mean? Sci Transl Med. 2016 Jun 1;8(341):341ps12. doi: 10.1126/scitranslmed.aaf5027 27252173 Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, du Sert NP, et al. A manifesto for reproducible science. Nature Human Behaviour. 2017 Jan 10;1(1):1–9. doi: 10.1038/s41562-016-0021 33954258 Plesser HE. Reproducibility vs. Replicability: A Brief History of a Confused Terminology. Front Neuroinform. 2018;11. Available from: http://dx.doi.org/10.3389/fninf.2017.00076 29403370 Claerbout JF, Karrenbach M. Electronic documents give reproducible research a new meaning. In: SEG Technical Program Expanded Abstracts 1992. Society of Exploration Geophysicists; 1992. p. 601–4. Stodden V, McNutt M, Bailey DH, Deelman E, Gil Y, Hanson B, et al. Enhancing reproducibility for computational methods. Science. 2016 Dec 9;354(6317):1240–1. doi: 10.1126/science.aah6168 27940837 Hutson M. Artificial intelligence faces reproducibility crisis. Science. 2018 Feb 16;359(6377):725–6. doi: 10.1126/science.359.6377.725 29449469 Kluyver T, Ragan-Kelley B, Pérez F, Granger B, Bussonnier M, Frederic J, et al. Jupyter Notebooks–a publishing format for reproducible computational workflows. In: Loizides F, Scmidt B, editors. Positioning and Power in Academic Publishing: Players, Agents and Agendas. IOS Press; 2016. p. 87–90. Xie Y, Allaire JJ, Grolemund G. R Markdown: The Definitive Guide. Taylor & Francis, CRC Press; 2018. 303 p. Rule A, Birmingham A, Zuniga C, Altintas I, Huang SC, Knight R, et al. Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks. PLoS Comput Biol. 2019 Jul;15(7). doi: 10.1371/journal.pcbi.1007007 31344036 Lau S, Drosos I, Markel JM, Guo PJ. The Design Space of Computational Notebooks: An Analysis of 60 Systems in Academia and Industry. In: 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 2020. p. 1–11. Xie Y. bookdown: Authoring Books and Technical Documents with R Markdown. 1st ed. Chapman and Hall/CRC; 2016. Executable Books Community. Jupyter Book [Internet]. 2020. https://zenodo.org/record/4539666 Howard J, Gugger S. Fastai: A Layered API for Deep Learning. Information. 2020 Feb 16;11(2):108. Gough B. GNU Scientific Library Reference Manual—Third Edition. 3rd ed. Network Theory Ltd.; 2009. Mölder F, Jablonski KP, Letcher B, Hall MB, Tomkins-Tinch CH, Sochat V, et al. Sustainable data analysis with Snakemake. F1000Res. 2021 Jan 18;10(33):33. doi: 10.12688/f1000research.29032.2 34035898 Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316–9. doi: 10.1038/nbt.3820 28398311 Reiter T, Brooks PT, Irber L, Joslin SEK, Reid CM, Scott C, et al. Streamlining data-intensive biology with workflow systems. Gigascience. 2021 Jan 13;10(1). Available from: http://dx.doi.org/10.1093/gigascience/giaa140 33438730 Gansner ER, North SC. An open graph visualization system and its applications to software engineering. Softw Pract Exp. 2000 Sep;30(11):1203–33. Amezquita RA, Lun ATL, Becht E, Carey VJ, Carpp LN, Geistlinger L, et al. Orchestrating single-cell analysis with Bioconductor. Nat Methods. 2019 Dec 2;17(2):137–45. doi: 10.1038/s41592-019-0654-x 31792435 Beaulieu-Jones BK, Greene CS. Reproducibility of computational workflows is automated using continuous analysis. Nat Biotechnol. 2017 Mar 13;35(4):342–6. doi: 10.1038/nbt.3780 28288103 Merkel D. Docker: lightweight linux containers for consistent development and deployment. Linux J. 2014;2014(239):2. Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459 28494014 Quarto [Internet]. [cited 2023 Feb 24]. https://quarto.org/ 10.1371/journal.pone.0289171.r001 Decision Letter 0 Bernasconi Anna Academic Editor 2023 Anna Bernasconi This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

9 May 2023

PONE-D-23-11089Scikick: a sidekick for workflow clarity and reproducibility during extensive data analysisPLOS ONE

Dear Dr. Oh,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jun 23 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Anna Bernasconi, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

3. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments (if provided):

Dear authors the paper contributes an important work. Please carefully revise the manuscript addressing the comments of Rev. 1 and 2. Provide a revised manuscript with changes/additions highlighted in different color.

We will be glad to review your revised manuscript once the points have been addressed.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: N/A

Reviewer #3: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: In this work, authors present Scikick, an interesting tool to create workflows composed by notebooks rather than from scripts or executable files as we all are used to do.

I have never heard about a similar tool and many times wondered how that could be done, so my overall opinion is that this is a good addition to the bioinformatics software ecosystem.

Authors have made a great effort to automatically run notebooks using Snakemake and provide exhaustive documentation to get started with the tool, including basic and advanced usage examples.

Regarding the paper, I have struggled a little bit with figure 2. I was able to fully understand it after browsing the example report online and navigating trough the different sections, so my suggestion is try improving the figure 2 legend so that readers can understand it easier. For instance, you could add a files tree of the project and briefly explain them.

In addition, three points:

- I'd suggest authors adding a new section (and maybe a figure) to explain the basic commands of the "sk" tool (init, add, status and run) as it is important for readers to see how the tool works and how dependencies are defined.

- Have you considered including tools or some mechanisms to obtain user-friendly readable diffs between notebook files? By default, "git diff" commands on Python notebooks are not very useful.

- Regarding figure 3, what does mean "verified archive"? I think that this figure can be also improved. For me, the flow would go from local development to the GitLab server and from that point the execution of the "continuous analysis" triggered. With the current image, one can think that the continuous analysis is performed in the same computer as the workflow is being developed.

Reviewer #2: The authors describe a command-line software package for tracking the sequence in which Jupyter notebooks are executed and "to define dependence between notebooks and to manage each notebook’s state." The goal is to prevent scientists from having to perform these tasks manually, which can be time consuming and error prone. For scientists who use notebooks, this tool may be helpful. Though in my own personal case, I have not yet found a scenario where I would use this tool.

The writing is clear, the software appears to be well designed, and it is open source. Below are some comments as suggestions for improving the paper.

- Figure 1 seems a bit superfluous to me. It attempts to provide a high-level justification for the tool, but in my opinion, it doesn't add anything to the paper beyond what is already in the text.

- The paper focuses mostly on justifying the need for the tool. But it provides little insight on *how* the tool works. For example, how does it collect the metadata and integrate everything? How does it integrate with snakemake? Etc. I don't think it's necessary to provide a lot more detail. But a little more would be helpful.

- The paper says, "Additionally, maintaining a simple project structure suitable for version control, further eases the adoption of Continuous Analysis on version control platforms (e.g., GitHub, GitLab, BitBucket, etc.)." I'm not sure if I understand exactly what this is saying. How does it support using version control?

- I'm probably missing it, but I don't see any explicit instructions about how to install the software. That's a pretty standard thing to see on documentation sites.

Reviewer #3: Computational analyses are getting a central part in an ever increasing number of scientific experiments and the amount and complexity of code that is produced is often difficult to maintain, distribute and replicate in a standard way.

In this manuscript the authors present a tools for easy integration, maintenance and organisation of multiple computational notebooks linked to complex analysis tasks. The tool functionalities are well structured and provide a way to add a layer of organisation to code without affecting the internal logic of existing components. The manuscript is clear and well written.

I think that, if adopted, the proposed tool can bring a considerable benefit to the research community.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Hugo López-Fernández

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

 10.1371/journal.pone.0289171.r002 Author response to Decision Letter 0 Submission Version 1

21 Jun 2023

Reviewer #1:

Comment: Regarding the paper, I have struggled a little bit with figure 2. I was able to fully understand it after browsing the example report online and navigating through the different sections, so my suggestion is try improving the figure 2 legend so that readers can understand it easier. For instance, you could add a files tree of the project and briefly explain them.

Response: We have taken the reviewers suggestion to better highlight the merits of Scikick being displayed in figure 2 by including the file tree and further describing the figure (L133-L139).

Comment: I'd suggest authors adding a new section (and maybe a figure) to explain the basic commands of the "sk" tool (init, add, status and run) as it is important for readers to see how the tool works and how dependencies are defined.

Response: In our initial submission, we have included detailed information regarding the execution in our supplemental information in an effort to reduce too much technicality in the manuscript. Based on the reviewer’s feedback, we have included a new section as an overview of the command-line interface (L94-L103), yet direct to the documentation for longer form tutorials and examples.

Comment: Have you considered including tools or some mechanisms to obtain user-friendly readable diffs between notebook files? By default, "git diff" commands on Python notebooks are not very useful.

Response: Thank you for the suggestion. While it is an interesting idea, we think it is beyond the scope of Scikick. There are some dedicated tools to specifically address this challenging issue (e.g., https://nbdime.readthedocs.io/en/latest/), and we feel these specialty tools and others like it will work in tandem with Scikick.

Comment: Regarding figure 3, what does mean "verified archive"? I think that this figure can be also improved. For me, the flow would go from local development to the GitLab server and from that point the execution of the "continuous analysis" triggered. With the current image, one can think that the continuous analysis is performed in the same computer as the workflow is being developed.

Response: We have replaced “verified archive” in Figure 3 with “GitLab Repository” in order to avoid the possible misunderstanding pointed out by the reviewer. We also have clarified what we mean by “verified” throughout the revised manuscript (L160-L161; L188-L190).

Reviewer #2:

Comment: Figure 1 seems a bit superfluous to me. It attempts to provide a high-level justification for the tool, but in my opinion, it doesn't add anything to the paper beyond what is already in the text.

Response: We agree with the reviewer that it is somewhat repetitive to what is described in the main text. However, we believe that an overview may serve a similar role to graphical abstracts for those who are trying to rapidly process information, as a visual aid.

Comment: The paper focuses mostly on justifying the need for the tool. But it provides little insight on *how* the tool works. For example, how does it collect the metadata and integrate everything? How does it integrate with snakemake? Etc. I don't think it's necessary to provide a lot more detail. But a little more would be helpful.

Response: Much of this information was included as supplemental information (found in the documentation “core design”) or was custom code that would be trivial to explain (e.g. “sys.time” for logging runtime). We used the article to highlight the need for tools like Scikick and the utility they add (i.e., the why rather than the how). We have made additions to the manuscript where appropriate to better allude to this content and to the underlying architecture of the software (L91; L109; L110; L112; L133-L136; L165-L166).

Comment: The paper says, "Additionally, maintaining a simple project structure suitable for version control, further eases the adoption of Continuous Analysis on version control platforms (e.g., GitHub, GitLab, BitBucket, etc.)." I'm not sure if I understand exactly what this is saying. How does it support using version control?

Response: This sentence combined too many concepts and has now been expanded for clarity (L165-L167).

Comment: I'm probably missing it, but I don't see any explicit instructions about how to install the software. That's a pretty standard thing to see on documentation sites.

Response: The installation instructions should have been found on the homepage (https://petronislab.camh.ca/pub/scikick). We have changed this link from “Start using Scikick” to “Installation and Usage” to avoid confusion and have added a “Quick-Start” installation to the GitHub homepage to further improve accessibility.

Submitted filename: response_to_the_reviewers_comments.docx

 10.1371/journal.pone.0289171.r003 Decision Letter 1 Bernasconi Anna Academic Editor 2023 Anna Bernasconi This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

13 Jul 2023

Scikick: a sidekick for workflow clarity and reproducibility during extensive data analysis

PONE-D-23-11089R1

Dear Dr. Oh,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Anna Bernasconi, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Dear authors,

your revised manuscript has been re-assessed by two reviewers and myself. We gladly recommend the paper for acceptance.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: All comments have been addressed so the paper can be accepted now. I do not have anything else to add.

Reviewer #2: I have no additional comments.This form is asking me to write a statement that is at least 100 characters long, but I have nothing else to say... :)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Hugo López-Fernández

Reviewer #2: Yes: Stephen Piccolo

**********

 10.1371/journal.pone.0289171.r004 Acceptance letter Bernasconi Anna Academic Editor 2023 Anna Bernasconi This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

18 Jul 2023

PONE-D-23-11089R1

Scikick: a sidekick for workflow clarity and reproducibility during extensive data analysis

Dear Dr. Oh:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Anna Bernasconi

Academic Editor

PLOS ONE