systemPipeRdata: Workflow templates and sample data
Last update: 03 May, 2021
Package
systemPipeRdata 1.18.2
Note: the most recent version of this vignette can be found here.
Note: if you use systemPipeR and systemPipeRdata in published research, please cite:
Backman, T.W.H and Girke, T. (2016). systemPipeR: Workflow and Report Generation Environment. BMC Bioinformatics, 17: 388. 10.1186/s12859-016-1241-0.
1 Introduction
systemPipeRdata is a helper package
to generate with a single command workflow templates that are intended to be
used by its parent package systemPipeR (H Backman and Girke 2016).
The systemPipeR project provides a suite of R/Bioconductor packages for designing,
building and running end-to-end analysis workflows on local machines, HPC clusters
and cloud systems, while generating at the same time publication quality analysis reports.
To test workflows quickly or design new ones from existing templates, users can
generate with a single command workflow instances fully populated with sample data
and parameter files required for running a chosen workflow.
Pre-configured directory structure of the workflow environment and the sample data
used by systemPipeRdata are described here.
2 Getting started
2.1 Installation
The systemPipeRdata package is available at Bioconductor and can be installed from within R as follows:
if (!requireNamespace("BiocManager", quietly = TRUE)) install.packages("BiocManager")
BiocManager::install("systemPipeRdata")Also, it is possible to install the development version from Bioconductor.
BiocManager::install("systemPipeRdata", version = "devel", build_vignettes = TRUE,
dependencies = TRUE) # Installs Devel version from Bioconductor2.2 Loading package and documentation
library("systemPipeRdata") # Loads the packagelibrary(help = "systemPipeRdata") # Lists package info
vignette("systemPipeRdata") # Opens vignette2.3 Starting with pre-configured workflow templates
Load one of the available workflows into your current working directory.
The following does this for the varseq workflow template. The name of the resulting
workflow directory can be specified under the mydirname argument. The default NULL
uses the name of the chosen workflow. An error is issued if a directory of the same
name and path exists already.
genWorkenvir(workflow = "varseq", mydirname = NULL)
setwd("varseq")On Linux and OS X systems the same can be achieved from the command-line of a terminal with the following commands.
$ Rscript -e "systemPipeRdata::genWorkenvir(workflow='varseq', mydirname=NULL)"3 Workflow templates collection
A collection of workflow templates are available, and it is possible to browse the current availability, as follows:
availableWF(github = TRUE)This function returns the list of workflow templates available within the package and systemPipeR Organization on GitHub. Each one listed template can be created as described above.
The workflow template choose from Github will be installed as an R package, and also it creates the environment with all the settings and files to run the demo analysis
genWorkenvir(workflow="systemPipeR/systemPipeVARseq", mydirname="NULL")
setwd("systemPipeVARseq")Besides, it is possible to choose different versions of the workflow template,
defined through other branches on the GitHub Repository. By default, the master
branch is selected, however, it is possible to define a different branch with the ref argument.
genWorkenvir(workflow="systemPipeR/systemPipeRNAseq", ref = "singleMachine")
setwd("systemPipeRNAseq")3.1 Download a specific R Markdown file
Also, it is possible to download a specific workflow script for your analysis.
The URL can be specified under url argument and the R Markdown file name in
the urlname argument. The default NULL copies the current version available in the chose template.
genWorkenvir(workflow="rnaseq", url = "https://raw.githubusercontent.com/systemPipeR/systemPipeRNAseq/cluster/vignettes/systemPipeRNAseq.Rmd",
urlname = "rnaseq_V-cluster.Rmd")
setwd("rnaseq")4 Dynamic generation of workflow template
It is possible to create a new workflow structure from RStudio
menu File -> New File -> R Markdown -> From Template -> systemPipeR New WorkFlow.
This interactive option creates the same environment as demonstrated above.
Figure 1: Selecting workflow template within RStudio.
5 Directory Structure
The workflow templates generated by genWorkenvir contain the following preconfigured directory structure:
- workflow/ (e.g. rnaseq/)
- This is the root directory of the R session running the workflow.
- Run script ( *.Rmd) and sample annotation (targets.txt) files are located here.
- Note, this directory can have any name (e.g. rnaseq, varseq). Changing its name does not require any modifications in the run script(s).
- Important subdirectories:
- param/
- Stores non-CWL parameter files such as: *.param, *.tmpl and *.run.sh. These files are only required for backwards compatibility to run old workflows using the previous custom command-line interface.
- param/cwl/: This subdirectory stores all the CWL parameter files. To organize workflows, each can have its own subdirectory, where all
CWL paramandinput.ymlfiles need to be in the same subdirectory.
- data/
- FASTQ files
- FASTA file of reference (e.g. reference genome)
- Annotation files
- etc.
- results/
- Analysis results are usually written to this directory, including: alignment, variant and peak files (BAM, VCF, BED); tabular result files; and image/plot files
- Note, the user has the option to organize results files for a given sample and analysis step in a separate subdirectory.
- param/
Note: Directory names are indicated in green. Users can change this structure as needed, but need to adjust the code in their workflows accordingly.
Figure 2: systemPipeR’s preconfigured directory structure.
6 Run workflows
Next, run from within R the chosen sample workflow by executing the code provided
in the corresponding *.Rmd template file.
Much more detailed information on running and customizing systemPipeR
workflows is available in its overview vignette here.
This vignette can also be opened from R with the following command.
library("systemPipeR") # Loads systemPipeR which needs to be installed via BiocManager::install() from Bioconductorvignette("systemPipeR", package = "systemPipeR")6.1 Return paths to sample data
The location of the sample data provided by systemPipeRdata can be returned as a list.
pathList()## $targets
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/param/targets.txt"
##
## $targetsPE
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/param/targetsPE.txt"
##
## $annotationdir
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/annotation/"
##
## $fastqdir
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/fastq/"
##
## $bamdir
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/bam/"
##
## $paramdir
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/param/"
##
## $workflows
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/workflows/"
##
## $chipseq
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/workflows/chipseq/"
##
## $rnaseq
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/workflows/rnaseq/"
##
## $riboseq
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/workflows/riboseq/"
##
## $varseq
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/workflows/varseq/"
##
## $new
## [1] "/tmp/RtmpAguIWs/Rinst15221740cd44/systemPipeRdata/extdata/workflows/new/"7 Version information
sessionInfo()## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.5 LTS
##
## Matrix products: default
## BLAS: /home/biocbuild/bbs-3.12-bioc/R/lib/libRblas.so
## LAPACK: /home/biocbuild/bbs-3.12-bioc/R/lib/libRlapack.so
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_US.UTF-8 LC_COLLATE=C
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## attached base packages:
## [1] stats4 parallel stats graphics grDevices
## [6] utils datasets methods base
##
## other attached packages:
## [1] systemPipeRdata_1.18.2 batchtools_0.9.15
## [3] ape_5.5 ggplot2_3.3.3
## [5] systemPipeR_1.24.6 ShortRead_1.48.0
## [7] GenomicAlignments_1.26.0 SummarizedExperiment_1.20.0
## [9] Biobase_2.50.0 MatrixGenerics_1.2.1
## [11] matrixStats_0.58.0 BiocParallel_1.24.1
## [13] Rsamtools_2.6.0 Biostrings_2.58.0
## [15] XVector_0.30.0 GenomicRanges_1.42.0
## [17] GenomeInfoDb_1.26.7 IRanges_2.24.1
## [19] S4Vectors_0.28.1 BiocGenerics_0.36.1
## [21] BiocStyle_2.18.1
##
## loaded via a namespace (and not attached):
## [1] colorspace_2.0-0 rjson_0.2.20
## [3] hwriter_1.3.2 ellipsis_0.3.2
## [5] remotes_2.3.0 bit64_4.0.5
## [7] AnnotationDbi_1.52.0 fansi_0.4.2
## [9] xml2_1.3.2 codetools_0.2-18
## [11] splines_4.0.5 cachem_1.0.4
## [13] knitr_1.33 jsonlite_1.7.2
## [15] annotate_1.68.0 GO.db_3.12.1
## [17] dbplyr_2.1.1 png_0.1-7
## [19] pheatmap_1.0.12 graph_1.68.0
## [21] BiocManager_1.30.12 compiler_4.0.5
## [23] httr_1.4.2 backports_1.2.1
## [25] GOstats_2.56.0 assertthat_0.2.1
## [27] Matrix_1.3-2 fastmap_1.1.0
## [29] limma_3.46.0 formatR_1.9
## [31] htmltools_0.5.1.1 prettyunits_1.1.1
## [33] tools_4.0.5 gtable_0.3.0
## [35] glue_1.4.2 GenomeInfoDbData_1.2.4
## [37] Category_2.56.0 dplyr_1.0.5
## [39] rsvg_2.1.1 rappdirs_0.3.3
## [41] V8_3.4.2 Rcpp_1.0.6
## [43] jquerylib_0.1.4 vctrs_0.3.8
## [45] nlme_3.1-152 debugme_1.1.0
## [47] rtracklayer_1.50.0 xfun_0.22
## [49] stringr_1.4.0 lifecycle_1.0.0
## [51] XML_3.99-0.6 edgeR_3.32.1
## [53] zlibbioc_1.36.0 scales_1.1.1
## [55] BSgenome_1.58.0 VariantAnnotation_1.36.0
## [57] hms_1.0.0 RBGL_1.66.0
## [59] RColorBrewer_1.1-2 yaml_2.2.1
## [61] curl_4.3.1 memoise_2.0.0
## [63] sass_0.3.1 biomaRt_2.46.3
## [65] latticeExtra_0.6-29 stringi_1.5.3
## [67] RSQLite_2.2.7 genefilter_1.72.1
## [69] checkmate_2.0.0 GenomicFeatures_1.42.3
## [71] DOT_0.1 rlang_0.4.11
## [73] pkgconfig_2.0.3 bitops_1.0-7
## [75] evaluate_0.14 lattice_0.20-44
## [77] purrr_0.3.4 bit_4.0.4
## [79] tidyselect_1.1.1 GSEABase_1.52.1
## [81] AnnotationForge_1.32.0 magrittr_2.0.1
## [83] bookdown_0.22 R6_2.5.0
## [85] generics_0.1.0 base64url_1.4
## [87] DelayedArray_0.16.3 DBI_1.1.1
## [89] withr_2.4.2 pillar_1.6.0
## [91] survival_3.2-11 RCurl_1.98-1.3
## [93] tibble_3.1.1 crayon_1.4.1
## [95] utf8_1.2.1 BiocFileCache_1.14.0
## [97] rmarkdown_2.7 jpeg_0.1-8.1
## [99] progress_1.2.2 locfit_1.5-9.4
## [101] grid_4.0.5 data.table_1.14.0
## [103] blob_1.2.1 Rgraphviz_2.34.0
## [105] digest_0.6.27 xtable_1.8-4
## [107] brew_1.0-6 openssl_1.4.4
## [109] munsell_0.5.0 bslib_0.2.4
## [111] askpass_1.18 Funding
This project was supported by funds from the National Institutes of Health (NIH) and the National Science Foundation (NSF).
References
H Backman, Tyler W, and Thomas Girke. 2016. “systemPipeR: NGS workflow and report generation environment.” BMC Bioinformatics 17 (1): 388. https://doi.org/10.1186/s12859-016-1241-0.