---
title: "User Guide for Rcwl"
author: "Qiang Hu"
date: "`r Sys.Date()`"
output:
  BiocStyle::html_document:
    toc: true
    toc_float: true
vignette: >
  %\VignetteIndexEntry{User Guide for Rcwl}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---
<!-- ```{r setup, include = FALSE} -->
<!-- knitr::opts_chunk$set( -->
<!--   collapse = TRUE, -->
<!--   comment = "#>" -->
<!-- ) -->
<!-- ``` -->

```{r setup, include=FALSE}
knitr::opts_knit$set(root.dir = tempdir())
```

The `Rcwl` package is aimed to be a simple and user-friendly way to manage command line tools and build data analysis pipelines in R using Common Workflow Language (CWL). It can be a bridge between heavy bioinformatics tools and pipeline to R/Bioconductor. More details about CWL can be found at <http://www.commonwl.org>.

# Installation
1. Download the package. 

```{r getPackage, eval=FALSE}
if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")
BiocManager::install("Rcwl")
```
The development version is also available to download from Github. 
```{r getDevel, eval=FALSE}
BiocManager::install("hubentu/Rcwl")
```

2. Load the package into the R session.
```{r Load, message=FALSE}
library(Rcwl)
```

# First Example
The main class and constructor function is `cwlParam`, which wrap a command line tool and its parameters in a `cwlParam` object. Let's start with a simple example, `echo hello world`.

First, we load the package and then define the input parameter for "echo", a string without a prefix. Just an `id` option required.
```{r}
input1 <- InputParam(id = "sth")
```

Second, create a `cwlParam` object with `baseCommand` for the command to execute and `InputParamList` for the input parameters.
```{r}
echo <- cwlParam(baseCommand = "echo", inputs = InputParamList(input1))
```

Now we have a command object to run. Let's send a string "Hello World!" to the object. Without defining the outputs, it will stream standard output to a temporary file by default.
```{r}
echo$sth <- "Hello World!"
echo
```

Let's run it. A list including the command executed, temporary output and logs. The output directory is the current folder by default, but it can be changed by setting `outdir` option. All standard out and standard error stream can also be printed by setting `stderr = ""`.
```{r}
r1 <- runCWL(echo, outdir = tempdir())
r1
readLines(r1$output)
```

The `writeCWL` function will write the cwlParam object to a CWL file for the command and YML for the inputs. Then it will invoke `cwl-runner` by default to execute the two files. Thus the command tool `cwl-runner` is required to be installed and available in the system path.

# Input Parameters
## Essential Input parameters
For the input parameters, three options need to be defined usually, *id*, *type*, and *prefix*. The type can be *string*, *int*, *long*, *float*, *double*, and so on. More detail can be found at: <https://www.commonwl.org/v1.0/CommandLineTool.html#CWLType>.

Here is an example from [CWL user guide](http://www.commonwl.org/user_guide/03-input/). Here we defined an `echo` with different type of input parameters by `InputParam`. The `stdout` option can be used to caputre the standard output stream to a file.
```{r}
e1 <- InputParam(id = "flag", type = "boolean", prefix = "-f")
e2 <- InputParam(id = "string", type = "string", prefix = "-s")
e3 <- InputParam(id = "int", type = "int", prefix = "-i")
e4 <- InputParam(id = "file", type = "File", prefix = "--file=", separate = FALSE)
echoA <- cwlParam(baseCommand = "echo",
                  inputs = InputParamList(e1, e2, e3, e4),
                  stdout = "output.txt")
```

Then we give it a try by setting values for the inputs.
```{r}
echoA$flag <- TRUE
echoA$string <- "Hello"
echoA$int <- 1

tmpfile <- tempfile()
write("World", tmpfile)
echoA$file <- tmpfile

r2 <- runCWL(echoA, outdir = tempdir())
r2$command
```

## Array Inputs
A similar example to CWL user guide. We can define three different type of array as inputs. 
```{r}
a1 <- InputParam(id = "A", type = "string[]", prefix = "-A")
a2 <- InputParam(id = "B",
                 type = InputArrayParam(items = "string",
                                        prefix="-B=", separate = FALSE))
a3 <- InputParam(id = "C", type = "string[]", prefix = "-C=",
                 itemSeparator = ",", separate = FALSE)
echoB <- cwlParam(baseCommand = "echo",
                 inputs = InputParamList(a1, a2, a3))
```

Then set values for the three inputs.
```{r}
echoB$A <- letters[1:3]
echoB$B <- letters[4:6]
echoB$C <- letters[7:9]
echoB
```

Now we can check whether the command behaves as we expected.
```{r}
r3 <- runCWL(echoB, outdir = tempdir())
r3$command
```

# Output Parameters
## Capturing Output
The outputs, similar to the inputs, is a list of output parameters. Three options *id*, *type* and *glob* can be defined. The glob option is used to define a pattern to find files relative to the output directory.

Here is an example to unzip a compressed `gz` file. First, we generate a compressed R script file.
```{r}
zzfil <- file.path(tempdir(), "sample.R.gz")
zz <- gzfile(zzfil, "w")
cat("sample(1:10, 5)", file = zz, sep = "\n")
close(zz)
```

We define a `cwlParam` object to use "gzip" to uncompress a input file.
```{r}
ofile <- "sample.R"
z1 <- InputParam(id = "uncomp", type = "boolean", prefix = "-d")
z2 <- InputParam(id = "out", type = "boolean", prefix = "-c")
z3 <- InputParam(id = "zfile", type = "File")
o1 <- OutputParam(id = "rfile", type = "File", glob = ofile)
gz <- cwlParam(baseCommand = "gzip",
               inputs = InputParamList(z1, z2, z3),
               outputs = OutputParamList(o1),
               stdout = ofile)
```

Now the `gz` object can be used to uncompress the previous generated compressed file.
```{r}
gz$uncomp <- TRUE
gz$out <- TRUE
gz$zfile <- zzfil
r4 <- runCWL(gz, outdir = tempdir())
r4$output
```

Or we can use `arguments` to set some default parameters.
```{r}
z1 <- InputParam(id = "zfile", type = "File")
o1 <- OutputParam(id = "rfile", type = "File", glob = ofile)
Gz <- cwlParam(baseCommand = "gzip",
               arguments = list("-d", "-c"),
               inputs = InputParamList(z1),
               outputs = OutputParamList(o1),
               stdout = ofile)
Gz
Gz$zfile <- zzfil
r4a <- runCWL(Gz, outdir = tempdir())
```

To make it for general usage, we can define a pattern with javascript to glob the output, which require `node` to be installed in your system PATH.
```{r}
pfile <- "$(inputs.zfile.path.split('/').slice(-1)[0].split('.').slice(0,-1).join('.'))"
```

Or we can use the CWL built in file property, `nameroot`, directly.
```{r}
pfile <- "$(inputs.zfile.nameroot)"
o2 <- OutputParam(id = "rfile", type = "File", glob = pfile)
req1 <- list(class = "InlineJavascriptRequirement")
GZ <- cwlParam(baseCommand = c("gzip", "-d", "-c"),
               requirements = list(), ## assign list(req1) if node installed.
               inputs = InputParamList(z1),
               outputs = OutputParamList(o2),
               stdout = pfile)
GZ$zfile <- zzfil
r4b <- runCWL(GZ, outdir = tempdir())
```

## Array Outputs

We can also capture multiple output files with `glob` pattern.
```{r}
a <- InputParam(id = "a", type = InputArrayParam(items = "string"))
b <- OutputParam(id = "b", type = OutputArrayParam(items = "File"), glob = "*.txt")
touch <- cwlParam(baseCommand = "touch", inputs = InputParamList(a), outputs = OutputParamList(b))
touch$a <- c("a.txt", "b.gz", "c.txt")
r5 <- runCWL(touch, outdir = tempdir())
r5$output
```

# Running Tools in Docker
The CWL can work with docker to simplify your software management and communicate files between host and container. The docker container can be defined by the `hints` or `requirements` option.
```{r}
d1 <- InputParam(id = "rfile", type = "File")
req1 <- list(class = "DockerRequirement",
             dockerPull = "r-base")
doc <- cwlParam(baseCommand = "Rscript",
                inputs = InputParamList(d1),
                stdout = "output.txt",
                hints = list(req1))
doc$rfile <- r4$output
```
```{r, eval=FALSE}
r6 <- runCWL(doc)
```

The tools defined with docker requirements can also be run locally by disabling the docker option. In case your `Rscript` depends some local libraries to run, an option from `cwltools`, "--preserve-entire-environment", can be used to pass all environment variables.
```{r}
r6a <- runCWL(doc, docker = FALSE, outdir = tempdir(),
              cwlArgs = "--preserve-entire-environment")
```

# Running Tools in Cluster server
The CWL also can work in high performance clusters with batch-queuing system, such as SGE, PBS, SLURM and so on, using the Bioconductor package `BiocParallel`. Here is an example to submit jobs with "Multiicore" and "SGE". A more detailed example can be found (<https://hubentu.github.io/others/Rcwl_RNASeq.html>).

```{r, eval=FALSE}
library(BiocParallel)
sth.list <- as.list(LETTERS)
names(sth.list) <- LETTERS

## submit with mutlicore
result1 <- runCWLBatch(cwl = echo, outdir = tempdir(), inputList = list(sth = sth.list),
                       BPPARAM = MulticoreParam(26))

## submit with SGE
result2 <- runCWLBatch(cwl = echo, outdir = tempdir(), inputList = list(sth = sth.list),
                       BPPARAM = BatchtoolsParam(workers = 26, cluster = "sge",
                                                 resources = list(queue = "all.q")))
```

# Writing Pipeline
We can connect multiple tools together into a pipeline. Here is an example to uncompress an R script and execute it with `Rscript`.

Here we define a simple `Rscript` tool without using docker.
```{r}
d1 <- InputParam(id = "rfile", type = "File")
Rs <- cwlParam(baseCommand = "Rscript",
               inputs = InputParamList(d1))
Rs
```

Test run:
```{r}
Rs$rfile <- r4$output
tres <- runCWL(Rs, outdir = tempdir())
readLines(tres$output)
```

The pipeline includes two steps, decompressed by `GZ` and compiled by `Rs`. The input file is a compressed file and the output would be the output `Rout` from `Rs`.

First we need to define the direct inputs and outputs from `GZ` and `Rs`.
```{r}
i1 <- InputParam(id = "cwl_zfile", type = "File")
o1 <- OutputParam(id = "cwl_cout", type = "File", outputSource = "Compile/output")
```
For the input "cwl_zifle", it refers to the `GZ` input `zfile`.  The output "cwl_cout" will be the outcome of `Rs` output `Rout`.

The `cwlStepParam` is used to define `inputs` and `outputs` from previous step. Then it connects with the two steps with `Step` function. The `run` option refer to the corresponding `cwlParam` object and the `In` option in steps should be linked to the input parameters defined by `cwlStepParam`. In the end, we use `+` to connect all steps.
```{r}
cwl <- cwlStepParam(inputs = InputParamList(i1),
                    outputs = OutputParamList(o1))
s1 <- Step(id = "Uncomp", run = GZ,
           In = list(zfile = "cwl_zfile"))
s2 <- Step(id = "Compile", run = Rs,
           In = list(rfile = "Uncomp/rfile"))
cwl <- cwl + s1 + s2
cwl
```

Let's run the pipeline.
```{r}
cwl$cwl_zfile <- zzfil
r7 <- runCWL(cwl, outdir = tempdir())
readLines(r7$output)
```

## Scattering pipeline
The scattering feature can specifies the associated workflow step or subworkflow to execute separately over a list of input elements. To use this feature, `ScatterFeatureRequirement` must be specified in the workflow requirements. Different `scatter` methods can be used in the associated step to decompose the input into a discrete set of jobs. More details can be found at: https://www.commonwl.org/v1.0/Workflow.html#WorkflowStep.

Here is an example to execute multiple R scripts. First, we need to set the input and output types to be array of "File", and add the requirments. In the "Compile" step, the scattering input is required to be set with the `scatter` option.
```{r}
i2 <- InputParam(id = "cwl_rfiles", type = "File[]")
o2 <- OutputParam(id = "cwl_couts", type = "File[]", outputSource = "Compile/output")
req1 <- list(class = "ScatterFeatureRequirement")

cwl2 <- cwlStepParam(requirements = list(req1),
                     inputs = InputParamList(i2),
                     outputs = OutputParamList(o2))
s1 <- Step(id = "Compile", run = Rs,
           In = list(rfile = "cwl_rfiles"),
           scatter = "rfile")
cwl2 <- cwl2 + s1
cwl2
```

Multiple R scripts can be assigned to the workflow inputs and executed.
```{r}
cwl2$cwl_rfiles <- c(r4b$output, r4b$output)
r8 <- runCWL(cwl2, outdir = tempdir())
r8$output
```

## Pipeline plot
The function `plotCWL` can be used to visualize the relationship of inputs, outputs and the analysis for a tool or pipeline.

```{r}
plotCWL(cwl)
```

# Web Application
## cwlParam example
Here we build a tool with different types of input parameters.
```{r}
e1 <- InputParam(id = "flag", type = "boolean",
                 prefix = "-f", doc = "boolean flag")
e2 <- InputParam(id = "string", type = "string", prefix = "-s")
e3 <- InputParam(id = "option", type = "string", prefix = "-o")
e4 <- InputParam(id = "int", type = "int", prefix = "-i", default = 123)
e5 <- InputParam(id = "file", type = "File",
                 prefix = "--file=", separate = FALSE)
e6 <- InputParam(id = "array", type = "string[]", prefix = "-A",
                 doc = "separated by comma")
mulEcho <- cwlParam(baseCommand = "echo", id = "mulEcho",
                 label = "Test parameter types",
                 inputs = InputParamList(e1, e2, e3, e4, e5, e6),
                 stdout = "output.txt")
mulEcho
```

## cwlParam to Shiny App
Some input parameters can be predefined in a list, which will be converted to select options in the webapp. An `upload` parameter can be used to defined wether to generate an upload interface for the file type option. If FALSE, the upload field will be text input (file path) instead of file input.
```{r, eval=FALSE}
inputList <- list(option = c("option1", "option2"))
app <- cwlShiny(mulEcho, inputList, upload = TRUE)
runApp(app)
```
![shinyApp](cwlShiny.png)

# Working with R functions
We can wrap an R function to `cwlParam` object by simply assigning the R function to `baseCommand`. This could be useful to summarize results from other tools in a pipeline. It can also be used to benchmark different parameters for a method written in R. Please note that this feature is only implemented by `Rcwl`, but not available in the common workflow language.

```{r}
fun1 <- function(x)x*2
testFun <- function(a, b){
    cat(fun1(a) + b^2, sep="\n")
}
assign("fun1", fun1, envir = .GlobalEnv)
assign("testFun", testFun, envir = .GlobalEnv)
p1 <- InputParam(id = "a", type = "int", prefix = "a=", separate = F)
p2 <- InputParam(id = "b", type = "int", prefix = "b=", separate = F)
o1 <- OutputParam(id = "o", type = "File", glob = "rout.txt")
TestFun <- cwlParam(baseCommand = testFun,
                    inputs = InputParamList(p1, p2),
                    outputs = OutputParamList(o1),
                    stdout = "rout.txt")
TestFun$a <- 1
TestFun$b <- 2
r1 <- runCWL(TestFun, cwlArgs = "--preserve-entire-environment")
readLines(r1$output)
```

The `runCWL` function wrote the `testFun` function and its dependencies into an R script file automatically and call `Rscript` to run the script with parameters. Each parameter requires a prefix from corresponding argument in the R function with "=" and without a separator. Here we assigned the R function and its dependencies into the global environment because it will start a new environment when the vignette compiled.

# Resources
## RcwlPipelines

The `Rcwl` package can be utilized to develop pipelines for best practices of reproducible research, especially for Bioinformatics study. Multiple Bioinformatics pipelines, such as RNASeq alignment, quality control and quantification, DNASeq alignment and variant calling, have been developed based on the tool in an R package `RcwlPipelines`, which contains the CWL recipes and the scripts to create the pipelines. Examples to analyze real data are also included. 

The package is currently available in github.

* <https://github.com/hubentu/RcwlPipelines>

To install the package.
```{r, eval=FALSE}
BiocManager::install("hubentu/RcwlPipelines")
```

More recipes will be collected in this package and we would like to invite community to submit more pipelines to this package.

## Tool collections in CWL format
Plenty of Bioinformatics tools and workflows can be found from github in CWL format. They can be imported to `cwlParam` object by `readCWL` function, or can be used directly.

* <https://github.com/common-workflow-language/workflows>
* <https://github.com/Duke-GCB/GGR-cwl>
* <https://github.com/pitagora-galaxy/cwl>

## Docker for Bioinformatics tools
Most of the Bioinformatics software are available in docker containers, which can be very convenient to be adopted to build portable CWL tools and pipelines.

* <https://dockstore.org>
* <https://biocontainers.pro>
  
# SessionInfo
```{r}
sessionInfo()
```