%\VignetteIndexEntry{3. Errors, Logs and Debugging}
%\VignetteKeywords{parallel, Infrastructure}
%\VignettePackage{BiocParallel}
%\VignetteEngine{knitr::knitr}


\documentclass{article}

<<style, eval=TRUE, echo=FALSE, results="asis">>=
BiocStyle::latex()
@

\newcommand{\BiocParallel}{\Biocpkg{BiocParallel}}

\title{Errors, Logs and Debugging in \BiocParallel}
\author{Valerie Obenchain and Martin Morgan}
\date{Edited: December 16, 2015; Compiled: \today}

\begin{document}

\maketitle

\tableofcontents

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

This vignette is part of the \BiocParallel{} package and focuses on error
handling and logging. A section at the end demonstrates how the two can be used
together as part of an effective debugging routine.

\BiocParallel{} provides a unified interface to the parallel infrastructure in
several packages including \CRANpkg{snow}, \CRANpkg{parallel},
\CRANpkg{batchtools} and \CRANpkg{foreach}.  When implementing error handling in
\BiocParallel{} the primary goals were to enable the return of partial results
when an error is thrown (vs just the error) and to establish logging on the
workers. In cases where error handling existed, such as \CRANpkg{batchtools} and
\CRANpkg{foreach}, those behaviors were preserved.  Clusters created with
\CRANpkg{snow} and \CRANpkg{parallel} now have flexible error handling and
logging available through \Rcode{SnowParam} and \Rcode{MulticoreParam} objects.

In this document the term ``job'' is used to describe a single call to a
bp*apply function (e.g., the \Rcode{X} in \Rcode{bplapply}). A ``job'' consists
of one or more ``tasks'', where each ``task'' is run separately on a worker.

The \Rpackage{BiocParallel} package is available at bioconductor.org
and can be downloaded via \Rcode{BiocManager::install}:

<<BiocManager, eval=FALSE>>=
if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")
BiocManager::install("BiocParallel")
@

Load the package:

<<load>>=
library(BiocParallel)
@

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Error Handling}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Messages and warnings}

\BiocParallel{} captures messages and warnings in each job, returning
the output to the manager and reporting these to the user after the
completion of the entire operation. Thus
<<messages, eval = FALSE>>=
res <- bplapply(1:2, function(i) { message(i); Sys.sleep(3) })
@
%%
reports messages only after the entire \Rcode{bplapply()} is
complete.

It may be desired to output messages immediatly. Do this using
\Rcode{sink()}, as in the following example:
<<messages-immediate, eval = FALSE>>=
res <- bplapply(1:2, function(i) {
    sink(NULL, type = "message")
    message(i)
    Sys.sleep(3)
})
@
%%
This could be confusing when multiple workers write messages at the
same time --the messages will be interleaved in an arbitrary way -- or
when the workers are not all running on the same computer (e.g., with
\Rcode{SnowParam()} or \Rcode{BatchjobsParaam()}) so should not be
used in package code.

\subsection{Catching errors}

By default, \BiocParallel{} attempts all computations and returns any warnings
and errors along with successful results.  The \Rcode{stop.on.error} field
controls if the job is terminated as soon as one task throws an error. This is
useful when debugging or when running large jobs (many tasks) and you want to
be notified of an error before all runs complete.

\Rcode{stop.on.error} is \Rcode{TRUE} by default.

<<errors_constructor>>=
param <- SnowParam()
param
@

The field can be set when constructing the param or modified with the
\Rcode{bpstopOnError} accessor.

<<errors_stopOnError>>=
param <- SnowParam(2, stop.on.error = TRUE)
param
bpstopOnError(param) <- FALSE
@

In this example \Rcode{X} is length 6. By default, the elements of \Rcode{X}
are divided as evenly as possible over the number of workers and run in chunks.
The number of tasks is set equal to the length of \Rcode{X} which forces
each element of \Rcode{X} to be executed separately (6 tasks).

<<errors_6tasksA_stopOnError>>=
X <- list(1, "2", 3, 4, 5, 6)
param <- SnowParam(3, tasks = length(X), stop.on.error = TRUE)
@

Tasks 1, 2, and 3 are assigned to the three workers, and are
evaluated. Task 2 fails, stopping further computation. All
successfully completed tasks are returned and can be accessed by `bpresult`.
Usually, this means that the results of tasks 1, 2, and 3
will be returned.

<<errors_6tasksA_stopOnError_output>>=
result <- tryCatch({
    bplapply(X, sqrt, BPPARAM = param)
}, error=identity)
result
bpresult(result)
@

Using \Rcode{stop.on.error=FALSE}, all tasks are evaluated.

<<errors_6tasks_nonstopOnError>>=
X <- list("1", 2, 3, 4, 5, 6)
param <- SnowParam(3, tasks = length(X), stop.on.error = FALSE)
result <- tryCatch({
    bplapply(X, sqrt, BPPARAM = param)
}, error=identity)
result
bpresult(result)
@

\Rcode{bptry()} is a convenient way of trying to evaluate a
\Rcode{bpapply}-like expression, returning the evaluated results
without signalling an error.

<<error_bptry>>=
bptry({
    bplapply(X, sqrt, BPPARAM=param)
})
@

In the next example the elements of \Rcode{X} are grouped instead
of run separately. The default value for \Rcode{tasks} is 0 which means 'X' is
split as evenly as possible across the number of workers. There are 3
workers so the first task consists of list(1, 2), the second is list("3", 4)
and the third is list(5, 6).

<<errors_3tasksA_stopOnError>>=
X <- list(1, 2, "3", 4, 5, 6)
param <- SnowParam(3, stop.on.error = TRUE)
@

The output shows an error in when evaluating the third element, but
also that the fourth element, in the same chunk as 3, was not
evaluated. All elements are evaluated because they were assigned to
workers before the first error occurred.

<<errors_3tasksA_stopOnError_output>>=
bptry(bplapply(X, sqrt, BPPARAM = param))
@

Side Note: Results are collected from workers as they finish which is not
necessarily the same order in which they were loaded.  Depending on how tasks
are divided it is possible that the task with the error completes after all
others so essentially all workers complete before the job is stopped.  In this
situation the output includes all results along with the error message and it
may appear that \Rcode{stop.on.error=TRUE} did not stop the job soon enough.
This is just a heads up that the usefulness of \Rcode{stop.on.error=TRUE} may
vary with run time and distribution of tasks over workers.

\subsection{Identify failures with \Rcode{bpok()}}

The \Rcode{bpok()} function is a quick way to determine which (if any)
tasks failed. In this example we use \Rcode{bptry()} to retrieve the
partially evaluated expression, including the failed elements.

<<errors_bpok_bplapply>>=
param <- SnowParam(2, stop.on.error=FALSE)
result <- bptry(bplapply(list(1, "2", 3), sqrt, BPPARAM=param))
@

\Rcode{bpok} returns TRUE if the task was successful.

<<errors_bpok>>=
bpok(result)
@

Once errors are identified with \Rcode{bpok} the traceback can be retrieved
with the \Rcode{attr} function. This is possible because errors are returned
as \Rcode{condition} objects with the traceback as an attribute.

<<errors_traceback>>=
tail(attr(result[[which(!bpok(result))]], "traceback"))
@

\subsection{Rerun failed tasks with \Rcode{BPREDO}}

Tasks can fail due to hardware problems or bugs in the input data.  The
\BiocParallel{} functions support a \Rcode{BPREDO} (re-do) argument for
recomputing only the tasks that failed. A list of partial results and errors is
supplied to \Rcode{BPREDO} in a second call to the function. The failed
elements are identified, recomputed and inserted into the original results.

The bug in this example is the second element of 'X' which is a character when
it should be numeric.

<<redo_error>>=
X <- list(1, "2", 3)
param <- SnowParam(2, stop.on.error=FALSE)
result <- bptry(bplapply(X, sqrt, BPPARAM=param))
result
@


First fix the input data.

<<errors_BPREDO_input>>=
X.redo <- list(1, 2, 3)
@

Repeat the call to \Rcode{bplapply} this time supplying the partial
results as \Rcode{BPREDO}. Only the failed calculations are computed,
in the present case requiring only one worker.

<<redo_run>>=
bplapply(X.redo, sqrt, BPREDO=result, BPPARAM=param)
@

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Logging}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

NOTE: Logging as described in this section is supported for SnowParam,
MulticoreParam and SerialParam.

\subsection{Parameters}

Logging in \BiocParallel{} is controlled by 3 fields in the
\Rcode{BiocParallelParam}:

\begin{verbatim}
  log:       TRUE or FALSE
  logdir:    location to write log file
  threshold: one of "TRACE", "DEBUG", "INFO", "WARN", "ERROR", "FATAL"
\end{verbatim}

When \Rcode{log = TRUE} the \CRANpkg{futile.logger} package is loaded on
each worker. \BiocParallel{} uses a custom script on the workers to
collect log messages as well as additional statistics such as gc, runtime
and node information. Output to stderr and stdout is also captured.

By default \Rcode{log} is FALSE and \Rcode{threshold} is {\it INFO}.

<<logs_constructor>>=
param <- SnowParam(stop.on.error=FALSE)
param
@

Turn logging on and set the threshold to {\it TRACE}.

<<logs_accessors>>=
bplog(param) <- TRUE
bpthreshold(param) <- "TRACE"
param
@

\subsection{Setting a threshold}

All thresholds defined in \CRANpkg{futile.logger} are supported: {\it
FATAL}, {\it ERROR}, {\it WARN}, {\it INFO}, {\it DEBUG} and {\it TRACE}.  All
messages greater than or equal to the severity of the threshold are shown. For
example, a threshold of {\it INFO} will print all messages tagged as {\it
FATAL}, {\it ERROR}, {\it WARN} and {\it INFO}.

Because the default threshold is {\it INFO} it catches the {\it ERROR}-level
message thrown when attempting the square root of a character ("2").

<<logs_bplapply>>=
tryCatch({
    bplapply(list(1, "2", 3), sqrt, BPPARAM = param)
}, error=function(e) invisible(e))
@

All user-supplied messages written in the \CRANpkg{futile.logger} syntax
are also captured. This function performs argument checking and includes
a couple of {\it WARN} and {\it DEBUG}-level messages.

<<logs_FUN>>=
FUN <- function(i) {
  futile.logger::flog.debug(paste("value of 'i':", i))

  if (!length(i)) {
      futile.logger::flog.warn("'i' has length 0")
      NA
  } else if (!is(i, "numeric")) {
      futile.logger::flog.debug("coercing 'i' to numeric")
      as.numeric(i)
  } else {
      i
  }
}
@

Turn logging on and set the threshold to {\it WARN}.

<<logs_FUN_WARN>>=
param <- SnowParam(2, log = TRUE, threshold = "WARN", stop.on.error=FALSE)
result <- bplapply(list(1, "2", integer()), FUN, BPPARAM = param)
simplify2array(result)
@

Changing the threshold to {\it DEBUG} catches both {\it WARN} and
{\it DEBUG} messages.

<<logs_FUN_DEBUG>>=
param <- SnowParam(2, log = TRUE, threshold = "DEBUG", stop.on.error=FALSE)
result <- bplapply(list(1, "2", integer()), FUN, BPPARAM = param)
simplify2array(result)
@

\subsection{Log files}

When \Rcode{log == TRUE}, log messages are written to the console by default.
If \Rcode{logdir} is given the output is written out to files, one per task.
File names are prefixed with the name in \Rcode{bpjobname(BPPARAM)}; default
is 'BPJOB'.

\begin{verbatim}
param <- SnowParam(2, log = TRUE, threshold = "DEBUG", logdir = tempdir())
res <- bplapply(list(1, "2", integer()), FUN, BPPARAM = param)
## loading futile.logger on workers
list.files(bplogdir(param))
## [1] "BPJOB.task1.log" "BPJOB.task2.log"
\end{verbatim}

Read in BPJOB.task2.log:

\begin{verbatim}
readLines(paste0(bplogdir(param), "/BPJOB.task2.log"))

##  [1] "############### LOG OUTPUT ###############"
##  [2] "Task: 2"
##  [3] "Node: 2"
##  [4] "Timestamp: 2015-07-08 09:03:59"
##  [5] "Success: TRUE"
##  [6] "Task duration: "
##  [7] "   user  system elapsed "
##  [8] "  0.009   0.000   0.011 "
##  [9] "Memory use (gc): "
## [10] "         used (Mb) gc trigger (Mb) max used (Mb)"
## [11] "Ncells 325664 17.4     592000 31.7   393522 21.1"
## [12] "Vcells 436181  3.4    1023718  7.9   530425  4.1"
## [13] "Log messages:"
## [14] "DEBUG [2015-07-08 09:03:59] value of 'i': 2"
## [15] "INFO [2015-07-08 09:03:59] coercing to numeric"
## [16] "DEBUG [2015-07-08 09:03:59] value of 'i': "
## [17] "WARN [2015-07-08 09:03:59] 'i' is missing"
## [18] ""
## [19] "stderr and stdout:"
## [20] "character(0)"
\end{verbatim}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Worker timeout}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

NOTE: \Rcode{timeout} is supported for SnowParam and MulticoreParam.

For long running jobs or untested code it can be useful to set a time limit.
The \Rcode{timeout} field is the time, in seconds, allowed for each worker to
complete a task; default is \Rcode{Inf}. If the task takes longer than
\Rcode{timeout} a timeout error is returned.

Time can be changed during param construction with the \Rcode{timeout} arg,

<<timeout_constructor>>=
param <- SnowParam(timeout = 20, stop.on.error=FALSE)
param
@

or with the \Rcode{bptimeout} setter:

<<timeout_setter>>=
param <- SnowParam(timeout = 2, stop.on.error=FALSE)
fun <- function(i) {
  Sys.sleep(i)
  i
}
bptry(bplapply(1:3, fun, BPPARAM = param))
@

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Debugging}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Effective debugging strategies vary by problem and often involve a combination
of error handling and logging techniques.  In general, when debugging
\R{}-generated errors the traceback is often the best place to start followed
by adding debug messages to the worker function. When trouble shooting
unexpected behavior (i.e., not a formal error or warning) adding debug messages
or switching to \Rcode{SerialParam} are good approaches. Below is an overview
of these different strategies.

\subsection{Accessing the traceback}

The traceback is a good place to start when tracking down \R{}-generated
errors. Because the function is executed on the workers it's not accessible for
interactive debugging with functions such as \Rcode{trace} or \Rcode{debug}.
The traceback provides a snapshot of the state of the worker at the time
the error was thrown.

This function takes the square root of the absolute value of a vector.

<<debug_sqrtabs>>=
fun1 <- function(x) {
    v <- abs(x)
    sapply(1:length(v), function(i) sqrt(v[i]))
}
@

Calling ``fun1'' with a character throws an error:

\begin{verbatim}
param <- SnowParam(stop.on.error=FALSE)
result <- bptry({
    bplapply(list(c(1,3), 5, "6"), fun1, BPPARAM = param)
}, error=identity)
result

## [[1]]
## [1] 1.000000 1.732051
##
## [[2]]
## [1] 2.236068
##
## [[3]]
## <remote_error in abs(x): non-numeric argument to mathematical function>
## traceback() available as 'attr(x, "traceback")'
\end{verbatim}

Identify which elements failed with \Rcode{bpok}:

\begin{verbatim}
bpok(result)

## [1]  TRUE  TRUE FALSE
\end{verbatim}

The error (i.e., third element of ``res'') is a \Rcode{condition} object:

\begin{verbatim}
is(result[[3]], "condition")

## [1] TRUE
\end{verbatim}


The traceback is an attribute of the \Rcode{condition} and can be accessed with
the \Rcode{attr} function.

\begin{verbatim}
noquote(tail(attr(result[[3]], "traceback")))

## [1]        call <- sapply(sys.calls(), deparse)
## [2]        e <- structure(e, class = c("remote_error", "condition"),
## [3]            traceback = capture.output(traceback(call)))
## [4]        invokeRestart("abort", e)
## [5]    }, "non-numeric argument to mathematical function", quote(abs(x)))
## [6] 1: h(simpleError(msg, call))
\end{verbatim}

\subsection{Adding debug messages}

When a \Rcode{numeric()} is passed to ``fun1'' no formal error is thrown
but the length of the second list element is 2 when it should be 1.

\begin{verbatim}
bplapply(list(c(1,3), numeric(), 6), fun1, BPPARAM = param)

## [[1]]
## [1] 1.000000 1.732051
##
## [[2]]
## [[2]][[1]]
## [1] NA
##
## [[2]][[2]]
## numeric(0)
##
## [[3]]
## [1] 2.44949
\end{verbatim}

Without a formal error we have no traceback so we'll add a few debug
messages. The \CRANpkg{futile.logger} syntax tags messages with
different levels of severity. A message created with \Rcode{flog.debug} will
only print if the threshold is {\it DEBUG} or lower. So in this case it will
catch both INFO and DEBUG messages.

``fun2'' has debug statements that show the value of `x', length of `v' and
the index `i'.

<<debug_fun1_debug>>=
fun2 <- function(x) {
    v <- abs(x)
    futile.logger::flog.debug(
      paste0("'x' = ", paste(x, collapse=","), ": length(v) = ", length(v))
    )
    sapply(1:length(v), function(i) {
      futile.logger::flog.info(paste0("'i' = ", i))
      sqrt(v[i])
    })
}
@

Create a param that logs at a threshold level of {\it DEBUG}.

<<debug_param_debug>>=
param <- SnowParam(3, log = TRUE, threshold = "DEBUG")
@

The debug messages reveal the problem occurs when `x' is \Rcode{numeric()}.
The index for \Rcode{sapply} is along `v' which in this case has length 0.
This forces `i' to take values of `1' and `0' giving an output of length 2 for
the second element (i.e., \Rcode{NA} and \Rcode{numeric(0)}).

<<debug_DEBUG>>=
res <- bplapply(list(c(1,3), numeric(), 6), fun2, BPPARAM = param)
res
@

``fun2'' can be fixed by using \Rcode{seq\_along(v)} to create the index
instead of \Rcode{1:length(v)}.

\subsection{Local debugging with \Rcode{SerialParam}}

Errors that occur on parallel workers can be difficult to debug. Often the
traceback sent back from the workers is too much to parse or not informative.
We are also limited in that our interactive strategies of
\Rcode{browser} and \Rcode{trace} are not available.

One option for further debugging is to run the code in serial with
\Rcode{SerialParam}.  This removes the ``parallel'' component and is
the same as running a straight \Rcode{*apply} function. This approach
may not help if the problem was hardware related but can be very
useful when the bug is in the \R{} code.

We use the now familiar square root example with a bug in the second element
of \Rcode{X}.

<<debug_sqrt>>=
res <- bptry({
    bplapply(list(1, "2", 3), sqrt,
             BPPARAM = SnowParam(3, stop.on.error=FALSE))
})
result
@

\Rcode{sqrt} is an internal function. The problem is likely with our data
going into the function and not the \Rcode{sqrt} function itself. We can
write a small wrapper around \Rcode{sqrt} so we can see the input.

<<debug_sqrt_wrap>>=
fun3 <- function(i) sqrt(i)
@

Debug the new function:

\begin{verbatim}
debug(fun3)
\end{verbatim}

We want to recompute only elements that failed and for that we use
the \Rcode{BPREDO} argument. The BPPARAM has been changed to
\Rcode{SerialParam} so the job is run in the local workspace in serial.

\begin{verbatim}
> bplapply(list(1, "2", 3), fun3, BPREDO = result, BPPARAM = SerialParam())
Resuming previous calculation ...
debugging in: FUN(...)
debug: sqrt(i)
Browse[2]> objects()
[1] "i"
Browse[2]> i
[1] "2"
Browse[2]>
\end{verbatim}

The local browsing allowed us to see the problem input was the
character "2".


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{\Rcode{sessionInfo()}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

<<sessionInfo, results="asis">>=
toLatex(sessionInfo())
@

\end{document}