TL;DR
The progressify package makes it extremely simple to report progress updates for your existing map-reduce calls. All you need to know is that there is a single function called progressify() that will take care of everything, e.g.
y <- lapply(x, fcn) |> progressify()
y <- map(x, fcn) |> progressify()
y <- foreach(x = xs) %do% { fcn(x) } |> progressify()
res <- boot::boot(data, statistic, R = 100) |> progressify()The progressify() function signals progress updates via the progressr package, meaning you can choose among the supported progressr handlers on how to render progress updates, whether it be via the terminal, a progress bar, or even a sound. The progressify package has only one hard dependency - the progressr package.
In addition to getting progress updates via progressr, by using progressify() you also get access to all the benefits that come with progressr. Notably, you have full control over when and how progress is reported, and it works seamlessly across different environments and frontends.
Supported map-reduce packages
The progressify package supports transpilation of functions from multiple packages. The table below summarizes the supported map-reduce functions. To programmatically see which packages are currently supported, use:
To see which functions are supported for a specific package, use:
progressify_supported_functions("purrr")| Package | Functions |
|---|---|
| base |
apply(), by(), eapply(), lapply(), .mapply(), mapply(), Map(), replicate(), sapply(), tapply(), vapply()
|
| stats | dendrapply() |
| future.apply |
future_apply(), future_by(), future_eapply(), future_lapply(), future_.mapply(), future_mapply(), future_Map(), future_replicate(), future_sapply(), future_tapply(), future_vapply()
|
| purrr |
map() and variants, walk() and variants, map2() and variants, walk2() and variants, pmap() and variants, pwalk(), imap() and variants, modify(), modify2(), imodify()
|
| crossmap |
xmap() and variants, xwalk(), map_vec(), map2_vec(), pmap_vec(), imap_vec(), plus future_*() variants |
| furrr |
future_map() and variants, future_walk() and variants, future_map2() and variants, future_walk2() and variants, future_pmap() and variants, future_pwalk(), future_imap() and variants, future_modify(), future_modify2(), future_imodify() and variants |
| foreach |
%do%, %dopar%
|
| doFuture | %dofuture% |
| plyr |
llply() and variants, mlply() and variants, rdply(), rlply(), raply(), r_ply()
|
Table 1: Map-reduce functions currently supported by progressify() for progress reporting.
Here are some examples:
library(progressify)
handlers(global = TRUE)
xs <- 1:10
ys <- lapply(xs, function(x) { Sys.sleep(0.1); sqrt(x) }) |> progressify()
d <- as.dendrogram(hclust(dist(USArrests)))
d2 <- dendrapply(d, function(n) { Sys.sleep(0.01); n }) |> progressify()
xs <- 1:10
ys <- purrr::map(xs, function(x) { Sys.sleep(0.1); sqrt(x) }) |> progressify()
xs <- list(1:5, 1:5)
ys <- crossmap::xmap(xs, ~ .y * .x) |> progressify()
library(foreach)
xs <- 1:10
ys <- foreach(x = xs) %do% { Sys.sleep(0.1); sqrt(x) } |> progressify()
xs <- 1:10
ys <- plyr::llply(xs, function(x) { Sys.sleep(0.1); sqrt(x) }) |> progressify()Supported domain-specific packages
You can also progressify calls from a growing set of domain-specific CRAN packages that have optional built-in support for parallelization.
CRAN packages with support for progressify
| Package | Functions |
|---|---|
| boot |
boot(), censboot(), tsboot()
|
| car | Boot() |
| fwb | fwb() |
| lme4 |
bootMer(), confint(..., method = "boot")
|
| partykit |
cforest(), varimp()
|
| sandwich |
vcovBS(), vcovJK()
|
| SimDesign | runSimulation() |
| Sim.DiffProc | MCM.sde() |
| tm |
tm_map(), TermDocumentMatrix(), DocumentTermMatrix(), tm_index()
|
Table 2: CRAN packages with domain-specific functions currently supported by progressify() for progress reporting.
Here are some examples:
res <- boot::boot(data, statistic, R = 100) |> progressify()
res <- car::Boot(fit, R = 999) |> progressify()
res <- fwb::fwb(data, statistic, R = 100) |> progressify()
res <- lme4::bootMer(fit, statistic, nsim = 100) |> progressify()
ci <- confint(fit, method = "boot", nsim = 100) |> progressify()
forest <- partykit::cforest(Survived ~ ., data = as.data.frame(Titanic), ntree = 50L) |> progressify()
vi <- partykit::varimp(forest) |> progressify()
v <- sandwich::vcovBS(fit) |> progressify()
res <- SimDesign::runSimulation(design, replications, generate, analyse, summarise) |> progressify()
res <- Sim.DiffProc::MCM.sde(model, statistic, R = 100) |> progressify()
tdm <- tm::TermDocumentMatrix(corpus) |> progressify()Compatible with futurize
The progressify package is compatible with the futurize package, which parallelizes code via the futureverse. You can combine the two, in either order, to get both progress reporting and parallelization:
library(progressify)
handlers(global = TRUE)
library(futurize)
plan(multisession)
xs <- 1:100
ys <- lapply(xs, slow_fcn) |> progressify() |> futurize()
ys <- purrr::map(xs, slow_fcn) |> progressify() |> futurize()
library(foreach)
ys <- foreach(x = xs) %do% { slow_fcn(x) } |> progressify() |> futurize()
ys <- plyr::llply(xs, slow_fcn) |> progressify() |> futurize()
res <- boot::boot(data, statistic, R = 100) |> progressify() |> futurize()An exception is functions where both progressify and futurize work by setting the same argument or option, which only one of them can control. This is the case for partykit (applyfun), sandwich (applyfun), and tm (tm_parLapply_engine()). These cannot yet be combined. With |> progressify() |> futurize(), the partykit and sandwich functions give an error and the tm functions run in parallel without reporting on progress. With |> futurize() |> progressify(), they report on progress, but run sequentially. For Sim.DiffProc, use |> futurize() |> progressify(), which reports on progress and runs in parallel, whereas |> progressify() |> futurize() gives an error.
