set.seed() not working with function that uses future_lapply()

[joelnitta]

Apologies - I haven't been able to come up with a simpler reprex. But I hope that perhaps the symptoms might ring a bell.

I am using future_lapply() inside of the cpr_rand_test() function in my package 'canaper'

cpr_rand_test() is an exported, user-facing function that calls some internal functions to conduct some randomizations and generate summary statistics. I have confirmed that each of those internal functions are reproducible with set.seed(). But the problem is, when I try to use set.seed() with cpr_rand_test(), it shows some odd behavior: the first call after plan() is not reproducible, but subsequent ones are. FWIW, this happens in sequential mode too.

(Issue in my package here)

Any ideas on how to fix this?

Thank you very much!

library(canaper)
library(future)
library(testthat)

plan(multisession, workers = 2)

set.seed(12345)
res_1 <- cpr_rand_test(biod_example$comm, biod_example$phy, null_model = "curveball", n_iterations = 10, n_reps = 10)

set.seed(67890)
res_2 <- cpr_rand_test(biod_example$comm, biod_example$phy, null_model = "curveball", n_iterations = 10, n_reps = 10)

set.seed(12345)
res_3 <- cpr_rand_test(biod_example$comm, biod_example$phy, null_model = "curveball", n_iterations = 10, n_reps = 10)

set.seed(67890)
res_4 <- cpr_rand_test(biod_example$comm, biod_example$phy, null_model = "curveball", n_iterations = 10, n_reps = 10)

set.seed(12345)
res_5 <- cpr_rand_test(biod_example$comm, biod_example$phy, null_model = "curveball", n_iterations = 10, n_reps = 10)

# the first cpr_rand_test() call is not reproducible
expect_true(isTRUE(all.equal(res_1, res_3)))
#> Error: isTRUE(all.equal(res_1, res_3)) is not TRUE
#> 
#> `actual`:   FALSE
#> `expected`: TRUE
expect_true(isTRUE(all.equal(res_1, res_5)))
#> Error: isTRUE(all.equal(res_1, res_5)) is not TRUE
#> 
#> `actual`:   FALSE
#> `expected`: TRUE

# but subsequent ones are
expect_true(isTRUE(all.equal(res_2, res_4)))
expect_true(isTRUE(all.equal(res_3, res_5)))

^{Created on 2021-10-27 by the reprex package (v2.0.0)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.1.1 (2021-08-10)
#>  os       macOS Catalina 10.15.7      
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       Asia/Tokyo                  
#>  date     2021-10-27                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version date       lib source                            
#>  abind          1.4-5   2016-07-21 [1] CRAN (R 4.1.0)                    
#>  ape            5.5     2021-04-25 [1] CRAN (R 4.1.0)                    
#>  assertr        2.8     2021-01-25 [1] CRAN (R 4.1.0)                    
#>  assertthat     0.2.1   2019-03-21 [1] CRAN (R 4.1.0)                    
#>  backports      1.2.1   2020-12-09 [1] CRAN (R 4.1.0)                    
#>  betapart       1.5.4   2021-04-11 [1] CRAN (R 4.1.0)                    
#>  canaper      * 0.0.1   2021-10-26 [1] Github (joelnitta/canaper@43a9f0a)
#>  cli            3.0.1   2021-07-17 [1] CRAN (R 4.1.0)                    
#>  cluster        2.1.2   2021-04-17 [1] CRAN (R 4.1.1)                    
#>  clustMixType   0.2-15  2021-08-18 [1] CRAN (R 4.1.0)                    
#>  codetools      0.2-18  2020-11-04 [1] CRAN (R 4.1.1)                    
#>  colorspace     2.0-2   2021-06-24 [1] CRAN (R 4.1.0)                    
#>  crayon         1.4.1   2021-02-08 [1] CRAN (R 4.1.0)                    
#>  DBI            1.1.1   2021-01-15 [1] CRAN (R 4.1.0)                    
#>  diffobj        0.3.5   2021-10-05 [1] CRAN (R 4.1.0)                    
#>  digest         0.6.28  2021-09-23 [1] CRAN (R 4.1.0)                    
#>  dismo          1.3-5   2021-10-11 [1] CRAN (R 4.1.0)                    
#>  doSNOW         1.0.19  2020-10-16 [1] CRAN (R 4.1.0)                    
#>  dplyr          1.0.7   2021-06-18 [1] CRAN (R 4.1.0)                    
#>  ellipsis       0.3.2   2021-04-29 [1] CRAN (R 4.1.0)                    
#>  evaluate       0.14    2019-05-28 [1] CRAN (R 4.1.0)                    
#>  fansi          0.5.0   2021-05-25 [1] CRAN (R 4.1.0)                    
#>  fastmap        1.1.0   2021-01-25 [1] CRAN (R 4.1.0)                    
#>  fastmatch      1.1-3   2021-07-23 [1] CRAN (R 4.1.0)                    
#>  foreach        1.5.1   2020-10-15 [1] CRAN (R 4.1.0)                    
#>  fs             1.5.0   2020-07-31 [1] CRAN (R 4.1.0)                    
#>  future       * 1.22.1  2021-08-25 [1] CRAN (R 4.1.0)                    
#>  future.apply   1.8.1   2021-08-10 [1] CRAN (R 4.1.0)                    
#>  generics       0.1.1   2021-10-25 [1] CRAN (R 4.1.1)                    
#>  geometry       0.4.5   2019-12-04 [1] CRAN (R 4.1.0)                    
#>  globals        0.14.0  2020-11-22 [1] CRAN (R 4.1.0)                    
#>  glue           1.4.2   2020-08-27 [1] CRAN (R 4.1.0)                    
#>  highr          0.9     2021-04-16 [1] CRAN (R 4.1.0)                    
#>  htmltools      0.5.2   2021-08-25 [1] CRAN (R 4.1.0)                    
#>  igraph         1.2.7   2021-10-15 [1] CRAN (R 4.1.0)                    
#>  iterators      1.0.13  2020-10-15 [1] CRAN (R 4.1.0)                    
#>  itertools      0.1-3   2014-03-12 [1] CRAN (R 4.1.0)                    
#>  knitr          1.36    2021-09-29 [1] CRAN (R 4.1.0)                    
#>  lattice        0.20-44 2021-05-02 [1] CRAN (R 4.1.1)                    
#>  lifecycle      1.0.1   2021-09-24 [1] CRAN (R 4.1.0)                    
#>  listenv        0.8.0   2019-12-05 [1] CRAN (R 4.1.0)                    
#>  magic          1.5-9   2018-09-17 [1] CRAN (R 4.1.0)                    
#>  magrittr       2.0.1   2020-11-17 [1] CRAN (R 4.1.0)                    
#>  MASS           7.3-54  2021-05-03 [1] CRAN (R 4.1.1)                    
#>  Matrix         1.3-4   2021-06-01 [1] CRAN (R 4.1.1)                    
#>  mgcv           1.8-36  2021-06-01 [1] CRAN (R 4.1.1)                    
#>  nlme           3.1-152 2021-02-04 [1] CRAN (R 4.1.1)                    
#>  parallelly     1.28.1  2021-09-09 [1] CRAN (R 4.1.0)                    
#>  permute        0.9-5   2019-03-12 [1] CRAN (R 4.1.0)                    
#>  phangorn       2.7.1   2021-07-13 [1] CRAN (R 4.1.0)                    
#>  phyloregion    1.0.6   2021-05-01 [1] CRAN (R 4.1.0)                    
#>  picante        1.8.2   2020-06-10 [1] CRAN (R 4.1.0)                    
#>  pillar         1.6.4   2021-10-18 [1] CRAN (R 4.1.0)                    
#>  pkgconfig      2.0.3   2019-09-22 [1] CRAN (R 4.1.0)                    
#>  progressr      0.9.0   2021-09-24 [1] CRAN (R 4.1.0)                    
#>  purrr          0.3.4   2020-04-17 [1] CRAN (R 4.1.0)                    
#>  quadprog       1.5-8   2019-11-20 [1] CRAN (R 4.1.0)                    
#>  R.cache        0.15.0  2021-04-30 [1] CRAN (R 4.1.0)                    
#>  R.methodsS3    1.8.1   2020-08-26 [1] CRAN (R 4.1.0)                    
#>  R.oo           1.24.0  2020-08-26 [1] CRAN (R 4.1.0)                    
#>  R.utils        2.10.1  2020-08-26 [1] CRAN (R 4.1.0)                    
#>  R6             2.5.1   2021-08-19 [1] CRAN (R 4.1.0)                    
#>  randomForest   4.6-14  2018-03-25 [1] CRAN (R 4.1.0)                    
#>  raster         3.5-2   2021-10-11 [1] CRAN (R 4.1.0)                    
#>  rcdd           1.4     2021-10-19 [1] CRAN (R 4.1.0)                    
#>  RColorBrewer   1.1-2   2014-12-07 [1] CRAN (R 4.1.0)                    
#>  Rcpp           1.0.7   2021-07-07 [1] CRAN (R 4.1.0)                    
#>  rematch2       2.1.2   2020-05-01 [1] CRAN (R 4.1.0)                    
#>  reprex         2.0.0   2021-04-02 [1] CRAN (R 4.1.0)                    
#>  rgeos          0.5-8   2021-09-22 [1] CRAN (R 4.1.0)                    
#>  rlang          0.4.12  2021-10-18 [1] CRAN (R 4.1.0)                    
#>  rmarkdown      2.11    2021-09-14 [1] CRAN (R 4.1.0)                    
#>  rstudioapi     0.13    2020-11-12 [1] CRAN (R 4.1.0)                    
#>  sessioninfo    1.1.1   2018-11-05 [1] CRAN (R 4.1.0)                    
#>  snow           0.4-3   2018-09-14 [1] CRAN (R 4.1.0)                    
#>  sp             1.4-5   2021-01-10 [1] CRAN (R 4.1.0)                    
#>  stringi        1.7.5   2021-10-04 [1] CRAN (R 4.1.0)                    
#>  stringr        1.4.0   2019-02-10 [1] CRAN (R 4.1.0)                    
#>  styler         1.6.2   2021-09-23 [1] CRAN (R 4.1.0)                    
#>  terra          1.4-11  2021-10-11 [1] CRAN (R 4.1.0)                    
#>  testthat     * 3.1.0   2021-10-04 [1] CRAN (R 4.1.0)                    
#>  tibble         3.1.5   2021-09-30 [1] CRAN (R 4.1.0)                    
#>  tidyselect     1.1.1   2021-04-30 [1] CRAN (R 4.1.0)                    
#>  utf8           1.2.2   2021-07-24 [1] CRAN (R 4.1.0)                    
#>  vctrs          0.3.8   2021-04-29 [1] CRAN (R 4.1.0)                    
#>  vegan          2.5-7   2020-11-28 [1] CRAN (R 4.1.0)                    
#>  waldo          0.3.1   2021-09-14 [1] CRAN (R 4.1.0)                    
#>  withr          2.4.2   2021-04-18 [1] CRAN (R 4.1.0)                    
#>  xfun           0.27    2021-10-18 [1] CRAN (R 4.1.0)                    
#>  yaml           2.2.1   2020-02-01 [1] CRAN (R 4.1.0)                    
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.1/Resources/library

[HenrikBengtsson]

My guess is that your calc_biodiv_random() function loads some package dependency, which somehow uses the random number generation (RNG) during load. I've seen this happen for packages that produces a random welcome message when attached.

See my blog post 'Detect When the Random Number Generator Was Used' for how to track that down. It's quite easy. Just cut'n'paste that task callback setup, and do library() on each package dependency one by one. That should help you hone in on the package that does this.

Let me know how it goes, and we can discuss workarounds if this is the cause.

[HenrikBengtsson]

You can also inject several:

message(sprintf("PID %d: RNG=%s", Sys.getpid(), digest::digest(.Random.seed)))

in your future_lapply() code to see at what step the .Random.seed is updated.

[HenrikBengtsson]

ANSWER: Further investigation shows that this happens when the betapart package is loaded on the worker, which in turn loads snow, which generates a random port number when loaded.

Here's a minimal, reproducible example using only future:

library(future)
plan(cluster, workers = 1)
set.seed(42); value(future(runif(1L), seed = TRUE))
#> [1] 0.432844
set.seed(42); value(future(runif(1L), seed = TRUE))
#> [1] 0.432844
set.seed(42); value(future(runif(1L), seed = TRUE))
#> [1] 0.432844

all good so far, but if we load the snow package on our workers, which will only happens once per worker, then that will cause the RNG state to go forward at the first call;

set.seed(42); value(future(runif(1L), seed = TRUE, packages = "snow"))
#> [1] 0.1667548
set.seed(42); value(future(runif(1L), seed = TRUE, packages = "snow"))
#> [1] 0.432844
set.seed(42); value(future(runif(1L), seed = TRUE, packages = "snow"))
#> [1] 0.432844

Solution

I thought about it, and this is not easy to work around. At least not without quite ugly hacks. However, it can be controlled at the core of the future package itself. I've now fixed this in future (>= 1.22.1-9005);

library(future)
packageVersion("future")
#> [1] '1.22.1.9005'
plan(cluster, workers = 1)
set.seed(42); value(future(runif(1L), seed = TRUE, packages = "snow"))
#> [1] 0.432844
set.seed(42); value(future(runif(1L), seed = TRUE, packages = "snow"))
#> [1] 0.432844
set.seed(42); value(future(runif(1L), seed = TRUE, packages = "snow"))
#> [1] 0.432844

See https://future.futureverse.org/#pre-release-version how to install it until available on CRAN.

BUG FIXES:

 * Now future(..., seed) will set the random seed as late as possible just
   before the future expression is evaluated.  Previously it was done before
   package dependencies where attached, which could lead to non-reproduce
   random numbers in case a package dependency would update the RNG seed
   when attached.

I'm not a big fan of packages using the RNG when loaded/attached, for this very reason, but I also understand they do exists. I only consider this a bug in future, because this could have worked around before.

This hopefully saves others from having to troubleshooting this, and, more importantly, it'll help guarantee reproducible results. Thanks for reporting!

Let me know if the updated version solves your specific problem.

[joelnitta]

Thanks so much for the prompt reply and bugfix!

So actually, even after upgrading to the dev version of future (1.22.1-9005), my example was still not fixed...

But you provided a very important hint in the form of the package that was resetting the RNG, betapart. I removed that package (actually, phyloregion, which imports betapart) from my imports, and that solved the problem. As far as I'm concerned, that takes care of it! If you still want to play around with my example above though, you can install my package at the commit it shows in session_info() (ropensci/canaper@43a9f0a) and see what else may be going on.

[HenrikBengtsson]

FOLLOW-UP, IMPROVED ANSWER: Thanks for testing, and thanks for confirming the updated version was not sufficient to resolve your problem. I did some more thinking and the problem you're having/had is of the same nature but trickier to protect against. The culprit is that the loading of the problematic package (here: snow) is deferred because it's part of a Suggests:ed dependency. Here's an example illustrating the problem:

library(future.apply)
plan(cluster, workers = 1)

foo <- function() {
  loadNamespace("snow") ## emulate a Suggests:ed package
  runif(1L)
}

future_sapply(1:3, FUN = function(x) {
  set.seed(42)
  foo()
}, future.seed = TRUE)

#> [1] 0.5547401 0.1738456 0.1738456

I'm actually not sure that I can protect or work around this type of problem in the core of future. Maybe, I could hack something really hacky together to detect the problem (= a package updates the RNG state on load).

The best solution to this type of problem right now is (i) to identify that the problem is there, (ii) understand the problem, (iii) track down the problematic package, and (iv) make sure to pre-load the problematic package. In future (>= 1.22.1-9005) the last part can be done by specifying the package via the packages argument (future.packages when using future.apply). Proof of concept:

library(future.apply)
packageVersion("future")
#> [1] '1.22.1.9005'
plan(cluster, workers = 1)

foo <- function() {
  loadNamespace("snow") ## emulated a Suggests:ed package
  runif(1L)
}

future_sapply(1:3, FUN = function(x) {
  set.seed(42)
  foo()
}, future.seed = TRUE, future.packages = "snow")

#> [1] 0.1738456 0.1738456 0.1738456

[joelnitta]

I see, thanks for looking into it more.

I can confirm that does fix the problem in my earlier code (joelnitta/canaper@43a9f0a). (by now though it's actually preferable to just bypass the package that was causing the issue... fewer dependencies that way anyways!)

I had almost come to the same solution by thrashing about... I tried passing the name of my package, canaper to future.packages, based on an intuition that something in the environment was different between the workers and the main process. I suppose that suffered from the same Suggests problem though... snow isn't actually loaded that way.