Singularity support #617

anton-khodak · 2018-01-19T17:33:34Z

initial code merged from https://github.com/johnfonner/cwltool/tree/feature-singularity
introduces --container-manageroption that allows choosing between Docker and Singularity. The default manager is Docker.

Think about --container-manager=singularity as this is rather verbose.
Should we auto switch to Singularity if there is no docker executable and a singularity executable is found?
document and confirm requirements. Current guess: Singularity v2.3.2 and Linux kernel v3.18 (or earlier with overlayfs support backported).

* merged from https://github.com/johnfonner/cwltool/tree/feature-singularity

anton-khodak · 2018-01-20T15:46:45Z

Think about --container-manager=singularity as this is rather verbose.

suggestions for a better naming?

Should we auto switch to Singularity if there is no docker executable and a singularity executable is found?

Might be a nice feature; currently, cwltool fails if docker command is not found even if the container runtime is singularity.

mr-c · 2018-01-22T16:15:49Z

--- Running conformance test v1.0 on /home/michael/cwltool/env/bin/cwltool ---
/home/michael/cwltool/env/bin/cwltool 1.0.20180119200559
Test [82/114] 

Test failed: /home/michael/cwltool/env/bin/cwltool --container-manager singularity --outdir=/tmp/tmp98bDHq --quiet v1.0/stagefile.cwl v1.0/stagefile-job.yml
Test writable staged files.
Returned non-zero
Got workflow error
Traceback (most recent call last):
  File "/home/michael/cwltool/cwltool/main.py", line 300, in single_job_executor
    r.run(**kwargs)
  File "/home/michael/cwltool/cwltool/job.py", line 527, in run
    self.add_volumes_singularity(self.generatemapper, runtime, True)
  File "/home/michael/cwltool/cwltool/job.py", line 387, in add_volumes_singularity
    shutil.copy(vol.resolved, vol.target)
  File "/usr/lib/python2.7/shutil.py", line 119, in copy
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: '/var/spool/cwl/bob.txt'
Workflow error, try again with --debug for more information:
[Errno 2] No such file or directory: '/var/spool/cwl/bob.txt'

0 tests passed, 1 failures, 0 unsupported features

1 tool tests failed

tetron · 2018-01-22T16:27:11Z

This might be a cwltool bug, I think there is another bug report about this (#614)

mr-c · 2018-01-22T16:34:59Z

@anton-khodak Please examine the code paths involving a DockerRequirement under hints and a writable InitialWorkDirRequirement entry

mr-c · 2018-01-22T17:18:29Z

@anton-khodak and yes, let's switch to --singularity

…arity-support

mr-c · 2018-01-25T11:47:37Z

cwltool/job.py

@@ -362,6 +363,44 @@ def add_volumes(self, pathmapper, runtime):
                        docker_windows_path_adjust(createtmp),
                        docker_windows_path_adjust(vol.target)))

+    def add_volumes_singularity(self, pathmapper, runtime, stage_output):


Would it make sense to move this to signularity.py?

mr-c · 2018-01-25T11:48:19Z

cwltool/job.py


        self._setup(kwargs)

-        if user_space_docker_cmd:
-            runtime = [user_space_docker_cmd, u"run"]
+        if use_singularity:


Likewise this block could also be nice to move to singularity.py

mr-c · 2018-01-28T11:54:24Z

cwltool/job.py

@@ -386,8 +348,7 @@ def run(self, pull_image=True, rm_container=True,
            try:
                env = cast(MutableMapping[Text, Text], os.environ)
                if docker_req and kwargs.get("use_container"):
-                    img_id = str(docker.get_from_requirements(
-                        docker_req, True, pull_image))
+                    img_id = self.get_from_requirements(docker_req, True, pull_image)


cwltool/job.py:351: error: Incompatible types in assignment (expression has type "unicode", variable has type "str")

thanks! forgot that the returned value was wrapped in string

# Conflicts: # cwltool/job.py

dleehr · 2018-01-31T20:58:28Z

Thanks @anton-khodak, very exciting! We've been investigating how to use singularity as a runtime for our CWL workflows and spent some time today with this branch on a handful of our tools from https://github.com/Duke-GCB/bespin-cwl. Some observations/questions from our tests:

This requires Singularity 2.4 for overlay support. cwltool sets up bind-mounts for files/directories that obviously don't exist in the pulled image FS (like /var/spool/cwl/ and /var/lib/cwl/some-uuid/inputfile.fastq).
Maybe a question for the standard, but is there a plan to support Singularity images (or singularity hub)?

Works well though, thanks!

mr-c · 2018-02-01T08:38:51Z

Thank you @dleehr for the review, we really appreciate it!

This requires Singularity 2.4 for overlay support.

According to https://github.com/singularityware/singularity/releases/tag/2.2 the overlayfs support is debuted in Singularity v2.2 (released 2016-10-11). But we should test and document the minimal required version, yes.
As for overlayfs itself, it seems that it debuted in the linux kernel v3.18, so that may also need to be documented.

Maybe a question for the standard, but is there a plan to support Singularity images (or singularity hub)?

I'm not a regular Singularity user, is there a particular advantage to using a Singularity image over converting a Docker image?
Singularity images aren't portable to Docker based systems, but the reverse is. So in the interest of portability that doesn't seem like a good addition to the specification. The good news is that Singularity has committed to supporting the OpenContainer Initiative standards starting with the OCI runtime standard for Singularity 3.0 (thus removing the need for special handling): Support for Singularity compliance with the OCI runtime apptainer/singularity#1235 (comment)

mr-c · 2018-02-01T13:08:25Z

Jenkins, test this please

mr-c · 2018-02-01T13:29:24Z

Jenkins, test this please

mr-c · 2018-02-01T13:53:53Z

@anton-khodak I can reproduce https://ci.commonwl.org/job/cwltool-pr-conformance-multiver/978/version=v1.0/testReport/py2/7_singularity/Test_empty_writable_dir_with_InitialWorkDirRequirement_inside_Docker/ locally, can you look into that?

mr-c · 2018-02-01T15:06:27Z

@anton-khodak There is some weirdness on Jenkins, here's a Vagrantfile that I've been using to test locally (as well as verify the system requirements): https://gist.github.com/mr-c/0ec90d717617d074017c0cb38b72d1a4

dleehr · 2018-02-01T15:35:05Z

overlayfs support is debuted in Singularity v2.2

Ah, good catch. I was conflating this with the Persistent Overlay feature in 2.4. I agree that the kernel support should be highlighted. Our cluster runs RHEL 6.8 with a 2.6.32 kernel. It doesn't support overlay, so even though we upgraded to Singularity 2.4.2, this still won't run.

I'm not a regular Singularity user, is there a particular advantage to using a Singularity image over converting a Docker image?

I suppose I assumed broad support from the branch singularity-support name and --singularity flag. What's here could be characterized as support for using singularity as the container runtime. It's incredibly useful and exactly what I'd want personally for running my Docker-first workflows on our HPC with singularity.

But to answer the question about why not Docker images -

I've found that some Docker images don't run under singularity due to permissions inside the container. Debugging these through the conversion step puts an extra layer in the process.
We're exploring how to promote good container practices on HPC where users don't have access to Docker. We're trying to make that accessible, and building native singularity images is a more direct and accessible solution than a separate docker build setup just to build images that need to be converted.

Singularity images aren't portable to Docker based systems, but the reverse is....Singularity has committed to supporting the OpenContainer Initiative standards starting with the OCI runtime standard for Singularity 3.0

At the end of the day, I agree that support for non-portable Singularity images doesn't seem like a good addition to the specification, especially with the promise of singularity supporting OCI.

mr-c · 2018-02-02T13:54:24Z

overlayfs support is debuted in Singularity v2.2

Ah, good catch. I was conflating this with the Persistent Overlay feature in 2.4. I agree that the kernel support should be highlighted.

As it turns out, we need Singularity 2.4 for their equivalent of docker pull anyhow :-)

Our cluster runs RHEL 6.8 with a 2.6.32 kernel. It doesn't support overlay, so even though we upgraded to Singularity 2.4.2, this still won't run.

Hmm... What if before runtime the relevant directories were added to the image?

I suppose I assumed broad support from the branch singularity-support name and --singularity flag. What's here could be characterized as support for using singularity as the container runtime. It's incredibly useful and exactly what I'd want personally for running my Docker-first workflows on our HPC with singularity.

Got it (and thanks!). Would --singularity-runtime be clearer or just more confusing?

Singularity images aren't portable to Docker based systems, but the reverse is....Singularity has committed to supporting the OpenContainer Initiative standards starting with the OCI runtime standard for Singularity 3.0

At the end of the day, I agree that support for non-portable Singularity images doesn't seem like a good addition to the specification, especially with the promise of singularity supporting OCI.

👍

dleehr · 2018-02-02T14:45:28Z

Hmm... What if before runtime the relevant directories were added to the image?

In theory, yes. I created /var/spool/cwl inside my image and got past one set of errors. But the input files get unique, file-specific mount points inside the container (e.g. /var/lib/cwl/abcde-123-fgh-456/input.txt), so those would need to exist too. I think it's fine to require overlayfs.

Would --singularity-runtime be clearer or just more confusing?

Hard to say. I think it's worth adding a second word there, but don't know if runtime is it. Maybe "runner"? If you think more runtimes will be supported, I'd suggest --container-runner=[docker|singularity|rkt|], where docker is the default.

mr-c · 2018-02-05T17:38:38Z

Odd that some of the tests are failing on Jenkins. Here's an Ubuntu based Vagrantfile where all the tests pass: https://gist.github.com/mr-c/0ec90d717617d074017c0cb38b72d1a4#file-vagrantfile-ubuntu

mr-c · 2018-02-05T17:40:45Z

Would --singularity-runtime be clearer or just more confusing?

Hard to say. I think it's worth adding a second word there, but don't know if runtime is it. Maybe "runner"? If you think more runtimes will be supported, I'd suggest --container-runner=[docker|singularity|rkt|], where docker is the default.

Currently we need two pieces of information: the type of docker-ish runner (docker, udocker/dx-docker, singularity) and the name of the runner's executable (though we can guess based upon the name). In a post-OCI runtime world I hope we can collapse that to just the name of the runner's executable :-)

uniqueg · 2018-02-09T09:51:21Z

This is great - we just wanted to start working on this yesterday and realized that it's already here! :-)

Well, almost anyway - because I have one question/issue: What if (like in our case) the HPC cluster (with Singularity installed) is inside a DMZ and thus unable to pull images from the internet? Is there a way to supply the path to a local Singularity image, ideally at runtime (it wouldn't be so nice to have to hard code paths inside the Command Line Tool descriptors)? So basically: is there a way to pull an image (whether Docker or Singularity, I guess) before the corresponding Command Line Tool is executed and ensuring that the local image is used during execution?

We are planning to use CWL/toil and an HPC (without internet access) managed by Slurm. The server running CWL/toil is outside the DMZ and has both Docker and Singularity installed. I'm not sure how common that setup is, but I could imagine we're not the only ones with this problem.

mr-c · 2018-02-09T10:11:21Z

Hello @uniqueg and thank you for your enthusiasm!

Does https://github.com/common-workflow-language/cwltool/pull/617/files#diff-d28bca6cb02397efd0ad40c0eefadbb8R40 give you what you need?

You should also look into Singularity's SINGULARITY_CACHEDIR feature: http://singularity.lbl.gov/docs-pull#pull-to-different-folder

To find out which images to download I would cwltool --pack your workflow and grep for dockerPull

anton-khodak · 2018-02-24T14:21:05Z

It successfully ran all conformance tests with Vagrant boxes from @mr-c comment

uniqueg · 2018-03-20T22:51:07Z

Hello @uniqueg and thank you for your enthusiasm!

Does https://github.com/common-workflow-language/cwltool/pull/617/files#diff-d28bca6cb02397efd0ad40c0eefadbb8R40 give you what you need?

You should also look into Singularity's SINGULARITY_CACHEDIR feature: http://singularity.lbl.gov/docs-pull#pull-to-different-folder

To find out which images to download I would cwltool --pack your workflow and grep for dockerPull

@mr-c: Thanks a lot for that! The approach works just great after some very slight modifications.
See here for the PR: #695

Singularity support basics

ddb22ab

* merged from https://github.com/johnfonner/cwltool/tree/feature-singularity

anton-khodak force-pushed the singularity-support branch from b3fcf15 to ddb22ab Compare January 19, 2018 17:35

mr-c and others added 3 commits January 19, 2018 15:05

Merge branch 'master' into singularity-support

0e8f1ef

pass CWL tests with singularity, except for #82

ffc8f21

Fix docker tests

01c02c1

Not require docker if runtime is singularity

047863d

mr-c and others added 4 commits January 24, 2018 11:53

Merge branch 'master' into singularity-support

db55d44

Change command line option for Singularity

b702298

Merge remote-tracking branch 'origin/singularity-support' into singul…

0bf444d

…arity-support

Merge branch 'master' into singularity-support

cd61ee6

mr-c reviewed Jan 25, 2018

View reviewed changes

Fix staging writable files for Singularity

3102a10

anton-khodak requested a review from tetron January 25, 2018 16:33

anton-khodak force-pushed the singularity-support branch 2 times, most recently from 2f5f1ab to ea4f587 Compare January 28, 2018 11:05

mr-c reviewed Jan 28, 2018

View reviewed changes

Create separate classes for docker & singularity command line jobs

602b4fe

anton-khodak force-pushed the singularity-support branch from ea4f587 to 602b4fe Compare January 28, 2018 12:13

anton-khodak changed the title ~~[WIP] Singularity support~~ Singularity support Jan 28, 2018

Anton Khodak added 2 commits January 31, 2018 18:43

Merge branch 'master' into singularity-support

7427030

# Conflicts: # cwltool/job.py

Merge branch 'master' into singularity-support

cda467d

# Conflicts: # cwltool/job.py

Merge branch 'master' into singularity-support

12cccc6

C_ALL=C to work around junit-xml ASCII only bug

b358808

mr-c and others added 2 commits February 1, 2018 15:56

try py2 instead of 2.7 to avoid issue with period

c04d991

Fix writable directory staging inside Singularity

2f0ad08

Merge branch 'master' into singularity-support

68dc21d

Merge branch 'master' into singularity-support

83272bc

DanThrane mentioned this pull request Feb 13, 2018

Investigate use of CWL for Apps SDU-eScience/UCloud#31

Closed

Anton Khodak and others added 5 commits February 24, 2018 13:19

Merge branch 'master' into singularity-support

40c09c7

Merge branch 'master' into singularity-support

6265ab2

Merge branch 'master' into singularity-support

031a0e0

Add Singularity notice to documentation

b79779b

Merge branch 'master' into singularity-support

16262ef

mr-c and others added 4 commits February 24, 2018 16:21

remove singularity testing for now

a8f8431

Merge branch 'master' into singularity-support

75a939a

cleanup of parameter descriptions, grouping

00c2e98

restore cwltool specific ability to omit the input object

10bef24

mr-c merged commit 353dbed into master Feb 25, 2018

mr-c deleted the singularity-support branch February 25, 2018 10:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Singularity support #617

Singularity support #617

anton-khodak commented Jan 19, 2018 •

edited by mr-c

Loading

anton-khodak commented Jan 20, 2018 •

edited

Loading

mr-c commented Jan 22, 2018

tetron commented Jan 22, 2018 •

edited

Loading

mr-c commented Jan 22, 2018

mr-c commented Jan 22, 2018

mr-c Jan 25, 2018

mr-c Jan 25, 2018

mr-c Jan 28, 2018

anton-khodak Jan 28, 2018 •

edited

Loading

dleehr commented Jan 31, 2018

mr-c commented Feb 1, 2018

mr-c commented Feb 1, 2018

mr-c commented Feb 1, 2018

mr-c commented Feb 1, 2018

mr-c commented Feb 1, 2018

dleehr commented Feb 1, 2018

mr-c commented Feb 2, 2018

dleehr commented Feb 2, 2018

mr-c commented Feb 5, 2018

mr-c commented Feb 5, 2018

uniqueg commented Feb 9, 2018

mr-c commented Feb 9, 2018

anton-khodak commented Feb 24, 2018

uniqueg commented Mar 20, 2018

Singularity support #617

Singularity support #617

Conversation

anton-khodak commented Jan 19, 2018 • edited by mr-c Loading

anton-khodak commented Jan 20, 2018 • edited Loading

mr-c commented Jan 22, 2018

tetron commented Jan 22, 2018 • edited Loading

mr-c commented Jan 22, 2018

mr-c commented Jan 22, 2018

mr-c Jan 25, 2018

Choose a reason for hiding this comment

mr-c Jan 25, 2018

Choose a reason for hiding this comment

mr-c Jan 28, 2018

Choose a reason for hiding this comment

anton-khodak Jan 28, 2018 • edited Loading

Choose a reason for hiding this comment

dleehr commented Jan 31, 2018

mr-c commented Feb 1, 2018

mr-c commented Feb 1, 2018

mr-c commented Feb 1, 2018

mr-c commented Feb 1, 2018

mr-c commented Feb 1, 2018

dleehr commented Feb 1, 2018

mr-c commented Feb 2, 2018

dleehr commented Feb 2, 2018

mr-c commented Feb 5, 2018

mr-c commented Feb 5, 2018

uniqueg commented Feb 9, 2018

mr-c commented Feb 9, 2018

anton-khodak commented Feb 24, 2018

uniqueg commented Mar 20, 2018

anton-khodak commented Jan 19, 2018 •

edited by mr-c

Loading

anton-khodak commented Jan 20, 2018 •

edited

Loading

tetron commented Jan 22, 2018 •

edited

Loading

anton-khodak Jan 28, 2018 •

edited

Loading