diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 000000000..72a09cde2
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1,4 @@
+*.txt text eol=lf
+*.fastq text eol=lf
+*.fai text eol=lf
+*.fasta text eol=lf
diff --git a/.travis.yml b/.travis.yml
new file mode 100644
index 000000000..309c6e049
--- /dev/null
+++ b/.travis.yml
@@ -0,0 +1,7 @@
+language: python
+python:
+ - 2.7
+before_script:
+ - git clone https://github.com/common-workflow-language/schema_salad.git salad && cd ./salad && pip install . && cd ..
+script:
+ - for target in v1.0/v1.0/*.cwl; do schema-salad-tool ./salad/schema_salad/tests/test_schema/CommonWorkflowLanguage.yml ${target} || exit 1; done
diff --git a/CITATION b/CITATION
index 337863fc7..3f9693101 100644
--- a/CITATION
+++ b/CITATION
@@ -3,10 +3,11 @@ To cite the Common Workflow Language standard in a publication, please use:
Amstutz, Peter; Crusoe, Michael R; Tijanić, Nebojša; Chapman, Brad;
Chilton, John; Heuer, Michael; Kartashov, Andrey; Kern, John; Leehr, Dan;
Ménager, Hervé; Nedeljkovich, Maya; Scales, Matt; Soiland-Reyes, Stian;
-Stojanovic, Luka (2016): Common Workflow Language, v1.0. figshare.
+Stojanovic, Luka (2016): Common Workflow Language, v1.0. Specification,
+Common Workflow Language working group. https://w3id.org/cwl/v1.0/
https://dx.doi.org/10.6084/m9.figshare.3115156.v2
-@data{common-workflow-language-draft3,
+@data{cwl,
doi = {10.6084/m9.figshare.3115156.v2},
url = {http://dx.doi.org/10.6084/m9.figshare.3115156.v2},
author = {Peter Amstutz; Michael R. Crusoe; Nebojša Tijanić; Brad Chapman;
@@ -15,6 +16,13 @@ Hervé Ménager; Maya Nedeljkovich; Matt Scales; Stian Soiland-Reyes;
Luka Stojanovic
},
publisher = {Figshare},
+ institution = {Common Workflow Language working group},
title = {Common Workflow Language, v1.0},
year = {2016}
}
+
+# Are you editing this file?
+# Synchronize any changes made with
+# README.md
+# and
+# https://github.com/common-workflow-language/user_guide/blob/gh-pages/CITATION
diff --git a/README.md b/README.md
index 57b92b7bd..6f9ca3254 100644
--- a/README.md
+++ b/README.md
@@ -1,7 +1,9 @@
Common Workflow Language
========================
-[](https://github.com/common-workflow-language/common-workflow-language/stargazers) [](https://gitter.im/common-workflow-language/common-workflow-language?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [**Support**](#Support)
+[**Support**](#Support) [](https://gitter.im/common-workflow-language/common-workflow-language?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[](https://travis-ci.org/common-workflow-language/common-workflow-language)
+[](https://github.com/common-workflow-language/common-workflow-language/stargazers)
`, for example `extension:cwl cwlVersion picard`.|
+|[Google](https://www.google.com/search?q=filetype%3Acwl+cwlVersion)|Search for CWL documents using `filetype:cwl cwlVersion + `, for example `filetype:cwl cwlVersion picard`.|
## Software for working with CWL
+### Editors and viewers
+
+|Software|Description|
+|--------|-----------|
+|[Rabix Composer](https://github.com/rabix/composer)|Graphical CWL editor|
+|[CWLviewer](https://view.commonwl.org/)|A web application to view and share Common Workflow Language workflows|
+|[atom-cwl](https://github.com/manabuishii/language-cwl)|CWL editing mode for Atom|
+|[vim-cwl](https://github.com/manabuishii/vim-cwl)|CWL editing mode for Vim|
+|[cwl-mode](https://github.com/tom-tan/cwl-mode)|CWL editing mode for Emacs (instructions [english](https://qiita.com/tm_tn/items/6c9653847412d115bec0), [日本語](https://qiita.com/tm_tn/items/79eec754338d152b092d)) |
+|[vscode-cwl](https://github.com/manabuishii/vscode-cwl)|CWL support in Visual Studio Code|
+|[IntelliJ CWL plugin](https://gitlab.com/AleksandrSl/cwl-plugin)|CWL plugin for [IntelliJ and other JetBrains editors](https://plugins.jetbrains.com/plugin/10040-cwl-plugin)|
+
+### Utilities
+
|Software|Description|
|--------|-----------|
|[cwltest](https://github.com/common-workflow-language/cwltest)|CWL testing framework, automated testing of tools and workflows written with CWL|
+|[cwl2zshcomp](https://github.com/kloetzl/cwl2zshcomp)|generates ZSH auto completions from CWL command line tool descriptions|
+|[Cerise](https://github.com/MD-Studio/cerise)|A REST service for running CWL workflows on remote clusters|
+|[cwl-inspector](https://github.com/tom-tan/cwl-inspector)|Tool to inspect properties of tools or workflows written in CWL|
+
+### Converters and code generators
+
+|Software|Description|
+|--------|-----------|
|[cwl-upgrader](https://github.com/common-workflow-language/cwl-upgrader)|Upgrade CWL documents from draft-3 to v1.0|
|[argparse2tool](https://github.com/erasche/argparse2tool#cwl-specific-functionality)|Generate CWL CommandLineTool wrappers (and/or Galaxy tool descriptions) from Python programs that use argparse. Also supports the [click](http://click.pocoo.org/5/) argument parser.|
|[cwl2argparse](https://github.com/common-workflow-language/cwl2argparse)|Generate Python argparse code from CWL CommandLineTool description.|
|[pypi2cwl](https://github.com/common-workflow-language/pypi2cwl)|Automatically run argparse2cwl on any package in PyPi|
-|[cwlavro](https://github.com/common-workflow-language/cwlavro)|Java classes for loading CWL documents|
-|[acd2cwl](https://github.com/common-workflow-language/acd2cwl)|CWL generator for ACD (EMBOSS) files |
-|[CWLviewer](https://view.commonwl.org/)|A web application to view and share Common Workflow Language workflows|
-|[cwl2zshcomp](https://github.com/kloetzl/cwl2zshcomp)|generates ZSH auto completions from CWL command line tool descriptions|
+|[acd2cwl](https://github.com/common-workflow-language/acd2cwl)|ACD (EMBOSS) to CWL generator|
+|[CTD converter](https://github.com/WorkflowConversion/CTDConverter)|Common Tool Definition (CTD) to CWL converter|
+|[WDL converter](https://github.com/common-workflow-language/wdl2cwl)|Workflow Definition Language (WDL) to CWL converter|
|[scriptcwl](https://github.com/NLeSC/scriptcwl)|Create CWL workflows by writing a simple Python script|
-|[python-cwlgen](https://github.com/common-workflow-language/python-cwlgen)|Generation of CWL programmatically from Python.|
+|[python-cwlgen](https://github.com/common-workflow-language/python-cwlgen)|Generate of CWL programmatically from Python.|
+|[cwl2nxf](https://github.com/nextflow-io/cwl2nxf)|Convert CWL to run on Nextflow|
+|[cwl-to-parsl](https://github.com/benhg/cwl-to-parsl)|Convert CWL to Parsl|
+|[Beatrice](https://github.com/Parsoa/Beatrice)|Pipeline Assembler For CWL|
+
+### Code libraries
+
+|Software|Description|
+|--------|-----------|
+|[cwltool](https://github.com/common-workflow-language/cwltool)|cwltool can be [imported as a Python module]((https://github.com/common-workflow-language/cwltool#import-as-a-module)) and [extended to create custom cwl runners](https://github.com/common-workflow-language/cwltool#extension-points)|
+|[schema salad](https://github.com/common-workflow-language/schema_salad)|Python module and tools for working with the CWL schema.|
+|[cwlavro](https://github.com/common-workflow-language/cwlavro)|Java classes for loading CWL documents|
+|[CWL for R](https://github.com/jefferys/cwl)|Parse and work with CWL from R|
+|[CWL for Julia](https://github.com/BioJulia/CWL.jl)|Utilities for working with CWL from Julia|
+|[CWL for Go](https://github.com/otiai10/cwl.go)|-|
+|[CWL for Scala](https://github.com/broadinstitute/wdl4s)|CWL object model for Scala|
+|[cwl-proto](https://github.com/broadinstitute/cwl-proto)|Reading and writing Common Workflow Language to Protocol Buffers|
## Projects the CWL community is participating in
@@ -88,8 +143,7 @@ Please cite https://dx.doi.org/10.6084/m9.figshare.3115156.v2
-* Robin Andeer
-* Brad Chapman
-* John Chilton
-* Michael R. Crusoe
-* Roman Valls Guimerà
+* Peter Amstutz, Curoverse Inc. / Arvados; https://orcid.org/0000-0003-3566-7705
+* Robin Andeer; https://orcid.org/0000-0003-1132-5305
+* Brad Chapman; https://orcid.org/0000-0002-3026-1856
+* John Chilton, Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-6794-0756
+* Michael R. Crusoe, CWL Project Lead; https://orcid.org/0000-0002-2961-9670
+* Roman Valls Guimerà; https://orcid.org/0000-0002-0034-9697
* Guillermo Carrasco Hernandez
-* Kenzo-Hugo Hillion
+* Kenzo-Hugo Hillion; https://orcid.org/0000-0002-6517-6934
* Sinisa Ivkovic
-* Sehrish Kanwal
-* Andrey Kartashov
-* John Kern
-* Farah Zaib Khan
-* Dan Leehr
-* Hervé Ménager
+* Sehrish Kanwal; https://orcid.org/0000-0002-5044-4692
+* Andrey Kartashov; https://orcid.org/0000-0001-9102-5681
+* John Kern; http://orcid.org/0000-0001-6977-458X
+* Farah Zaib Khan; https://orcid.org/0000-0002-6337-3037
+* Dan Leehr; https://orcid.org/0000-0003-3221-9579
+* Hervé Ménager, Institut Pasteur; https://orcid.org/0000-0002-7552-1009
* Maxim Mikheev
-* Michael Miller
+* Michael Miller
* Tim Pierce
-* Josh Randall
-* Mark Robinson
+* Josh Randall; https://orcid.org/0000-0003-1540-203X
+* Mark Robinson; https://orcid.org/0000-0002-8184-7507
* Janko Simonović
-* Stian Soiland-Reyes
+* Stian Soiland-Reyes, University of Manchester; https://orcid.org/0000-0001-9842-9718
* Luka Stojanovic
* Nebojša Tijanić
+## CWL Advisors
+
+(Alphabetical)
+
+* Peter Amstutz, Curoverse Inc. / Arvados; https://orcid.org/0000-0003-3566-7705
+* Artem Barski, Cincinnati Children's Hospital Medical Center / University of Cincinnati College of Medicine; https://orcid.org/0000-0002-1861-5316
+* John Chilton, Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-6794-0756
+* Kyle Cranmer, New York University; https://orcid.org/0000-0002-5769-7094
+* Michael R. Crusoe, CWL Project Lead; https://orcid.org/0000-0002-2961-9670
+* Brandi Davis Dusenbery, Seven Bridges Genomics, Inc.; https://orcid.org/0000-0001-7811-8613
+* Niels Drost, Netherland eScience Center; https://orcid.org/0000-0001-9795-7981
+* Geet Duggal, DNAnexus; https://orcid.org/0000-0003-3485-359X
+* Rob Finn, EMBL-EBI; https://orcid.org/0000-0001-8626-2148
+* Marc Fiume, DNAstack; https://orcid.org/0000-0002-9769-375X
+* Jeff Gentry, Broad Institute; https://orcid.org/0000-0001-5351-8442
+* Kaushik Ghose, Seven Bridges Genomics, Inc; https://orcid.org/0000-0003-2933-1260
+* Carole Goble, The University of Manchester; https://orcid.org/0000-0003-1219-2137
+* Oliver Hofmann, University of Melbourne / bcbio-nextgen; https://orcid.org/0000-0002-7738-1513
+* Hervé Ménager, Institut Pasteur; https://orcid.org/0000-0002-7552-1009
+* Folker Meyer, Argonne / University of Chicago; https://orcid.org/0000-0003-1112-2284
+* Tom Morris, Curoverse; https://orcid.org/0000-0003-0435-7851
+* Anton Nekrutenko, The Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-5987-8032
+* Brian O'Connor, University of California Santa Cruz; https://orcid.org/0000-0002-7681-6415
+* Tibor Simko, CERN, https://orcid.org/0000-0001-7202-5803
+* Nihar Sheth, DNAnexus; https://orcid.org/0000-0003-4128-4364
+* Stian Soiland-Reyes, University of Manchester; https://orcid.org/0000-0001-9842-9718
+* James Taylor, Johns Hopkins University / Galaxy Project; https://orcid.org/0000-0001-5079-840X
+* Nebojša Tijanić, Seven Bridges
+* Ward Vandewege, Curoverse Inc. / Arvados; https://orcid.org/0000-0002-2527-6949
+* Alexander Wait Zaranek, Curoverse Inc. / Arvados; https://orcid.org/0000-0002-0415-9655
+
+## CWL Leadership Team
+
+(Alphabetical)
-The CWL Community Engineer, Michael R. Crusoe, publishes a blog about his work
-with updates at http://mrc.commonwl.org.
+* Peter Amstutz, Curoverse Inc. / Arvados; https://orcid.org/0000-0003-3566-7705
+* John Chilton, Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-6794-0756
+* Michael R. Crusoe, CWL Project Lead; https://orcid.org/0000-0002-2961-9670
+* Brandi Davis Dusenbery, Seven Bridges Genomics, Inc.; https://orcid.org/0000-0001-7811-8613
+* Jeff Gentry, Broad Institute; https://orcid.org/0000-0001-5351-8442
+* Hervé Ménager, Institut Pasteur; https://orcid.org/0000-0002-7552-1009
+* Stian Soiland-Reyes, University of Manchester; https://orcid.org/0000-0001-9842-9718
diff --git a/draft-3/README.md b/draft-3/README.md
index 96aad3b4b..f00800524 100644
--- a/draft-3/README.md
+++ b/draft-3/README.md
@@ -1,20 +1,23 @@
-# Common Workflow Language Specifications, draft-3
+# Outdated, please refer to CWL version 1.0 specifications
-The CWL specifications are divided up into several documents.
+This URL is for the outdated "draft-3" version of the Common Workflow Language standards.
-The [User Guide](UserGuide.html) provides a gentle introduction to writing CWL
-command line tools and workflows.
+Please go to https://w3id.org/cwl/ for the latest stable version.
-The [Command Line Tool Description Specification](CommandLineTool.html)
+## Common Workflow Language Specifications, draft-3
+
+The CWL draft-3 specifications are divided up into several documents.
+
+The [CWL draft-3 Command Line Tool Description Specification](CommandLineTool.html)
specifies the document schema and execution semantics for wrapping and
executing command line tools.
-The [Workflow Description Specification](Workflow.html) specifies the document
+The [CWL draft-3 Workflow Description Specification](Workflow.html) specifies the document
schema and execution semantics for composing workflows from components such as
command line tools and other workflows.
The
-[Semantic Annotations for Linked Avro Data (SALAD) Specification](SchemaSalad.html)
+[draft-3 version Semantic Annotations for Linked Avro Data (SALAD) Specification](SchemaSalad.html)
specifies the preprocessing steps that must be applied when loading CWL
documents and the schema language used to write the above specifications.
diff --git a/draft-3/UserGuide.yml b/draft-3/UserGuide.yml
index 2afbe0385..f74b3d73e 100644
--- a/draft-3/UserGuide.yml
+++ b/draft-3/UserGuide.yml
@@ -45,7 +45,7 @@
class: CommandLineTool
```
- The 'cwlVersion` field indicates the version of the CWL spec used by the
+ The `cwlVersion` field indicates the version of the CWL spec used by the
document. The `class` field indicates this document describes a command
line tool.
diff --git a/draft-3/userguide-intro.md b/draft-3/userguide-intro.md
index dbcd3e968..dcec161d5 100644
--- a/draft-3/userguide-intro.md
+++ b/draft-3/userguide-intro.md
@@ -2,27 +2,5 @@
Hello!
-This guide will introduce you to writing tool wrappers and workflows using the
-Common Workflow Language (CWL). This guide describes the current stable
-specification, draft 3.
-
-Note: This document is a work in progress. Not all features are covered, yet.
-
-
-
-# Introduction
-
-CWL is a way to describe command line tools and connect them together to create
-workflows. Because CWL is a specification and not a specific piece of
-software, tools and workflows described using CWL are portable across a variety
-of platforms that support the CWL standard.
-
-CWL has roots in "make" and many similar tools that determine order of
-execution based on dependencies between tasks. However unlike "make", CWL
-tasks are isolated and you must be explicit about your inputs and outputs. The
-benefit of explicitness and isolation are flexibility, portability, and
-scalability: tools and workflows described with CWL can transparently leverage
-technologies such as Docker, be used with CWL implementations from different
-vendors, and is well suited for describing large-scale workflows in cluster,
-cloud and high performance computing environments where tasks are scheduled in
-parallel across many nodes.
+The Common Workflow Language user guide has moved to
+[http://www.commonwl.org/user_guide/](http://www.commonwl.org/user_guide/)
diff --git a/run_test.sh b/run_test.sh
index 3c4129842..dc2a1e637 100755
--- a/run_test.sh
+++ b/run_test.sh
@@ -5,10 +5,24 @@ $(basename $0): Run common workflow tool description language conformance tests.
Syntax:
$(basename $0) [RUNNER=/path/to/cwl-runner] [DRAFT=cwl-draft-version]
+ [EXTRA=--optional-arguments-to-cwl-runner]
Options:
- -nT Run a specific test.
- -l List tests
+ -nT Run a specific test.
+ -l List tests
+ -jJ Specifies the number of tests to run simultaneously
+ (defaults to one).
+ --only-tools Only test CommandLineTools
+ --junit-xml=FILENAME Store results in JUnit XML format using the given
+ FILENAME
+ --classname=CLASSNAME In the JUnit XML, tag the results with the given
+ CLASSNAME
+ --verbose Print the cwltest invocation and pass --verbose to
+ cwltest
+
+Note:
+ EXTRA is useful for passing --enable-dev to the CWL reference runner:
+ Example: RUNNER=cwltool EXTRA=--enable-dev
EOF
DRAFT=v1.0
@@ -18,6 +32,8 @@ RUNNER=cwl-runner
PLATFORM=$(uname -s)
COVERAGE="python"
EXTRA=""
+CLASS=""
+VERBOSE=""
while [[ -n "$1" ]]
do
@@ -31,21 +47,32 @@ do
-n*)
TEST_N=$arg
;;
+ -j*)
+ TEST_J=$arg
+ ;;
-l)
TEST_L=-l
;;
--only-tools)
- ONLY_TOOLS=--only-tools
+ ONLY_TOOLS=$arg
;;
--junit-xml=*)
JUNIT_XML=$arg
;;
+ --classname=*)
+ CLASS=$arg
+ ;;
+ --verbose)
+ VERBOSE=$arg
+ ;;
*=*)
eval $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2-)\"
;;
esac
done
+DRAFT_DIR="$(cd $(dirname $0); pwd)/${DRAFT}"
+
if ! runner="$(which $RUNNER)" ; then
echo >&2 "$helpmessage"
echo >&2
@@ -68,10 +95,13 @@ runtest() {
"$1" --version
runs=$((runs+1))
- (cd $DRAFT
- cwltest --tool "$1" \
- --test=conformance_test_${DRAFT}.yaml ${TEST_N} \
- ${TEST_L} ${ONLY_TOOLS} ${JUNIT_XML} --basedir ${DRAFT} -- ${EXTRA}
+ (cd $DRAFT_DIR
+ COMMAND="cwltest --tool $1 \
+ --test=conformance_test_${DRAFT}.yaml ${CLASS} ${TEST_N} \
+ ${VERBOSE} ${TEST_L} ${TEST_J} ${ONLY_TOOLS} ${JUNIT_XML} \
+ --basedir ${DRAFT_DIR} -- ${EXTRA}"
+ if [[ $VERBOSE == "--verbose" ]]; then echo ${COMMAND}; fi
+ ${COMMAND}
)
checkexit
}
diff --git a/site/cwlsite-draft3-job.json b/site/cwlsite-draft3-job.json
index 24fc20202..9f332f9f0 100644
--- a/site/cwlsite-draft3-job.json
+++ b/site/cwlsite-draft3-job.json
@@ -8,7 +8,8 @@
"redirect": [],
"brandlink": "../index.html",
"brandimg": ", Arvados Project, Curoverse
-* Michael R. Crusoe , Common Workflow Language
+* Peter Amstutz , Arvados Project, Veritas Genetics
+* Michael R. Crusoe , Common Workflow Language
project
* Nebojša Tijanić , Seven Bridges Genomics
@@ -9,11 +9,11 @@ Contributors:
* Brad Chapman , Harvard Chan School of Public Health
* John Chilton , Galaxy Project, Pennsylvania State University
-* Michael Heuer ,UC Berkeley AMPLab
+* Michael Heuer , UC Berkeley AMPLab
* Andrey Kartashov , Cincinnati Children's Hospital
* Dan Leehr , Duke University
* Hervé Ménager , Institut Pasteur
* Maya Nedeljkovich , Seven Bridges Genomics
* Matt Scales , Institute of Cancer Research, London
-* Stian Soiland-Reyes [soiland-reyes@cs.manchester.ac.uk](mailto:soiland-reyes@cs.manchester.ac.uk), University of Manchester
+* Stian Soiland-Reyes , University of Manchester
* Luka Stojanovic , Seven Bridges Genomics
diff --git a/v1.0/salad/.gitignore b/v1.0/salad/.gitignore
index 85014c5d4..3d7187d48 100644
--- a/v1.0/salad/.gitignore
+++ b/v1.0/salad/.gitignore
@@ -2,3 +2,11 @@
.eggs
*.egg-info/
*pyc
+
+build/
+dist/
+ruamel
+typeshed/2and3/ruamel/yaml
+# virtualenv
+venv/
+.cache/
diff --git a/v1.0/salad/.travis.yml b/v1.0/salad/.travis.yml
index ae3d95487..f59182cde 100644
--- a/v1.0/salad/.travis.yml
+++ b/v1.0/salad/.travis.yml
@@ -1,20 +1,16 @@
+sudo: false
language: python
-python: 2.7
+python:
+ - 2.7
+ - 3.3
+ - 3.4
+ - 3.5
+ - 3.6
os:
- linux
-env:
- - TOX_ENV=py27-lint
- - TOX_ENV=py27-unit
- - TOX_ENV=py34-mypy
-# - TOX_ENV=py34-lint
-# - TOX_ENV=py33-lint
-# - TOX_ENV=py34-unit
-# - TOX_ENV=py33-unit
install:
- - pip install tox
-
-script: tox -e $TOX_ENV
-
+ - pip install tox-travis
+script: tox
notifications:
email: false
diff --git a/v1.0/salad/MANIFEST.in b/v1.0/salad/MANIFEST.in
index 624b1d12d..c3870ab6e 100644
--- a/v1.0/salad/MANIFEST.in
+++ b/v1.0/salad/MANIFEST.in
@@ -1 +1,8 @@
-include gittaggers.py ez_setup.py
+include gittaggers.py Makefile
+include schema_salad/tests/*
+include schema_salad/tests/test_schema/*.md
+include schema_salad/tests/test_schema/*.yml
+include schema_salad/tests/test_schema/*.cwl
+include schema_salad/metaschema/*
+global-exclude *~
+global-exclude *.pyc
diff --git a/v1.0/salad/Makefile b/v1.0/salad/Makefile
index f5dfaaa29..9fe16fd96 100644
--- a/v1.0/salad/Makefile
+++ b/v1.0/salad/Makefile
@@ -26,7 +26,9 @@ MODULE=schema_salad
# `SHELL=bash` Will break Titus's laptop, so don't use BASH-isms like
# `[[` conditional expressions.
PYSOURCES=$(wildcard ${MODULE}/**.py tests/*.py) setup.py
-DEVPKGS=pep8 diff_cover autopep8 pylint coverage pep257
+DEVPKGS=pep8 diff_cover autopep8 pylint coverage pep257 pytest flake8
+COVBASE=coverage run --branch --append --source=${MODULE} \
+ --omit=schema_salad/tests/*
VERSION=$(shell git describe --tags --dirty | sed s/v//)
@@ -46,7 +48,7 @@ install-dependencies:
## install : install the ${MODULE} module and schema-salad-tool
install: FORCE
- ./setup.py build install
+ pip install .
## dist : create a module package for distribution
dist: dist/${MODULE}-$(VERSION).tar.gz
@@ -103,37 +105,31 @@ diff_pylint_report: pylint_report.txt
diff-quality --violations=pylint pylint_report.txt
.coverage: $(PYSOURCES)
- coverage run --branch --source=${MODULE} setup.py test
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ rm -f .coverage
+ $(COVBASE) setup.py test
+ $(COVBASE) -m schema_salad.main \
--print-jsonld-context schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-rdfs schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-avro schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-rdf schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-pre schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-index schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-metadata schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.makedoc schema_salad/metaschema/metaschema.yml \
+ $(COVBASE) -m schema_salad.makedoc \
+ schema_salad/metaschema/metaschema.yml \
> /dev/null
coverage.xml: .coverage
@@ -169,17 +165,33 @@ list-author-emails:
@echo 'name, E-Mail Address'
@git log --format='%aN,%aE' | sort -u | grep -v 'root'
-mypy: ${PYSOURCES}
- MYPYPATH=typeshed/2.7 mypy --py2 --disallow-untyped-calls schema_salad
+mypy2: ${PYSOURCES}
+ rm -Rf typeshed/2and3/ruamel/yaml
+ ln -s $(shell python -c 'from __future__ import print_function; import ruamel.yaml; import os.path; print(os.path.dirname(ruamel.yaml.__file__))') \
+ typeshed/2and3/ruamel/
+ MYPYPATH=$MYPYPATH:typeshed/2.7:typeshed/2and3 mypy --py2 --disallow-untyped-calls \
+ --warn-redundant-casts \
+ schema_salad
+
+mypy3: ${PYSOURCES}
+ rm -Rf typeshed/2and3/ruamel/yaml
+ ln -s $(shell python -c 'from __future__ import print_function; import ruamel.yaml; import os.path; print(os.path.dirname(ruamel.yaml.__file__))') \
+ typeshed/2and3/ruamel/
+ MYPYPATH=$MYPYPATH:typeshed/3:typeshed/2and3 mypy --disallow-untyped-calls \
+ --warn-redundant-casts \
+ schema_salad
jenkins:
- if ! test -d env ; then virtualenv env ; fi
+ rm -Rf env && virtualenv env
. env/bin/activate ; \
pip install -U setuptools pip wheel ; \
${MAKE} install-dep coverage.html coverage.xml pep257_report.txt \
sloccount.sc pep8_report.txt pylint_report.txt
if ! test -d env3 ; then virtualenv -p python3 env3 ; fi
. env3/bin/activate ; \
- pip install -U mypy-lang; ${MAKE} mypy
+ pip install -U setuptools pip wheel ; \
+ ${MAKE} install-dep ; \
+ pip install -U -r mypy_requirements.txt ; ${MAKE} mypy2
+ # pip install -U -r mypy_requirements.txt ; ${MAKE} mypy3
FORCE:
diff --git a/v1.0/salad/README.rst b/v1.0/salad/README.rst
index e66427cdd..38ffb4298 100644
--- a/v1.0/salad/README.rst
+++ b/v1.0/salad/README.rst
@@ -27,6 +27,13 @@ Usage
$ python
>>> import schema_salad
+To install from source::
+
+ git clone https://github.com/common-workflow-language/schema_salad
+ cd schema_salad
+ python setup.py install
+
+
Documentation
-------------
diff --git a/v1.0/salad/mypy.ini b/v1.0/salad/mypy.ini
new file mode 100644
index 000000000..3dccfb5bc
--- /dev/null
+++ b/v1.0/salad/mypy.ini
@@ -0,0 +1,2 @@
+[mypy-ruamel.*]
+ignore_errors = True
\ No newline at end of file
diff --git a/v1.0/salad/mypy_requirements.txt b/v1.0/salad/mypy_requirements.txt
new file mode 100644
index 000000000..508a7cbf6
--- /dev/null
+++ b/v1.0/salad/mypy_requirements.txt
@@ -0,0 +1 @@
+mypy==0.511 ; python_version>="3"
\ No newline at end of file
diff --git a/v1.0/salad/release-test.sh b/v1.0/salad/release-test.sh
new file mode 100755
index 000000000..e7fc832ec
--- /dev/null
+++ b/v1.0/salad/release-test.sh
@@ -0,0 +1,69 @@
+#!/bin/bash
+
+set -e
+set -x
+
+package=schema-salad
+module=schema_salad
+repo=https://github.com/common-workflow-language/schema_salad.git
+run_tests="py.test --pyarg ${module}"
+pipver=8.0.1 # minimum required version of pip
+setupver=20.10.1 # minimum required version of setuptools
+
+rm -Rf testenv? || /bin/true
+
+export HEAD=`git rev-parse HEAD`
+virtualenv testenv1
+virtualenv testenv2
+virtualenv testenv3
+virtualenv testenv4
+
+# First we test the head
+source testenv1/bin/activate
+rm testenv1/lib/python-wheels/setuptools* \
+ && pip install --force-reinstall -U pip==${pipver} \
+ && pip install setuptools==${setupver} wheel
+make install-dependencies
+make test
+pip uninstall -y ${package} || true; pip uninstall -y ${package} || true; make install
+mkdir testenv1/not-${module}
+# if there is a subdir named '${module}' py.test will execute tests
+# there instead of the installed module's tests
+pushd testenv1/not-${module}; ../bin/${run_tests}; popd
+
+
+# Secondly we test via pip
+
+cd testenv2
+source bin/activate
+rm lib/python-wheels/setuptools* \
+ && pip install --force-reinstall -U pip==${pipver} \
+ && pip install setuptools==${setupver} wheel
+pip install -e git+${repo}@${HEAD}#egg=${package}
+cd src/${package}
+make install-dependencies
+make dist
+make test
+cp dist/${package}*tar.gz ../../../testenv3/
+pip uninstall -y ${package} || true; pip uninstall -y ${package} || true; make install
+cd ../.. # no subdir named ${proj} here, safe for py.testing the installed module
+bin/${run_tests}
+
+# Is the distribution in testenv2 complete enough to build another
+# functional distribution?
+
+cd ../testenv3/
+source bin/activate
+rm lib/python-wheels/setuptools* \
+ && pip install --force-reinstall -U pip==${pipver} \
+ && pip install setuptools==${setupver} wheel
+pip install ${package}*tar.gz
+pip install pytest
+mkdir out
+tar --extract --directory=out -z -f ${package}*.tar.gz
+cd out/${package}*
+make dist
+make test
+pip uninstall -y ${package} || true; pip uninstall -y ${package} || true; make install
+mkdir ../not-${module}
+pushd ../not-${module} ; ../../bin/${run_tests}; popd
diff --git a/v1.0/salad/requirements.txt b/v1.0/salad/requirements.txt
index 8c64938e7..4a580453f 100644
--- a/v1.0/salad/requirements.txt
+++ b/v1.0/salad/requirements.txt
@@ -1,9 +1,10 @@
-requests
-ruamel.yaml
-rdflib >= 4.1.
-rdflib-jsonld >= 0.3.0
-mistune
-typing ; python_version>="2.7"
-avro ; python_version<"3"
-avro-python3 ; python_version>="3"
-
+typing==3.5.3
+avro-cwl ; python_version>="3"
+future ; python_version>="3"
+avro==1.8.1 ; python_version<"3"
+ruamel.yaml>=0.12.4, <0.15
+rdflib==4.2.2
+rdflib-jsonld==0.4.0
+mistune==0.7.3
+CacheControl==0.11.7
+lockfile==0.12.2
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/__init__.py b/v1.0/salad/schema_salad/__init__.py
index 381ec7664..a751d64be 100644
--- a/v1.0/salad/schema_salad/__init__.py
+++ b/v1.0/salad/schema_salad/__init__.py
@@ -1,3 +1,4 @@
+from __future__ import absolute_import
import logging
import sys
import typing
diff --git a/v1.0/salad/schema_salad/__main__.py b/v1.0/salad/schema_salad/__main__.py
index 4bf3d7eae..5890f6f28 100644
--- a/v1.0/salad/schema_salad/__main__.py
+++ b/v1.0/salad/schema_salad/__main__.py
@@ -1,3 +1,4 @@
+from __future__ import absolute_import
from . import main
import sys
import typing
diff --git a/v1.0/salad/schema_salad/add_dictlist.py b/v1.0/salad/schema_salad/add_dictlist.py
deleted file mode 100644
index 53bd4d48d..000000000
--- a/v1.0/salad/schema_salad/add_dictlist.py
+++ /dev/null
@@ -1,7 +0,0 @@
-import sys
-from typing import Any, Dict
-
-def add_dictlist(di, key, val): # type: (Dict, Any, Any) -> None
- if key not in di:
- di[key] = []
- di[key].append(val)
diff --git a/v1.0/salad/schema_salad/aslist.py b/v1.0/salad/schema_salad/aslist.py
deleted file mode 100644
index 0332a2be5..000000000
--- a/v1.0/salad/schema_salad/aslist.py
+++ /dev/null
@@ -1,10 +0,0 @@
-import sys
-from typing import Any, List
-
-def aslist(l): # type: (Any) -> List
- """Convenience function to wrap single items and lists, and return lists unchanged."""
-
- if isinstance(l, list):
- return l
- else:
- return [l]
diff --git a/v1.0/salad/schema_salad/jsonld_context.py b/v1.0/salad/schema_salad/jsonld_context.py
index 225b686f5..57e6ff47c 100755
--- a/v1.0/salad/schema_salad/jsonld_context.py
+++ b/v1.0/salad/schema_salad/jsonld_context.py
@@ -1,6 +1,11 @@
+from __future__ import absolute_import
import collections
import shutil
import json
+
+import six
+from six.moves import urllib
+
import ruamel.yaml as yaml
try:
from ruamel.yaml import CSafeLoader as SafeLoader
@@ -16,36 +21,44 @@
from rdflib import Graph, URIRef
import rdflib.namespace
from rdflib.namespace import RDF, RDFS
-import urlparse
import logging
-from .aslist import aslist
-from typing import Any, cast, Dict, Iterable, Tuple, Union
-from .ref_resolver import Loader
+from schema_salad.utils import aslist
+from typing import (cast, Any, Dict, Iterable, List, Optional, Text, Tuple,
+ Union)
+from .ref_resolver import Loader, ContextType
_logger = logging.getLogger("salad")
-def pred(datatype, field, name, context, defaultBase, namespaces):
- # type: (Dict[str, Union[Dict, str]], Dict, str, Loader.ContextType, str, Dict[str, rdflib.namespace.Namespace]) -> Union[Dict, str]
- split = urlparse.urlsplit(name)
+def pred(datatype, # type: Dict[str, Union[Dict, str]]
+ field, # type: Optional[Dict]
+ name, # type: str
+ context, # type: ContextType
+ defaultBase, # type: str
+ namespaces # type: Dict[str, rdflib.namespace.Namespace]
+ ):
+ # type: (...) -> Union[Dict, Text]
+ split = urllib.parse.urlsplit(name)
- vee = None # type: Union[str, unicode]
+ vee = None # type: Optional[Text]
- if split.scheme:
+ if split.scheme != '':
vee = name
- (ns, ln) = rdflib.namespace.split_uri(unicode(vee))
+ (ns, ln) = rdflib.namespace.split_uri(six.text_type(vee))
name = ln
if ns[0:-1] in namespaces:
- vee = unicode(namespaces[ns[0:-1]][ln])
+ vee = six.text_type(namespaces[ns[0:-1]][ln])
_logger.debug("name, v %s %s", name, vee)
- v = None # type: Any
+ v = None # type: Optional[Dict]
- if field and "jsonldPredicate" in field:
+ if field is not None and "jsonldPredicate" in field:
if isinstance(field["jsonldPredicate"], dict):
v = {}
for k, val in field["jsonldPredicate"].items():
v[("@" + k[1:] if k.startswith("_") else k)] = val
+ if "@id" not in v:
+ v["@id"] = vee
else:
v = field["jsonldPredicate"]
elif "jsonldPredicate" in datatype:
@@ -60,11 +73,6 @@ def pred(datatype, field, name, context, defaultBase, namespaces):
"Dictionaries")
else:
raise Exception("jsonldPredicate must be a List of Dictionaries.")
- # if not v:
- # if field and "jsonldPrefix" in field:
- # defaultBase = field["jsonldPrefix"]
- # elif "jsonldPrefix" in datatype:
- # defaultBase = datatype["jsonldPrefix"]
ret = v or vee
@@ -82,8 +90,14 @@ def pred(datatype, field, name, context, defaultBase, namespaces):
return ret
-def process_type(t, g, context, defaultBase, namespaces, defaultPrefix):
- # type: (Dict[str, Any], Graph, Loader.ContextType, str, Dict[str, rdflib.namespace.Namespace], str) -> None
+def process_type(t, # type: Dict[str, Any]
+ g, # type: Graph
+ context, # type: ContextType
+ defaultBase, # type: str
+ namespaces, # type: Dict[str, rdflib.namespace.Namespace]
+ defaultPrefix # type: str
+ ):
+ # type: (...) -> None
if t["type"] == "record":
recordname = t["name"]
@@ -92,15 +106,15 @@ def process_type(t, g, context, defaultBase, namespaces, defaultPrefix):
classnode = URIRef(recordname)
g.add((classnode, RDF.type, RDFS.Class))
- split = urlparse.urlsplit(recordname)
- if "jsonldPrefix" in t:
- predicate = "%s:%s" % (t["jsonldPrefix"], recordname)
- elif split.scheme:
- (ns, ln) = rdflib.namespace.split_uri(unicode(recordname))
- predicate = recordname
- recordname = ln
- else:
- predicate = "%s:%s" % (defaultPrefix, recordname)
+ split = urllib.parse.urlsplit(recordname)
+ predicate = recordname
+ if t.get("inVocab", True):
+ if split.scheme:
+ (ns, ln) = rdflib.namespace.split_uri(six.text_type(recordname))
+ predicate = recordname
+ recordname = ln
+ else:
+ predicate = "%s:%s" % (defaultPrefix, recordname)
if context.get(recordname, predicate) != predicate:
raise Exception("Predicate collision on '%s', '%s' != '%s'" % (
@@ -118,15 +132,16 @@ def process_type(t, g, context, defaultBase, namespaces, defaultPrefix):
_logger.debug("Processing field %s", i)
- v = pred(t, i, fieldname, context, defaultPrefix, namespaces)
+ v = pred(t, i, fieldname, context, defaultPrefix,
+ namespaces) # type: Union[Dict[Any, Any], Text, None]
- if isinstance(v, basestring):
+ if isinstance(v, six.string_types):
v = v if v[0] != "@" else None
- else:
+ elif v is not None:
v = v["_@id"] if v.get("_@id", "@")[0] != "@" else None
- if v:
- (ns, ln) = rdflib.namespace.split_uri(unicode(v))
+ if bool(v):
+ (ns, ln) = rdflib.namespace.split_uri(six.text_type(v))
if ns[0:-1] in namespaces:
propnode = namespaces[ns[0:-1]][ln]
else:
@@ -152,8 +167,8 @@ def process_type(t, g, context, defaultBase, namespaces, defaultPrefix):
def salad_to_jsonld_context(j, schema_ctx):
- # type: (Iterable, Dict[str, Any]) -> Tuple[Loader.ContextType, Graph]
- context = {} # type: Loader.ContextType
+ # type: (Iterable, Dict[str, Any]) -> Tuple[ContextType, Graph]
+ context = {} # type: ContextType
namespaces = {}
g = Graph()
defaultPrefix = ""
@@ -176,8 +191,11 @@ def salad_to_jsonld_context(j, schema_ctx):
return (context, g)
-def fix_jsonld_ids(obj, ids):
- # type: (Union[Dict[unicode, Any], List[Dict[unicode, Any]]], List[unicode]) -> None
+
+def fix_jsonld_ids(obj, # type: Union[Dict[Text, Any], List[Dict[Text, Any]]]
+ ids # type: List[Text]
+ ):
+ # type: (...) -> None
if isinstance(obj, dict):
for i in ids:
if i in obj:
@@ -188,39 +206,47 @@ def fix_jsonld_ids(obj, ids):
for entry in obj:
fix_jsonld_ids(entry, ids)
-def makerdf(workflow, wf, ctx):
- # type: (Union[str, unicode], Union[List[Dict[unicode, Any]], Dict[unicode, Any]], Loader.ContextType) -> Graph
+
+def makerdf(workflow, # type: Text
+ wf, # type: Union[List[Dict[Text, Any]], Dict[Text, Any]]
+ ctx, # type: ContextType
+ graph=None # type: Graph
+ ):
+ # type: (...) -> Graph
prefixes = {}
idfields = []
- for k,v in ctx.iteritems():
+ for k, v in six.iteritems(ctx):
if isinstance(v, dict):
url = v["@id"]
else:
url = v
if url == "@id":
idfields.append(k)
- doc_url, frg = urlparse.urldefrag(url)
+ doc_url, frg = urllib.parse.urldefrag(url)
if "/" in frg:
- p, _ = frg.split("/")
+ p = frg.split("/")[0]
prefixes[p] = u"%s#%s/" % (doc_url, p)
+ fix_jsonld_ids(wf, idfields)
+
+ if graph is None:
+ g = Graph()
+ else:
+ g = graph
+
if isinstance(wf, list):
- wf = {
- "@context": ctx,
- "@graph": wf
- }
+ for w in wf:
+ w["@context"] = ctx
+ g.parse(data=json.dumps(w), format='json-ld', publicID=str(workflow))
else:
wf["@context"] = ctx
-
- fix_jsonld_ids(wf, idfields)
-
- g = Graph().parse(data=json.dumps(wf), format='json-ld', location=workflow)
+ g.parse(data=json.dumps(wf), format='json-ld', publicID=str(workflow))
# Bug in json-ld loader causes @id fields to be added to the graph
- for s,p,o in g.triples((None, URIRef("@id"), None)):
- g.remove((s, p, o))
+ for sub, pred, obj in g.triples((None, URIRef("@id"), None)):
+ g.remove((sub, pred, obj))
- for k2,v2 in prefixes.iteritems():
+ for k2, v2 in six.iteritems(prefixes):
g.namespace_manager.bind(k2, v2)
return g
diff --git a/v1.0/salad/schema_salad/main.py b/v1.0/salad/schema_salad/main.py
index 3c9a634e5..e9535a54e 100644
--- a/v1.0/salad/schema_salad/main.py
+++ b/v1.0/salad/schema_salad/main.py
@@ -1,29 +1,40 @@
from __future__ import print_function
+from __future__ import absolute_import
import argparse
import logging
import sys
-import pkg_resources # part of setuptools
-from . import schema
-from . import jsonld_context
-from . import makedoc
+import traceback
import json
+import os
+
+import six
+from six.moves import urllib
+
+import pkg_resources # part of setuptools
+
+from typing import Any, Dict, List, Union, Text
+
from rdflib import Graph, plugin
from rdflib.serializer import Serializer
-import os
-import urlparse
-from .ref_resolver import Loader
+from . import schema
+from . import jsonld_context
+from . import makedoc
from . import validate
-from typing import Any, Dict, List, Union
-
+from .sourceline import strip_dup_lineno
+from .ref_resolver import Loader, file_uri
_logger = logging.getLogger("salad")
from rdflib.plugin import register, Parser
register('json-ld', Parser, 'rdflib_jsonld.parser', 'JsonLDParser')
-def printrdf(workflow, wf, ctx, sr):
- # type: (str, Union[List[Dict[unicode, Any]], Dict[unicode, Any]], Dict[unicode, Any], str) -> None
+def printrdf(workflow, # type: str
+ wf, # type: Union[List[Dict[Text, Any]], Dict[Text, Any]]
+ ctx, # type: Dict[Text, Any]
+ sr # type: str
+ ):
+ # type: (...) -> None
g = jsonld_context.makerdf(workflow, wf, ctx)
print(g.serialize(format=sr))
@@ -53,8 +64,6 @@ def main(argsl=None): # type: (List[str]) -> int
"--print-index", action="store_true", help="Print node index")
exgroup.add_argument("--print-metadata",
action="store_true", help="Print document metadata")
- exgroup.add_argument("--version", action="store_true",
- help="Print version")
exgroup = parser.add_mutually_exclusive_group()
exgroup.add_argument("--strict", action="store_true", help="Strict validation (unrecognized or out of place fields are error)",
@@ -70,11 +79,18 @@ def main(argsl=None): # type: (List[str]) -> int
exgroup.add_argument("--debug", action="store_true",
help="Print even more logging")
- parser.add_argument("schema", type=str)
+ parser.add_argument("schema", type=str, nargs="?", default=None)
parser.add_argument("document", type=str, nargs="?", default=None)
+ parser.add_argument("--version", "-v", action="store_true",
+ help="Print version", default=None)
+
args = parser.parse_args(argsl)
+ if args.version is None and args.schema is None:
+ print('%s: error: too few arguments' % sys.argv[0])
+ return 1
+
if args.quiet:
_logger.setLevel(logging.WARN)
if args.debug:
@@ -83,10 +99,10 @@ def main(argsl=None): # type: (List[str]) -> int
pkg = pkg_resources.require("schema_salad")
if pkg:
if args.version:
- print("%s %s" % (sys.argv[0], pkg[0].version))
+ print("%s Current version: %s" % (sys.argv[0], pkg[0].version))
return 0
else:
- _logger.info("%s %s", sys.argv[0], pkg[0].version)
+ _logger.info("%s Current version: %s", sys.argv[0], pkg[0].version)
# Get the metaschema to validate the schema
metaschema_names, metaschema_doc, metaschema_loader = schema.get_metaschema()
@@ -94,8 +110,8 @@ def main(argsl=None): # type: (List[str]) -> int
# Load schema document and resolve refs
schema_uri = args.schema
- if not urlparse.urlparse(schema_uri)[0]:
- schema_uri = "file://" + os.path.abspath(schema_uri)
+ if not (urllib.parse.urlparse(schema_uri)[0] and urllib.parse.urlparse(schema_uri)[0] in [u'http', u'https', u'file']):
+ schema_uri = file_uri(os.path.abspath(schema_uri))
schema_raw_doc = metaschema_loader.fetch(schema_uri)
try:
@@ -103,9 +119,13 @@ def main(argsl=None): # type: (List[str]) -> int
schema_raw_doc, schema_uri)
except (validate.ValidationException) as e:
_logger.error("Schema `%s` failed link checking:\n%s",
- args.schema, e, exc_info=(e if args.debug else False))
- _logger.debug("Index is %s", metaschema_loader.idx.keys())
- _logger.debug("Vocabulary is %s", metaschema_loader.vocab.keys())
+ args.schema, e, exc_info=(True if args.debug else False))
+ _logger.debug("Index is %s", list(metaschema_loader.idx.keys()))
+ _logger.debug("Vocabulary is %s", list(metaschema_loader.vocab.keys()))
+ return 1
+ except (RuntimeError) as e:
+ _logger.error("Schema `%s` read error:\n%s",
+ args.schema, e, exc_info=(True if args.debug else False))
return 1
# Optionally print the schema after ref resolution
@@ -114,13 +134,14 @@ def main(argsl=None): # type: (List[str]) -> int
return 0
if not args.document and args.print_index:
- print(json.dumps(metaschema_loader.idx.keys(), indent=4))
+ print(json.dumps(list(metaschema_loader.idx.keys()), indent=4))
return 0
# Validate the schema document against the metaschema
try:
schema.validate_doc(metaschema_names, schema_doc,
- metaschema_loader, args.strict)
+ metaschema_loader, args.strict,
+ source=schema_metadata.get("name"))
except validate.ValidationException as e:
_logger.error("While validating schema `%s`:\n%s" %
(args.schema, str(e)))
@@ -132,7 +153,11 @@ def main(argsl=None): # type: (List[str]) -> int
metactx = schema_raw_doc.get("$namespaces", {})
if "$base" in schema_raw_doc:
metactx["@base"] = schema_raw_doc["$base"]
- (schema_ctx, rdfs) = jsonld_context.salad_to_jsonld_context(schema_doc, metactx)
+ if schema_doc is not None:
+ (schema_ctx, rdfs) = jsonld_context.salad_to_jsonld_context(
+ schema_doc, metactx)
+ else:
+ raise Exception("schema_doc is None??")
# Create the loader that will be used to load the target document.
document_loader = Loader(schema_ctx)
@@ -148,7 +173,8 @@ def main(argsl=None): # type: (List[str]) -> int
if isinstance(avsc_names, Exception):
_logger.error("Schema `%s` error:\n%s", args.schema,
- avsc_names, exc_info=(avsc_names if args.debug else False))
+ avsc_names, exc_info=((type(avsc_names), avsc_names,
+ None) if args.debug else None))
if args.print_avro:
print(json.dumps(avsc_obj, indent=4))
return 1
@@ -181,12 +207,12 @@ def main(argsl=None): # type: (List[str]) -> int
# Load target document and resolve refs
try:
uri = args.document
- if not urlparse.urlparse(uri)[0]:
+ if not urllib.parse.urlparse(uri)[0]:
doc = "file://" + os.path.abspath(uri)
document, doc_metadata = document_loader.resolve_ref(uri)
except (validate.ValidationException, RuntimeError) as e:
_logger.error("Document `%s` failed validation:\n%s",
- args.document, e, exc_info=(e if args.debug else False))
+ args.document, strip_dup_lineno(six.text_type(e)), exc_info=args.debug)
return 1
# Optionally print the document after ref resolution
@@ -195,7 +221,7 @@ def main(argsl=None): # type: (List[str]) -> int
return 0
if args.print_index:
- print(json.dumps(document_loader.idx.keys(), indent=4))
+ print(json.dumps(list(document_loader.idx.keys()), indent=4))
return 0
# Validate the schema document against the metaschema
diff --git a/v1.0/salad/schema_salad/makedoc.py b/v1.0/salad/schema_salad/makedoc.py
index 3ab4eaced..6f287b28b 100644
--- a/v1.0/salad/schema_salad/makedoc.py
+++ b/v1.0/salad/schema_salad/makedoc.py
@@ -1,23 +1,28 @@
+from __future__ import absolute_import
+
import mistune
-from . import schema
+import argparse
import json
import os
import copy
import re
import sys
-import StringIO
import logging
-import urlparse
-from .aslist import aslist
-from .add_dictlist import add_dictlist
-import re
-import argparse
-from typing import Any, IO, Union
+from io import open
+
+from . import schema
+from .utils import add_dictlist, aslist
+
+import six
+from six.moves import range
+from six.moves import urllib
+from six import StringIO
+from typing import cast, Any, Dict, IO, List, Optional, Set, Text, Union
_logger = logging.getLogger("salad")
-def has_types(items): # type: (Any) -> List[basestring]
+def has_types(items): # type: (Any) -> List[Text]
r = [] # type: List
if isinstance(items, dict):
if items["type"] == "https://w3id.org/cwl/salad#record":
@@ -30,13 +35,13 @@ def has_types(items): # type: (Any) -> List[basestring]
for i in items:
r.extend(has_types(i))
return r
- if isinstance(items, basestring):
+ if isinstance(items, six.string_types):
return [items]
return []
-def linkto(item):
- _, frg = urlparse.urldefrag(item)
+def linkto(item): # type: (Text) -> Text
+ _, frg = urllib.parse.urldefrag(item)
return "[%s](#%s)" % (frg, to_id(frg))
@@ -46,11 +51,17 @@ def __init__(self): # type: () -> None
super(mistune.Renderer, self).__init__()
self.options = {}
- def header(self, text, level, raw=None):
+ def header(self, text, level, raw=None): # type: (Text, int, Any) -> Text
return """%s """ % (level, to_id(text), text, level)
+ def table(self, header, body): # type: (Text, Text) -> Text
+ return (
+ '\n'
+ ) % (header, body)
+
-def to_id(text): # type: (Union[str, unicode]) -> Union[str, unicode]
+def to_id(text): # type: (Text) -> Text
textid = text
if text[0] in ("0", "1", "2", "3", "4", "5", "6", "7", "8", "9"):
try:
@@ -104,6 +115,7 @@ def contents(self, idn): # type: (str) -> str
c += """"""
return c
+
basicTypes = ("https://w3id.org/cwl/salad#null",
"http://www.w3.org/2001/XMLSchema#boolean",
"http://www.w3.org/2001/XMLSchema#int",
@@ -129,7 +141,7 @@ def number_headings(toc, maindoc): # type: (ToC, str) -> str
if not skip:
m = re.match(r'^(#+) (.*)', line)
- if m:
+ if m is not None:
num = toc.add_entry(len(m.group(1)), m.group(2))
line = "%s %s %s" % (m.group(1), num, m.group(2))
line = re.sub(r'^(https?://\S+)', r'[\1](\1)', line)
@@ -151,16 +163,17 @@ def fix_doc(doc): # type: (Union[List[str], str]) -> str
class RenderType(object):
- def __init__(self, toc, j, renderlist, redirects):
- # type: (ToC, List[Dict], str, Dict) -> None
- self.typedoc = StringIO.StringIO()
+ def __init__(self, toc, j, renderlist, redirects, primitiveType):
+ # type: (ToC, List[Dict], str, Dict, str) -> None
+ self.typedoc = StringIO()
self.toc = toc
self.subs = {} # type: Dict[str, str]
self.docParent = {} # type: Dict[str, List]
self.docAfter = {} # type: Dict[str, List]
self.rendered = set() # type: Set[str]
self.redirects = redirects
- self.title = None # type: str
+ self.title = None # type: Optional[str]
+ self.primitiveType = primitiveType
for t in j:
if "extends" in t:
@@ -196,8 +209,8 @@ def __init__(self, toc, j, renderlist, redirects):
if tp not in self.uses:
self.uses[tp] = []
if (t["name"], f["name"]) not in self.uses[tp]:
- _, frg1 = urlparse.urldefrag(t["name"])
- _, frg2 = urlparse.urldefrag(f["name"])
+ _, frg1 = urllib.parse.urldefrag(t["name"])
+ _, frg2 = urllib.parse.urldefrag(f["name"])
self.uses[tp].append((frg1, frg2))
if tp not in basicTypes and tp not in self.record_refs[t["name"]]:
self.record_refs[t["name"]].append(tp)
@@ -213,9 +226,13 @@ def __init__(self, toc, j, renderlist, redirects):
("docAfter" not in f))):
self.render_type(f, 1)
- def typefmt(self, tp, redirects, nbsp=False, jsonldPredicate=None):
- # type: (Any, Dict[str, str], bool, Dict[str, str]) -> Union[str, unicode]
- global primitiveType
+ def typefmt(self,
+ tp, # type: Any
+ redirects, # type: Dict[str, str]
+ nbsp=False, # type: bool
+ jsonldPredicate=None # type: Optional[Dict[str, str]]
+ ):
+ # type: (...) -> Text
if isinstance(tp, list):
if nbsp and len(tp) <= 3:
return " | ".join([self.typefmt(n, redirects, jsonldPredicate=jsonldPredicate) for n in tp])
@@ -223,19 +240,23 @@ def typefmt(self, tp, redirects, nbsp=False, jsonldPredicate=None):
return " | ".join([self.typefmt(n, redirects) for n in tp])
if isinstance(tp, dict):
if tp["type"] == "https://w3id.org/cwl/salad#array":
- ar = "array<%s>" % (self.typefmt(tp["items"], redirects, nbsp=True))
- if jsonldPredicate and "mapSubject" in jsonldPredicate:
+ ar = "array<%s>" % (self.typefmt(
+ tp["items"], redirects, nbsp=True))
+ if jsonldPredicate is not None and "mapSubject" in jsonldPredicate:
if "mapPredicate" in jsonldPredicate:
ar += " | map<%s.%s, %s.%s>" % (self.typefmt(tp["items"], redirects),
- jsonldPredicate["mapSubject"],
- self.typefmt(tp["items"], redirects),
- jsonldPredicate["mapPredicate"])
+ jsonldPredicate[
+ "mapSubject"],
+ self.typefmt(
+ tp["items"], redirects),
+ jsonldPredicate["mapPredicate"])
ar += " | map<%s.%s, %s>" % (self.typefmt(tp["items"], redirects),
- jsonldPredicate["mapSubject"],
- self.typefmt(tp["items"], redirects))
+ jsonldPredicate[
+ "mapSubject"],
+ self.typefmt(tp["items"], redirects))
return ar
if tp["type"] in ("https://w3id.org/cwl/salad#record", "https://w3id.org/cwl/salad#enum"):
- frg = schema.avro_name(tp["name"])
+ frg = cast(Text, schema.avro_name(tp["name"]))
if tp["name"] in redirects:
return """%s """ % (redirects[tp["name"]], frg)
elif tp["name"] in self.typemap:
@@ -248,18 +269,22 @@ def typefmt(self, tp, redirects, nbsp=False, jsonldPredicate=None):
if str(tp) in redirects:
return """%s """ % (redirects[tp], redirects[tp])
elif str(tp) in basicTypes:
- return """%s """ % (primitiveType, schema.avro_name(str(tp)))
+ return """%s """ % (self.primitiveType, schema.avro_name(str(tp)))
else:
- _, frg = urlparse.urldefrag(tp)
- if frg:
+ _, frg = urllib.parse.urldefrag(tp)
+ if frg is not '':
tp = frg
return """%s """ % (to_id(tp), tp)
+ raise Exception("We should not be here!")
def render_type(self, f, depth): # type: (Dict[str, Any], int) -> None
if f["name"] in self.rendered or f["name"] in self.redirects:
return
self.rendered.add(f["name"])
+ if f.get("abstract"):
+ return
+
if "doc" not in f:
f["doc"] = ""
@@ -305,16 +330,18 @@ def extendsfrom(item, ex):
lines.append(l)
f["doc"] = "\n".join(lines)
- _, frg = urlparse.urldefrag(f["name"])
+ _, frg = urllib.parse.urldefrag(f["name"])
num = self.toc.add_entry(depth, frg)
- doc = "## %s %s\n" % (num, frg)
+ doc = u"%s %s %s\n" % (("#" * depth), num, frg)
else:
- doc = ""
+ doc = u""
if self.title is None and f["doc"]:
- self.title = f["doc"][0:f["doc"].index("\n")]
- if self.title.startswith('# '):
- self.title = self.title[2:]
+ title = f["doc"][0:f["doc"].index("\n")]
+ if title.startswith('# '):
+ self.title = title[2:]
+ else:
+ self.title = title
if f["type"] == "documentation":
f["doc"] = number_headings(self.toc, f["doc"])
@@ -357,7 +384,7 @@ def extendsfrom(item, ex):
"%s " % (
rfrg, self.typefmt(tp, self.redirects,
jsonldPredicate=i.get("jsonldPredicate")),
- "Optional" if opt else "Required",
+ opt,
mistune.markdown(desc))
if opt:
required.append(tr)
@@ -394,19 +421,20 @@ def extendsfrom(item, ex):
self.render_type(self.typemap[s], depth)
-def avrold_doc(j, outdoc, renderlist, redirects, brand, brandlink):
- # type: (List[Dict[unicode, Any]], IO[Any], str, Dict, str, str) -> None
+def avrold_doc(j, outdoc, renderlist, redirects, brand, brandlink, primtype):
+ # type: (List[Dict[Text, Any]], IO[Any], str, Dict, str, str, str) -> None
toc = ToC()
toc.start_numbering = False
- rt = RenderType(toc, j, renderlist, redirects)
- content = rt.typedoc.getvalue() # type: unicode
+ rt = RenderType(toc, j, renderlist, redirects, primtype)
+ content = rt.typedoc.getvalue() # type: Text
outdoc.write("""
`, ``, and ``) or characters
+ [not allowed](http://www.iana.org/assignments/idna-tables-6.3.0/idna-tables-6.3.0.xhtml)
+ for [Internationalized Domain Names for Applications](https://tools.ietf.org/html/rfc6452)
+ then implementations may terminate the process with a
+ `permanentFailure`.
+ jsonldPredicate:
+ "_id": "cwl:path"
+ "_type": "@id"
+ - name: basename
+ type: string?
+ doc: |
+ The base name of the file, that is, the name of the file without any
+ leading directory path. The base name must not contain a slash `/`.
+
+ If not provided, the implementation must set this field based on the
+ `location` field by taking the final path component after parsing
+ `location` as an IRI. If `basename` is provided, it is not required to
+ match the value from `location`.
+
+ When this file is made available to a CommandLineTool, it must be named
+ with `basename`, i.e. the final component of the `path` field must match
+ `basename`.
+ jsonldPredicate: "cwl:basename"
+ - name: dirname
+ type: string?
+ doc: |
+ The name of the directory containing file, that is, the path leading up
+ to the final slash in the path such that `dirname + '/' + basename ==
+ path`.
+
+ The implementation must set this field based on the value of `path`
+ prior to evaluating parameter references or expressions in a
+ CommandLineTool document. This field must not be used in any other
+ context.
+ - name: nameroot
+ type: string?
+ doc: |
+ The basename root such that `nameroot + nameext == basename`, and
+ `nameext` is empty or begins with a period and contains at most one
+ period. For the purposess of path splitting leading periods on the
+ basename are ignored; a basename of `.cshrc` will have a nameroot of
+ `.cshrc`.
+
+ The implementation must set this field automatically based on the value
+ of `basename` prior to evaluating parameter references or expressions.
+ - name: nameext
+ type: string?
+ doc: |
+ The basename extension such that `nameroot + nameext == basename`, and
+ `nameext` is empty or begins with a period and contains at most one
+ period. Leading periods on the basename are ignored; a basename of
+ `.cshrc` will have an empty `nameext`.
+
+ The implementation must set this field automatically based on the value
+ of `basename` prior to evaluating parameter references or expressions.
+ - name: checksum
+ type: string?
+ doc: |
+ Optional hash code for validating file integrity. Currently must be in the form
+ "sha1$ + hexadecimal string" using the SHA-1 algorithm.
+ - name: size
+ type: long?
+ doc: Optional file size
+ - name: "secondaryFiles"
+ type:
+ - "null"
+ - type: array
+ items: [File, Directory]
+ jsonldPredicate: "cwl:secondaryFiles"
+ doc: |
+ A list of additional files that are associated with the primary file
+ and must be transferred alongside the primary file. Examples include
+ indexes of the primary file, or external references which must be
+ included when loading primary document. A file object listed in
+ `secondaryFiles` may itself include `secondaryFiles` for which the same
+ rules apply.
+ - name: format
+ type: string?
+ jsonldPredicate:
+ _id: cwl:format
+ _type: "@id"
+ identity: true
+ doc: |
+ The format of the file: this must be an IRI of a concept node that
+ represents the file format, preferrably defined within an ontology.
+ If no ontology is available, file formats may be tested by exact match.
+
+ Reasoning about format compatability must be done by checking that an
+ input file format is the same, `owl:equivalentClass` or
+ `rdfs:subClassOf` the format required by the input parameter.
+ `owl:equivalentClass` is transitive with `rdfs:subClassOf`, e.g. if
+ ` owl:equivalentClass ` and ` owl:subclassOf ` then infer
+ ` owl:subclassOf `.
+
+ File format ontologies may be provided in the "$schema" metadata at the
+ root of the document. If no ontologies are specified in `$schema`, the
+ runtime may perform exact file format matches.
+ - name: contents
+ type: string?
+ doc: |
+ File contents literal. Maximum of 64 KiB.
+
+ If neither `location` nor `path` is provided, `contents` must be
+ non-null. The implementation must assign a unique identifier for the
+ `location` field. When the file is staged as input to CommandLineTool,
+ the value of `contents` must be written to a file.
+
+ If `loadContents` of `inputBinding` or `outputBinding` is true and
+ `location` is valid, the implementation must read up to the first 64
+ KiB of text from the file and place it in the "contents" field.
+
+
+- name: Directory
+ type: record
+ docAfter: "#File"
+ doc: |
+ Represents a directory to present to a command line tool.
+ fields:
+ - name: class
+ type:
+ type: enum
+ name: Directory_class
+ symbols:
+ - cwl:Directory
+ jsonldPredicate:
+ _id: "@type"
+ _type: "@vocab"
+ doc: Must be `Directory` to indicate this object describes a Directory.
+ - name: location
+ type: string?
+ doc: |
+ An IRI that identifies the directory resource. This may be a relative
+ reference, in which case it must be resolved using the base IRI of the
+ document. The location may refer to a local or remote resource. If
+ the `listing` field is not set, the implementation must use the
+ location IRI to retrieve directory listing. If an implementation is
+ unable to retrieve the directory listing stored at a remote resource (due to
+ unsupported protocol, access denied, or other issue) it must signal an
+ error.
+
+ If the `location` field is not provided, the `listing` field must be
+ provided. The implementation must assign a unique identifier for
+ the `location` field.
+
+ If the `path` field is provided but the `location` field is not, an
+ implementation may assign the value of the `path` field to `location`,
+ then follow the rules above.
+ jsonldPredicate:
+ _id: "@id"
+ _type: "@id"
+ - name: path
+ type: string?
+ doc: |
+ The local path where the Directory is made available prior to executing a
+ CommandLineTool. This must be set by the implementation. This field
+ must not be used in any other context. The command line tool being
+ executed must be able to to access the directory at `path` using the POSIX
+ `opendir(2)` syscall.
+
+ If the `path` contains [POSIX shell metacharacters](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02)
+ (`|`,`&`, `;`, `<`, `>`, `(`,`)`, `$`,`` ` ``, `\`, `"`, `'`,
+ ``, ``, and ``) or characters
+ [not allowed](http://www.iana.org/assignments/idna-tables-6.3.0/idna-tables-6.3.0.xhtml)
+ for [Internationalized Domain Names for Applications](https://tools.ietf.org/html/rfc6452)
+ then implementations may terminate the process with a
+ `permanentFailure`.
+ jsonldPredicate:
+ _id: "cwl:path"
+ _type: "@id"
+ - name: basename
+ type: string?
+ doc: |
+ The base name of the directory, that is, the name of the file without any
+ leading directory path. The base name must not contain a slash `/`.
+
+ If not provided, the implementation must set this field based on the
+ `location` field by taking the final path component after parsing
+ `location` as an IRI. If `basename` is provided, it is not required to
+ match the value from `location`.
+
+ When this file is made available to a CommandLineTool, it must be named
+ with `basename`, i.e. the final component of the `path` field must match
+ `basename`.
+ jsonldPredicate: "cwl:basename"
+ - name: listing
+ type:
+ - "null"
+ - type: array
+ items: [File, Directory]
+ doc: |
+ List of files or subdirectories contained in this directory. The name
+ of each file or subdirectory is determined by the `basename` field of
+ each `File` or `Directory` object. It is an error if a `File` shares a
+ `basename` with any other entry in `listing`. If two or more
+ `Directory` object share the same `basename`, this must be treated as
+ equivalent to a single subdirectory with the listings recursively
+ merged.
+ jsonldPredicate:
+ _id: "cwl:listing"
+
+- name: SchemaBase
+ type: record
+ abstract: true
+ fields:
+ - name: label
+ type:
+ - "null"
+ - string
+ jsonldPredicate: "rdfs:label"
+ doc: "A short, human-readable label of this object."
+
+
+- name: Parameter
+ type: record
+ extends: SchemaBase
+ abstract: true
+ doc: |
+ Define an input or output parameter to a process.
+
+ fields:
+ - name: secondaryFiles
+ type:
+ - "null"
+ - string
+ - Expression
+ - type: array
+ items: [string, Expression]
+ jsonldPredicate: "cwl:secondaryFiles"
+ doc: |
+ Only valid when `type: File` or is an array of `items: File`.
+
+ Describes files that must be included alongside the primary file(s).
+
+ If the value is an expression, the value of `self` in the expression
+ must be the primary input or output File to which this binding applies.
+
+ If the value is a string, it specifies that the following pattern
+ should be applied to the primary file:
+
+ 1. If string begins with one or more caret `^` characters, for each
+ caret, remove the last file extension from the path (the last
+ period `.` and all following characters). If there are no file
+ extensions, the path is unchanged.
+ 2. Append the remainder of the string to the end of the file path.
+
+ - name: format
+ type:
+ - "null"
+ - string
+ - type: array
+ items: string
+ - Expression
+ jsonldPredicate:
+ _id: cwl:format
+ _type: "@id"
+ identity: true
+ doc: |
+ Only valid when `type: File` or is an array of `items: File`.
+
+ For input parameters, this must be one or more IRIs of concept nodes
+ that represents file formats which are allowed as input to this
+ parameter, preferrably defined within an ontology. If no ontology is
+ available, file formats may be tested by exact match.
+
+ For output parameters, this is the file format that will be assigned to
+ the output parameter.
+
+ - name: streamable
+ type: boolean?
+ doc: |
+ Only valid when `type: File` or is an array of `items: File`.
+
+ A value of `true` indicates that the file is read or written
+ sequentially without seeking. An implementation may use this flag to
+ indicate whether it is valid to stream file contents using a named
+ pipe. Default: `false`.
+
+ - name: doc
+ type:
+ - string?
+ - string[]?
+ doc: "A documentation string for this type, or an array of strings which should be concatenated."
+ jsonldPredicate: "rdfs:comment"
+
+
+- type: enum
+ name: Expression
+ doc: |
+ 'Expression' is not a real type. It indicates that a field must allow
+ runtime parameter references. If [InlineJavascriptRequirement](#InlineJavascriptRequirement)
+ is declared and supported by the platform, the field must also allow
+ Javascript expressions.
+ symbols:
+ - cwl:ExpressionPlaceholder
+
+
+- name: InputBinding
+ type: record
+ abstract: true
+ fields:
+ - name: loadContents
+ type:
+ - "null"
+ - boolean
+ jsonldPredicate: "cwl:loadContents"
+ doc: |
+ Only valid when `type: File` or is an array of `items: File`.
+
+ Read up to the first 64 KiB of text from the file and place it in the
+ "contents" field of the file object for use by expressions.
+
+
+- name: OutputBinding
+ type: record
+ abstract: true
+
+
+- name: InputSchema
+ extends: SchemaBase
+ type: record
+ abstract: true
+
+
+- name: OutputSchema
+ extends: SchemaBase
+ type: record
+ abstract: true
+
+
+- name: InputRecordField
+ type: record
+ extends: "sld:RecordField"
+ specialize:
+ - specializeFrom: "sld:RecordSchema"
+ specializeTo: InputRecordSchema
+ - specializeFrom: "sld:EnumSchema"
+ specializeTo: InputEnumSchema
+ - specializeFrom: "sld:ArraySchema"
+ specializeTo: InputArraySchema
+ - specializeFrom: "sld:PrimitiveType"
+ specializeTo: CWLType
+ fields:
+ - name: inputBinding
+ type: InputBinding?
+ jsonldPredicate: "cwl:inputBinding"
+ - name: label
+ type: string?
+ jsonldPredicate: "rdfs:label"
+ doc: "A short, human-readable label of this process object."
+
+
+- name: InputRecordSchema
+ type: record
+ extends: ["sld:RecordSchema", InputSchema]
+ specialize:
+ - specializeFrom: "sld:RecordField"
+ specializeTo: InputRecordField
+
+
+- name: InputEnumSchema
+ type: record
+ extends: ["sld:EnumSchema", InputSchema]
+ fields:
+ - name: inputBinding
+ type: InputBinding?
+ jsonldPredicate: "cwl:inputBinding"
+
+
+- name: InputArraySchema
+ type: record
+ extends: ["sld:ArraySchema", InputSchema]
+ specialize:
+ - specializeFrom: "sld:RecordSchema"
+ specializeTo: InputRecordSchema
+ - specializeFrom: "sld:EnumSchema"
+ specializeTo: InputEnumSchema
+ - specializeFrom: "sld:ArraySchema"
+ specializeTo: InputArraySchema
+ - specializeFrom: "sld:PrimitiveType"
+ specializeTo: CWLType
+ fields:
+ - name: inputBinding
+ type: InputBinding?
+ jsonldPredicate: "cwl:inputBinding"
+
+
+- name: OutputRecordField
+ type: record
+ extends: "sld:RecordField"
+ specialize:
+ - specializeFrom: "sld:RecordSchema"
+ specializeTo: OutputRecordSchema
+ - specializeFrom: "sld:EnumSchema"
+ specializeTo: OutputEnumSchema
+ - specializeFrom: "sld:ArraySchema"
+ specializeTo: OutputArraySchema
+ - specializeFrom: "sld:PrimitiveType"
+ specializeTo: CWLType
+ fields:
+ - name: outputBinding
+ type: OutputBinding?
+ jsonldPredicate: "cwl:outputBinding"
+
+
+- name: OutputRecordSchema
+ type: record
+ extends: ["sld:RecordSchema", "#OutputSchema"]
+ docParent: "#OutputParameter"
+ specialize:
+ - specializeFrom: "sld:RecordField"
+ specializeTo: OutputRecordField
+
+
+- name: OutputEnumSchema
+ type: record
+ extends: ["sld:EnumSchema", OutputSchema]
+ docParent: "#OutputParameter"
+ fields:
+ - name: outputBinding
+ type: OutputBinding?
+ jsonldPredicate: "cwl:outputBinding"
+
+- name: OutputArraySchema
+ type: record
+ extends: ["sld:ArraySchema", OutputSchema]
+ docParent: "#OutputParameter"
+ specialize:
+ - specializeFrom: "sld:RecordSchema"
+ specializeTo: OutputRecordSchema
+ - specializeFrom: "sld:EnumSchema"
+ specializeTo: OutputEnumSchema
+ - specializeFrom: "sld:ArraySchema"
+ specializeTo: OutputArraySchema
+ - specializeFrom: "sld:PrimitiveType"
+ specializeTo: CWLType
+ fields:
+ - name: outputBinding
+ type: OutputBinding?
+ jsonldPredicate: "cwl:outputBinding"
+
+
+- name: InputParameter
+ type: record
+ extends: Parameter
+ fields:
+ - name: id
+ type: string
+ jsonldPredicate: "@id"
+ doc: "The unique identifier for this parameter object."
+
+ - name: inputBinding
+ type: InputBinding?
+ jsonldPredicate: "cwl:inputBinding"
+ doc: |
+ Describes how to handle the inputs of a process and convert them
+ into a concrete form for execution, such as command line parameters.
+
+ - name: default
+ type: Any?
+ jsonldPredicate: "cwl:default"
+ doc: |
+ The default value for this parameter if not provided in the input
+ object.
+
+ - name: type
+ type:
+ - "null"
+ - CWLType
+ - InputRecordSchema
+ - InputEnumSchema
+ - InputArraySchema
+ - string
+ - type: array
+ items:
+ - CWLType
+ - InputRecordSchema
+ - InputEnumSchema
+ - InputArraySchema
+ - string
+ jsonldPredicate:
+ "_id": "sld:type"
+ "_type": "@vocab"
+ refScope: 2
+ typeDSL: True
+ doc: |
+ Specify valid types of data that may be assigned to this parameter.
+
+- name: OutputParameter
+ type: record
+ extends: Parameter
+ fields:
+ - name: id
+ type: string
+ jsonldPredicate: "@id"
+ doc: "The unique identifier for this parameter object."
+ - name: outputBinding
+ type: OutputBinding?
+ jsonldPredicate: "cwl:outputBinding"
+ doc: |
+ Describes how to handle the outputs of a process.
+
+
+- type: record
+ name: ProcessRequirement
+ abstract: true
+ doc: |
+ A process requirement declares a prerequisite that may or must be fulfilled
+ before executing a process. See [`Process.hints`](#process) and
+ [`Process.requirements`](#process).
+
+ Process requirements are the primary mechanism for specifying extensions to
+ the CWL core specification.
+
+
+- type: record
+ name: Process
+ abstract: true
+ doc: |
+
+ The base executable type in CWL is the `Process` object defined by the
+ document. Note that the `Process` object is abstract and cannot be
+ directly executed.
+
+ fields:
+ - name: id
+ type: string?
+ jsonldPredicate: "@id"
+ doc: "The unique identifier for this process object."
+ - name: inputs
+ type:
+ type: array
+ items: InputParameter
+ jsonldPredicate:
+ _id: "cwl:inputs"
+ mapSubject: id
+ mapPredicate: type
+ doc: |
+ Defines the input parameters of the process. The process is ready to
+ run when all required input parameters are associated with concrete
+ values. Input parameters include a schema for each parameter which is
+ used to validate the input object. It may also be used to build a user
+ interface for constructing the input object.
+ - name: outputs
+ type:
+ type: array
+ items: OutputParameter
+ jsonldPredicate:
+ _id: "cwl:outputs"
+ mapSubject: id
+ mapPredicate: type
+ doc: |
+ Defines the parameters representing the output of the process. May be
+ used to generate and/or validate the output object.
+ - name: requirements
+ type: ProcessRequirement[]?
+ jsonldPredicate:
+ _id: "cwl:requirements"
+ mapSubject: class
+ doc: |
+ Declares requirements that apply to either the runtime environment or the
+ workflow engine that must be met in order to execute this process. If
+ an implementation cannot satisfy all requirements, or a requirement is
+ listed which is not recognized by the implementation, it is a fatal
+ error and the implementation must not attempt to run the process,
+ unless overridden at user option.
+ - name: hints
+ type: Any[]?
+ doc: |
+ Declares hints applying to either the runtime environment or the
+ workflow engine that may be helpful in executing this process. It is
+ not an error if an implementation cannot satisfy all hints, however
+ the implementation may report a warning.
+ jsonldPredicate:
+ _id: cwl:hints
+ noLinkCheck: true
+ mapSubject: class
+ - name: label
+ type: string?
+ jsonldPredicate: "rdfs:label"
+ doc: "A short, human-readable label of this process object."
+ - name: doc
+ type: string?
+ jsonldPredicate: "rdfs:comment"
+ doc: "A long, human-readable description of this process object."
+ - name: cwlVersion
+ type: CWLVersion?
+ doc: |
+ CWL document version. Always required at the document root. Not
+ required for a Process embedded inside another Process.
+ jsonldPredicate:
+ "_id": "cwl:cwlVersion"
+ "_type": "@vocab"
+
+- name: InlineJavascriptRequirement
+ type: record
+ extends: ProcessRequirement
+ doc: |
+ Indicates that the workflow platform must support inline Javascript expressions.
+ If this requirement is not present, the workflow platform must not perform expression
+ interpolatation.
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'InlineJavascriptRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: expressionLib
+ type: string[]?
+ doc: |
+ Additional code fragments that will also be inserted
+ before executing the expression code. Allows for function definitions that may
+ be called from CWL expressions.
+
+
+- name: SchemaDefRequirement
+ type: record
+ extends: ProcessRequirement
+ doc: |
+ This field consists of an array of type definitions which must be used when
+ interpreting the `inputs` and `outputs` fields. When a `type` field
+ contain a IRI, the implementation must check if the type is defined in
+ `schemaDefs` and use that definition. If the type is not found in
+ `schemaDefs`, it is an error. The entries in `schemaDefs` must be
+ processed in the order listed such that later schema definitions may refer
+ to earlier schema definitions.
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'SchemaDefRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: types
+ type:
+ type: array
+ items: InputSchema
+ doc: The list of type definitions.
diff --git a/v1.0/salad/schema_salad/tests/test_schema/Workflow.yml b/v1.0/salad/schema_salad/tests/test_schema/Workflow.yml
new file mode 100644
index 000000000..26bde8e29
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/Workflow.yml
@@ -0,0 +1,582 @@
+$base: "https://w3id.org/cwl/cwl#"
+
+$namespaces:
+ cwl: "https://w3id.org/cwl/cwl#"
+
+$graph:
+
+- name: "WorkflowDoc"
+ type: documentation
+ doc:
+ - |
+ # Common Workflow Language (CWL) Workflow Description, v1.0
+
+ This version:
+ * https://w3id.org/cwl/v1.0/
+
+ Current version:
+ * https://w3id.org/cwl/
+ - "\n\n"
+ - {$include: contrib.md}
+ - "\n\n"
+ - |
+ # Abstract
+
+ One way to define a workflow is: an analysis task represented by a
+ directed graph describing a sequence of operations that transform an
+ input data set to output. This specification defines the Common Workflow
+ Language (CWL) Workflow description, a vendor-neutral standard for
+ representing workflows intended to be portable across a variety of
+ computing platforms.
+
+ - {$include: intro.md}
+
+ - |
+
+ ## Introduction to v1.0
+
+ This specification represents the first full release from the CWL group.
+ Since draft-3, this draft introduces the following changes and additions:
+
+ * The `inputs` and `outputs` fields have been renamed `in` and `out`.
+ * Syntax simplifcations: denoted by the `map<>` syntax. Example: `in`
+ contains a list of items, each with an id. Now one can specify
+ a mapping of that identifier to the corresponding
+ `InputParameter`.
+ ```
+ in:
+ - id: one
+ type: string
+ doc: First input parameter
+ - id: two
+ type: int
+ doc: Second input parameter
+ ```
+ can be
+ ```
+ in:
+ one:
+ type: string
+ doc: First input parameter
+ two:
+ type: int
+ doc: Second input parameter
+ ```
+ * The common field `description` has been renamed to `doc`.
+
+ ## Purpose
+
+ The Common Workflow Language Command Line Tool Description express
+ workflows for data-intensive science, such as Bioinformatics, Chemistry,
+ Physics, and Astronomy. This specification is intended to define a data
+ and execution model for Workflows that can be implemented on top of a
+ variety of computing platforms, ranging from an individual workstation to
+ cluster, grid, cloud, and high performance computing systems.
+
+ - {$include: concepts.md}
+
+- name: ExpressionToolOutputParameter
+ type: record
+ extends: OutputParameter
+ fields:
+ - name: type
+ type:
+ - "null"
+ - "#CWLType"
+ - "#OutputRecordSchema"
+ - "#OutputEnumSchema"
+ - "#OutputArraySchema"
+ - string
+ - type: array
+ items:
+ - "#CWLType"
+ - "#OutputRecordSchema"
+ - "#OutputEnumSchema"
+ - "#OutputArraySchema"
+ - string
+ jsonldPredicate:
+ "_id": "sld:type"
+ "_type": "@vocab"
+ refScope: 2
+ typeDSL: True
+ doc: |
+ Specify valid types of data that may be assigned to this parameter.
+
+- type: record
+ name: ExpressionTool
+ extends: Process
+ specialize:
+ - specializeFrom: "#OutputParameter"
+ specializeTo: "#ExpressionToolOutputParameter"
+ documentRoot: true
+ doc: |
+ Execute an expression as a Workflow step.
+ fields:
+ - name: "class"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ type: string
+ - name: expression
+ type: [string, Expression]
+ doc: |
+ The expression to execute. The expression must return a JSON object which
+ matches the output parameters of the ExpressionTool.
+
+- name: LinkMergeMethod
+ type: enum
+ docParent: "#WorkflowStepInput"
+ doc: The input link merge method, described in [WorkflowStepInput](#WorkflowStepInput).
+ symbols:
+ - merge_nested
+ - merge_flattened
+
+
+- name: WorkflowOutputParameter
+ type: record
+ extends: OutputParameter
+ docParent: "#Workflow"
+ doc: |
+ Describe an output parameter of a workflow. The parameter must be
+ connected to one or more parameters defined in the workflow that will
+ provide the value of the output parameter.
+ fields:
+ - name: outputSource
+ doc: |
+ Specifies one or more workflow parameters that supply the value of to
+ the output parameter.
+ jsonldPredicate:
+ "_id": "cwl:outputSource"
+ "_type": "@id"
+ refScope: 0
+ type:
+ - string?
+ - string[]?
+ - name: linkMerge
+ type: ["null", "#LinkMergeMethod"]
+ jsonldPredicate: "cwl:linkMerge"
+ doc: |
+ The method to use to merge multiple sources into a single array.
+ If not specified, the default method is "merge_nested".
+ - name: type
+ type:
+ - "null"
+ - "#CWLType"
+ - "#OutputRecordSchema"
+ - "#OutputEnumSchema"
+ - "#OutputArraySchema"
+ - string
+ - type: array
+ items:
+ - "#CWLType"
+ - "#OutputRecordSchema"
+ - "#OutputEnumSchema"
+ - "#OutputArraySchema"
+ - string
+ jsonldPredicate:
+ "_id": "sld:type"
+ "_type": "@vocab"
+ refScope: 2
+ typeDSL: True
+ doc: |
+ Specify valid types of data that may be assigned to this parameter.
+
+
+- name: Sink
+ type: record
+ abstract: true
+ fields:
+ - name: source
+ doc: |
+ Specifies one or more workflow parameters that will provide input to
+ the underlying step parameter.
+ jsonldPredicate:
+ "_id": "cwl:source"
+ "_type": "@id"
+ refScope: 2
+ type:
+ - string?
+ - string[]?
+ - name: linkMerge
+ type: LinkMergeMethod?
+ jsonldPredicate: "cwl:linkMerge"
+ doc: |
+ The method to use to merge multiple inbound links into a single array.
+ If not specified, the default method is "merge_nested".
+
+
+- type: record
+ name: WorkflowStepInput
+ extends: Sink
+ docParent: "#WorkflowStep"
+ doc: |
+ The input of a workflow step connects an upstream parameter (from the
+ workflow inputs, or the outputs of other workflows steps) with the input
+ parameters of the underlying step.
+
+ ## Input object
+
+ A WorkflowStepInput object must contain an `id` field in the form
+ `#fieldname` or `#stepname.fieldname`. When the `id` field contains a
+ period `.` the field name consists of the characters following the final
+ period. This defines a field of the workflow step input object with the
+ value of the `source` parameter(s).
+
+ ## Merging
+
+ To merge multiple inbound data links,
+ [MultipleInputFeatureRequirement](#MultipleInputFeatureRequirement) must be specified
+ in the workflow or workflow step requirements.
+
+ If the sink parameter is an array, or named in a [workflow
+ scatter](#WorkflowStep) operation, there may be multiple inbound data links
+ listed in the `source` field. The values from the input links are merged
+ depending on the method specified in the `linkMerge` field. If not
+ specified, the default method is "merge_nested".
+
+ * **merge_nested**
+
+ The input must be an array consisting of exactly one entry for each
+ input link. If "merge_nested" is specified with a single link, the value
+ from the link must be wrapped in a single-item list.
+
+ * **merge_flattened**
+
+ 1. The source and sink parameters must be compatible types, or the source
+ type must be compatible with single element from the "items" type of
+ the destination array parameter.
+ 2. Source parameters which are arrays are concatenated.
+ Source parameters which are single element types are appended as
+ single elements.
+
+ fields:
+ - name: id
+ type: string
+ jsonldPredicate: "@id"
+ doc: "A unique identifier for this workflow input parameter."
+ - name: default
+ type: ["null", Any]
+ doc: |
+ The default value for this parameter if there is no `source`
+ field.
+ jsonldPredicate: "cwl:default"
+ - name: valueFrom
+ type:
+ - "null"
+ - "string"
+ - "#Expression"
+ jsonldPredicate: "cwl:valueFrom"
+ doc: |
+ To use valueFrom, [StepInputExpressionRequirement](#StepInputExpressionRequirement) must
+ be specified in the workflow or workflow step requirements.
+
+ If `valueFrom` is a constant string value, use this as the value for
+ this input parameter.
+
+ If `valueFrom` is a parameter reference or expression, it must be
+ evaluated to yield the actual value to be assiged to the input field.
+
+ The `self` value of in the parameter reference or expression must be
+ the value of the parameter(s) specified in the `source` field, or
+ null if there is no `source` field.
+
+ The value of `inputs` in the parameter reference or expression must be
+ the input object to the workflow step after assigning the `source`
+ values and then scattering. The order of evaluating `valueFrom` among
+ step input parameters is undefined and the result of evaluating
+ `valueFrom` on a parameter must not be visible to evaluation of
+ `valueFrom` on other parameters.
+
+
+- type: record
+ name: WorkflowStepOutput
+ docParent: "#WorkflowStep"
+ doc: |
+ Associate an output parameter of the underlying process with a workflow
+ parameter. The workflow parameter (given in the `id` field) be may be used
+ as a `source` to connect with input parameters of other workflow steps, or
+ with an output parameter of the process.
+ fields:
+ - name: id
+ type: string
+ jsonldPredicate: "@id"
+ doc: |
+ A unique identifier for this workflow output parameter. This is the
+ identifier to use in the `source` field of `WorkflowStepInput` to
+ connect the output value to downstream parameters.
+
+
+- name: ScatterMethod
+ type: enum
+ docParent: "#WorkflowStep"
+ doc: The scatter method, as described in [workflow step scatter](#WorkflowStep).
+ symbols:
+ - dotproduct
+ - nested_crossproduct
+ - flat_crossproduct
+
+
+- name: WorkflowStep
+ type: record
+ docParent: "#Workflow"
+ doc: |
+ A workflow step is an executable element of a workflow. It specifies the
+ underlying process implementation (such as `CommandLineTool` or another
+ `Workflow`) in the `run` field and connects the input and output parameters
+ of the underlying process to workflow parameters.
+
+ # Scatter/gather
+
+ To use scatter/gather,
+ [ScatterFeatureRequirement](#ScatterFeatureRequirement) must be specified
+ in the workflow or workflow step requirements.
+
+ A "scatter" operation specifies that the associated workflow step or
+ subworkflow should execute separately over a list of input elements. Each
+ job making up a scatter operation is independent and may be executed
+ concurrently.
+
+ The `scatter` field specifies one or more input parameters which will be
+ scattered. An input parameter may be listed more than once. The declared
+ type of each input parameter is implicitly wrapped in an array for each
+ time it appears in the `scatter` field. As a result, upstream parameters
+ which are connected to scattered parameters may be arrays.
+
+ All output parameter types are also implicitly wrapped in arrays. Each job
+ in the scatter results in an entry in the output array.
+
+ If `scatter` declares more than one input parameter, `scatterMethod`
+ describes how to decompose the input into a discrete set of jobs.
+
+ * **dotproduct** specifies that each of the input arrays are aligned and one
+ element taken from each array to construct each job. It is an error
+ if all input arrays are not the same length.
+
+ * **nested_crossproduct** specifies the Cartesian product of the inputs,
+ producing a job for every combination of the scattered inputs. The
+ output must be nested arrays for each level of scattering, in the
+ order that the input arrays are listed in the `scatter` field.
+
+ * **flat_crossproduct** specifies the Cartesian product of the inputs,
+ producing a job for every combination of the scattered inputs. The
+ output arrays must be flattened to a single level, but otherwise listed in the
+ order that the input arrays are listed in the `scatter` field.
+
+ # Subworkflows
+
+ To specify a nested workflow as part of a workflow step,
+ [SubworkflowFeatureRequirement](#SubworkflowFeatureRequirement) must be
+ specified in the workflow or workflow step requirements.
+
+ fields:
+ - name: id
+ type: string
+ jsonldPredicate: "@id"
+ doc: "The unique identifier for this workflow step."
+ - name: in
+ type: WorkflowStepInput[]
+ jsonldPredicate:
+ _id: "cwl:in"
+ mapSubject: id
+ mapPredicate: source
+ doc: |
+ Defines the input parameters of the workflow step. The process is ready to
+ run when all required input parameters are associated with concrete
+ values. Input parameters include a schema for each parameter which is
+ used to validate the input object. It may also be used build a user
+ interface for constructing the input object.
+ - name: out
+ type:
+ - type: array
+ items: [string, WorkflowStepOutput]
+ jsonldPredicate:
+ _id: "cwl:out"
+ _type: "@id"
+ identity: true
+ doc: |
+ Defines the parameters representing the output of the process. May be
+ used to generate and/or validate the output object.
+ - name: requirements
+ type: ProcessRequirement[]?
+ jsonldPredicate:
+ _id: "cwl:requirements"
+ mapSubject: class
+ doc: |
+ Declares requirements that apply to either the runtime environment or the
+ workflow engine that must be met in order to execute this workflow step. If
+ an implementation cannot satisfy all requirements, or a requirement is
+ listed which is not recognized by the implementation, it is a fatal
+ error and the implementation must not attempt to run the process,
+ unless overridden at user option.
+ - name: hints
+ type: Any[]?
+ jsonldPredicate:
+ _id: "cwl:hints"
+ noLinkCheck: true
+ mapSubject: class
+ doc: |
+ Declares hints applying to either the runtime environment or the
+ workflow engine that may be helpful in executing this workflow step. It is
+ not an error if an implementation cannot satisfy all hints, however
+ the implementation may report a warning.
+ - name: label
+ type: string?
+ jsonldPredicate: "rdfs:label"
+ doc: "A short, human-readable label of this process object."
+ - name: doc
+ type: string?
+ jsonldPredicate: "rdfs:comment"
+ doc: "A long, human-readable description of this process object."
+ - name: run
+ type: [string, Process]
+ jsonldPredicate:
+ "_id": "cwl:run"
+ "_type": "@id"
+ doc: |
+ Specifies the process to run.
+ - name: scatter
+ type:
+ - string?
+ - string[]?
+ jsonldPredicate:
+ "_id": "cwl:scatter"
+ "_type": "@id"
+ "_container": "@list"
+ refScope: 0
+ - name: scatterMethod
+ doc: |
+ Required if `scatter` is an array of more than one element.
+ type: ScatterMethod?
+ jsonldPredicate:
+ "_id": "cwl:scatterMethod"
+ "_type": "@vocab"
+
+
+- name: Workflow
+ type: record
+ extends: "#Process"
+ documentRoot: true
+ specialize:
+ - specializeFrom: "#OutputParameter"
+ specializeTo: "#WorkflowOutputParameter"
+ doc: |
+ A workflow describes a set of **steps** and the **dependencies** between
+ those steps. When a step produces output that will be consumed by a
+ second step, the first step is a dependency of the second step.
+
+ When there is a dependency, the workflow engine must execute the preceeding
+ step and wait for it to successfully produce output before executing the
+ dependent step. If two steps are defined in the workflow graph that
+ are not directly or indirectly dependent, these steps are **independent**,
+ and may execute in any order or execute concurrently. A workflow is
+ complete when all steps have been executed.
+
+ Dependencies between parameters are expressed using the `source` field on
+ [workflow step input parameters](#WorkflowStepInput) and [workflow output
+ parameters](#WorkflowOutputParameter).
+
+ The `source` field expresses the dependency of one parameter on another
+ such that when a value is associated with the parameter specified by
+ `source`, that value is propagated to the destination parameter. When all
+ data links inbound to a given step are fufilled, the step is ready to
+ execute.
+
+ ## Workflow success and failure
+
+ A completed step must result in one of `success`, `temporaryFailure` or
+ `permanentFailure` states. An implementation may choose to retry a step
+ execution which resulted in `temporaryFailure`. An implementation may
+ choose to either continue running other steps of a workflow, or terminate
+ immediately upon `permanentFailure`.
+
+ * If any step of a workflow execution results in `permanentFailure`, then
+ the workflow status is `permanentFailure`.
+
+ * If one or more steps result in `temporaryFailure` and all other steps
+ complete `success` or are not executed, then the workflow status is
+ `temporaryFailure`.
+
+ * If all workflow steps are executed and complete with `success`, then the
+ workflow status is `success`.
+
+ # Extensions
+
+ [ScatterFeatureRequirement](#ScatterFeatureRequirement) and
+ [SubworkflowFeatureRequirement](#SubworkflowFeatureRequirement) are
+ available as standard [extensions](#Extensions_and_Metadata) to core
+ workflow semantics.
+
+ fields:
+ - name: "class"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ type: string
+ - name: steps
+ doc: |
+ The individual steps that make up the workflow. Each step is executed when all of its
+ input data links are fufilled. An implementation may choose to execute
+ the steps in a different order than listed and/or execute steps
+ concurrently, provided that dependencies between steps are met.
+ type:
+ - type: array
+ items: "#WorkflowStep"
+ jsonldPredicate:
+ mapSubject: id
+
+
+- type: record
+ name: SubworkflowFeatureRequirement
+ extends: ProcessRequirement
+ doc: |
+ Indicates that the workflow platform must support nested workflows in
+ the `run` field of [WorkflowStep](#WorkflowStep).
+ fields:
+ - name: "class"
+ type: "string"
+ doc: "Always 'SubworkflowFeatureRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+
+- name: ScatterFeatureRequirement
+ type: record
+ extends: ProcessRequirement
+ doc: |
+ Indicates that the workflow platform must support the `scatter` and
+ `scatterMethod` fields of [WorkflowStep](#WorkflowStep).
+ fields:
+ - name: "class"
+ type: "string"
+ doc: "Always 'ScatterFeatureRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+
+- name: MultipleInputFeatureRequirement
+ type: record
+ extends: ProcessRequirement
+ doc: |
+ Indicates that the workflow platform must support multiple inbound data links
+ listed in the `source` field of [WorkflowStepInput](#WorkflowStepInput).
+ fields:
+ - name: "class"
+ type: "string"
+ doc: "Always 'MultipleInputFeatureRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+
+- type: record
+ name: StepInputExpressionRequirement
+ extends: ProcessRequirement
+ doc: |
+ Indicate that the workflow platform must support the `valueFrom` field
+ of [WorkflowStepInput](#WorkflowStepInput).
+ fields:
+ - name: "class"
+ type: "string"
+ doc: "Always 'StepInputExpressionRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
diff --git a/v1.0/salad/schema_salad/tests/test_schema/concepts.md b/v1.0/salad/schema_salad/tests/test_schema/concepts.md
new file mode 100644
index 000000000..8b1378917
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/concepts.md
@@ -0,0 +1 @@
+
diff --git a/v1.0/salad/schema_salad/tests/test_schema/contrib.md b/v1.0/salad/schema_salad/tests/test_schema/contrib.md
new file mode 100644
index 000000000..8b1378917
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/contrib.md
@@ -0,0 +1 @@
+
diff --git a/v1.0/salad/schema_salad/tests/test_schema/intro.md b/v1.0/salad/schema_salad/tests/test_schema/intro.md
new file mode 100644
index 000000000..8b1378917
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/intro.md
@@ -0,0 +1 @@
+
diff --git a/v1.0/salad/schema_salad/tests/test_schema/invocation.md b/v1.0/salad/schema_salad/tests/test_schema/invocation.md
new file mode 100644
index 000000000..8b1378917
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/invocation.md
@@ -0,0 +1 @@
+
diff --git a/v1.0/salad/schema_salad/tests/test_schema/metaschema_base.yml b/v1.0/salad/schema_salad/tests/test_schema/metaschema_base.yml
new file mode 100644
index 000000000..73511d141
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/metaschema_base.yml
@@ -0,0 +1,164 @@
+$base: "https://w3id.org/cwl/salad#"
+
+$namespaces:
+ sld: "https://w3id.org/cwl/salad#"
+ dct: "http://purl.org/dc/terms/"
+ rdf: "http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+ rdfs: "http://www.w3.org/2000/01/rdf-schema#"
+ xsd: "http://www.w3.org/2001/XMLSchema#"
+
+$graph:
+- name: PrimitiveType
+ type: enum
+ symbols:
+ - "sld:null"
+ - "xsd:boolean"
+ - "xsd:int"
+ - "xsd:long"
+ - "xsd:float"
+ - "xsd:double"
+ - "xsd:string"
+ doc:
+ - |
+ Salad data types are based on Avro schema declarations. Refer to the
+ [Avro schema declaration documentation](https://avro.apache.org/docs/current/spec.html#schemas) for
+ detailed information.
+ - "null: no value"
+ - "boolean: a binary value"
+ - "int: 32-bit signed integer"
+ - "long: 64-bit signed integer"
+ - "float: single precision (32-bit) IEEE 754 floating-point number"
+ - "double: double precision (64-bit) IEEE 754 floating-point number"
+ - "string: Unicode character sequence"
+
+
+- name: Any
+ type: enum
+ symbols: ["#Any"]
+ doc: |
+ The **Any** type validates for any non-null value.
+
+
+- name: RecordField
+ type: record
+ doc: A field of a record.
+ fields:
+ - name: name
+ type: string
+ jsonldPredicate: "@id"
+ doc: |
+ The name of the field
+
+ - name: doc
+ type: string?
+ doc: |
+ A documentation string for this field
+ jsonldPredicate: "rdfs:comment"
+
+ - name: type
+ type:
+ - PrimitiveType
+ - RecordSchema
+ - EnumSchema
+ - ArraySchema
+ - string
+ - type: array
+ items:
+ - PrimitiveType
+ - RecordSchema
+ - EnumSchema
+ - ArraySchema
+ - string
+ jsonldPredicate:
+ _id: sld:type
+ _type: "@vocab"
+ typeDSL: true
+ refScope: 2
+ doc: |
+ The field type
+
+
+- name: RecordSchema
+ type: record
+ fields:
+ type:
+ doc: "Must be `record`"
+ type:
+ name: Record_symbol
+ type: enum
+ symbols:
+ - "sld:record"
+ jsonldPredicate:
+ _id: "sld:type"
+ _type: "@vocab"
+ typeDSL: true
+ refScope: 2
+ fields:
+ type: RecordField[]?
+ jsonldPredicate:
+ _id: sld:fields
+ mapSubject: name
+ mapPredicate: type
+ doc: "Defines the fields of the record."
+
+
+- name: EnumSchema
+ type: record
+ doc: |
+ Define an enumerated type.
+ fields:
+ type:
+ doc: "Must be `enum`"
+ type:
+ name: Enum_symbol
+ type: enum
+ symbols:
+ - "sld:enum"
+ jsonldPredicate:
+ _id: "sld:type"
+ _type: "@vocab"
+ typeDSL: true
+ refScope: 2
+ symbols:
+ type: string[]
+ jsonldPredicate:
+ _id: "sld:symbols"
+ _type: "@id"
+ identity: true
+ doc: "Defines the set of valid symbols."
+
+
+- name: ArraySchema
+ type: record
+ fields:
+ type:
+ doc: "Must be `array`"
+ type:
+ name: Array_symbol
+ type: enum
+ symbols:
+ - "sld:array"
+ jsonldPredicate:
+ _id: "sld:type"
+ _type: "@vocab"
+ typeDSL: true
+ refScope: 2
+ items:
+ type:
+ - PrimitiveType
+ - RecordSchema
+ - EnumSchema
+ - ArraySchema
+ - string
+ - type: array
+ items:
+ - PrimitiveType
+ - RecordSchema
+ - EnumSchema
+ - ArraySchema
+ - string
+ jsonldPredicate:
+ _id: "sld:items"
+ _type: "@vocab"
+ refScope: 2
+ doc: "Defines the type of the array elements."
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test1.cwl b/v1.0/salad/schema_salad/tests/test_schema/test1.cwl
new file mode 100644
index 000000000..2406c8648
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test1.cwl
@@ -0,0 +1 @@
+class: Workflow
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test10.cwl b/v1.0/salad/schema_salad/tests/test_schema/test10.cwl
new file mode 100644
index 000000000..28608072e
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test10.cwl
@@ -0,0 +1,10 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps:
+ step1:
+ scatterMethod: [record]
+ in: []
+ out: [out]
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test11.cwl b/v1.0/salad/schema_salad/tests/test_schema/test11.cwl
new file mode 100644
index 000000000..43281fbc5
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test11.cwl
@@ -0,0 +1,10 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps:
+ step1:
+ run: blub.cwl
+ in: []
+ out: [out]
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test12.cwl b/v1.0/salad/schema_salad/tests/test_schema/test12.cwl
new file mode 100644
index 000000000..d994e7cbc
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test12.cwl
@@ -0,0 +1,16 @@
+cwlVersion: v1.0
+class: CommandLineTool
+baseCommand: echo
+inputs:
+ - id: example_flag
+ type: boolean
+ inputBinding:
+ position: 1
+ prefix: -f
+ - id: example_flag
+ type: int
+ inputBinding:
+ position: 3
+ prefix: --example-string
+
+outputs: []
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test13.cwl b/v1.0/salad/schema_salad/tests/test_schema/test13.cwl
new file mode 100644
index 000000000..caa274d2a
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test13.cwl
@@ -0,0 +1,20 @@
+cwlVersion: v1.0
+class: Workflow
+inputs:
+ example_flag:
+ type: boolean
+ inputBinding:
+ position: 1
+ prefix: -f
+
+outputs: []
+
+steps:
+ example_flag:
+ in: []
+ out: []
+ run:
+ id: blah
+ class: CommandLineTool
+ inputs: []
+ outputs: []
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test14.cwl b/v1.0/salad/schema_salad/tests/test_schema/test14.cwl
new file mode 100644
index 000000000..729ee83df
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test14.cwl
@@ -0,0 +1,11 @@
+cwlVersion: v1.0
+class: CommandLineTool
+baseCommand: echo
+inputs:
+ example_flag:
+ type: boolean
+ inputBinding:
+ position: 1
+ prefix: -f
+outputs:
+ example_flag: int
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test2.cwl b/v1.0/salad/schema_salad/tests/test_schema/test2.cwl
new file mode 100644
index 000000000..96ae14014
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test2.cwl
@@ -0,0 +1 @@
+class: xWorkflow
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test3.cwl b/v1.0/salad/schema_salad/tests/test_schema/test3.cwl
new file mode 100644
index 000000000..517e920d2
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test3.cwl
@@ -0,0 +1,6 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: xstring
+steps: []
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test4.cwl b/v1.0/salad/schema_salad/tests/test_schema/test4.cwl
new file mode 100644
index 000000000..e57292d36
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test4.cwl
@@ -0,0 +1,6 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: 12
+steps: []
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test5.cwl b/v1.0/salad/schema_salad/tests/test_schema/test5.cwl
new file mode 100644
index 000000000..8a7ba2220
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test5.cwl
@@ -0,0 +1,6 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps: [12]
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test6.cwl b/v1.0/salad/schema_salad/tests/test_schema/test6.cwl
new file mode 100644
index 000000000..eff4ac5c7
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test6.cwl
@@ -0,0 +1,5 @@
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps: [12]
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test7.cwl b/v1.0/salad/schema_salad/tests/test_schema/test7.cwl
new file mode 100644
index 000000000..0e12c1295
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test7.cwl
@@ -0,0 +1,10 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps:
+ step1:
+ scatter_method: blub
+ in: []
+ out: [out]
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test8.cwl b/v1.0/salad/schema_salad/tests/test_schema/test8.cwl
new file mode 100644
index 000000000..128cb4a75
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test8.cwl
@@ -0,0 +1,10 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps:
+ step1:
+ scatterMethod: abc
+ in: []
+ out: [out]
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/test_schema/test9.cwl b/v1.0/salad/schema_salad/tests/test_schema/test9.cwl
new file mode 100644
index 000000000..2d7ff4cd9
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/test_schema/test9.cwl
@@ -0,0 +1,10 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps:
+ step1:
+ scatterMethod: 12
+ in: []
+ out: [out]
\ No newline at end of file
diff --git a/v1.0/salad/schema_salad/tests/util.py b/v1.0/salad/schema_salad/tests/util.py
new file mode 100644
index 000000000..0f71eced9
--- /dev/null
+++ b/v1.0/salad/schema_salad/tests/util.py
@@ -0,0 +1,15 @@
+from __future__ import absolute_import
+from pkg_resources import Requirement, resource_filename, ResolutionError # type: ignore
+from typing import Optional, Text
+import os
+
+def get_data(filename): # type: (Text) -> Optional[Text]
+ filepath = None
+ try:
+ filepath = resource_filename(
+ Requirement.parse("schema-salad"), filename)
+ except ResolutionError:
+ pass
+ if not filepath or not os.path.isfile(filepath):
+ filepath = os.path.join(os.path.dirname(__file__), os.pardir, filename)
+ return filepath
diff --git a/v1.0/salad/schema_salad/flatten.py b/v1.0/salad/schema_salad/utils.py
similarity index 56%
rename from v1.0/salad/schema_salad/flatten.py
rename to v1.0/salad/schema_salad/utils.py
index c9d9b4636..78a0c90ab 100644
--- a/v1.0/salad/schema_salad/flatten.py
+++ b/v1.0/salad/schema_salad/utils.py
@@ -1,7 +1,24 @@
-import sys
-from typing import Any, Tuple
+from __future__ import absolute_import
+
+from typing import Any, Dict, List
+
+
+def add_dictlist(di, key, val): # type: (Dict, Any, Any) -> None
+ if key not in di:
+ di[key] = []
+ di[key].append(val)
+
+
+def aslist(l): # type: (Any) -> List
+ """Convenience function to wrap single items and lists, and return lists unchanged."""
+
+ if isinstance(l, list):
+ return l
+ else:
+ return [l]
# http://rightfootin.blogspot.com/2006/09/more-on-python-flatten.html
+
def flatten(l, ltypes=(list, tuple)):
# type: (Any, Any) -> Any
if l is None:
diff --git a/v1.0/salad/schema_salad/validate.py b/v1.0/salad/schema_salad/validate.py
index 90b02e97f..2fab41596 100644
--- a/v1.0/salad/schema_salad/validate.py
+++ b/v1.0/salad/schema_salad/validate.py
@@ -1,29 +1,45 @@
+from __future__ import absolute_import
import pprint
import avro.schema
+from avro.schema import Schema
import sys
-import urlparse
-from typing import Any
+import re
+import logging
+
+import six
+from six.moves import urllib
+from six.moves import range
+
+from typing import Any, List, Set, Union, Text
+from .sourceline import SourceLine, lineno_re, bullets, indent
+
+_logger = logging.getLogger("salad")
class ValidationException(Exception):
pass
-def validate(expected_schema, datum, identifiers=set(), strict=False, foreign_properties=set()):
- # type: (avro.schema.Schema, Any, Set[unicode], bool, Set[unicode]) -> bool
- try:
- return validate_ex(expected_schema, datum, identifiers, strict=strict, foreign_properties=foreign_properties)
- except ValidationException:
- return False
+
+class ClassValidationException(ValidationException):
+ pass
+
+
+def validate(expected_schema, # type: Schema
+ datum, # type: Any
+ identifiers=[], # type: List[Text]
+ strict=False, # type: bool
+ foreign_properties=set() # type: Set[Text]
+ ):
+ # type: (...) -> bool
+ return validate_ex(
+ expected_schema, datum, identifiers, strict=strict,
+ foreign_properties=foreign_properties, raise_ex=False)
+
INT_MIN_VALUE = -(1 << 31)
INT_MAX_VALUE = (1 << 31) - 1
LONG_MIN_VALUE = -(1 << 63)
LONG_MAX_VALUE = (1 << 63) - 1
-def indent(v, nolead=False): # type: (str, bool) -> str
- if nolead:
- return v.splitlines()[0] + "\n".join([" " + l for l in v.splitlines()[1:]])
- else:
- return "\n".join([" " + l for l in v.splitlines()])
def friendly(v): # type: (Any) -> Any
if isinstance(v, avro.schema.NamedSchema):
@@ -37,11 +53,6 @@ def friendly(v): # type: (Any) -> Any
else:
return v
-def multi(v, q=""): # type: (str, str) -> str
- if '\n' in v:
- return "%s%s%s\n" % (q, v, q)
- else:
- return "%s%s%s" % (q, v, q)
def vpformat(datum): # type: (Any) -> str
a = pprint.pformat(datum)
@@ -49,13 +60,21 @@ def vpformat(datum): # type: (Any) -> str
a = a[0:160] + "[...]"
return a
-def validate_ex(expected_schema, datum, identifiers=None, strict=False,
- foreign_properties=None):
- # type: (avro.schema.Schema, Any, Set[unicode], bool, Set[unicode]) -> bool
+
+def validate_ex(expected_schema, # type: Schema
+ datum, # type: Any
+ identifiers=None, # type: List[Text]
+ strict=False, # type: bool
+ foreign_properties=None, # type: Set[Text]
+ raise_ex=True, # type: bool
+ strict_foreign_properties=False, # type: bool
+ logger=_logger # type: logging.Logger
+ ):
+ # type: (...) -> bool
"""Determine if a python datum is an instance of a schema."""
if not identifiers:
- identifiers = set()
+ identifiers = []
if not foreign_properties:
foreign_properties = set()
@@ -66,93 +85,188 @@ def validate_ex(expected_schema, datum, identifiers=None, strict=False,
if datum is None:
return True
else:
- raise ValidationException("the value `%s` is not null" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(u"the value is not null")
+ else:
+ return False
elif schema_type == 'boolean':
if isinstance(datum, bool):
return True
else:
- raise ValidationException("the value `%s` is not boolean" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(u"the value is not boolean")
+ else:
+ return False
elif schema_type == 'string':
- if isinstance(datum, basestring):
+ if isinstance(datum, six.string_types):
return True
elif isinstance(datum, bytes):
- datum = datum.decode("utf-8")
+ datum = datum.decode(u"utf-8")
return True
else:
- raise ValidationException("the value `%s` is not string" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(u"the value is not string")
+ else:
+ return False
elif schema_type == 'bytes':
if isinstance(datum, str):
return True
else:
- raise ValidationException("the value `%s` is not bytes" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(
+ u"the value `%s` is not bytes" % vpformat(datum))
+ else:
+ return False
elif schema_type == 'int':
- if ((isinstance(datum, int) or isinstance(datum, long))
- and INT_MIN_VALUE <= datum <= INT_MAX_VALUE):
+ if (isinstance(datum, six.integer_types)
+ and INT_MIN_VALUE <= datum <= INT_MAX_VALUE):
return True
else:
- raise ValidationException("`%s` is not int" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(u"`%s` is not int" % vpformat(datum))
+ else:
+ return False
elif schema_type == 'long':
- if ((isinstance(datum, int) or isinstance(datum, long))
- and LONG_MIN_VALUE <= datum <= LONG_MAX_VALUE):
+ if ((isinstance(datum, six.integer_types))
+ and LONG_MIN_VALUE <= datum <= LONG_MAX_VALUE):
return True
else:
- raise ValidationException("the value `%s` is not long" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(
+ u"the value `%s` is not long" % vpformat(datum))
+ else:
+ return False
elif schema_type in ['float', 'double']:
- if (isinstance(datum, int) or isinstance(datum, long)
- or isinstance(datum, float)):
- return True
- else:
- raise ValidationException("the value `%s` is not float or double" % vpformat(datum))
- elif isinstance(expected_schema, avro.schema.FixedSchema):
- if isinstance(datum, str) and len(datum) == expected_schema.size:
+ if (isinstance(datum, six.integer_types)
+ or isinstance(datum, float)):
return True
else:
- raise ValidationException("the value `%s` is not fixed" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(
+ u"the value `%s` is not float or double" % vpformat(datum))
+ else:
+ return False
elif isinstance(expected_schema, avro.schema.EnumSchema):
if expected_schema.name == "Any":
if datum is not None:
return True
else:
- raise ValidationException("Any type must be non-null")
+ if raise_ex:
+ raise ValidationException(u"'Any' type must be non-null")
+ else:
+ return False
+ if not isinstance(datum, six.string_types):
+ if raise_ex:
+ raise ValidationException(
+ u"value is a %s but expected a string" % (type(datum).__name__))
+ else:
+ return False
if datum in expected_schema.symbols:
return True
else:
- raise ValidationException("the value `%s`\n is not a valid symbol in enum %s, expected one of %s" % (vpformat(datum), expected_schema.name, "'" + "', '".join(expected_schema.symbols) + "'"))
+ if raise_ex:
+ raise ValidationException(u"the value %s is not a valid %s, expected %s%s" % (vpformat(datum), expected_schema.name,
+ "one of " if len(
+ expected_schema.symbols) > 1 else "",
+ "'" + "', '".join(expected_schema.symbols) + "'"))
+ else:
+ return False
elif isinstance(expected_schema, avro.schema.ArraySchema):
if isinstance(datum, list):
for i, d in enumerate(datum):
try:
- validate_ex(expected_schema.items, d, identifiers, strict=strict, foreign_properties=foreign_properties)
+ sl = SourceLine(datum, i, ValidationException)
+ if not validate_ex(expected_schema.items, d, identifiers,
+ strict=strict,
+ foreign_properties=foreign_properties,
+ raise_ex=raise_ex,
+ strict_foreign_properties=strict_foreign_properties,
+ logger=logger):
+ return False
except ValidationException as v:
- raise ValidationException("At position %i\n%s" % (i, indent(str(v))))
- return True
- else:
- raise ValidationException("the value `%s` is not a list, expected list of %s" % (vpformat(datum), friendly(expected_schema.items)))
- elif isinstance(expected_schema, avro.schema.MapSchema):
- if (isinstance(datum, dict) and
- False not in [isinstance(k, basestring) for k in datum.keys()] and
- False not in [validate(expected_schema.values, v, strict=strict) for v in datum.values()]):
+ if raise_ex:
+ raise sl.makeError(
+ six.text_type("item is invalid because\n%s" % (indent(str(v)))))
+ else:
+ return False
return True
else:
- raise ValidationException("`%s` is not a valid map value, expected\n %s" % (vpformat(datum), vpformat(expected_schema.values)))
+ if raise_ex:
+ raise ValidationException(u"the value %s is not a list, expected list of %s" % (
+ vpformat(datum), friendly(expected_schema.items)))
+ else:
+ return False
elif isinstance(expected_schema, avro.schema.UnionSchema):
- if True in [validate(s, datum, identifiers, strict=strict) for s in expected_schema.schemas]:
- return True
+ for s in expected_schema.schemas:
+ if validate_ex(s, datum, identifiers, strict=strict, raise_ex=False,
+ strict_foreign_properties=strict_foreign_properties,
+ logger=logger):
+ return True
+
+ if not raise_ex:
+ return False
+
+ errors = [] # type: List[Text]
+ checked = []
+ for s in expected_schema.schemas:
+ if isinstance(datum, list) and not isinstance(s, avro.schema.ArraySchema):
+ continue
+ elif isinstance(datum, dict) and not isinstance(s, avro.schema.RecordSchema):
+ continue
+ elif isinstance(datum, (bool, six.integer_types, float, six.string_types)) and isinstance(s, (avro.schema.ArraySchema, avro.schema.RecordSchema)):
+ continue
+ elif datum is not None and s.type == "null":
+ continue
+
+ checked.append(s)
+ try:
+ validate_ex(s, datum, identifiers, strict=strict,
+ foreign_properties=foreign_properties,
+ raise_ex=True,
+ strict_foreign_properties=strict_foreign_properties,
+ logger=logger)
+ except ClassValidationException as e:
+ raise
+ except ValidationException as e:
+ errors.append(six.text_type(e))
+ if bool(errors):
+ raise ValidationException(bullets(["tried %s but\n%s" % (friendly(
+ checked[i]), indent(errors[i])) for i in range(0, len(errors))], "- "))
else:
- errors = []
- for s in expected_schema.schemas:
- try:
- validate_ex(s, datum, identifiers, strict=strict, foreign_properties=foreign_properties)
- except ValidationException as e:
- errors.append(str(e))
- raise ValidationException("the value %s is not a valid type in the union, expected one of:\n%s" % (multi(vpformat(datum), '`'), "\n".join(["- %s, but\n %s" % (friendly(expected_schema.schemas[i]), indent(multi(errors[i]))) for i in range(0, len(expected_schema.schemas))])))
+ raise ValidationException("value is a %s, expected %s" % (
+ type(datum).__name__, friendly(expected_schema)))
elif isinstance(expected_schema, avro.schema.RecordSchema):
if not isinstance(datum, dict):
- raise ValidationException("`%s`\n is not a dict" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(u"is not a dict")
+ else:
+ return False
+
+ classmatch = None
+ for f in expected_schema.fields:
+ if f.name in ("class",):
+ d = datum.get(f.name)
+ if not d:
+ if raise_ex:
+ raise ValidationException(
+ u"Missing '%s' field" % (f.name))
+ else:
+ return False
+ if expected_schema.name != d:
+ if raise_ex:
+ raise ValidationException(
+ u"Expected class '%s' but this is '%s'" % (expected_schema.name, d))
+ else:
+ return False
+ classmatch = d
+ break
errors = []
for f in expected_schema.fields:
+ if f.name in ("class",):
+ continue
+
if f.name in datum:
fieldval = datum[f.name]
else:
@@ -162,28 +276,60 @@ def validate_ex(expected_schema, datum, identifiers=None, strict=False,
fieldval = None
try:
- validate_ex(f.type, fieldval, identifiers, strict=strict, foreign_properties=foreign_properties)
+ sl = SourceLine(datum, f.name, six.text_type)
+ if not validate_ex(f.type, fieldval, identifiers, strict=strict,
+ foreign_properties=foreign_properties,
+ raise_ex=raise_ex,
+ strict_foreign_properties=strict_foreign_properties,
+ logger=logger):
+ return False
except ValidationException as v:
if f.name not in datum:
- errors.append("missing required field `%s`" % f.name)
+ errors.append(u"missing required field `%s`" % f.name)
else:
- errors.append("could not validate field `%s` because\n%s" % (f.name, multi(indent(str(v)))))
- if strict:
- for d in datum:
- found = False
- for f in expected_schema.fields:
- if d == f.name:
- found = True
- if not found:
- if d not in identifiers and d not in foreign_properties and d[0] not in ("@", "$"):
- split = urlparse.urlsplit(d)
- if split.scheme:
- errors.append("could not validate extension field `%s` because it is not recognized and strict is True. Did you include a $schemas section?" % (d))
+ errors.append(sl.makeError(u"the `%s` field is not valid because\n%s" % (
+ f.name, indent(str(v)))))
+
+ for d in datum:
+ found = False
+ for f in expected_schema.fields:
+ if d == f.name:
+ found = True
+ if not found:
+ sl = SourceLine(datum, d, six.text_type)
+ if d not in identifiers and d not in foreign_properties and d[0] not in ("@", "$"):
+ if (d not in identifiers and strict) and (
+ d not in foreign_properties and strict_foreign_properties) and not raise_ex:
+ return False
+ split = urllib.parse.urlsplit(d)
+ if split.scheme:
+ err = sl.makeError(u"unrecognized extension field `%s`%s."
+ " Did you include "
+ "a $schemas section?" % (
+ d, " and strict_foreign_properties is True" if strict_foreign_properties else ""))
+ if strict_foreign_properties:
+ errors.append(err)
else:
- errors.append("could not validate field `%s` because it is not recognized and strict is True, valid fields are: %s" % (d, ", ".join(fn.name for fn in expected_schema.fields)))
+ logger.warn(err)
+ else:
+ err = sl.makeError(u"invalid field `%s`, expected one of: %s" % (
+ d, ", ".join("'%s'" % fn.name for fn in expected_schema.fields)))
+ if strict:
+ errors.append(err)
+ else:
+ logger.warn(err)
- if errors:
- raise ValidationException("\n".join(errors))
+ if bool(errors):
+ if raise_ex:
+ if classmatch:
+ raise ClassValidationException(bullets(errors, "* "))
+ else:
+ raise ValidationException(bullets(errors, "* "))
+ else:
+ return False
else:
return True
- raise ValidationException("Unrecognized schema_type %s" % schema_type)
+ if raise_ex:
+ raise ValidationException(u"Unrecognized schema_type %s" % schema_type)
+ else:
+ return False
diff --git a/v1.0/salad/setup.cfg b/v1.0/salad/setup.cfg
index 44c6bf126..903b12df4 100644
--- a/v1.0/salad/setup.cfg
+++ b/v1.0/salad/setup.cfg
@@ -3,3 +3,9 @@ ignore=E124,E128,E129,E201,E202,E225,E226,E231,E265,E271,E302,E303,F401,E402,E50
[bdist_wheel]
universal = 1
+
+[aliases]
+test=pytest
+
+[tool:pytest]
+addopts=--pyarg schema_salad
diff --git a/v1.0/salad/setup.py b/v1.0/salad/setup.py
index cb796f62f..8dffa5e41 100755
--- a/v1.0/salad/setup.py
+++ b/v1.0/salad/setup.py
@@ -2,10 +2,8 @@
import os
import sys
-import shutil
import setuptools.command.egg_info as egg_info_cmd
-
from setuptools import setup, find_packages
SETUP_DIR = os.path.dirname(__file__)
@@ -17,6 +15,9 @@
except ImportError:
tagger = egg_info_cmd.egg_info
+needs_pytest = {'pytest', 'test', 'ptr'}.intersection(sys.argv)
+pytest_runner = ['pytest-runner'] if needs_pytest else []
+
if os.path.exists("requirements.txt"):
requirements = [
r for r in open("requirements.txt").read().split("\n") if ";" not in r]
@@ -25,23 +26,24 @@
requirements = []
install_requires = [
- 'requests',
- 'ruamel.yaml == 0.11.11',
- 'rdflib >= 4.1.0',
- 'rdflib-jsonld >= 0.3.0',
- 'mistune',
- 'typing']
-
-install_requires.append("avro") # TODO: remove me once cwltool is
-# available in Debian Stable, Ubuntu 12.04 LTS
+ 'setuptools',
+ 'requests >= 1.0',
+ 'ruamel.yaml >= 0.12.4, < 0.15',
+ 'rdflib >= 4.2.2, < 4.3.0',
+ 'rdflib-jsonld >= 0.3.0, < 0.5.0',
+ 'mistune >= 0.7.3, < 0.8',
+ 'typing >= 3.5.3',
+ 'CacheControl >= 0.11.7, < 0.12',
+ 'lockfile >= 0.9',
+ 'six >= 1.8.0']
-# extras_require={ # TODO: uncomment me, same conditions as above
-# ':python_version<"3"': ['avro'],
-# ':python_version>="3"': ['avro-python3']}
-extras_require = {} # TODO: to be removed when the above is added
+extras_require={
+ ':python_version<"3"': ['avro == 1.8.1'],
+ ':python_version>="3"': ['future', 'avro-cwl == 1.8.3'] # fork of avro for working with python3
+}
setup(name='schema-salad',
- version='1.14',
+ version='2.6',
description='Schema Annotations for Linked Avro Data (SALAD)',
long_description=open(README).read(),
author='Common workflow language working group',
@@ -49,14 +51,16 @@
url="https://github.com/common-workflow-language/common-workflow-language",
download_url="https://github.com/common-workflow-language/common-workflow-language",
license='Apache 2.0',
- packages=["schema_salad"],
+ setup_requires=[] + pytest_runner,
+ packages=["schema_salad", "schema_salad.tests"],
package_data={'schema_salad': ['metaschema/*']},
+ include_package_data=True,
install_requires=install_requires,
extras_require=extras_require,
test_suite='tests',
- tests_require=[],
+ tests_require=['pytest'],
entry_points={
- 'console_scripts': ["schema-salad-tool=schema_salad.main:main"]
+ 'console_scripts': ["schema-salad-tool=schema_salad.main:main", "schema-salad-doc=schema_salad.makedoc:main"]
},
zip_safe=True,
cmdclass={'egg_info': tagger},
@@ -67,8 +71,9 @@
"Operating System :: MacOS :: MacOS X",
"Development Status :: 4 - Beta",
"Programming Language :: Python :: 2.7",
- #"Programming Language :: Python :: 3.3", # TODO: uncomment these
- #"Programming Language :: Python :: 3.4", # lines
- #"Programming Language :: Python :: 3.5"
+ "Programming Language :: Python :: 3.3",
+ "Programming Language :: Python :: 3.4",
+ "Programming Language :: Python :: 3.5",
+ "Programming Language :: Python :: 3.6"
]
)
diff --git a/v1.0/salad/tests/test_examples.py b/v1.0/salad/tests/test_examples.py
deleted file mode 100644
index 1c1a51b79..000000000
--- a/v1.0/salad/tests/test_examples.py
+++ /dev/null
@@ -1,355 +0,0 @@
-import unittest
-import schema_salad.ref_resolver
-import schema_salad.main
-import schema_salad.schema
-from schema_salad.jsonld_context import makerdf
-import rdflib
-import ruamel.yaml as yaml
-import json
-import os
-
-try:
- from ruamel.yaml import CSafeLoader as SafeLoader
-except ImportError:
- from ruamel.yaml import SafeLoader
-
-
-class TestSchemas(unittest.TestCase):
- def test_schemas(self):
- l = schema_salad.ref_resolver.Loader({})
-
- ra, _ = l.resolve_all({
- u"$schemas": [u"tests/EDAM.owl"],
- u"$namespaces": {u"edam": u"http://edamontology.org/"},
- u"edam:has_format": u"edam:format_1915"
- }, "")
-
- self.assertEqual({
- u"$schemas": [u"tests/EDAM.owl"],
- u"$namespaces": {u"edam": u"http://edamontology.org/"},
- u'http://edamontology.org/has_format': u'http://edamontology.org/format_1915'
- }, ra)
-
- # def test_domain(self):
- # l = schema_salad.ref_resolver.Loader({})
-
- # l.idx["http://example.com/stuff"] = {
- # "$schemas": ["tests/EDAM.owl"],
- # "$namespaces": {"edam": "http://edamontology.org/"},
- # }
-
- # ra, _ = l.resolve_all({
- # "$import": "http://example.com/stuff",
- # "edam:has_format": "edam:format_1915"
- # }, "")
-
- # self.assertEquals(ra, {
- # "$schemas": ["tests/EDAM.owl"],
- # "$namespaces": {"edam": "http://edamontology.org/"},
- # 'http://edamontology.org/has_format': 'http://edamontology.org/format_1915'
- # })
-
- def test_self_validate(self):
- self.assertEqual(0, schema_salad.main.main(argsl=["schema_salad/metaschema/metaschema.yml"]))
- self.assertEqual(0, schema_salad.main.main(argsl=["schema_salad/metaschema/metaschema.yml",
- "schema_salad/metaschema/metaschema.yml"]))
-
- def test_avro_regression(self):
- self.assertEqual(0, schema_salad.main.main(argsl=["tests/Process.yml"]))
-
- def test_jsonld_ctx(self):
- ldr, _, _, _ = schema_salad.schema.load_schema({
- "$base": "Y",
- "name": "X",
- "$namespaces": {
- "foo": "http://example.com/foo#"
- },
- "$graph": [{
- "name": "ExampleType",
- "type": "enum",
- "symbols": ["asym", "bsym"]}]
- })
-
- ra, _ = ldr.resolve_all({"foo:bar": "asym"}, "X")
-
- self.assertEqual(ra, {
- 'http://example.com/foo#bar': 'asym'
- })
-
- maxDiff = None
-
- def test_idmap(self):
- ldr = schema_salad.ref_resolver.Loader({})
- ldr.add_context({
- "inputs": {
- "@id": "http://example.com/inputs",
- "mapSubject": "id",
- "mapPredicate": "a"
- },
- "outputs": {
- "@type": "@id",
- "identity": True,
- },
- "id": "@id"})
-
- ra, _ = ldr.resolve_all({
- "id": "stuff",
- "inputs": {
- "zip": 1,
- "zing": 2
- },
- "outputs": ["out"],
- "other": {
- 'n': 9
- }
- }, "http://example2.com/")
-
- self.assertEqual("http://example2.com/#stuff", ra["id"])
- for item in ra["inputs"]:
- if item["a"] == 2:
- self.assertEquals('http://example2.com/#stuff/zing', item["id"])
- else:
- self.assertEquals('http://example2.com/#stuff/zip', item["id"])
- self.assertEquals(['http://example2.com/#stuff/out'], ra['outputs'])
- self.assertEquals({'n': 9}, ra['other'])
-
- def test_scoped_ref(self):
- ldr = schema_salad.ref_resolver.Loader({})
- ldr.add_context({
- "scatter": {
- "@type": "@id",
- "refScope": 0,
- },
- "source": {
- "@type": "@id",
- "refScope": 2,
- },
- "in": {
- "mapSubject": "id",
- "mapPredicate": "source"
- },
- "out": {
- "@type": "@id",
- "identity": True
- },
- "inputs": {
- "mapSubject": "id",
- "mapPredicate": "type"
- },
- "outputs": {
- "mapSubject": "id",
- },
- "steps": {
- "mapSubject": "id"
- },
- "id": "@id"})
-
- ra, _ = ldr.resolve_all({
- "inputs": {
- "inp": "string",
- "inp2": "string"
- },
- "outputs": {
- "out": {
- "type": "string",
- "source": "step2/out"
- }
- },
- "steps": {
- "step1": {
- "in": {
- "inp": "inp",
- "inp2": "#inp2",
- "inp3": ["inp", "inp2"]
- },
- "out": ["out"],
- "scatter": "inp"
- },
- "step2": {
- "in": {
- "inp": "step1/out"
- },
- "scatter": "inp",
- "out": ["out"]
- }
- }
- }, "http://example2.com/")
-
- self.assertEquals(
- {'inputs': [{
- 'id': 'http://example2.com/#inp',
- 'type': 'string'
- }, {
- 'id': 'http://example2.com/#inp2',
- 'type': 'string'
- }],
- 'outputs': [{
- 'id': 'http://example2.com/#out',
- 'type': 'string',
- 'source': 'http://example2.com/#step2/out'
- }],
- 'steps': [{
- 'id': 'http://example2.com/#step1',
- 'scatter': 'http://example2.com/#step1/inp',
- 'in': [{
- 'id': 'http://example2.com/#step1/inp',
- 'source': 'http://example2.com/#inp'
- }, {
- 'id': 'http://example2.com/#step1/inp2',
- 'source': 'http://example2.com/#inp2'
- }, {
- 'id': 'http://example2.com/#step1/inp3',
- 'source': ['http://example2.com/#inp', 'http://example2.com/#inp2']
- }],
- "out": ["http://example2.com/#step1/out"],
- }, {
- 'id': 'http://example2.com/#step2',
- 'scatter': 'http://example2.com/#step2/inp',
- 'in': [{
- 'id': 'http://example2.com/#step2/inp',
- 'source': 'http://example2.com/#step1/out'
- }],
- "out": ["http://example2.com/#step2/out"],
- }]
- }, ra)
-
-
- def test_examples(self):
- self.maxDiff = None
- for a in ["field_name", "ident_res", "link_res", "vocab_res"]:
- ldr, _, _, _ = schema_salad.schema.load_schema(
- "schema_salad/metaschema/%s_schema.yml" % a)
- with open("schema_salad/metaschema/%s_src.yml" % a) as src_fp:
- src = ldr.resolve_all(
- yaml.load(src_fp, Loader=SafeLoader), "", checklinks=False)[0]
- with open("schema_salad/metaschema/%s_proc.yml" % a) as src_proc:
- proc = yaml.load(src_proc, Loader=SafeLoader)
- self.assertEqual(proc, src)
-
- def test_yaml_float_test(self):
- self.assertEqual(yaml.load("float-test: 2e-10")["float-test"], 2e-10)
-
- def test_typedsl_ref(self):
- ldr = schema_salad.ref_resolver.Loader({})
- ldr.add_context({
- "File": "http://example.com/File",
- "null": "http://example.com/null",
- "array": "http://example.com/array",
- "type": {
- "@type": "@vocab",
- "typeDSL": True
- }
- })
-
- ra, _ = ldr.resolve_all({"type": "File"}, "")
- self.assertEqual({'type': 'File'}, ra)
-
- ra, _ = ldr.resolve_all({"type": "File?"}, "")
- self.assertEqual({'type': ['null', 'File']}, ra)
-
- ra, _ = ldr.resolve_all({"type": "File[]"}, "")
- self.assertEqual({'type': {'items': 'File', 'type': 'array'}}, ra)
-
- ra, _ = ldr.resolve_all({"type": "File[]?"}, "")
- self.assertEqual({'type': ['null', {'items': 'File', 'type': 'array'}]}, ra)
-
- def test_scoped_id(self):
- ldr = schema_salad.ref_resolver.Loader({})
- ctx = {
- "id": "@id",
- "location": {
- "@id": "@id",
- "@type": "@id"
- },
- "bar": "http://example.com/bar",
- "ex": "http://example.com/"
- }
- ldr.add_context(ctx)
-
- ra, _ = ldr.resolve_all({
- "id": "foo",
- "bar": {
- "id": "baz"
- }
- }, "http://example.com")
- self.assertEqual({'id': 'http://example.com/#foo',
- 'bar': {
- 'id': 'http://example.com/#foo/baz'},
- }, ra)
-
- g = makerdf(None, ra, ctx)
- print(g.serialize(format="n3"))
-
- ra, _ = ldr.resolve_all({
- "location": "foo",
- "bar": {
- "location": "baz"
- }
- }, "http://example.com", checklinks=False)
- self.assertEqual({'location': 'http://example.com/foo',
- 'bar': {
- 'location': 'http://example.com/baz'},
- }, ra)
-
- g = makerdf(None, ra, ctx)
- print(g.serialize(format="n3"))
-
- ra, _ = ldr.resolve_all({
- "id": "foo",
- "bar": {
- "location": "baz"
- }
- }, "http://example.com", checklinks=False)
- self.assertEqual({'id': 'http://example.com/#foo',
- 'bar': {
- 'location': 'http://example.com/baz'},
- }, ra)
-
- g = makerdf(None, ra, ctx)
- print(g.serialize(format="n3"))
-
- ra, _ = ldr.resolve_all({
- "location": "foo",
- "bar": {
- "id": "baz"
- }
- }, "http://example.com", checklinks=False)
- self.assertEqual({'location': 'http://example.com/foo',
- 'bar': {
- 'id': 'http://example.com/#baz'},
- }, ra)
-
- g = makerdf(None, ra, ctx)
- print(g.serialize(format="n3"))
-
-
- def test_mixin(self):
- ldr = schema_salad.ref_resolver.Loader({})
- ra = ldr.resolve_ref({"$mixin": "mixin.yml", "one": "five"},
- base_url="file://"+os.getcwd()+"/tests/")
- self.assertEqual({'id': 'four', 'one': 'five'}, ra[0])
-
- ldr = schema_salad.ref_resolver.Loader({"id": "@id"})
- base_url="file://"+os.getcwd()+"/tests/"
- ra = ldr.resolve_all([{
- "id": "a",
- "m": {"$mixin": "mixin.yml"}
- }, {
- "id": "b",
- "m": {"$mixin": "mixin.yml"}
- }], base_url=base_url)
- self.assertEqual([{
- 'id': base_url+'#a',
- 'm': {
- 'id': base_url+u'#a/four',
- 'one': 'two'
- },
- }, {
- 'id': base_url+'#b',
- 'm': {
- 'id': base_url+u'#b/four',
- 'one': 'two'}
- }], ra[0])
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/v1.0/salad/tox.ini b/v1.0/salad/tox.ini
index 481fb1eab..2a50b71e6 100644
--- a/v1.0/salad/tox.ini
+++ b/v1.0/salad/tox.ini
@@ -1,44 +1,49 @@
[tox]
-#envlist = py35-lint,py34-lint,py33-lint,py27-lint,py35-unit,py34-unit,py33-unit,py27-unit
-envlist = py27-lint, py27-unit, py34-mypy
-skipsdist = True
-
-[testenv]
-deps = -rrequirements.txt
-
-[testenv:py34-mypy]
-commands = make mypy
-whitelist_externals = make
-deps = mypy-lang>=0.4
+envlist =
+ py{27,33,34,35,36}-lint,
+ py{27,33,34,35,36}-unit,
+ py35-mypy{2,3},
+ py27-pipconflictchecker
-[testenv:py35-lint]
-commands = flake8 schema_salad setup.py
-whitelist_externals = flake8
-deps = flake8
-
-[testenv:py34-lint]
-commands = flake8 schema_salad setup.py
-whitelist_externals = flake8
-deps = flake8
+skipsdist = True
+skip_missing_interpreters = True
-[testenv:py33-lint]
-commands = flake8 schema_salad setup.py
-whitelist_externals = flake8
-deps = flake8
+[tox:travis]
+2.7 = py27
+3.3 = py33
+3.4 = py34
+3.5 = py35
+3.6 = py36
-[testenv:py27-lint]
-commands = flake8 schema_salad setup.py
-whitelist_externals = flake8
-deps = flake8
+[testenv]
+deps =
+ -rrequirements.txt
+ py{27,33,34,35,36}-lint: flake8
-[testenv:py35-unit]
-commands = python setup.py test
+commands =
+ py{27,33,34,35,36}-unit: python setup.py test
+ py{27,33,34,35,36}-lint: flake8 schema_salad setup.py
-[testenv:py34-unit]
-commands = python setup.py test
+whitelist_externals =
+ py{27,33,34,35,36}-lint: flake8
-[testenv:py33-unit]
-commands = python setup.py test
+[testenv:py35-mypy2]
+commands =
+ make mypy2
+whitelist_externals = make
+deps =
+ -rmypy_requirements.txt
+ -rrequirements.txt
-[testenv:py27-unit]
-commands = python setup.py test
+[testenv:py35-mypy3]
+commands =
+ make mypy3
+whitelist_externals = make
+deps =
+ -rmypy_requirements.txt
+ -rrequirements.txt
+
+[testenv:py27-pipconflictchecker]
+commands = pipconflictchecker
+whitelist_externals = pipconflictchecker
+deps = pip-conflict-checker
\ No newline at end of file
diff --git a/v1.0/salad/typeshed/2.7/argparse.pyi b/v1.0/salad/typeshed/2.7/argparse.pyi
deleted file mode 100644
index 13a7b15e5..000000000
--- a/v1.0/salad/typeshed/2.7/argparse.pyi
+++ /dev/null
@@ -1,176 +0,0 @@
-# Stubs for argparse (Python 2)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any, Callable, Dict, List, IO, Iterable, Sequence, Union
-
-SUPPRESS = ... # type: Any
-OPTIONAL = ... # type: Any
-ZERO_OR_MORE = ... # type: Any
-ONE_OR_MORE = ... # type: Any
-PARSER = ... # type: Any
-REMAINDER = ... # type: Any
-
-class _AttributeHolder: ...
-
-class HelpFormatter:
- def __init__(self, prog, indent_increment=..., max_help_position=..., width=...) -> None: ...
- class _Section:
- formatter = ... # type: Any
- parent = ... # type: Any
- heading = ... # type: Any
- items = ... # type: Any
- def __init__(self, formatter, parent, heading=...) -> None: ...
- def format_help(self): ...
- def start_section(self, heading): ...
- def end_section(self): ...
- def add_text(self, text): ...
- def add_usage(self, usage, actions, groups, prefix=...): ...
- def add_argument(self, action): ...
- def add_arguments(self, actions): ...
- def format_help(self): ...
-
-class RawDescriptionHelpFormatter(HelpFormatter): ...
-class RawTextHelpFormatter(RawDescriptionHelpFormatter): ...
-class ArgumentDefaultsHelpFormatter(HelpFormatter): ...
-
-class ArgumentError(Exception):
- argument_name = ... # type: Any
- message = ... # type: Any
- def __init__(self, argument, message) -> None: ...
-
-class ArgumentTypeError(Exception): ...
-
-class Action(_AttributeHolder):
- option_strings = ... # type: Any
- dest = ... # type: Any
- nargs = ... # type: Any
- const = ... # type: Any
- default = ... # type: Any
- type = ... # type: Any
- choices = ... # type: Any
- required = ... # type: Any
- help = ... # type: Any
- metavar = ... # type: Any
- def __init__(self, option_strings: List[str], dest=str, nargs: Union[int, str]=..., const: Any =..., default: Any =..., type: Callable[[str], Any] =..., choices: Iterable[Any] =..., required: bool=..., help: str=..., metavar: str =...) -> None: ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _StoreAction(Action):
- def __init__(self, option_strings, dest, nargs=..., const=..., default=..., type=...,
- choices=..., required=..., help=..., metavar=...): ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _StoreConstAction(Action):
- def __init__(self, option_strings, dest, const, default=..., required=..., help=...,
- metavar=...): ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _StoreTrueAction(_StoreConstAction):
- def __init__(self, option_strings, dest, default=..., required=..., help=...) -> None: ...
-
-class _StoreFalseAction(_StoreConstAction):
- def __init__(self, option_strings, dest, default=..., required=..., help=...) -> None: ...
-
-class _AppendAction(Action):
- def __init__(self, option_strings, dest, nargs=..., const=..., default=..., type=...,
- choices=..., required=..., help=..., metavar=...): ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _AppendConstAction(Action):
- def __init__(self, option_strings, dest, const, default=..., required=..., help=...,
- metavar=...): ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _CountAction(Action):
- def __init__(self, option_strings, dest, default=..., required=..., help=...) -> None: ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _HelpAction(Action):
- def __init__(self, option_strings, dest=..., default=..., help=...) -> None: ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _VersionAction(Action):
- version = ... # type: Any
- def __init__(self, option_strings, version=..., dest=..., default=..., help=...) -> None: ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _SubParsersAction(Action):
- class _ChoicesPseudoAction(Action):
- def __init__(self, name, help) -> None: ...
- def __init__(self, option_strings, prog, parser_class, dest=..., help=..., metavar=...) -> None: ...
- def add_parser(self, name, **kwargs): ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class FileType:
- def __init__(self, mode=..., bufsize=...) -> None: ...
- def __call__(self, string): ...
-
-class Namespace(_AttributeHolder):
- def __init__(self, **kwargs) -> None: ...
- __hash__ = ... # type: Any
- def __eq__(self, other): ...
- def __ne__(self, other): ...
- def __contains__(self, key): ...
- def __getattr__(self, name: str) -> Any: ...
-
-class _ActionsContainer:
- description = ... # type: Any
- argument_default = ... # type: Any
- prefix_chars = ... # type: Any
- conflict_handler = ... # type: Any
- def __init__(self, description, prefix_chars, argument_default, conflict_handler) -> None: ...
- def register(self, registry_name, value, object): ...
- def set_defaults(self, **kwargs): ...
- def get_default(self, dest): ...
- def add_argument(self,
- *args: Union[str, unicode],
- action: Union[str, Action] = ...,
- nargs: str = ...,
- const: Any = ...,
- default: Any = ...,
- type: Any = ...,
- choices: Any = ..., # TODO: Container?
- required: bool = ...,
- help: str = ...,
- metavar: str = ...,
- dest: str = ...,
- version: str = ...
- ) -> None: ...
- def add_argument_group(self, *args, **kwargs): ...
- def add_mutually_exclusive_group(self, **kwargs) -> _MutuallyExclusiveGroup: ...
-
-class _ArgumentGroup(_ActionsContainer):
- title = ... # type: Any
- def __init__(self, container, title=..., description=..., **kwargs) -> None: ...
-
-class _MutuallyExclusiveGroup(_ArgumentGroup):
- required = ... # type: Any
- def __init__(self, container, required=...) -> None: ...
-
-class ArgumentParser(_AttributeHolder, _ActionsContainer):
- prog = ... # type: Any
- usage = ... # type: Any
- epilog = ... # type: Any
- version = ... # type: Any
- formatter_class = ... # type: Any
- fromfile_prefix_chars = ... # type: Any
- add_help = ... # type: Any
- def __init__(self, prog: str=..., usage: str=..., description: str=...,
- epilog: str=..., version: None=...,
- parents: Iterable[ArgumentParser]=...,
- formatter_class: HelpFormatter=..., prefix_chars: str=...,
- fromfile_prefix_chars: str=...,
- argument_default: str=..., conflict_handler: str=...,
- add_help: bool=...) -> None: ...
- def add_subparsers(self, **kwargs): ...
- def parse_args(self, args: Sequence[str] = ..., namespace=...): ...
- def parse_known_args(self, args=..., namespace=...): ...
- def convert_arg_line_to_args(self, arg_line): ...
- def format_usage(self): ...
- def format_help(self): ...
- def format_version(self): ...
- def print_usage(self, file=...): ...
- def print_help(self, file: IO[Any] = None) -> None: ...
- def print_version(self, file=...): ...
- def exit(self, status=..., message=...): ...
- def error(self, message): ...
diff --git a/v1.0/salad/typeshed/2.7/mistune.pyi b/v1.0/salad/typeshed/2.7/mistune.pyi
deleted file mode 100644
index 0b9b0d95d..000000000
--- a/v1.0/salad/typeshed/2.7/mistune.pyi
+++ /dev/null
@@ -1,164 +0,0 @@
-# Stubs for mistune (Python 2)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-
-def escape(text, quote=False, smart_amp=True): ...
-
-class BlockGrammar:
- def_links = ... # type: Any
- def_footnotes = ... # type: Any
- newline = ... # type: Any
- block_code = ... # type: Any
- fences = ... # type: Any
- hrule = ... # type: Any
- heading = ... # type: Any
- lheading = ... # type: Any
- block_quote = ... # type: Any
- list_block = ... # type: Any
- list_item = ... # type: Any
- list_bullet = ... # type: Any
- paragraph = ... # type: Any
- block_html = ... # type: Any
- table = ... # type: Any
- nptable = ... # type: Any
- text = ... # type: Any
-
-class BlockLexer:
- grammar_class = ... # type: Any
- default_rules = ... # type: Any
- list_rules = ... # type: Any
- footnote_rules = ... # type: Any
- tokens = ... # type: Any
- def_links = ... # type: Any
- def_footnotes = ... # type: Any
- rules = ... # type: Any
- def __init__(self, rules=None, **kwargs): ...
- def __call__(self, text, rules=None): ...
- def parse(self, text, rules=None): ...
- def parse_newline(self, m): ...
- def parse_block_code(self, m): ...
- def parse_fences(self, m): ...
- def parse_heading(self, m): ...
- def parse_lheading(self, m): ...
- def parse_hrule(self, m): ...
- def parse_list_block(self, m): ...
- def parse_block_quote(self, m): ...
- def parse_def_links(self, m): ...
- def parse_def_footnotes(self, m): ...
- def parse_table(self, m): ...
- def parse_nptable(self, m): ...
- def parse_block_html(self, m): ...
- def parse_paragraph(self, m): ...
- def parse_text(self, m): ...
-
-class InlineGrammar:
- escape = ... # type: Any
- inline_html = ... # type: Any
- autolink = ... # type: Any
- link = ... # type: Any
- reflink = ... # type: Any
- nolink = ... # type: Any
- url = ... # type: Any
- double_emphasis = ... # type: Any
- emphasis = ... # type: Any
- code = ... # type: Any
- linebreak = ... # type: Any
- strikethrough = ... # type: Any
- footnote = ... # type: Any
- text = ... # type: Any
- def hard_wrap(self): ...
-
-class InlineLexer:
- grammar_class = ... # type: Any
- default_rules = ... # type: Any
- inline_html_rules = ... # type: Any
- renderer = ... # type: Any
- links = ... # type: Any
- footnotes = ... # type: Any
- footnote_index = ... # type: Any
- rules = ... # type: Any
- def __init__(self, renderer, rules=None, **kwargs): ...
- def __call__(self, text, rules=None): ...
- def setup(self, links, footnotes): ...
- line_match = ... # type: Any
- line_started = ... # type: Any
- def output(self, text, rules=None): ...
- def output_escape(self, m): ...
- def output_autolink(self, m): ...
- def output_url(self, m): ...
- def output_inline_html(self, m): ...
- def output_footnote(self, m): ...
- def output_link(self, m): ...
- def output_reflink(self, m): ...
- def output_nolink(self, m): ...
- def output_double_emphasis(self, m): ...
- def output_emphasis(self, m): ...
- def output_code(self, m): ...
- def output_linebreak(self, m): ...
- def output_strikethrough(self, m): ...
- def output_text(self, m): ...
-
-class Renderer:
- options = ... # type: Any
- def __init__(self, **kwargs) -> None: ...
- def placeholder(self): ...
- def block_code(self, code, lang=None): ...
- def block_quote(self, text): ...
- def block_html(self, html): ...
- def header(self, text, level, raw=None): ...
- def hrule(self): ...
- def list(self, body, ordered=True): ...
- def list_item(self, text): ...
- def paragraph(self, text): ...
- def table(self, header, body): ...
- def table_row(self, content): ...
- def table_cell(self, content, **flags): ...
- def double_emphasis(self, text): ...
- def emphasis(self, text): ...
- def codespan(self, text): ...
- def linebreak(self): ...
- def strikethrough(self, text): ...
- def text(self, text): ...
- def autolink(self, link, is_email=False): ...
- def link(self, link, title, text): ...
- def image(self, src, title, text): ...
- def inline_html(self, html): ...
- def newline(self): ...
- def footnote_ref(self, key, index): ...
- def footnote_item(self, key, text): ...
- def footnotes(self, text): ...
-
-class Markdown:
- renderer = ... # type: Any
- inline = ... # type: Any
- block = ... # type: Any
- footnotes = ... # type: Any
- tokens = ... # type: Any
- def __init__(self, renderer=None, inline=None, block=None, **kwargs): ...
- def __call__(self, text): ...
- def render(self, text): ...
- def parse(self, text): ...
- token = ... # type: Any
- def pop(self): ...
- def peek(self): ...
- def output(self, text, rules=None): ...
- def tok(self): ...
- def tok_text(self): ...
- def output_newline(self): ...
- def output_hrule(self): ...
- def output_heading(self): ...
- def output_code(self): ...
- def output_table(self): ...
- def output_block_quote(self): ...
- def output_list(self): ...
- def output_list_item(self): ...
- def output_loose_item(self): ...
- def output_footnote(self): ...
- def output_close_html(self): ...
- def output_open_html(self): ...
- def output_paragraph(self): ...
- def output_text(self): ...
-
-def markdown(text: str, escape: bool=True, **kwargs) -> str: ...
diff --git a/v1.0/salad/typeshed/2.7/pathlib2.pyi b/v1.0/salad/typeshed/2.7/pathlib2.pyi
new file mode 100644
index 000000000..30bbe2cbd
--- /dev/null
+++ b/v1.0/salad/typeshed/2.7/pathlib2.pyi
@@ -0,0 +1,188 @@
+# Stubs for pathlib2 (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Any, AnyStr, Type, TypeVar, Optional, Union
+from collections import Sequence
+
+_P = TypeVar('_P', bound='PurePath')
+
+intern = ... # type: Any
+basestring = ... # type: Any
+supports_symlinks = ... # type: bool
+nt = ... # type: Any
+
+class _Flavour:
+ join = ... # type: Any
+ def __init__(self) -> None: ...
+ def parse_parts(self, parts): ...
+ def join_parsed_parts(self, drv, root, parts, drv2, root2, parts2): ...
+
+class _WindowsFlavour(_Flavour):
+ sep = ... # type: str
+ altsep = ... # type: str
+ has_drv = ... # type: bool
+ pathmod = ... # type: Any
+ is_supported = ... # type: Any
+ drive_letters = ... # type: Any
+ ext_namespace_prefix = ... # type: str
+ reserved_names = ... # type: Any
+ def splitroot(self, part, sep: Any = ...): ...
+ def casefold(self, s): ...
+ def casefold_parts(self, parts): ...
+ def resolve(self, path): ...
+ def is_reserved(self, parts): ...
+ def make_uri(self, path): ...
+ def gethomedir(self, username): ...
+
+class _PosixFlavour(_Flavour):
+ sep = ... # type: str
+ altsep = ... # type: str
+ has_drv = ... # type: bool
+ pathmod = ... # type: Any
+ is_supported = ... # type: Any
+ def splitroot(self, part, sep: Any = ...): ...
+ def casefold(self, s): ...
+ def casefold_parts(self, parts): ...
+ def resolve(self, path): ...
+ def is_reserved(self, parts): ...
+ def make_uri(self, path): ...
+ def gethomedir(self, username): ...
+
+class _Accessor: ...
+
+class _NormalAccessor(_Accessor):
+ stat = ... # type: Any
+ lstat = ... # type: Any
+ open = ... # type: Any
+ listdir = ... # type: Any
+ chmod = ... # type: Any
+ lchmod = ... # type: Any
+ #def lchmod(self, pathobj, mode): ...
+ mkdir = ... # type: Any
+ unlink = ... # type: Any
+ rmdir = ... # type: Any
+ rename = ... # type: Any
+ replace = ... # type: Any
+ symlink = ... # type: Any
+ #def symlink(a, b, target_is_directory): ...
+ #@staticmethod
+ #def symlink(a, b, target_is_directory): ...
+ utime = ... # type: Any
+ def readlink(self, path): ...
+
+class _Selector:
+ child_parts = ... # type: Any
+ successor = ... # type: Any
+ def __init__(self, child_parts) -> None: ...
+ def select_from(self, parent_path): ...
+
+class _TerminatingSelector: ...
+
+class _PreciseSelector(_Selector):
+ name = ... # type: Any
+ def __init__(self, name, child_parts) -> None: ...
+
+class _WildcardSelector(_Selector):
+ pat = ... # type: Any
+ def __init__(self, pat, child_parts) -> None: ...
+
+class _RecursiveWildcardSelector(_Selector):
+ def __init__(self, pat, child_parts) -> None: ...
+
+class _PathParents(Sequence):
+ def __init__(self, path) -> None: ...
+ def __len__(self): ...
+ def __getitem__(self, idx): ...
+
+class PurePath:
+ def __new__(cls, *args): ...
+ def __reduce__(self): ...
+ def as_posix(self): ...
+ def __bytes__(self): ...
+ def as_uri(self) -> str: ...
+ def __eq__(self, other): ...
+ def __ne__(self, other): ...
+ def __hash__(self): ...
+ def __lt__(self, other): ...
+ def __le__(self, other): ...
+ def __gt__(self, other): ...
+ def __ge__(self, other): ...
+ drive = ... # type: Any
+ root = ... # type: Any
+ @property
+ def anchor(self): ...
+ @property
+ def name(self): ...
+ @property
+ def suffix(self): ...
+ @property
+ def suffixes(self): ...
+ @property
+ def stem(self): ...
+ def with_name(self, name): ...
+ def with_suffix(self, suffix): ...
+ def relative_to(self, *other): ...
+ @property
+ def parts(self): ...
+ def joinpath(self, *args): ...
+ def __truediv__(self, key): ...
+ def __rtruediv__(self, key): ...
+ __div__ = ... # type: Any
+ __rdiv__ = ... # type: Any
+ @property
+ def parent(self): ...
+ @property
+ def parents(self): ...
+ def is_absolute(self): ...
+ def is_reserved(self): ...
+ def match(self, path_pattern): ...
+
+class PurePosixPath(PurePath): ...
+class PureWindowsPath(PurePath): ...
+
+class Path(PurePath):
+ def __new__(cls: Type[_P], *args: Union[AnyStr, PurePath],
+ **kwargs: Any) -> _P: ...
+ def __enter__(self): ...
+ def __exit__(self, t, v, tb): ...
+ @classmethod
+ def cwd(cls): ...
+ @classmethod
+ def home(cls): ...
+ def samefile(self, other_path): ...
+ def iterdir(self): ...
+ def glob(self, pattern): ...
+ def rglob(self, pattern): ...
+ def absolute(self): ...
+ def resolve(self): ...
+ def stat(self): ...
+ def owner(self): ...
+ def group(self): ...
+ def open(self, mode: str = ..., buffering: int = ..., encoding: Optional[Any] = ..., errors: Optional[Any] = ..., newline: Optional[Any] = ...): ...
+ def read_bytes(self): ...
+ def read_text(self, encoding: Optional[Any] = ..., errors: Optional[Any] = ...): ...
+ def write_bytes(self, data): ...
+ def write_text(self, data, encoding: Optional[Any] = ..., errors: Optional[Any] = ...): ...
+ def touch(self, mode: int = ..., exist_ok: bool = ...): ...
+ def mkdir(self, mode: int = ..., parents: bool = ..., exist_ok: bool = ...): ...
+ def chmod(self, mode): ...
+ def lchmod(self, mode): ...
+ def unlink(self): ...
+ def rmdir(self): ...
+ def lstat(self): ...
+ def rename(self, target): ...
+ def replace(self, target): ...
+ def symlink_to(self, target, target_is_directory: bool = ...): ...
+ def exists(self): ...
+ def is_dir(self): ...
+ def is_file(self): ...
+ def is_symlink(self): ...
+ def is_block_device(self): ...
+ def is_char_device(self): ...
+ def is_fifo(self): ...
+ def is_socket(self): ...
+ def expanduser(self): ...
+
+class PosixPath(Path, PurePosixPath): ...
+class WindowsPath(Path, PureWindowsPath): ...
diff --git a/v1.0/salad/typeshed/2.7/pprint.pyi b/v1.0/salad/typeshed/2.7/pprint.pyi
deleted file mode 100644
index 1452a69ab..000000000
--- a/v1.0/salad/typeshed/2.7/pprint.pyi
+++ /dev/null
@@ -1,21 +0,0 @@
-# Stubs for pprint (Python 2)
-#
-# NOTE: Based on a dynamically typed automatically generated by stubgen.
-
-from typing import IO, Any
-
-def pprint(object: Any, stream: IO[Any] = ..., indent: int = ..., width: int = ...,
- depth: int = ...) -> None: ...
-def pformat(object: Any, indent: int =..., width: int =..., depth: int =...) -> str: ...
-def saferepr(object): ...
-def isreadable(object): ...
-def isrecursive(object): ...
-
-class PrettyPrinter:
- def __init__(self, indent: int = ..., width: int = ..., depth: int = ...,
- stream: IO[Any] = ...) -> None: ...
- def pprint(self, object: Any) -> str: ...
- def pformat(self, object): ...
- def isrecursive(self, object): ...
- def isreadable(self, object): ...
- def format(self, object, context, maxlevels, level): ...
diff --git a/v1.0/salad/typeshed/2.7/re.pyi b/v1.0/salad/typeshed/2.7/re.pyi
deleted file mode 100644
index 8b88a7822..000000000
--- a/v1.0/salad/typeshed/2.7/re.pyi
+++ /dev/null
@@ -1,99 +0,0 @@
-# Stubs for re
-# Ron Murawski
-# 'bytes' support added by Jukka Lehtosalo
-
-# based on: http://docs.python.org/2.7/library/re.html
-
-from typing import (
- List, Iterator, overload, Callable, Tuple, Sequence, Dict,
- Generic, AnyStr, Match, Pattern, Any
-)
-
-# ----- re variables and constants -----
-DEBUG = 0
-I = 0
-IGNORECASE = 0
-L = 0
-LOCALE = 0
-M = 0
-MULTILINE = 0
-S = 0
-DOTALL = 0
-X = 0
-VERBOSE = 0
-U = 0
-UNICODE = 0
-T = 0
-TEMPLATE = 0
-
-class error(Exception): ...
-
-@overload
-def compile(pattern: AnyStr, flags: int = ...) -> Pattern[AnyStr]: ...
-@overload
-def compile(pattern: Pattern[AnyStr], flags: int = ...) -> Pattern[AnyStr]: ...
-
-@overload
-def search(pattern: AnyStr, string: AnyStr, flags: int = ...) -> Match[AnyStr]: ...
-@overload
-def search(pattern: Pattern[AnyStr], string: AnyStr, flags: int = ...) -> Match[AnyStr]: ...
-
-@overload
-def match(pattern: AnyStr, string: AnyStr, flags: int = ...) -> Match[AnyStr]: ...
-@overload
-def match(pattern: Pattern[AnyStr], string: AnyStr, flags: int = ...) -> Match[AnyStr]: ...
-
-@overload
-def split(pattern: AnyStr, string: AnyStr,
- maxsplit: int = ..., flags: int = ...) -> List[AnyStr]: ...
-@overload
-def split(pattern: Pattern[AnyStr], string: AnyStr,
- maxsplit: int = ..., flags: int = ...) -> List[AnyStr]: ...
-
-@overload
-def findall(pattern: AnyStr, string: AnyStr, flags: int = ...) -> List[Any]: ...
-@overload
-def findall(pattern: Pattern[AnyStr], string: AnyStr, flags: int = ...) -> List[Any]: ...
-
-# Return an iterator yielding match objects over all non-overlapping matches
-# for the RE pattern in string. The string is scanned left-to-right, and
-# matches are returned in the order found. Empty matches are included in the
-# result unless they touch the beginning of another match.
-@overload
-def finditer(pattern: AnyStr, string: AnyStr,
- flags: int = ...) -> Iterator[Match[AnyStr]]: ...
-@overload
-def finditer(pattern: Pattern[AnyStr], string: AnyStr,
- flags: int = ...) -> Iterator[Match[AnyStr]]: ...
-
-@overload
-def sub(pattern: AnyStr, repl: AnyStr, string: AnyStr, count: int = ...,
- flags: int = ...) -> AnyStr: ...
-@overload
-def sub(pattern: AnyStr, repl: Callable[[Match[AnyStr]], AnyStr],
- string: AnyStr, count: int = ..., flags: int = ...) -> AnyStr: ...
-@overload
-def sub(pattern: Pattern[AnyStr], repl: AnyStr, string: AnyStr, count: int = ...,
- flags: int = ...) -> AnyStr: ...
-@overload
-def sub(pattern: Pattern[AnyStr], repl: Callable[[Match[AnyStr]], AnyStr],
- string: AnyStr, count: int = ..., flags: int = ...) -> AnyStr: ...
-
-@overload
-def subn(pattern: AnyStr, repl: AnyStr, string: AnyStr, count: int = ...,
- flags: int = ...) -> Tuple[AnyStr, int]: ...
-@overload
-def subn(pattern: AnyStr, repl: Callable[[Match[AnyStr]], AnyStr],
- string: AnyStr, count: int = ...,
- flags: int = ...) -> Tuple[AnyStr, int]: ...
-@overload
-def subn(pattern: Pattern[AnyStr], repl: AnyStr, string: AnyStr, count: int = ...,
- flags: int = ...) -> Tuple[AnyStr, int]: ...
-@overload
-def subn(pattern: Pattern[AnyStr], repl: Callable[[Match[AnyStr]], AnyStr],
- string: AnyStr, count: int = ...,
- flags: int = ...) -> Tuple[AnyStr, int]: ...
-
-def escape(string: AnyStr) -> AnyStr: ...
-
-def purge() -> None: ...
diff --git a/v1.0/salad/typeshed/2.7/requests/__init__.pyi b/v1.0/salad/typeshed/2.7/requests/__init__.pyi
deleted file mode 100644
index 6ea56efcc..000000000
--- a/v1.0/salad/typeshed/2.7/requests/__init__.pyi
+++ /dev/null
@@ -1,38 +0,0 @@
-# Stubs for requests (based on version 2.6.0, Python 3)
-
-from typing import Any
-from requests import models
-from requests import api
-from requests import sessions
-from requests import status_codes
-from requests import exceptions
-import logging
-
-__title__ = ... # type: Any
-__build__ = ... # type: Any
-__license__ = ... # type: Any
-__copyright__ = ... # type: Any
-
-Request = models.Request
-Response = models.Response
-PreparedRequest = models.PreparedRequest
-request = api.request
-get = api.get
-head = api.head
-post = api.post
-patch = api.patch
-put = api.put
-delete = api.delete
-options = api.options
-session = sessions.session
-Session = sessions.Session
-codes = status_codes.codes
-RequestException = exceptions.RequestException
-Timeout = exceptions.Timeout
-URLRequired = exceptions.URLRequired
-TooManyRedirects = exceptions.TooManyRedirects
-HTTPError = exceptions.HTTPError
-ConnectionError = exceptions.ConnectionError
-
-class NullHandler(logging.Handler):
- def emit(self, record): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/adapters.pyi b/v1.0/salad/typeshed/2.7/requests/adapters.pyi
deleted file mode 100644
index 109dc9a3e..000000000
--- a/v1.0/salad/typeshed/2.7/requests/adapters.pyi
+++ /dev/null
@@ -1,69 +0,0 @@
-# Stubs for requests.adapters (Python 3)
-
-from typing import Any
-from . import models
-from .packages.urllib3 import poolmanager
-from .packages.urllib3 import response
-from .packages.urllib3.util import retry
-from . import compat
-from . import utils
-from . import structures
-from .packages.urllib3 import exceptions as urllib3_exceptions
-from . import cookies
-from . import exceptions
-from . import auth
-
-Response = models.Response
-PoolManager = poolmanager.PoolManager
-proxy_from_url = poolmanager.proxy_from_url
-HTTPResponse = response.HTTPResponse
-Retry = retry.Retry
-DEFAULT_CA_BUNDLE_PATH = utils.DEFAULT_CA_BUNDLE_PATH
-get_encoding_from_headers = utils.get_encoding_from_headers
-prepend_scheme_if_needed = utils.prepend_scheme_if_needed
-get_auth_from_url = utils.get_auth_from_url
-urldefragauth = utils.urldefragauth
-CaseInsensitiveDict = structures.CaseInsensitiveDict
-ConnectTimeoutError = urllib3_exceptions.ConnectTimeoutError
-MaxRetryError = urllib3_exceptions.MaxRetryError
-ProtocolError = urllib3_exceptions.ProtocolError
-ReadTimeoutError = urllib3_exceptions.ReadTimeoutError
-ResponseError = urllib3_exceptions.ResponseError
-extract_cookies_to_jar = cookies.extract_cookies_to_jar
-ConnectionError = exceptions.ConnectionError
-ConnectTimeout = exceptions.ConnectTimeout
-ReadTimeout = exceptions.ReadTimeout
-SSLError = exceptions.SSLError
-ProxyError = exceptions.ProxyError
-RetryError = exceptions.RetryError
-
-DEFAULT_POOLBLOCK = ... # type: Any
-DEFAULT_POOLSIZE = ... # type: Any
-DEFAULT_RETRIES = ... # type: Any
-
-class BaseAdapter:
- def __init__(self) -> None: ...
- # TODO: "request" parameter not actually supported, added to please mypy.
- def send(self, request=...): ...
- def close(self): ...
-
-class HTTPAdapter(BaseAdapter):
- __attrs__ = ... # type: Any
- max_retries = ... # type: Any
- config = ... # type: Any
- proxy_manager = ... # type: Any
- def __init__(self, pool_connections=..., pool_maxsize=..., max_retries=...,
- pool_block=...): ...
- poolmanager = ... # type: Any
- def init_poolmanager(self, connections, maxsize, block=..., **pool_kwargs): ...
- def proxy_manager_for(self, proxy, **proxy_kwargs): ...
- def cert_verify(self, conn, url, verify, cert): ...
- def build_response(self, req, resp): ...
- def get_connection(self, url, proxies=...): ...
- def close(self): ...
- def request_url(self, request, proxies): ...
- def add_headers(self, request, **kwargs): ...
- def proxy_headers(self, proxy): ...
- # TODO: "request" is not actually optional, modified to please mypy.
- def send(self, request=..., stream=..., timeout=..., verify=..., cert=...,
- proxies=...): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/api.pyi b/v1.0/salad/typeshed/2.7/requests/api.pyi
deleted file mode 100644
index 44853f72b..000000000
--- a/v1.0/salad/typeshed/2.7/requests/api.pyi
+++ /dev/null
@@ -1,14 +0,0 @@
-# Stubs for requests.api (Python 3)
-
-from typing import Union
-
-from .models import Response
-
-def request(method: str, url: str, **kwargs) -> Response: ...
-def get(url: Union[str, unicode], **kwargs) -> Response: ...
-def options(url: Union[str, unicode], **kwargs) -> Response: ...
-def head(url: Union[str, unicode], **kwargs) -> Response: ...
-def post(url: Union[str, unicode], data=..., json=..., **kwargs) -> Response: ...
-def put(url: Union[str, unicode], data=..., **kwargs) -> Response: ...
-def patch(url: Union[str, unicode], data=..., **kwargs) -> Response: ...
-def delete(url: Union[str, unicode], **kwargs) -> Response: ...
diff --git a/v1.0/salad/typeshed/2.7/requests/auth.pyi b/v1.0/salad/typeshed/2.7/requests/auth.pyi
deleted file mode 100644
index 8eea2b0e0..000000000
--- a/v1.0/salad/typeshed/2.7/requests/auth.pyi
+++ /dev/null
@@ -1,41 +0,0 @@
-# Stubs for requests.auth (Python 3)
-
-from typing import Any
-from . import compat
-from . import cookies
-from . import utils
-from . import status_codes
-
-extract_cookies_to_jar = cookies.extract_cookies_to_jar
-parse_dict_header = utils.parse_dict_header
-to_native_string = utils.to_native_string
-codes = status_codes.codes
-
-CONTENT_TYPE_FORM_URLENCODED = ... # type: Any
-CONTENT_TYPE_MULTI_PART = ... # type: Any
-
-class AuthBase:
- def __call__(self, r): ...
-
-class HTTPBasicAuth(AuthBase):
- username = ... # type: Any
- password = ... # type: Any
- def __init__(self, username, password) -> None: ...
- def __call__(self, r): ...
-
-class HTTPProxyAuth(HTTPBasicAuth):
- def __call__(self, r): ...
-
-class HTTPDigestAuth(AuthBase):
- username = ... # type: Any
- password = ... # type: Any
- last_nonce = ... # type: Any
- nonce_count = ... # type: Any
- chal = ... # type: Any
- pos = ... # type: Any
- num_401_calls = ... # type: Any
- def __init__(self, username, password) -> None: ...
- def build_digest_header(self, method, url): ...
- def handle_redirect(self, r, **kwargs): ...
- def handle_401(self, r, **kwargs): ...
- def __call__(self, r): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/compat.pyi b/v1.0/salad/typeshed/2.7/requests/compat.pyi
deleted file mode 100644
index 63b92f6fe..000000000
--- a/v1.0/salad/typeshed/2.7/requests/compat.pyi
+++ /dev/null
@@ -1,6 +0,0 @@
-# Stubs for requests.compat (Python 3.4)
-
-from typing import Any
-import collections
-
-OrderedDict = collections.OrderedDict
diff --git a/v1.0/salad/typeshed/2.7/requests/cookies.pyi b/v1.0/salad/typeshed/2.7/requests/cookies.pyi
deleted file mode 100644
index 6f56c82fb..000000000
--- a/v1.0/salad/typeshed/2.7/requests/cookies.pyi
+++ /dev/null
@@ -1,61 +0,0 @@
-# Stubs for requests.cookies (Python 3)
-
-from typing import Any, MutableMapping
-import collections
-from . import compat
-
-class MockRequest:
- type = ... # type: Any
- def __init__(self, request) -> None: ...
- def get_type(self): ...
- def get_host(self): ...
- def get_origin_req_host(self): ...
- def get_full_url(self): ...
- def is_unverifiable(self): ...
- def has_header(self, name): ...
- def get_header(self, name, default=...): ...
- def add_header(self, key, val): ...
- def add_unredirected_header(self, name, value): ...
- def get_new_headers(self): ...
- @property
- def unverifiable(self): ...
- @property
- def origin_req_host(self): ...
- @property
- def host(self): ...
-
-class MockResponse:
- def __init__(self, headers) -> None: ...
- def info(self): ...
- def getheaders(self, name): ...
-
-def extract_cookies_to_jar(jar, request, response): ...
-def get_cookie_header(jar, request): ...
-def remove_cookie_by_name(cookiejar, name, domain=..., path=...): ...
-
-class CookieConflictError(RuntimeError): ...
-
-class RequestsCookieJar(MutableMapping):
- def get(self, name, default=..., domain=..., path=...): ...
- def set(self, name, value, **kwargs): ...
- def iterkeys(self): ...
- def keys(self): ...
- def itervalues(self): ...
- def values(self): ...
- def iteritems(self): ...
- def items(self): ...
- def list_domains(self): ...
- def list_paths(self): ...
- def multiple_domains(self): ...
- def get_dict(self, domain=..., path=...): ...
- def __getitem__(self, name): ...
- def __setitem__(self, name, value): ...
- def __delitem__(self, name): ...
- def set_cookie(self, cookie, *args, **kwargs): ...
- def update(self, other): ...
- def copy(self): ...
-
-def create_cookie(name, value, **kwargs): ...
-def morsel_to_cookie(morsel): ...
-def cookiejar_from_dict(cookie_dict, cookiejar=..., overwrite=...): ...
-def merge_cookies(cookiejar, cookies): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/exceptions.pyi b/v1.0/salad/typeshed/2.7/requests/exceptions.pyi
deleted file mode 100644
index ff0c32883..000000000
--- a/v1.0/salad/typeshed/2.7/requests/exceptions.pyi
+++ /dev/null
@@ -1,26 +0,0 @@
-# Stubs for requests.exceptions (Python 3)
-
-from typing import Any
-from .packages.urllib3.exceptions import HTTPError as BaseHTTPError
-
-class RequestException(IOError):
- response = ... # type: Any
- request = ... # type: Any
- def __init__(self, *args, **kwargs) -> None: ...
-
-class HTTPError(RequestException): ...
-class ConnectionError(RequestException): ...
-class ProxyError(ConnectionError): ...
-class SSLError(ConnectionError): ...
-class Timeout(RequestException): ...
-class ConnectTimeout(ConnectionError, Timeout): ...
-class ReadTimeout(Timeout): ...
-class URLRequired(RequestException): ...
-class TooManyRedirects(RequestException): ...
-class MissingSchema(RequestException, ValueError): ...
-class InvalidSchema(RequestException, ValueError): ...
-class InvalidURL(RequestException, ValueError): ...
-class ChunkedEncodingError(RequestException): ...
-class ContentDecodingError(RequestException, BaseHTTPError): ...
-class StreamConsumedError(RequestException, TypeError): ...
-class RetryError(RequestException): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/hooks.pyi b/v1.0/salad/typeshed/2.7/requests/hooks.pyi
deleted file mode 100644
index 3367d9a48..000000000
--- a/v1.0/salad/typeshed/2.7/requests/hooks.pyi
+++ /dev/null
@@ -1,8 +0,0 @@
-# Stubs for requests.hooks (Python 3)
-
-from typing import Any
-
-HOOKS = ... # type: Any
-
-def default_hooks(): ...
-def dispatch_hook(key, hooks, hook_data, **kwargs): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/models.pyi b/v1.0/salad/typeshed/2.7/requests/models.pyi
deleted file mode 100644
index d400d4a06..000000000
--- a/v1.0/salad/typeshed/2.7/requests/models.pyi
+++ /dev/null
@@ -1,133 +0,0 @@
-# Stubs for requests.models (Python 3)
-
-from typing import Any, List, MutableMapping, Iterator, Dict
-import datetime
-
-from . import hooks
-from . import structures
-from . import auth
-from . import cookies
-from .cookies import RequestsCookieJar
-from .packages.urllib3 import fields
-from .packages.urllib3 import filepost
-from .packages.urllib3 import util
-from .packages.urllib3 import exceptions as urllib3_exceptions
-from . import exceptions
-from . import utils
-from . import compat
-from . import status_codes
-
-default_hooks = hooks.default_hooks
-CaseInsensitiveDict = structures.CaseInsensitiveDict
-HTTPBasicAuth = auth.HTTPBasicAuth
-cookiejar_from_dict = cookies.cookiejar_from_dict
-get_cookie_header = cookies.get_cookie_header
-RequestField = fields.RequestField
-encode_multipart_formdata = filepost.encode_multipart_formdata
-DecodeError = urllib3_exceptions.DecodeError
-ReadTimeoutError = urllib3_exceptions.ReadTimeoutError
-ProtocolError = urllib3_exceptions.ProtocolError
-LocationParseError = urllib3_exceptions.LocationParseError
-HTTPError = exceptions.HTTPError
-MissingSchema = exceptions.MissingSchema
-InvalidURL = exceptions.InvalidURL
-ChunkedEncodingError = exceptions.ChunkedEncodingError
-ContentDecodingError = exceptions.ContentDecodingError
-ConnectionError = exceptions.ConnectionError
-StreamConsumedError = exceptions.StreamConsumedError
-guess_filename = utils.guess_filename
-get_auth_from_url = utils.get_auth_from_url
-requote_uri = utils.requote_uri
-stream_decode_response_unicode = utils.stream_decode_response_unicode
-to_key_val_list = utils.to_key_val_list
-parse_header_links = utils.parse_header_links
-iter_slices = utils.iter_slices
-guess_json_utf = utils.guess_json_utf
-super_len = utils.super_len
-to_native_string = utils.to_native_string
-codes = status_codes.codes
-
-REDIRECT_STATI = ... # type: Any
-DEFAULT_REDIRECT_LIMIT = ... # type: Any
-CONTENT_CHUNK_SIZE = ... # type: Any
-ITER_CHUNK_SIZE = ... # type: Any
-json_dumps = ... # type: Any
-
-class RequestEncodingMixin:
- @property
- def path_url(self): ...
-
-class RequestHooksMixin:
- def register_hook(self, event, hook): ...
- def deregister_hook(self, event, hook): ...
-
-class Request(RequestHooksMixin):
- hooks = ... # type: Any
- method = ... # type: Any
- url = ... # type: Any
- headers = ... # type: Any
- files = ... # type: Any
- data = ... # type: Any
- json = ... # type: Any
- params = ... # type: Any
- auth = ... # type: Any
- cookies = ... # type: Any
- def __init__(self, method=..., url=..., headers=..., files=..., data=..., params=...,
- auth=..., cookies=..., hooks=..., json=...): ...
- def prepare(self): ...
-
-class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
- method = ... # type: Any
- url = ... # type: Any
- headers = ... # type: Any
- body = ... # type: Any
- hooks = ... # type: Any
- def __init__(self) -> None: ...
- def prepare(self, method=..., url=..., headers=..., files=..., data=..., params=...,
- auth=..., cookies=..., hooks=..., json=...): ...
- def copy(self): ...
- def prepare_method(self, method): ...
- def prepare_url(self, url, params): ...
- def prepare_headers(self, headers): ...
- def prepare_body(self, data, files, json=...): ...
- def prepare_content_length(self, body): ...
- def prepare_auth(self, auth, url=...): ...
- def prepare_cookies(self, cookies): ...
- def prepare_hooks(self, hooks): ...
-
-class Response:
- __attrs__ = ... # type: Any
- status_code = ... # type: int
- headers = ... # type: MutableMapping[str, str]
- raw = ... # type: Any
- url = ... # type: str
- encoding = ... # type: str
- history = ... # type: List[Response]
- reason = ... # type: str
- cookies = ... # type: RequestsCookieJar
- elapsed = ... # type: datetime.timedelta
- request = ... # type: PreparedRequest
- def __init__(self) -> None: ...
- def __bool__(self) -> bool: ...
- def __nonzero__(self) -> bool: ...
- def __iter__(self) -> Iterator[str]: ...
- @property
- def ok(self) -> bool: ...
- @property
- def is_redirect(self) -> bool: ...
- @property
- def is_permanent_redirect(self) -> bool: ...
- @property
- def apparent_encoding(self) -> str: ...
- def iter_content(self, chunk_size: int = ...,
- decode_unicode: bool = ...) -> Iterator[Any]: ...
- def iter_lines(self, chunk_size=..., decode_unicode=..., delimiter=...): ...
- @property
- def content(self) -> str: ...
- @property
- def text(self) -> str: ...
- def json(self, **kwargs) -> Any: ...
- @property
- def links(self) -> Dict[Any, Any]: ...
- def raise_for_status(self) -> None: ...
- def close(self) -> None: ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/__init__.pyi b/v1.0/salad/typeshed/2.7/requests/packages/__init__.pyi
deleted file mode 100644
index 2b1bff828..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/__init__.pyi
+++ /dev/null
@@ -1,8 +0,0 @@
-# Stubs for requests.packages (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-class VendorAlias:
- def __init__(self, package_names) -> None: ...
- def find_module(self, fullname, path=...): ...
- def load_module(self, name): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/__init__.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/__init__.pyi
deleted file mode 100644
index 38cf6729c..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/__init__.pyi
+++ /dev/null
@@ -1,12 +0,0 @@
-# Stubs for requests.packages.urllib3 (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-import logging
-
-class NullHandler(logging.Handler):
- def emit(self, record): ...
-
-def add_stderr_logger(level=...): ...
-def disable_warnings(category=...): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/_collections.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/_collections.pyi
deleted file mode 100644
index 58aa94422..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/_collections.pyi
+++ /dev/null
@@ -1,51 +0,0 @@
-# Stubs for requests.packages.urllib3._collections (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from collections import MutableMapping
-
-class RLock:
- def __enter__(self): ...
- def __exit__(self, exc_type, exc_value, traceback): ...
-
-class RecentlyUsedContainer(MutableMapping):
- ContainerCls = ... # type: Any
- dispose_func = ... # type: Any
- lock = ... # type: Any
- def __init__(self, maxsize=..., dispose_func=...) -> None: ...
- def __getitem__(self, key): ...
- def __setitem__(self, key, value): ...
- def __delitem__(self, key): ...
- def __len__(self): ...
- def __iter__(self): ...
- def clear(self): ...
- def keys(self): ...
-
-class HTTPHeaderDict(dict):
- def __init__(self, headers=..., **kwargs) -> None: ...
- def __setitem__(self, key, val): ...
- def __getitem__(self, key): ...
- def __delitem__(self, key): ...
- def __contains__(self, key): ...
- def __eq__(self, other): ...
- def __ne__(self, other): ...
- values = ... # type: Any
- get = ... # type: Any
- update = ... # type: Any
- iterkeys = ... # type: Any
- itervalues = ... # type: Any
- def pop(self, key, default=...): ...
- def discard(self, key): ...
- def add(self, key, val): ...
- def extend(*args, **kwargs): ...
- def getlist(self, key): ...
- getheaders = ... # type: Any
- getallmatchingheaders = ... # type: Any
- iget = ... # type: Any
- def copy(self): ...
- def iteritems(self): ...
- def itermerged(self): ...
- def items(self): ...
- @classmethod
- def from_httplib(cls, message, duplicates=...): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/connection.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/connection.pyi
deleted file mode 100644
index 289fd1836..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/connection.pyi
+++ /dev/null
@@ -1,51 +0,0 @@
-# Stubs for requests.packages.urllib3.connection (Python 3.4)
-
-from typing import Any
-from httplib import HTTPException
-from . import packages
-from . import exceptions
-from . import util
-
-class DummyConnection: ...
-
-ConnectTimeoutError = exceptions.ConnectTimeoutError
-SystemTimeWarning = exceptions.SystemTimeWarning
-SecurityWarning = exceptions.SecurityWarning
-
-port_by_scheme = ... # type: Any
-RECENT_DATE = ... # type: Any
-
-class HTTPConnection(object):
- default_port = ... # type: Any
- default_socket_options = ... # type: Any
- is_verified = ... # type: Any
- source_address = ... # type: Any
- socket_options = ... # type: Any
- def __init__(self, *args, **kw) -> None: ...
- def connect(self): ...
-
-class HTTPSConnection(HTTPConnection):
- default_port = ... # type: Any
- key_file = ... # type: Any
- cert_file = ... # type: Any
- def __init__(self, host, port=..., key_file=..., cert_file=..., strict=..., timeout=..., **kw) -> None: ...
- sock = ... # type: Any
- def connect(self): ...
-
-class VerifiedHTTPSConnection(HTTPSConnection):
- cert_reqs = ... # type: Any
- ca_certs = ... # type: Any
- ssl_version = ... # type: Any
- assert_fingerprint = ... # type: Any
- key_file = ... # type: Any
- cert_file = ... # type: Any
- assert_hostname = ... # type: Any
- def set_cert(self, key_file=..., cert_file=..., cert_reqs=..., ca_certs=..., assert_hostname=..., assert_fingerprint=...): ...
- sock = ... # type: Any
- auto_open = ... # type: Any
- is_verified = ... # type: Any
- def connect(self): ...
-
-UnverifiedHTTPSConnection = ... # type: Any
-
-class ConnectionError(Exception): pass
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/connectionpool.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/connectionpool.pyi
deleted file mode 100644
index 03c3140f2..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/connectionpool.pyi
+++ /dev/null
@@ -1,87 +0,0 @@
-# Stubs for requests.packages.urllib3.connectionpool (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from ssl import SSLError as BaseSSLError
-from . import exceptions
-from .packages import ssl_match_hostname
-from . import packages
-from . import connection
-from . import request
-from . import response
-from .util import connection as _connection
-from .util import retry
-from .util import timeout
-from .util import url
-
-ClosedPoolError = exceptions.ClosedPoolError
-ProtocolError = exceptions.ProtocolError
-EmptyPoolError = exceptions.EmptyPoolError
-HostChangedError = exceptions.HostChangedError
-LocationValueError = exceptions.LocationValueError
-MaxRetryError = exceptions.MaxRetryError
-ProxyError = exceptions.ProxyError
-ReadTimeoutError = exceptions.ReadTimeoutError
-SSLError = exceptions.SSLError
-TimeoutError = exceptions.TimeoutError
-InsecureRequestWarning = exceptions.InsecureRequestWarning
-CertificateError = ssl_match_hostname.CertificateError
-port_by_scheme = connection.port_by_scheme
-DummyConnection = connection.DummyConnection
-HTTPConnection = connection.HTTPConnection
-HTTPSConnection = connection.HTTPSConnection
-VerifiedHTTPSConnection = connection.VerifiedHTTPSConnection
-HTTPException = connection.HTTPException
-ConnectionError = connection.ConnectionError
-RequestMethods = request.RequestMethods
-HTTPResponse = response.HTTPResponse
-is_connection_dropped = _connection.is_connection_dropped
-Retry = retry.Retry
-Timeout = timeout.Timeout
-get_host = url.get_host
-
-xrange = ... # type: Any
-log = ... # type: Any
-
-class ConnectionPool:
- scheme = ... # type: Any
- QueueCls = ... # type: Any
- host = ... # type: Any
- port = ... # type: Any
- def __init__(self, host, port=...) -> None: ...
- def __enter__(self): ...
- def __exit__(self, exc_type, exc_val, exc_tb): ...
- def close(self): ...
-
-class HTTPConnectionPool(ConnectionPool, RequestMethods):
- scheme = ... # type: Any
- ConnectionCls = ... # type: Any
- strict = ... # type: Any
- timeout = ... # type: Any
- retries = ... # type: Any
- pool = ... # type: Any
- block = ... # type: Any
- proxy = ... # type: Any
- proxy_headers = ... # type: Any
- num_connections = ... # type: Any
- num_requests = ... # type: Any
- conn_kw = ... # type: Any
- def __init__(self, host, port=..., strict=..., timeout=..., maxsize=..., block=..., headers=..., retries=..., _proxy=..., _proxy_headers=..., **conn_kw) -> None: ...
- def close(self): ...
- def is_same_host(self, url): ...
- def urlopen(self, method, url, body=..., headers=..., retries=..., redirect=..., assert_same_host=..., timeout=..., pool_timeout=..., release_conn=..., **response_kw): ...
-
-class HTTPSConnectionPool(HTTPConnectionPool):
- scheme = ... # type: Any
- ConnectionCls = ... # type: Any
- key_file = ... # type: Any
- cert_file = ... # type: Any
- cert_reqs = ... # type: Any
- ca_certs = ... # type: Any
- ssl_version = ... # type: Any
- assert_hostname = ... # type: Any
- assert_fingerprint = ... # type: Any
- def __init__(self, host, port=..., strict=..., timeout=..., maxsize=..., block=..., headers=..., retries=..., _proxy=..., _proxy_headers=..., key_file=..., cert_file=..., cert_reqs=..., ca_certs=..., ssl_version=..., assert_hostname=..., assert_fingerprint=..., **conn_kw) -> None: ...
-
-def connection_from_url(url, **kw): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/contrib/__init__.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/contrib/__init__.pyi
deleted file mode 100644
index 17d26bb13..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/contrib/__init__.pyi
+++ /dev/null
@@ -1,4 +0,0 @@
-# Stubs for requests.packages.urllib3.contrib (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/exceptions.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/exceptions.pyi
deleted file mode 100644
index 3e7d0f6c9..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/exceptions.pyi
+++ /dev/null
@@ -1,54 +0,0 @@
-# Stubs for requests.packages.urllib3.exceptions (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-
-class HTTPError(Exception): ...
-class HTTPWarning(Warning): ...
-
-class PoolError(HTTPError):
- pool = ... # type: Any
- def __init__(self, pool, message) -> None: ...
- def __reduce__(self): ...
-
-class RequestError(PoolError):
- url = ... # type: Any
- def __init__(self, pool, url, message) -> None: ...
- def __reduce__(self): ...
-
-class SSLError(HTTPError): ...
-class ProxyError(HTTPError): ...
-class DecodeError(HTTPError): ...
-class ProtocolError(HTTPError): ...
-
-ConnectionError = ... # type: Any
-
-class MaxRetryError(RequestError):
- reason = ... # type: Any
- def __init__(self, pool, url, reason=...) -> None: ...
-
-class HostChangedError(RequestError):
- retries = ... # type: Any
- def __init__(self, pool, url, retries=...) -> None: ...
-
-class TimeoutStateError(HTTPError): ...
-class TimeoutError(HTTPError): ...
-class ReadTimeoutError(TimeoutError, RequestError): ...
-class ConnectTimeoutError(TimeoutError): ...
-class EmptyPoolError(PoolError): ...
-class ClosedPoolError(PoolError): ...
-class LocationValueError(ValueError, HTTPError): ...
-
-class LocationParseError(LocationValueError):
- location = ... # type: Any
- def __init__(self, location) -> None: ...
-
-class ResponseError(HTTPError):
- GENERIC_ERROR = ... # type: Any
- SPECIFIC_ERROR = ... # type: Any
-
-class SecurityWarning(HTTPWarning): ...
-class InsecureRequestWarning(SecurityWarning): ...
-class SystemTimeWarning(SecurityWarning): ...
-class InsecurePlatformWarning(SecurityWarning): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/fields.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/fields.pyi
deleted file mode 100644
index cdc7734e2..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/fields.pyi
+++ /dev/null
@@ -1,16 +0,0 @@
-# Stubs for requests.packages.urllib3.fields (Python 3.4)
-
-from typing import Any
-from . import packages
-
-def guess_content_type(filename, default=...): ...
-def format_header_param(name, value): ...
-
-class RequestField:
- data = ... # type: Any
- headers = ... # type: Any
- def __init__(self, name, data, filename=..., headers=...) -> None: ...
- @classmethod
- def from_tuples(cls, fieldname, value): ...
- def render_headers(self): ...
- def make_multipart(self, content_disposition=..., content_type=..., content_location=...): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/filepost.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/filepost.pyi
deleted file mode 100644
index c6fefa618..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/filepost.pyi
+++ /dev/null
@@ -1,19 +0,0 @@
-# Stubs for requests.packages.urllib3.filepost (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from . import packages
-#from .packages import six
-from . import fields
-
-#six = packages.six
-#b = six.b
-RequestField = fields.RequestField
-
-writer = ... # type: Any
-
-def choose_boundary(): ...
-def iter_field_objects(fields): ...
-def iter_fields(fields): ...
-def encode_multipart_formdata(fields, boundary=...): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/packages/__init__.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/packages/__init__.pyi
deleted file mode 100644
index 231463649..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/packages/__init__.pyi
+++ /dev/null
@@ -1,4 +0,0 @@
-# Stubs for requests.packages.urllib3.packages (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/__init__.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/__init__.pyi
deleted file mode 100644
index 05c03dc08..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/__init__.pyi
+++ /dev/null
@@ -1 +0,0 @@
-class CertificateError(ValueError): pass
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.pyi
deleted file mode 100644
index 5abbc9dd5..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.pyi
+++ /dev/null
@@ -1,7 +0,0 @@
-# Stubs for requests.packages.urllib3.packages.ssl_match_hostname._implementation (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-class CertificateError(ValueError): ...
-
-def match_hostname(cert, hostname): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/poolmanager.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/poolmanager.pyi
deleted file mode 100644
index a65f66497..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/poolmanager.pyi
+++ /dev/null
@@ -1,31 +0,0 @@
-# Stubs for requests.packages.urllib3.poolmanager (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from .request import RequestMethods
-
-class PoolManager(RequestMethods):
- proxy = ... # type: Any
- connection_pool_kw = ... # type: Any
- pools = ... # type: Any
- def __init__(self, num_pools=..., headers=..., **connection_pool_kw) -> None: ...
- def __enter__(self): ...
- def __exit__(self, exc_type, exc_val, exc_tb): ...
- def clear(self): ...
- def connection_from_host(self, host, port=..., scheme=...): ...
- def connection_from_url(self, url): ...
- # TODO: This was the original signature -- copied another one from base class to fix complaint.
- # def urlopen(self, method, url, redirect=True, **kw): ...
- def urlopen(self, method, url, body=..., headers=..., encode_multipart=..., multipart_boundary=..., **kw): ...
-
-class ProxyManager(PoolManager):
- proxy = ... # type: Any
- proxy_headers = ... # type: Any
- def __init__(self, proxy_url, num_pools=..., headers=..., proxy_headers=..., **connection_pool_kw) -> None: ...
- def connection_from_host(self, host, port=..., scheme=...): ...
- # TODO: This was the original signature -- copied another one from base class to fix complaint.
- # def urlopen(self, method, url, redirect=True, **kw): ...
- def urlopen(self, method, url, body=..., headers=..., encode_multipart=..., multipart_boundary=..., **kw): ...
-
-def proxy_from_url(url, **kw): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/request.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/request.pyi
deleted file mode 100644
index 788c759c5..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/request.pyi
+++ /dev/null
@@ -1,13 +0,0 @@
-# Stubs for requests.packages.urllib3.request (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-
-class RequestMethods:
- headers = ... # type: Any
- def __init__(self, headers=...) -> None: ...
- def urlopen(self, method, url, body=..., headers=..., encode_multipart=..., multipart_boundary=..., **kw): ...
- def request(self, method, url, fields=..., headers=..., **urlopen_kw): ...
- def request_encode_url(self, method, url, fields=..., **urlopen_kw): ...
- def request_encode_body(self, method, url, fields=..., headers=..., encode_multipart=..., multipart_boundary=..., **urlopen_kw): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/response.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/response.pyi
deleted file mode 100644
index c84f7e91f..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/response.pyi
+++ /dev/null
@@ -1,58 +0,0 @@
-# Stubs for requests.packages.urllib3.response (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any, IO
-import io
-from . import _collections
-from . import exceptions
-#from .packages import six
-from . import connection
-from .util import response
-
-HTTPHeaderDict = _collections.HTTPHeaderDict
-ProtocolError = exceptions.ProtocolError
-DecodeError = exceptions.DecodeError
-ReadTimeoutError = exceptions.ReadTimeoutError
-binary_type = str # six.binary_type
-PY3 = True # six.PY3
-is_fp_closed = response.is_fp_closed
-
-class DeflateDecoder:
- def __init__(self) -> None: ...
- def __getattr__(self, name): ...
- def decompress(self, data): ...
-
-class GzipDecoder:
- def __init__(self) -> None: ...
- def __getattr__(self, name): ...
- def decompress(self, data): ...
-
-class HTTPResponse(IO[Any]):
- CONTENT_DECODERS = ... # type: Any
- REDIRECT_STATUSES = ... # type: Any
- headers = ... # type: Any
- status = ... # type: Any
- version = ... # type: Any
- reason = ... # type: Any
- strict = ... # type: Any
- decode_content = ... # type: Any
- def __init__(self, body=..., headers=..., status=..., version=..., reason=..., strict=..., preload_content=..., decode_content=..., original_response=..., pool=..., connection=...) -> None: ...
- def get_redirect_location(self): ...
- def release_conn(self): ...
- @property
- def data(self): ...
- def tell(self): ...
- def read(self, amt=..., decode_content=..., cache_content=...): ...
- def stream(self, amt=..., decode_content=...): ...
- @classmethod
- def from_httplib(ResponseCls, r, **response_kw): ...
- def getheaders(self): ...
- def getheader(self, name, default=...): ...
- def close(self): ...
- @property
- def closed(self): ...
- def fileno(self): ...
- def flush(self): ...
- def readable(self): ...
- def readinto(self, b): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/__init__.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/__init__.pyi
deleted file mode 100644
index eca2ea93d..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/__init__.pyi
+++ /dev/null
@@ -1,7 +0,0 @@
-# Stubs for requests.packages.urllib3.util (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from . import connection
-from . import request
-
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/connection.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/connection.pyi
deleted file mode 100644
index cd673098c..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/connection.pyi
+++ /dev/null
@@ -1,11 +0,0 @@
-# Stubs for requests.packages.urllib3.util.connection (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-
-poll = ... # type: Any
-select = ... # type: Any
-
-def is_connection_dropped(conn): ...
-def create_connection(address, timeout=..., source_address=..., socket_options=...): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/request.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/request.pyi
deleted file mode 100644
index 20a6ea277..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/request.pyi
+++ /dev/null
@@ -1,12 +0,0 @@
-# Stubs for requests.packages.urllib3.util.request (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-#from ..packages import six
-
-#b = six.b
-
-ACCEPT_ENCODING = ... # type: Any
-
-def make_headers(keep_alive=..., accept_encoding=..., user_agent=..., basic_auth=..., proxy_basic_auth=..., disable_cache=...): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/response.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/response.pyi
deleted file mode 100644
index 761a00679..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/response.pyi
+++ /dev/null
@@ -1,5 +0,0 @@
-# Stubs for requests.packages.urllib3.util.response (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-def is_fp_closed(obj): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/retry.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/retry.pyi
deleted file mode 100644
index e958d9061..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/retry.pyi
+++ /dev/null
@@ -1,36 +0,0 @@
-# Stubs for requests.packages.urllib3.util.retry (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from .. import exceptions
-from .. import packages
-
-ConnectTimeoutError = exceptions.ConnectTimeoutError
-MaxRetryError = exceptions.MaxRetryError
-ProtocolError = exceptions.ProtocolError
-ReadTimeoutError = exceptions.ReadTimeoutError
-ResponseError = exceptions.ResponseError
-
-log = ... # type: Any
-
-class Retry:
- DEFAULT_METHOD_WHITELIST = ... # type: Any
- BACKOFF_MAX = ... # type: Any
- total = ... # type: Any
- connect = ... # type: Any
- read = ... # type: Any
- redirect = ... # type: Any
- status_forcelist = ... # type: Any
- method_whitelist = ... # type: Any
- backoff_factor = ... # type: Any
- raise_on_redirect = ... # type: Any
- def __init__(self, total=..., connect=..., read=..., redirect=..., method_whitelist=..., status_forcelist=..., backoff_factor=..., raise_on_redirect=..., _observed_errors=...) -> None: ...
- def new(self, **kw): ...
- @classmethod
- def from_int(cls, retries, redirect=..., default=...): ...
- def get_backoff_time(self): ...
- def sleep(self): ...
- def is_forced_retry(self, method, status_code): ...
- def is_exhausted(self): ...
- def increment(self, method=..., url=..., response=..., error=..., _pool=..., _stacktrace=...): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/timeout.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/timeout.pyi
deleted file mode 100644
index 0a7653c5f..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/timeout.pyi
+++ /dev/null
@@ -1,24 +0,0 @@
-# Stubs for requests.packages.urllib3.util.timeout (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from .. import exceptions
-
-TimeoutStateError = exceptions.TimeoutStateError
-
-def current_time(): ...
-
-class Timeout:
- DEFAULT_TIMEOUT = ... # type: Any
- total = ... # type: Any
- def __init__(self, total=..., connect=..., read=...) -> None: ...
- @classmethod
- def from_float(cls, timeout): ...
- def clone(self): ...
- def start_connect(self): ...
- def get_connect_duration(self): ...
- @property
- def connect_timeout(self): ...
- @property
- def read_timeout(self): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/url.pyi b/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/url.pyi
deleted file mode 100644
index 9877b4a17..000000000
--- a/v1.0/salad/typeshed/2.7/requests/packages/urllib3/util/url.pyi
+++ /dev/null
@@ -1,26 +0,0 @@
-# Stubs for requests.packages.urllib3.util.url (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from .. import exceptions
-
-LocationParseError = exceptions.LocationParseError
-
-url_attrs = ... # type: Any
-
-class Url:
- slots = ... # type: Any
- def __new__(cls, scheme=..., auth=..., host=..., port=..., path=..., query=..., fragment=...): ...
- @property
- def hostname(self): ...
- @property
- def request_uri(self): ...
- @property
- def netloc(self): ...
- @property
- def url(self): ...
-
-def split_first(s, delims): ...
-def parse_url(url): ...
-def get_host(url): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/sessions.pyi b/v1.0/salad/typeshed/2.7/requests/sessions.pyi
deleted file mode 100644
index 17912833c..000000000
--- a/v1.0/salad/typeshed/2.7/requests/sessions.pyi
+++ /dev/null
@@ -1,92 +0,0 @@
-# Stubs for requests.sessions (Python 3)
-
-from typing import Any, Union, MutableMapping
-from . import auth
-from . import compat
-from . import cookies
-from . import models
-from .models import Response
-from . import hooks
-from . import utils
-from . import exceptions
-from .packages.urllib3 import _collections
-from . import structures
-from . import adapters
-from . import status_codes
-
-OrderedDict = compat.OrderedDict
-cookiejar_from_dict = cookies.cookiejar_from_dict
-extract_cookies_to_jar = cookies.extract_cookies_to_jar
-RequestsCookieJar = cookies.RequestsCookieJar
-merge_cookies = cookies.merge_cookies
-Request = models.Request
-PreparedRequest = models.PreparedRequest
-DEFAULT_REDIRECT_LIMIT = models.DEFAULT_REDIRECT_LIMIT
-default_hooks = hooks.default_hooks
-dispatch_hook = hooks.dispatch_hook
-to_key_val_list = utils.to_key_val_list
-default_headers = utils.default_headers
-to_native_string = utils.to_native_string
-TooManyRedirects = exceptions.TooManyRedirects
-InvalidSchema = exceptions.InvalidSchema
-ChunkedEncodingError = exceptions.ChunkedEncodingError
-ContentDecodingError = exceptions.ContentDecodingError
-RecentlyUsedContainer = _collections.RecentlyUsedContainer
-CaseInsensitiveDict = structures.CaseInsensitiveDict
-HTTPAdapter = adapters.HTTPAdapter
-requote_uri = utils.requote_uri
-get_environ_proxies = utils.get_environ_proxies
-get_netrc_auth = utils.get_netrc_auth
-should_bypass_proxies = utils.should_bypass_proxies
-get_auth_from_url = utils.get_auth_from_url
-codes = status_codes.codes
-REDIRECT_STATI = models.REDIRECT_STATI
-
-REDIRECT_CACHE_SIZE = ... # type: Any
-
-def merge_setting(request_setting, session_setting, dict_class=...): ...
-def merge_hooks(request_hooks, session_hooks, dict_class=...): ...
-
-class SessionRedirectMixin:
- def resolve_redirects(self, resp, req, stream=..., timeout=..., verify=..., cert=...,
- proxies=...): ...
- def rebuild_auth(self, prepared_request, response): ...
- def rebuild_proxies(self, prepared_request, proxies): ...
-
-class Session(SessionRedirectMixin):
- __attrs__ = ... # type: Any
- headers = ... # type: MutableMapping[str, str]
- auth = ... # type: Any
- proxies = ... # type: Any
- hooks = ... # type: Any
- params = ... # type: Any
- stream = ... # type: Any
- verify = ... # type: Any
- cert = ... # type: Any
- max_redirects = ... # type: Any
- trust_env = ... # type: Any
- cookies = ... # type: Any
- adapters = ... # type: Any
- redirect_cache = ... # type: Any
- def __init__(self) -> None: ...
- def __enter__(self) -> 'Session': ...
- def __exit__(self, *args) -> None: ...
- def prepare_request(self, request): ...
- def request(self, method: str, url: str, params=..., data=..., headers=...,
- cookies=..., files=..., auth=..., timeout=..., allow_redirects=...,
- proxies=..., hooks=..., stream=..., verify=..., cert=...,
- json=...) -> Response: ...
- def get(self, url: str, **kwargs) -> Response: ...
- def options(self, url: str, **kwargs) -> Response: ...
- def head(self, url: str, **kwargs) -> Response: ...
- def post(self, url: str, data=..., json=..., **kwargs) -> Response: ...
- def put(self, url: str, data=..., **kwargs) -> Response: ...
- def patch(self, url: str, data=..., **kwargs) -> Response: ...
- def delete(self, url: str, **kwargs) -> Response: ...
- def send(self, request, **kwargs): ...
- def merge_environment_settings(self, url, proxies, stream, verify, cert): ...
- def get_adapter(self, url): ...
- def close(self) -> None: ...
- def mount(self, prefix, adapter): ...
-
-def session() -> Session: ...
diff --git a/v1.0/salad/typeshed/2.7/requests/status_codes.pyi b/v1.0/salad/typeshed/2.7/requests/status_codes.pyi
deleted file mode 100644
index e3035eb91..000000000
--- a/v1.0/salad/typeshed/2.7/requests/status_codes.pyi
+++ /dev/null
@@ -1,8 +0,0 @@
-# Stubs for requests.status_codes (Python 3)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from .structures import LookupDict
-
-codes = ... # type: Any
diff --git a/v1.0/salad/typeshed/2.7/requests/structures.pyi b/v1.0/salad/typeshed/2.7/requests/structures.pyi
deleted file mode 100644
index 837cf2501..000000000
--- a/v1.0/salad/typeshed/2.7/requests/structures.pyi
+++ /dev/null
@@ -1,21 +0,0 @@
-# Stubs for requests.structures (Python 3)
-
-from typing import Any
-import collections
-
-class CaseInsensitiveDict(collections.MutableMapping):
- def __init__(self, data=..., **kwargs) -> None: ...
- def __setitem__(self, key, value): ...
- def __getitem__(self, key): ...
- def __delitem__(self, key): ...
- def __iter__(self): ...
- def __len__(self): ...
- def lower_items(self): ...
- def __eq__(self, other): ...
- def copy(self): ...
-
-class LookupDict(dict):
- name = ... # type: Any
- def __init__(self, name=...) -> None: ...
- def __getitem__(self, key): ...
- def get(self, key, default=...): ...
diff --git a/v1.0/salad/typeshed/2.7/requests/utils.pyi b/v1.0/salad/typeshed/2.7/requests/utils.pyi
deleted file mode 100644
index 945277afc..000000000
--- a/v1.0/salad/typeshed/2.7/requests/utils.pyi
+++ /dev/null
@@ -1,52 +0,0 @@
-# Stubs for requests.utils (Python 3)
-
-from typing import Any
-from . import compat
-from . import cookies
-from . import structures
-from . import exceptions
-
-OrderedDict = compat.OrderedDict
-RequestsCookieJar = cookies.RequestsCookieJar
-cookiejar_from_dict = cookies.cookiejar_from_dict
-CaseInsensitiveDict = structures.CaseInsensitiveDict
-InvalidURL = exceptions.InvalidURL
-
-NETRC_FILES = ... # type: Any
-DEFAULT_CA_BUNDLE_PATH = ... # type: Any
-
-def dict_to_sequence(d): ...
-def super_len(o): ...
-def get_netrc_auth(url): ...
-def guess_filename(obj): ...
-def from_key_val_list(value): ...
-def to_key_val_list(value): ...
-def parse_list_header(value): ...
-def parse_dict_header(value): ...
-def unquote_header_value(value, is_filename=...): ...
-def dict_from_cookiejar(cj): ...
-def add_dict_to_cookiejar(cj, cookie_dict): ...
-def get_encodings_from_content(content): ...
-def get_encoding_from_headers(headers): ...
-def stream_decode_response_unicode(iterator, r): ...
-def iter_slices(string, slice_length): ...
-def get_unicode_from_response(r): ...
-
-UNRESERVED_SET = ... # type: Any
-
-def unquote_unreserved(uri): ...
-def requote_uri(uri): ...
-def address_in_network(ip, net): ...
-def dotted_netmask(mask): ...
-def is_ipv4_address(string_ip): ...
-def is_valid_cidr(string_network): ...
-def should_bypass_proxies(url): ...
-def get_environ_proxies(url): ...
-def default_user_agent(name=...): ...
-def default_headers(): ...
-def parse_header_links(value): ...
-def guess_json_utf(data): ...
-def prepend_scheme_if_needed(url, new_scheme): ...
-def get_auth_from_url(url): ...
-def to_native_string(string, encoding=...): ...
-def urldefragauth(url): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/__init__.pyi b/v1.0/salad/typeshed/2and3/cachecontrol/__init__.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/__init__.pyi
rename to v1.0/salad/typeshed/2and3/cachecontrol/__init__.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/adapter.pyi b/v1.0/salad/typeshed/2and3/cachecontrol/adapter.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/adapter.pyi
rename to v1.0/salad/typeshed/2and3/cachecontrol/adapter.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/cache.pyi b/v1.0/salad/typeshed/2and3/cachecontrol/cache.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/cache.pyi
rename to v1.0/salad/typeshed/2and3/cachecontrol/cache.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/caches/__init__.pyi b/v1.0/salad/typeshed/2and3/cachecontrol/caches/__init__.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/caches/__init__.pyi
rename to v1.0/salad/typeshed/2and3/cachecontrol/caches/__init__.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/caches/file_cache.pyi b/v1.0/salad/typeshed/2and3/cachecontrol/caches/file_cache.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/caches/file_cache.pyi
rename to v1.0/salad/typeshed/2and3/cachecontrol/caches/file_cache.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/compat.pyi b/v1.0/salad/typeshed/2and3/cachecontrol/compat.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/compat.pyi
rename to v1.0/salad/typeshed/2and3/cachecontrol/compat.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/controller.pyi b/v1.0/salad/typeshed/2and3/cachecontrol/controller.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/controller.pyi
rename to v1.0/salad/typeshed/2and3/cachecontrol/controller.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/filewrapper.pyi b/v1.0/salad/typeshed/2and3/cachecontrol/filewrapper.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/filewrapper.pyi
rename to v1.0/salad/typeshed/2and3/cachecontrol/filewrapper.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/serialize.pyi b/v1.0/salad/typeshed/2and3/cachecontrol/serialize.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/serialize.pyi
rename to v1.0/salad/typeshed/2and3/cachecontrol/serialize.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/wrapper.pyi b/v1.0/salad/typeshed/2and3/cachecontrol/wrapper.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/cachecontrol/wrapper.pyi
rename to v1.0/salad/typeshed/2and3/cachecontrol/wrapper.pyi
diff --git a/v1.0/salad/typeshed/2and3/mistune.pyi b/v1.0/salad/typeshed/2and3/mistune.pyi
new file mode 100644
index 000000000..d267d9216
--- /dev/null
+++ b/v1.0/salad/typeshed/2and3/mistune.pyi
@@ -0,0 +1,271 @@
+__author__ = "Aleksandr Slepchenkov"
+__email__ = "Sl.aleksandr28@gmail.com"
+
+from typing import Any, Optional, Pattern, List, Text, Tuple, Dict, Match, Type, Sequence, Iterable
+
+Tokens = List[Dict[Text, Any]]
+# There are too much levels of optional unions of lists of text in cell and align 385 and 396 lines in mistune
+
+
+def escape(text: Text, quote: bool = ..., smart_amp: bool = ...) -> Text: ...
+
+
+class BlockGrammar:
+ def_links = ... # type: Pattern
+ def_footnotes = ... # type: Pattern
+ newline = ... # type: Pattern
+ block_code = ... # type: Pattern
+ fences = ... # type: Pattern
+ hrule = ... # type: Pattern
+ heading = ... # type: Pattern
+ lheading = ... # type: Pattern
+ block_quote = ... # type: Pattern
+ list_block = ... # type: Pattern
+ list_item = ... # type: Pattern
+ list_bullet = ... # type: Pattern
+ paragraph = ... # type: Pattern
+ block_html = ... # type: Pattern
+ table = ... # type: Pattern
+ nptable = ... # type: Pattern
+ text = ... # type: Pattern
+
+
+class BlockLexer:
+ grammar_class = ... # type: Type[BlockGrammar]
+ default_rules = ... # type: List[Text]
+ list_rules = ... # type: Tuple[Text]
+ footnote_rules = ... # type: Tuple[Text]
+ tokens = ... # type: Tokens
+ def_links = ... # type: Dict[Text, Dict[Text, Text]]
+ def_footnotes = ... # type: Dict[Text, int]
+ rules = ... # type: BlockGrammar
+
+ def __init__(self, rules: Optional[BlockGrammar] = ..., **kwargs) -> None: ...
+
+ def __call__(self, text: Text, rules: Optional[Sequence[Text]] = ...) -> Tokens: ...
+
+ def parse(self, text: Text, rules: Optional[Sequence[Text]] = ...) -> Tokens: ...
+
+ def parse_newline(self, m: Match) -> None: ...
+
+ def parse_block_code(self, m: Match) -> None: ...
+
+ def parse_fences(self, m: Match) -> None: ...
+
+ def parse_heading(self, m: Match) -> None: ...
+
+ def parse_lheading(self, m: Match) -> None: ...
+
+ def parse_hrule(self, m: Match) -> None: ...
+
+ def parse_list_block(self, m: Match) -> None: ...
+
+ def parse_block_quote(self, m: Match) -> None: ...
+
+ def parse_def_links(self, m: Match) -> None: ...
+
+ def parse_def_footnotes(self, m: Match) -> None: ...
+
+ def parse_table(self, m: Match) -> None: ...
+
+ def parse_nptable(self, m: Match) -> None: ...
+
+ def parse_block_html(self, m: Match) -> None: ...
+
+ def parse_paragraph(self, m: Match) -> None: ...
+
+ def parse_text(self, m: Match) -> None: ...
+
+
+class InlineGrammar:
+ escape = ... # type: Pattern
+ inline_html = ... # type: Pattern
+ autolink = ... # type: Pattern
+ link = ... # type: Pattern
+ reflink = ... # type: Pattern
+ nolink = ... # type: Pattern
+ url = ... # type: Pattern
+ double_emphasis = ... # type: Pattern
+ emphasis = ... # type: Pattern
+ code = ... # type: Pattern
+ linebreak = ... # type: Pattern
+ strikethrough = ... # type: Pattern
+ footnote = ... # type: Pattern
+ text = ... # type: Pattern
+
+ def hard_wrap(self) -> None: ...
+
+
+class InlineLexer:
+ grammar_class = ... # type: Type[InlineGrammar]
+ default_rules = ... # type: List[Text]
+ inline_html_rules = ... # type: List[Text]
+ renderer = ... # type: Renderer
+ links = ... # type: Dict[Any, Dict]
+ footnotes = ... # type: Dict[Text, int]
+ footnote_index = ... # type: int
+ _in_link = ... # type: bool
+ _in_footnote = ... # type: bool
+ _parse_inline_html = ... # type: bool
+ rules = ... # type: InlineGrammar
+
+ def __init__(self, renderer: Renderer, rules: Optional[InlineGrammar] = ..., **kwargs) -> None: ...
+
+ def __call__(self, text: Text, rules: Optional[Sequence[Text]] = ...) -> Text: ...
+
+ def setup(self, links: Optional[Dict[Any, Dict]], footnotes: Optional[Dict[Text, int]]) -> None: ...
+
+ line_match = ... # type: Match
+ line_started = ... # type: bool
+
+ def output(self, text: Text, rules: Optional[Sequence[Text]] = ...) -> Text: ...
+
+ def output_escape(self, m: Match) -> Text: ...
+
+ def output_autolink(self, m: Match) -> Text: ...
+
+ def output_url(self, m: Match) -> Text: ...
+
+ def output_inline_html(self, m: Match) -> Text: ...
+
+ def output_footnote(self, m: Match) -> Optional[Text]: ...
+
+ def output_link(self, m: Match) -> Text: ...
+
+ def output_reflink(self, m: Match) -> Optional[Text]: ...
+
+ def output_nolink(self, m: Match) -> Optional[Text]: ...
+
+ def output_double_emphasis(self, m: Match) -> Text: ...
+
+ def output_emphasis(self, m: Match) -> Text: ...
+
+ def output_code(self, m: Match) -> Text: ...
+
+ def output_linebreak(self, m: Match) -> Text: ...
+
+ def output_strikethrough(self, m: Match) -> Text: ...
+
+ def output_text(self, m: Match) -> Text: ...
+
+
+class Renderer:
+ options = ... # type: Dict
+
+ def __init__(self, **kwargs) -> None: ...
+
+ def placeholder(self) -> Text: ...
+
+ def block_code(self, code: Text,
+ lang: Any = ...) -> Text: ... # It seems that lang should be string, however other types are valid as well
+
+ def block_quote(self, text: Text) -> Text: ...
+
+ def block_html(self, html: Text) -> Text: ...
+
+ def header(self, text: Text, level: int, raw: Optional[Text] = ...) -> Text: ...
+
+ def hrule(self) -> Text: ...
+
+ def list(self, body: Any,
+ ordered: bool = ...) -> Text: ... # body - same reason as for lang above, and for other Any in this class
+
+ def list_item(self, text: Any) -> Text: ...
+
+ def paragraph(self, text: Text) -> Text: ...
+
+ def table(self, header: Any, body: Any) -> Text: ...
+
+ def table_row(self, content: Any) -> Text: ...
+
+ def table_cell(self, content: Any, **flags) -> Text: ...
+
+ def double_emphasis(self, text: Any) -> Text: ...
+
+ def emphasis(self, text: Any) -> Text: ...
+
+ def codespan(self, text: Text) -> Text: ...
+
+ def linebreak(self) -> Text: ...
+
+ def strikethrough(self, text: Any) -> Text: ...
+
+ def text(self, text: Any) -> Text: ...
+
+ def escape(self, text: Any) -> Text: ...
+
+ def autolink(self, link: Any, is_email: bool = ...) -> Text: ...
+
+ def link(self, link: Any, title: Any, text: Any) -> Text: ...
+
+ def image(self, src: Any, title: Any, text: Any) -> Text: ...
+
+ def inline_html(self, html: Any) -> Text: ...
+
+ def newline(self) -> Text: ...
+
+ def footnote_ref(self, key: Any, index: int) -> Text: ...
+
+ def footnote_item(self, key: Any, text: Text) -> Text: ...
+
+ def footnotes(self, text: Any) -> Text: ...
+
+
+class Markdown:
+ renderer = ... # type: Renderer
+ inline = ... # type: InlineLexer
+ block = ... # type: BlockLexer
+ footnotes = ... # type: List[Dict[Text, Any]]
+ tokens = ... # type: Tokens
+
+ def __init__(self, renderer: Optional[Renderer] = ..., inline: Optional[InlineLexer] = ...,
+ block: Optional[BlockLexer] = ..., **kwargs) -> None: ...
+
+ def __call__(self, text: Text) -> Text: ...
+
+ def render(self, text: Text) -> Text: ...
+
+ def parse(self, text: Text) -> Text: ...
+
+ token = ... # type: Dict[Text, Any]
+
+ def pop(self) -> Optional[Dict[Text, Any]]: ...
+
+ def peek(self) -> Optional[Dict[Text, Any]]: ...
+
+ def output(self, text: Text, rules: Optional[Sequence[Text]] = ...): ...
+
+ def tok(self) -> Text: ...
+
+ def tok_text(self) -> Text: ...
+
+ def output_newline(self) -> Text: ...
+
+ def output_hrule(self) -> Text: ...
+
+ def output_heading(self) -> Text: ...
+
+ def output_code(self) -> Text: ...
+
+ def output_table(self) -> Text: ...
+
+ def output_block_quote(self) -> Text: ...
+
+ def output_list(self) -> Text: ...
+
+ def output_list_item(self) -> Text: ...
+
+ def output_loose_item(self) -> Text: ...
+
+ def output_footnote(self) -> Text: ...
+
+ def output_close_html(self) -> Text: ...
+
+ def output_open_html(self) -> Text: ...
+
+ def output_paragraph(self) -> Text: ...
+
+ def output_text(self) -> Text: ...
+
+
+def markdown(text: Text, escape: bool = ..., **kwargs) -> Text: ...
\ No newline at end of file
diff --git a/v1.0/salad/typeshed/2.7/rdflib/__init__.pyi b/v1.0/salad/typeshed/2and3/rdflib/__init__.pyi
similarity index 100%
rename from v1.0/salad/typeshed/2.7/rdflib/__init__.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/__init__.pyi
diff --git a/v1.0/salad/typeshed/2.7/rdflib/events.pyi b/v1.0/salad/typeshed/2and3/rdflib/events.pyi
similarity index 100%
rename from v1.0/salad/typeshed/2.7/rdflib/events.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/events.pyi
diff --git a/v1.0/salad/typeshed/2.7/rdflib/exceptions.pyi b/v1.0/salad/typeshed/2and3/rdflib/exceptions.pyi
similarity index 100%
rename from v1.0/salad/typeshed/2.7/rdflib/exceptions.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/exceptions.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/graph.pyi b/v1.0/salad/typeshed/2and3/rdflib/graph.pyi
similarity index 94%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/graph.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/graph.pyi
index 6a36968bd..6972611ea 100644
--- a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/graph.pyi
+++ b/v1.0/salad/typeshed/2and3/rdflib/graph.pyi
@@ -2,8 +2,8 @@
#
# NOTE: This dynamically typed stub was automatically generated by stubgen.
-from typing import Any, AnyStr, Union, IO, Tuple, Iterator
-from StringIO import StringIO as BytesIO
+from typing import Any, AnyStr, Dict, Union, IO, Tuple, Iterator, Text
+from io import StringIO as BytesIO
from rdflib.term import Node, URIRef
from rdflib.store import Store
from rdflib.namespace import NamespaceManager
@@ -26,7 +26,7 @@ class Graph(Node):
def add(self, __tuple_arg_2: Tuple[Node, Node, Node]) -> None: ...
def addN(self, quads): ...
def remove(self, __tuple_arg_2: Tuple[Union[AnyStr, URIRef], AnyStr, Union[AnyStr, URIRef]]) -> None: ...
- def triples(self, __tuple_arg_2: Tuple[Union[AnyStr, URIRef], AnyStr, Union[AnyStr, URIRef]]) -> Iterator[Tuple[AnyStr, AnyStr, AnyStr]]: ...
+ def triples(self, __tuple_arg_2: Tuple[Union[AnyStr, URIRef, None], Union[AnyStr, None], Union[AnyStr, URIRef, None]]) -> Iterator[Tuple[AnyStr, AnyStr, AnyStr]]: ...
def __getitem__(self, item): ...
def __len__(self): ...
def __iter__(self): ...
@@ -71,8 +71,8 @@ class Graph(Node):
def absolutize(self, uri, defrag=1): ...
def serialize(self, destination: Union[str, IO[Any]]=None, format: str='', base: str=None, encoding: str=None, **args) -> Union[bytes, None]: ...
def parse(self, source: str = None, publicID: str = None,
- format: Union[str, unicode] = None,
- location: Union[str, unicode] = None, file: IO[Any] = None,
+ format: Text = None,
+ location: Text = None, file: IO[Any] = None,
data: str = None, **args): ...
def load(self, source, publicID=None, format=''): ...
def query(self, query_object, processor: str = '', result: str = '', initNs: Dict = None, initBindings: Dict = None, use_store_provided: bool = True, **kwargs) -> Result: ...
diff --git a/v1.0/salad/typeshed/2.7/rdflib/namespace.pyi b/v1.0/salad/typeshed/2and3/rdflib/namespace.pyi
similarity index 90%
rename from v1.0/salad/typeshed/2.7/rdflib/namespace.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/namespace.pyi
index 01197c745..2a0a6816d 100644
--- a/v1.0/salad/typeshed/2.7/rdflib/namespace.pyi
+++ b/v1.0/salad/typeshed/2and3/rdflib/namespace.pyi
@@ -2,9 +2,9 @@
#
# NOTE: This dynamically typed stub was automatically generated by stubgen.
-from typing import Any, Tuple, Union
+from typing import Any, Tuple, Union, Text
-class Namespace(unicode):
+class Namespace(Text):
__doc__ = ... # type: Any
def __new__(cls, value): ...
@property
@@ -13,7 +13,7 @@ class Namespace(unicode):
def __getitem__(self, key, default=None): ...
def __getattr__(self, name): ...
-class URIPattern(unicode):
+class URIPattern(Text):
__doc__ = ... # type: Any
def __new__(cls, value): ...
def __mod__(self, *args, **kwargs): ...
@@ -57,4 +57,4 @@ def is_ncname(name): ...
XMLNS = ... # type: Any
-def split_uri(uri: Union[str, unicode]) -> Tuple[str, str]: ...
+def split_uri(uri: Text) -> Tuple[str, str]: ...
diff --git a/v1.0/salad/typeshed/2.7/rdflib/parser.pyi b/v1.0/salad/typeshed/2and3/rdflib/parser.pyi
similarity index 84%
rename from v1.0/salad/typeshed/2.7/rdflib/parser.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/parser.pyi
index 3ff074d8c..fa1ec9a74 100644
--- a/v1.0/salad/typeshed/2.7/rdflib/parser.pyi
+++ b/v1.0/salad/typeshed/2and3/rdflib/parser.pyi
@@ -3,7 +3,7 @@
# NOTE: This dynamically typed stub was automatically generated by stubgen.
from typing import Any
-from StringIO import StringIO as BytesIO
+from io import StringIO as BytesIO
class Parser:
def __init__(self): ...
diff --git a/v1.0/salad/typeshed/2.7/rdflib/plugin.pyi b/v1.0/salad/typeshed/2and3/rdflib/plugin.pyi
similarity index 100%
rename from v1.0/salad/typeshed/2.7/rdflib/plugin.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/plugin.pyi
diff --git a/v1.0/salad/typeshed/2.7/rdflib/plugins/__init__.pyi b/v1.0/salad/typeshed/2and3/rdflib/plugins/__init__.pyi
similarity index 100%
rename from v1.0/salad/typeshed/2.7/rdflib/plugins/__init__.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/plugins/__init__.pyi
diff --git a/v1.0/salad/typeshed/2.7/rdflib/plugins/parsers/__init__.pyi b/v1.0/salad/typeshed/2and3/rdflib/plugins/parsers/__init__.pyi
similarity index 100%
rename from v1.0/salad/typeshed/2.7/rdflib/plugins/parsers/__init__.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/plugins/parsers/__init__.pyi
diff --git a/v1.0/salad/typeshed/2.7/rdflib/plugins/parsers/notation3.pyi b/v1.0/salad/typeshed/2and3/rdflib/plugins/parsers/notation3.pyi
similarity index 100%
rename from v1.0/salad/typeshed/2.7/rdflib/plugins/parsers/notation3.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/plugins/parsers/notation3.pyi
diff --git a/v1.0/salad/typeshed/2.7/rdflib/query.pyi b/v1.0/salad/typeshed/2and3/rdflib/query.pyi
similarity index 97%
rename from v1.0/salad/typeshed/2.7/rdflib/query.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/query.pyi
index db002b8b0..fad3c3770 100644
--- a/v1.0/salad/typeshed/2.7/rdflib/query.pyi
+++ b/v1.0/salad/typeshed/2and3/rdflib/query.pyi
@@ -3,7 +3,7 @@
# NOTE: This dynamically typed stub was automatically generated by stubgen.
from typing import Any
-from StringIO import StringIO as BytesIO
+from io import StringIO as BytesIO
import collections
class Processor:
diff --git a/v1.0/salad/typeshed/2.7/rdflib/serializer.pyi b/v1.0/salad/typeshed/2and3/rdflib/serializer.pyi
similarity index 100%
rename from v1.0/salad/typeshed/2.7/rdflib/serializer.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/serializer.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/store.pyi b/v1.0/salad/typeshed/2and3/rdflib/store.pyi
similarity index 97%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/store.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/store.pyi
index 4a98a44b0..6886321b0 100644
--- a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/store.pyi
+++ b/v1.0/salad/typeshed/2and3/rdflib/store.pyi
@@ -4,7 +4,7 @@
from typing import Any
from rdflib.events import Event
-from cStringIO import StringIO as BytesIO
+from io import StringIO as BytesIO
class StoreCreatedEvent(Event): ...
class TripleAddedEvent(Event): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/term.pyi b/v1.0/salad/typeshed/2and3/rdflib/term.pyi
similarity index 97%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/term.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/term.pyi
index 698218123..0021d14be 100644
--- a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/term.pyi
+++ b/v1.0/salad/typeshed/2and3/rdflib/term.pyi
@@ -2,11 +2,11 @@
#
# NOTE: This dynamically typed stub was automatically generated by stubgen.
-from typing import Any
+from typing import Any, Text
class Node: ...
-class Identifier(Node, unicode):
+class Identifier(Node, Text):
def __new__(cls, value): ...
def eq(self, other): ...
def neq(self, other): ...
diff --git a/v1.0/salad/typeshed/2.7/rdflib/util.pyi b/v1.0/salad/typeshed/2and3/rdflib/util.pyi
similarity index 100%
rename from v1.0/salad/typeshed/2.7/rdflib/util.pyi
rename to v1.0/salad/typeshed/2and3/rdflib/util.pyi
diff --git a/v1.0/salad/typeshed/2.7/ruamel/__init__.py b/v1.0/salad/typeshed/2and3/ruamel/__init__.py
similarity index 100%
rename from v1.0/salad/typeshed/2.7/ruamel/__init__.py
rename to v1.0/salad/typeshed/2and3/ruamel/__init__.py
diff --git a/v1.0/salad/typeshed/3/avro/__init__.pyi b/v1.0/salad/typeshed/3/avro/__init__.pyi
new file mode 100644
index 000000000..0a82029d3
--- /dev/null
+++ b/v1.0/salad/typeshed/3/avro/__init__.pyi
@@ -0,0 +1,10 @@
+# Stubs for avro (Python 3.5)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+# Names in __all__ with no definition:
+# datafile
+# io
+# ipc
+# protocol
+# schema
diff --git a/v1.0/salad/typeshed/3/avro/schema.pyi b/v1.0/salad/typeshed/3/avro/schema.pyi
new file mode 100644
index 000000000..5276b29a3
--- /dev/null
+++ b/v1.0/salad/typeshed/3/avro/schema.pyi
@@ -0,0 +1,180 @@
+# Stubs for avro.schema (Python 3.5)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Any, Optional
+
+logger = ... # type: Any
+DEBUG_VERBOSE = ... # type: int
+NULL = ... # type: str
+BOOLEAN = ... # type: str
+STRING = ... # type: str
+BYTES = ... # type: str
+INT = ... # type: str
+LONG = ... # type: str
+FLOAT = ... # type: str
+DOUBLE = ... # type: str
+FIXED = ... # type: str
+ENUM = ... # type: str
+RECORD = ... # type: str
+ERROR = ... # type: str
+ARRAY = ... # type: str
+MAP = ... # type: str
+UNION = ... # type: str
+REQUEST = ... # type: str
+ERROR_UNION = ... # type: str
+PRIMITIVE_TYPES = ... # type: Any
+NAMED_TYPES = ... # type: Any
+VALID_TYPES = ... # type: Any
+SCHEMA_RESERVED_PROPS = ... # type: Any
+FIELD_RESERVED_PROPS = ... # type: Any
+VALID_FIELD_SORT_ORDERS = ... # type: Any
+
+class Error(Exception): ...
+class AvroException(Error): ...
+class SchemaParseException(AvroException): ...
+
+class ImmutableDict(dict):
+ def __init__(self, items: Optional[Any] = ..., **kwargs) -> None: ...
+ def __setitem__(self, key, value): ...
+ def __delitem__(self, key): ...
+ def clear(self): ...
+ def update(self, **kwargs): ...
+ def pop(self, key, default: Optional[Any] = ...): ...
+ def popitem(self): ...
+
+class Schema:
+ def __init__(self, type, other_props: Optional[Any] = ...) -> None: ...
+ @property
+ def name(self): ...
+ @property
+ def fullname(self): ...
+ @property
+ def namespace(self): ...
+ @property
+ def type(self): ...
+ @property
+ def doc(self): ...
+ @property
+ def props(self): ...
+ @property
+ def other_props(self): ...
+ def to_json(self, names): ...
+
+class Name:
+ def __init__(self, name, namespace: Optional[Any] = ...) -> None: ...
+ def __eq__(self, other): ...
+ @property
+ def simple_name(self): ...
+ @property
+ def namespace(self): ...
+ @property
+ def fullname(self): ...
+
+class Names:
+ def __init__(self, default_namespace: Optional[Any] = ..., names: Optional[Any] = ...) -> None: ...
+ @property
+ def names(self): ...
+ @property
+ def default_namespace(self): ...
+ def NewWithDefaultNamespace(self, namespace): ...
+ def GetName(self, name, namespace: Optional[Any] = ...): ...
+ def has_name(self, name, namespace: Optional[Any] = ...): ...
+ def get_name(self, name, namespace: Optional[Any] = ...): ...
+ def GetSchema(self, name, namespace: Optional[Any] = ...): ...
+ def prune_namespace(self, properties): ...
+ def Register(self, schema): ...
+
+class NamedSchema(Schema):
+ def __init__(self, type, name, namespace: Optional[Any] = ..., names: Optional[Any] = ..., other_props: Optional[Any] = ...) -> None: ...
+ @property
+ def avro_name(self): ...
+ @property
+ def name(self): ...
+ @property
+ def namespace(self): ...
+ @property
+ def fullname(self): ...
+ def name_ref(self, names): ...
+
+class Field:
+ def __init__(self, type, name, index, has_default, default: Any = ..., order: Optional[Any] = ..., names: Optional[Any] = ..., doc: Optional[Any] = ..., other_props: Optional[Any] = ...) -> None: ...
+ @property
+ def type(self): ...
+ @property
+ def name(self): ...
+ @property
+ def index(self): ...
+ @property
+ def default(self): ...
+ @property
+ def has_default(self): ...
+ @property
+ def order(self): ...
+ @property
+ def doc(self): ...
+ @property
+ def props(self): ...
+ @property
+ def other_props(self): ...
+ def to_json(self, names: Optional[Any] = ...): ...
+ def __eq__(self, that): ...
+
+class PrimitiveSchema(Schema):
+ def __init__(self, type, other_props: Optional[Any] = ...) -> None: ...
+ @property
+ def name(self): ...
+ def to_json(self, names: Optional[Any] = ...): ...
+ def __eq__(self, that): ...
+
+class FixedSchema(NamedSchema):
+ def __init__(self, name, namespace, size, names: Optional[Any] = ..., other_props: Optional[Any] = ...) -> None: ...
+ @property
+ def size(self): ...
+ def to_json(self, names: Optional[Any] = ...): ...
+ def __eq__(self, that): ...
+
+class EnumSchema(NamedSchema):
+ def __init__(self, name, namespace, symbols, names: Optional[Any] = ..., doc: Optional[Any] = ..., other_props: Optional[Any] = ...) -> None: ...
+ @property
+ def symbols(self): ...
+ def to_json(self, names: Optional[Any] = ...): ...
+ def __eq__(self, that): ...
+
+class ArraySchema(Schema):
+ def __init__(self, items, other_props: Optional[Any] = ...) -> None: ...
+ @property
+ def items(self): ...
+ def to_json(self, names: Optional[Any] = ...): ...
+ def __eq__(self, that): ...
+
+class MapSchema(Schema):
+ def __init__(self, values, other_props: Optional[Any] = ...) -> None: ...
+ @property
+ def values(self): ...
+ def to_json(self, names: Optional[Any] = ...): ...
+ def __eq__(self, that): ...
+
+class UnionSchema(Schema):
+ def __init__(self, schemas) -> None: ...
+ @property
+ def schemas(self): ...
+ def to_json(self, names: Optional[Any] = ...): ...
+ def __eq__(self, that): ...
+
+class ErrorUnionSchema(UnionSchema):
+ def __init__(self, schemas) -> None: ...
+ def to_json(self, names: Optional[Any] = ...): ...
+
+class RecordSchema(NamedSchema):
+ def __init__(self, name, namespace, fields: Optional[Any] = ..., make_fields: Optional[Any] = ..., names: Optional[Any] = ..., record_type: Any = ..., doc: Optional[Any] = ..., other_props: Optional[Any] = ...) -> None: ...
+ @property
+ def fields(self): ...
+ @property
+ def field_map(self): ...
+ def to_json(self, names: Optional[Any] = ...): ...
+ def __eq__(self, that): ...
+
+def FilterKeysOut(items, keys): ...
+def SchemaFromJSONData(json_data, names: Optional[Any] = ...): ...
+def Parse(json_string): ...
diff --git a/v1.0/userguide-intro.md b/v1.0/userguide-intro.md
index 342cd9022..dcec161d5 100644
--- a/v1.0/userguide-intro.md
+++ b/v1.0/userguide-intro.md
@@ -2,27 +2,5 @@
Hello!
-This guide will introduce you to writing tool wrappers and workflows using the
-Common Workflow Language (CWL). This guide describes the current stable
-specification, version 1.0.
-
-Note: This document is a work in progress. Not all features are covered, yet.
-
-
-
-# Introduction
-
-CWL is a way to describe command line tools and connect them together to create
-workflows. Because CWL is a specification and not a specific piece of
-software, tools and workflows described using CWL are portable across a variety
-of platforms that support the CWL standard.
-
-CWL has roots in "make" and many similar tools that determine order of
-execution based on dependencies between tasks. However unlike "make", CWL
-tasks are isolated and you must be explicit about your inputs and outputs. The
-benefit of explicitness and isolation are flexibility, portability, and
-scalability: tools and workflows described with CWL can transparently leverage
-technologies such as Docker, be used with CWL implementations from different
-vendors, and is well suited for describing large-scale workflows in cluster,
-cloud and high performance computing environments where tasks are scheduled in
-parallel across many nodes.
+The Common Workflow Language user guide has moved to
+[http://www.commonwl.org/user_guide/](http://www.commonwl.org/user_guide/)
diff --git a/v1.0/examples/Hello.java b/v1.0/v1.0/Hello.java
similarity index 100%
rename from v1.0/examples/Hello.java
rename to v1.0/v1.0/Hello.java
diff --git a/v1.0/v1.0/any-type-compat.cwl b/v1.0/v1.0/any-type-compat.cwl
new file mode 100644
index 000000000..0eaa71590
--- /dev/null
+++ b/v1.0/v1.0/any-type-compat.cwl
@@ -0,0 +1,22 @@
+cwlVersion: v1.0
+class: Workflow
+
+steps: []
+inputs:
+ input1:
+ type: Any
+ input2:
+ type: Any[]
+ input3:
+ type: Any
+
+outputs:
+ - id: output1
+ type: string[]
+ outputSource: input1
+ - id: output2
+ type: string[]
+ outputSource: input2
+ - id: output3
+ type: string
+ outputSource: input3
\ No newline at end of file
diff --git a/v1.0/v1.0/any-type-job.json b/v1.0/v1.0/any-type-job.json
new file mode 100644
index 000000000..d5309e47d
--- /dev/null
+++ b/v1.0/v1.0/any-type-job.json
@@ -0,0 +1,5 @@
+{
+ "input1": ["hello", "world"],
+ "input2": ["foo", "bar"],
+ "input3": "hello"
+}
\ No newline at end of file
diff --git a/v1.0/examples/arguments-job.yml b/v1.0/v1.0/arguments-job.yml
similarity index 100%
rename from v1.0/examples/arguments-job.yml
rename to v1.0/v1.0/arguments-job.yml
diff --git a/v1.0/v1.0/array-of-strings-job.yml b/v1.0/v1.0/array-of-strings-job.yml
new file mode 100644
index 000000000..113dbbfe2
--- /dev/null
+++ b/v1.0/v1.0/array-of-strings-job.yml
@@ -0,0 +1,5 @@
+array_input:
+ - class: File
+ path: ./hello.txt
+ - class: File
+ path: ./hello.2.txt
diff --git a/v1.0/v1.0/basename-fields-job.yml b/v1.0/v1.0/basename-fields-job.yml
new file mode 100644
index 000000000..b9fd71586
--- /dev/null
+++ b/v1.0/v1.0/basename-fields-job.yml
@@ -0,0 +1,5 @@
+cwlVersion: v1.0
+tool:
+ class: File
+ path: echo-tool.cwl # could have been any file, this isn't a secret CWL feature :-)
+
diff --git a/v1.0/v1.0/basename-fields-test.cwl b/v1.0/v1.0/basename-fields-test.cwl
new file mode 100644
index 000000000..b46cc47a0
--- /dev/null
+++ b/v1.0/v1.0/basename-fields-test.cwl
@@ -0,0 +1,33 @@
+cwlVersion: v1.0
+class: Workflow
+
+requirements:
+ - class: StepInputExpressionRequirement
+
+inputs:
+ tool: File
+
+outputs:
+ rootFile:
+ type: File
+ outputSource: root/out
+ extFile:
+ type: File
+ outputSource: ext/out
+
+steps:
+ root:
+ run: echo-file-tool.cwl
+ in:
+ tool: tool
+ in:
+ valueFrom: $(inputs.tool.nameroot)
+ out: [out]
+ ext:
+ run: echo-file-tool.cwl
+ in:
+ tool: tool
+ in:
+ valueFrom: $(inputs.tool.nameext)
+ out: [out]
+
diff --git a/v1.0/v1.0/binding-test.cwl b/v1.0/v1.0/binding-test.cwl
index 99f812fb1..d2e2e5471 100755
--- a/v1.0/v1.0/binding-test.cwl
+++ b/v1.0/v1.0/binding-test.cwl
@@ -2,7 +2,9 @@
class: CommandLineTool
cwlVersion: v1.0
-
+hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
inputs:
- id: reference
type: File
diff --git a/v1.0/v1.0/bool-empty-inputbinding-job.json b/v1.0/v1.0/bool-empty-inputbinding-job.json
new file mode 100644
index 000000000..a89b76acd
--- /dev/null
+++ b/v1.0/v1.0/bool-empty-inputbinding-job.json
@@ -0,0 +1,3 @@
+{
+ "flag": true
+}
\ No newline at end of file
diff --git a/v1.0/v1.0/bool-empty-inputbinding.cwl b/v1.0/v1.0/bool-empty-inputbinding.cwl
new file mode 100644
index 000000000..3b1953ce1
--- /dev/null
+++ b/v1.0/v1.0/bool-empty-inputbinding.cwl
@@ -0,0 +1,19 @@
+#!/usr/bin/env cwl-runner
+class: CommandLineTool
+cwlVersion: v1.0
+inputs:
+- id: flag
+ type: boolean
+ inputBinding: {}
+- id: "args.py"
+ type: File
+ default:
+ class: File
+ location: args.py
+ inputBinding:
+ position: -1
+outputs:
+- id: args
+ type: string[]
+baseCommand: python
+arguments: []
\ No newline at end of file
diff --git a/v1.0/v1.0/bwa-mem-tool.cwl b/v1.0/v1.0/bwa-mem-tool.cwl
index 1d838b267..8c5adf736 100755
--- a/v1.0/v1.0/bwa-mem-tool.cwl
+++ b/v1.0/v1.0/bwa-mem-tool.cwl
@@ -6,7 +6,7 @@ class: CommandLineTool
hints:
- class: ResourceRequirement
- coresMin: 4
+ coresMin: 2
- class: DockerRequirement
dockerPull: python:2-slim
diff --git a/v1.0/v1.0/cat-tool.cwl b/v1.0/v1.0/cat-tool.cwl
new file mode 100644
index 000000000..3d92a5fc4
--- /dev/null
+++ b/v1.0/v1.0/cat-tool.cwl
@@ -0,0 +1,17 @@
+#!/usr/bin/env cwl-runner
+
+class: CommandLineTool
+cwlVersion: v1.0
+
+inputs:
+ file1: File
+
+outputs:
+ output:
+ type: File
+ outputBinding: { glob: output }
+
+baseCommand: [cat]
+
+stdin: $(inputs.file1.path)
+stdout: output
diff --git a/v1.0/v1.0/cat1-testcli.cwl b/v1.0/v1.0/cat1-testcli.cwl
index 9558ed82c..26927b749 100755
--- a/v1.0/v1.0/cat1-testcli.cwl
+++ b/v1.0/v1.0/cat1-testcli.cwl
@@ -3,6 +3,12 @@
"class": "CommandLineTool",
"cwlVersion": "v1.0",
"doc": "Print the contents of a file to stdout using 'cat' running in a docker container.",
+ "hints": [
+ {
+ "class": "DockerRequirement",
+ "dockerPull": "python:2-slim"
+ }
+ ],
"inputs": [
{
"id": "file1",
diff --git a/v1.0/v1.0/cat1-tool.cwl b/v1.0/v1.0/cat1-tool.cwl
deleted file mode 100755
index 2ae424a99..000000000
--- a/v1.0/v1.0/cat1-tool.cwl
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/usr/bin/env cwl-runner
-cwlVersion: v1.0
-class: CommandLineTool
-doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
-hints:
- DockerRequirement:
- dockerPull: debian:wheezy
- SoftwareRequirement:
- packages:
- - name: cat
-inputs:
- file1:
- type: File
- inputBinding: {position: 1}
- numbering:
- type: boolean?
- inputBinding:
- position: 0
- prefix: -n
-outputs: []
-baseCommand: cat
diff --git a/v1.0/v1.0/cat3-nodocker.cwl b/v1.0/v1.0/cat3-nodocker.cwl
new file mode 100755
index 000000000..7007aa153
--- /dev/null
+++ b/v1.0/v1.0/cat3-nodocker.cwl
@@ -0,0 +1,16 @@
+#!/usr/bin/env cwl-runner
+class: CommandLineTool
+cwlVersion: v1.0
+doc: "Print the contents of a file to stdout using 'cat'."
+inputs:
+ file1:
+ type: File
+ label: Input File
+ doc: "The file that will be copied using 'cat'"
+ inputBinding: {position: 1}
+outputs:
+ output_file:
+ type: File
+ outputBinding: {glob: output.txt}
+baseCommand: cat
+stdout: output.txt
diff --git a/v1.0/v1.0/cat3-tool-mediumcut.cwl b/v1.0/v1.0/cat3-tool-mediumcut.cwl
index 81958c530..9a4132126 100755
--- a/v1.0/v1.0/cat3-tool-mediumcut.cwl
+++ b/v1.0/v1.0/cat3-tool-mediumcut.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.0
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1:
type: File
diff --git a/v1.0/v1.0/cat3-tool-shortcut.cwl b/v1.0/v1.0/cat3-tool-shortcut.cwl
index 0f1694bc2..9d6244636 100755
--- a/v1.0/v1.0/cat3-tool-shortcut.cwl
+++ b/v1.0/v1.0/cat3-tool-shortcut.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.0
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1:
type: File
diff --git a/v1.0/v1.0/cat3-tool.cwl b/v1.0/v1.0/cat3-tool.cwl
index c019c88cc..2b7e75525 100755
--- a/v1.0/v1.0/cat3-tool.cwl
+++ b/v1.0/v1.0/cat3-tool.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.0
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1:
type: File
diff --git a/v1.0/v1.0/cat4-tool.cwl b/v1.0/v1.0/cat4-tool.cwl
index f3d8956c0..d05c0ae3b 100755
--- a/v1.0/v1.0/cat4-tool.cwl
+++ b/v1.0/v1.0/cat4-tool.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.0
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1: File
outputs:
diff --git a/v1.0/v1.0/cat5-tool.cwl b/v1.0/v1.0/cat5-tool.cwl
index c106cd602..46c041068 100755
--- a/v1.0/v1.0/cat5-tool.cwl
+++ b/v1.0/v1.0/cat5-tool.cwl
@@ -6,7 +6,7 @@ class: CommandLineTool
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: "debian:wheezy"
+ dockerPull: "debian:stretch-slim"
ex:BlibberBlubberFakeRequirement:
fakeField: fraggleFroogle
inputs:
diff --git a/v1.0/v1.0/count-lines10-wf.cwl b/v1.0/v1.0/count-lines10-wf.cwl
index 63d9f0ff1..781127154 100755
--- a/v1.0/v1.0/count-lines10-wf.cwl
+++ b/v1.0/v1.0/count-lines10-wf.cwl
@@ -4,11 +4,11 @@ cwlVersion: v1.0
inputs:
file1: File
outputs:
- count_output: {type: int, outputSource: step1/count_output}
+ count_output: {type: int, outputSource: step0/count_output}
requirements:
SubworkflowFeatureRequirement: {}
steps:
- step1:
+ step0:
in: {file1: file1}
out: [count_output]
run:
diff --git a/v1.0/v1.0/count-lines11-wf.cwl b/v1.0/v1.0/count-lines11-wf.cwl
new file mode 100755
index 000000000..17e1af955
--- /dev/null
+++ b/v1.0/v1.0/count-lines11-wf.cwl
@@ -0,0 +1,28 @@
+#!/usr/bin/env cwl-runner
+class: Workflow
+cwlVersion: v1.0
+
+inputs:
+ file1: File?
+
+outputs:
+ count_output:
+ type: int
+ outputSource: step2/output
+
+steps:
+ step1:
+ run: wc-tool.cwl
+ in:
+ file1:
+ source: file1
+ default:
+ class: File
+ location: whale.txt
+ out: [output]
+
+ step2:
+ run: parseInt-tool.cwl
+ in:
+ file1: step1/output
+ out: [output]
diff --git a/v1.0/v1.0/count-lines12-wf.cwl b/v1.0/v1.0/count-lines12-wf.cwl
new file mode 100755
index 000000000..533d9ed5e
--- /dev/null
+++ b/v1.0/v1.0/count-lines12-wf.cwl
@@ -0,0 +1,27 @@
+#!/usr/bin/env cwl-runner
+class: Workflow
+cwlVersion: v1.0
+requirements:
+ - class: MultipleInputFeatureRequirement
+
+inputs:
+ file1:
+ - type: array
+ items: File
+ file2:
+ - type: array
+ items: File
+
+outputs:
+ count_output:
+ type: int
+ outputSource: step1/output
+
+steps:
+ step1:
+ run: wc3-tool.cwl
+ in:
+ file1:
+ source: [file1, file2]
+ linkMerge: merge_flattened
+ out: [output]
diff --git a/v1.0/v1.0/count-lines13-wf.cwl b/v1.0/v1.0/count-lines13-wf.cwl
new file mode 100644
index 000000000..474aa0323
--- /dev/null
+++ b/v1.0/v1.0/count-lines13-wf.cwl
@@ -0,0 +1,20 @@
+#!/usr/bin/env cwl-runner
+class: Workflow
+cwlVersion: v1.0
+
+inputs:
+ file1: File[]
+ file2: File[]
+
+outputs:
+ count_output:
+ type: int
+ outputSource: step1/output
+
+steps:
+ step1:
+ run: wc3-tool.cwl
+ in:
+ file1:
+ source: [file1]
+ out: [output]
diff --git a/v1.0/v1.0/dir2.cwl b/v1.0/v1.0/dir2.cwl
index 5a3d5dbc7..7136f1b14 100644
--- a/v1.0/v1.0/dir2.cwl
+++ b/v1.0/v1.0/dir2.cwl
@@ -2,7 +2,7 @@ class: CommandLineTool
cwlVersion: v1.0
hints:
DockerRequirement:
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
ShellCommandRequirement: {}
inputs:
indir: Directory
diff --git a/v1.0/v1.0/dir5.cwl b/v1.0/v1.0/dir5.cwl
index ac22a758b..f81a9083a 100644
--- a/v1.0/v1.0/dir5.cwl
+++ b/v1.0/v1.0/dir5.cwl
@@ -11,7 +11,7 @@ outputs:
type: File
outputBinding:
glob: output.txt
-arguments: ["find", "-L", ".",
+arguments: ["find", "-L", ".", "!", "-path", "*.txt",
{shellQuote: false, valueFrom: "|"},
"sort"]
stdout: output.txt
\ No newline at end of file
diff --git a/v1.0/v1.0/docker-array-secondaryfiles.cwl b/v1.0/v1.0/docker-array-secondaryfiles.cwl
index 4b1597563..7ed1468cf 100644
--- a/v1.0/v1.0/docker-array-secondaryfiles.cwl
+++ b/v1.0/v1.0/docker-array-secondaryfiles.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.0
requirements:
- class: DockerRequirement
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
- class: InlineJavascriptRequirement
- class: ShellCommandRequirement
diff --git a/v1.0/v1.0/docker-output-dir.cwl b/v1.0/v1.0/docker-output-dir.cwl
index f2bc3c5fc..7f392f095 100644
--- a/v1.0/v1.0/docker-output-dir.cwl
+++ b/v1.0/v1.0/docker-output-dir.cwl
@@ -2,7 +2,7 @@ class: CommandLineTool
cwlVersion: v1.0
requirements:
DockerRequirement:
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
dockerOutputDirectory: /other
inputs: []
outputs:
diff --git a/v1.0/v1.0/dynresreq-job.json b/v1.0/v1.0/dynresreq-job.json
new file mode 100644
index 000000000..9f8c018ca
--- /dev/null
+++ b/v1.0/v1.0/dynresreq-job.json
@@ -0,0 +1,3 @@
+special_file:
+ class: File
+ path: special_file
diff --git a/v1.0/v1.0/dynresreq-workflow.cwl b/v1.0/v1.0/dynresreq-workflow.cwl
new file mode 100644
index 000000000..e38ba17c5
--- /dev/null
+++ b/v1.0/v1.0/dynresreq-workflow.cwl
@@ -0,0 +1,24 @@
+#!/usr/bin/env cwl-runner
+class: Workflow
+cwlVersion: v1.0
+
+inputs:
+ special_file: File
+
+outputs:
+ cores:
+ type: File
+ outputSource: report/output
+
+steps:
+ count:
+ in:
+ special_file: special_file
+ out: [output]
+ run: dynresreq.cwl
+
+ report:
+ in:
+ file1: count/output
+ out: [output]
+ run: cat-tool.cwl
diff --git a/v1.0/v1.0/dynresreq.cwl b/v1.0/v1.0/dynresreq.cwl
new file mode 100644
index 000000000..9af95330f
--- /dev/null
+++ b/v1.0/v1.0/dynresreq.cwl
@@ -0,0 +1,21 @@
+#!/usr/bin/env cwl-runner
+class: CommandLineTool
+cwlVersion: v1.0
+
+requirements:
+ ResourceRequirement:
+ coresMin: $(inputs.special_file.size)
+ coresMax: $(inputs.special_file.size)
+
+inputs:
+ special_file: File
+
+outputs:
+ output:
+ type: stdout
+
+baseCommand: echo
+
+stdout: cores.txt
+
+arguments: [ $(runtime.cores) ]
diff --git a/v1.0/v1.0/echo-file-tool.cwl b/v1.0/v1.0/echo-file-tool.cwl
new file mode 100644
index 000000000..189a4c8f4
--- /dev/null
+++ b/v1.0/v1.0/echo-file-tool.cwl
@@ -0,0 +1,12 @@
+cwlVersion: v1.0
+class: CommandLineTool
+baseCommand: [echo]
+inputs:
+ in:
+ type: string
+ inputBinding:
+ position: 1
+outputs:
+ out:
+ type: stdout
+
diff --git a/v1.0/v1.0/echo-tool-default.cwl b/v1.0/v1.0/echo-tool-default.cwl
new file mode 100644
index 000000000..f88cd7672
--- /dev/null
+++ b/v1.0/v1.0/echo-tool-default.cwl
@@ -0,0 +1,19 @@
+#!/usr/bin/env cwl-runner
+
+class: CommandLineTool
+cwlVersion: v1.0
+inputs:
+ in:
+ type: string
+ default: tool_default
+ inputBinding:
+ position: 1
+outputs:
+ out:
+ type: string
+ outputBinding:
+ glob: out.txt
+ loadContents: true
+ outputEval: $(self[0].contents)
+baseCommand: [echo, -n]
+stdout: out.txt
diff --git a/v1.0/v1.0/echo-wf-default.cwl b/v1.0/v1.0/echo-wf-default.cwl
new file mode 100644
index 000000000..0dfedd24d
--- /dev/null
+++ b/v1.0/v1.0/echo-wf-default.cwl
@@ -0,0 +1,19 @@
+
+class: Workflow
+cwlVersion: v1.0
+
+inputs: []
+
+steps:
+ step1:
+ run: echo-tool-default.cwl
+ in:
+ in:
+ default: workflow_default
+ out: [out]
+
+outputs:
+ default_output:
+ type: string
+ outputSource: step1/out
+
diff --git a/v1.0/v1.0/empty-array-input.cwl b/v1.0/v1.0/empty-array-input.cwl
new file mode 100644
index 000000000..d2a7f9897
--- /dev/null
+++ b/v1.0/v1.0/empty-array-input.cwl
@@ -0,0 +1,32 @@
+#!/usr/bin/env cwl-runner
+
+cwlVersion: v1.0
+class: CommandLineTool
+
+hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
+
+inputs:
+ - id: array
+ type: { type: array, items: int }
+ inputBinding:
+ position: 1
+ prefix: -I
+ itemSeparator: ","
+
+ - id: args.py
+ type: File
+ default:
+ class: File
+ location: args.py
+ inputBinding:
+ position: -1
+
+outputs:
+ - id: args
+ type:
+ type: array
+ items: string
+
+baseCommand: python
diff --git a/v1.0/v1.0/empty-array-job.json b/v1.0/v1.0/empty-array-job.json
new file mode 100644
index 000000000..038229f2b
--- /dev/null
+++ b/v1.0/v1.0/empty-array-job.json
@@ -0,0 +1,3 @@
+{
+ "array": []
+}
\ No newline at end of file
diff --git a/v1.0/v1.0/envvar2.cwl b/v1.0/v1.0/envvar2.cwl
index a89e62e23..d688174e2 100644
--- a/v1.0/v1.0/envvar2.cwl
+++ b/v1.0/v1.0/envvar2.cwl
@@ -6,7 +6,7 @@ requirements:
ShellCommandRequirement: {}
hints:
DockerRequirement:
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
arguments: [
echo, {valueFrom: '"HOME=$HOME"', shellQuote: false}, {valueFrom: '"TMPDIR=$TMPDIR"', shellQuote: false},
{valueFrom: '&&', shellQuote: false},
diff --git a/v1.0/v1.0/exit-success.cwl b/v1.0/v1.0/exit-success.cwl
new file mode 100644
index 000000000..9e66483eb
--- /dev/null
+++ b/v1.0/v1.0/exit-success.cwl
@@ -0,0 +1,11 @@
+#!/usr/bin/env cwl-runner
+cwlVersion: v1.0
+class: CommandLineTool
+
+inputs: []
+baseCommand: "false"
+outputs: []
+
+successCodes: [ 1 ]
+permanentFailCodes: [ 0 ]
+temporaryFailCodes: [ 42 ]
diff --git a/v1.0/v1.0/file1-null.json b/v1.0/v1.0/file1-null.json
new file mode 100644
index 000000000..cc6b36193
--- /dev/null
+++ b/v1.0/v1.0/file1-null.json
@@ -0,0 +1,3 @@
+{
+ "file1": null
+}
diff --git a/v1.0/v1.0/formattest2.cwl b/v1.0/v1.0/formattest2.cwl
index 54acd4a24..9e40cd5ad 100644
--- a/v1.0/v1.0/formattest2.cwl
+++ b/v1.0/v1.0/formattest2.cwl
@@ -5,6 +5,9 @@ $schemas:
class: CommandLineTool
cwlVersion: v1.0
doc: "Reverse each line using the `rev` command"
+hints:
+ DockerRequirement:
+ dockerPull: "debian:stretch-slim"
inputs:
input:
diff --git a/v1.0/v1.0/formattest3.cwl b/v1.0/v1.0/formattest3.cwl
index d0baf604a..2e4a1fca5 100644
--- a/v1.0/v1.0/formattest3.cwl
+++ b/v1.0/v1.0/formattest3.cwl
@@ -7,6 +7,9 @@ $schemas:
class: CommandLineTool
cwlVersion: v1.0
doc: "Reverse each line using the `rev` command"
+hints:
+ DockerRequirement:
+ dockerPull: "debian:stretch-slim"
inputs:
input:
diff --git a/v1.0/v1.0/glob_test.cwl b/v1.0/v1.0/glob_test.cwl
new file mode 100644
index 000000000..59f5af8a3
--- /dev/null
+++ b/v1.0/v1.0/glob_test.cwl
@@ -0,0 +1,10 @@
+#!/usr/bin/env cwl-runner
+cwlVersion: v1.0
+class: CommandLineTool
+
+inputs: []
+baseCommand: [touch, z, y, x, w, c, b, a]
+outputs:
+ letters:
+ type: File[]
+ outputBinding: { glob: '*' }
diff --git a/v1.0/v1.0/hello.2.txt b/v1.0/v1.0/hello.2.txt
new file mode 100644
index 000000000..cd0875583
--- /dev/null
+++ b/v1.0/v1.0/hello.2.txt
@@ -0,0 +1 @@
+Hello world!
diff --git a/v1.0/v1.0/index.py b/v1.0/v1.0/index.py
index 65acd921b..77918b5e5 100755
--- a/v1.0/v1.0/index.py
+++ b/v1.0/v1.0/index.py
@@ -31,3 +31,7 @@
open(sys.argv[1] + ".idx3", "w")
open(sys.argv[1] + ".idx4", "w")
open(sys.argv[1] + ".idx5", "w")
+open(os.path.splitext(sys.argv[1])[0] + ".idx6" + os.path.splitext(sys.argv[1])[1], "w")
+open(sys.argv[1] + ".idx7", "w")
+os.mkdir(sys.argv[1] + "_idx8")
+open(sys.argv[1] + "_idx8/index", "w")
diff --git a/v1.0/v1.0/initialwork-path.cwl b/v1.0/v1.0/initialwork-path.cwl
new file mode 100644
index 000000000..d01b30ad1
--- /dev/null
+++ b/v1.0/v1.0/initialwork-path.cwl
@@ -0,0 +1,15 @@
+class: CommandLineTool
+cwlVersion: v1.0
+requirements:
+ InitialWorkDirRequirement:
+ listing:
+ - entry: $(inputs.file1)
+ entryname: bob.txt
+ ShellCommandRequirement: {}
+inputs:
+ file1: File
+outputs: []
+arguments:
+ - shellQuote: false
+ valueFrom: |
+ test "$(inputs.file1.path)" = "$(runtime.outdir)/bob.txt"
diff --git a/v1.0/v1.0/initialworkdirrequirement-docker-out.cwl b/v1.0/v1.0/initialworkdirrequirement-docker-out.cwl
index 4ba0d998d..366c18b48 100644
--- a/v1.0/v1.0/initialworkdirrequirement-docker-out.cwl
+++ b/v1.0/v1.0/initialworkdirrequirement-docker-out.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.0
requirements:
- class: DockerRequirement
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
- class: InitialWorkDirRequirement
listing:
- $(inputs.INPUT)
diff --git a/v1.0/v1.0/inline-js.cwl b/v1.0/v1.0/inline-js.cwl
index 2460a063d..585a3dc22 100755
--- a/v1.0/v1.0/inline-js.cwl
+++ b/v1.0/v1.0/inline-js.cwl
@@ -1,9 +1,11 @@
cwlVersion: v1.0
class: CommandLineTool
-baseCommand: echo
requirements:
- class: InlineJavascriptRequirement
+hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
inputs:
- id: args.py
@@ -36,3 +38,8 @@ arguments:
}
return r;
}
+ # Test errors similar to https://github.com/common-workflow-language/cwltool/issues/648 are fixed
+ - prefix: -D
+ valueFrom: $(true)
+ - prefix: -E
+ valueFrom: $(false)
diff --git a/v1.0/v1.0/iwdr-entry.cwl b/v1.0/v1.0/iwdr-entry.cwl
new file mode 100644
index 000000000..e7553a253
--- /dev/null
+++ b/v1.0/v1.0/iwdr-entry.cwl
@@ -0,0 +1,20 @@
+#!/usr/bin/env cwl-runner
+
+class: CommandLineTool
+cwlVersion: v1.0
+baseCommand: ["cat", "example.conf"]
+
+requirements:
+ InitialWorkDirRequirement:
+ listing:
+ - entryname: example.conf
+ entry: |
+ CONFIGVAR=$(inputs.message)
+
+inputs:
+ message: string
+outputs:
+ out:
+ type: File
+ outputBinding:
+ glob: example.conf
\ No newline at end of file
diff --git a/v1.0/v1.0/iwdr_with_nested_dirs.cwl b/v1.0/v1.0/iwdr_with_nested_dirs.cwl
new file mode 100644
index 000000000..2ff899ce3
--- /dev/null
+++ b/v1.0/v1.0/iwdr_with_nested_dirs.cwl
@@ -0,0 +1,41 @@
+#!/usr/bin/env cwl-runner
+cwlVersion: v1.0
+class: Workflow
+
+inputs: []
+outputs:
+ ya_empty:
+ type: File
+ outputSource: second/ya
+
+steps:
+ first:
+ run:
+ class: CommandLineTool
+ baseCommand: [ mkdir, -p, deeply/nested/dir/structure ]
+ inputs: []
+ outputs:
+ deep_dir:
+ type: Directory
+ outputBinding: { glob: deeply }
+ in: {}
+ out: [ deep_dir ]
+
+ second:
+ run:
+ class: CommandLineTool
+ baseCommand: [ touch, deeply/nested/dir/structure/ya ]
+ requirements:
+ InitialWorkDirRequirement:
+ listing:
+ - entry: $(inputs.dir)
+ writable: true
+ inputs:
+ dir: Directory
+ outputs:
+ ya:
+ type: File
+ outputBinding: { glob: deeply/nested/dir/structure/ya }
+
+ in: { dir: first/deep_dir }
+ out: [ ya ]
diff --git a/v1.0/examples/linkfile.cwl b/v1.0/v1.0/linkfile.cwl
similarity index 100%
rename from v1.0/examples/linkfile.cwl
rename to v1.0/v1.0/linkfile.cwl
diff --git a/v1.0/v1.0/metadata.cwl b/v1.0/v1.0/metadata.cwl
index c80e5b6c0..f0dd01c1d 100644
--- a/v1.0/v1.0/metadata.cwl
+++ b/v1.0/v1.0/metadata.cwl
@@ -18,7 +18,7 @@ dct:creator:
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1:
type: File
diff --git a/v1.0/v1.0/null-defined.cwl b/v1.0/v1.0/null-defined.cwl
new file mode 100644
index 000000000..097d7822f
--- /dev/null
+++ b/v1.0/v1.0/null-defined.cwl
@@ -0,0 +1,15 @@
+cwlVersion: v1.0
+class: CommandLineTool
+requirements:
+ InlineJavascriptRequirement: {}
+inputs:
+ file1: File?
+outputs:
+ out:
+ type: string
+ outputBinding:
+ glob: out.txt
+ loadContents: true
+ outputEval: $(self[0].contents)
+stdout: out.txt
+arguments: [echo, '$(inputs.file1 === null ? "t" : "f")']
diff --git a/v1.0/v1.0/optional-output.cwl b/v1.0/v1.0/optional-output.cwl
index 4172436b1..16cf36a1a 100644
--- a/v1.0/v1.0/optional-output.cwl
+++ b/v1.0/v1.0/optional-output.cwl
@@ -4,7 +4,7 @@ cwlVersion: "v1.0"
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1:
type: File
diff --git a/v1.0/v1.0/params_inc.yml b/v1.0/v1.0/params_inc.yml
index 9ecffdb74..930d2e89c 100644
--- a/v1.0/v1.0/params_inc.yml
+++ b/v1.0/v1.0/params_inc.yml
@@ -108,3 +108,13 @@
type: Any
outputBinding:
outputEval: $(inputs.bar.buz[1]) $(inputs.bar.buz[1])
+
+ - id: t27
+ type: "null"
+ outputBinding:
+ outputEval: $(null)
+
+ - id: t28
+ type: int
+ outputBinding:
+ outputEval: $(inputs.bar.buz.length)
diff --git a/v1.0/v1.0/recursive-input-directory.cwl b/v1.0/v1.0/recursive-input-directory.cwl
new file mode 100644
index 000000000..a7aaf4736
--- /dev/null
+++ b/v1.0/v1.0/recursive-input-directory.cwl
@@ -0,0 +1,34 @@
+cwlVersion: v1.0
+class: CommandLineTool
+requirements:
+ - class: InlineJavascriptRequirement
+ - class: InitialWorkDirRequirement
+ listing:
+ - entry: $(inputs.input_dir)
+ entryname: work_dir
+ writable: true
+ - class: ShellCommandRequirement
+stdout: output.txt
+arguments:
+ - shellQuote: false
+ valueFrom: |
+ touch work_dir/e;
+ if [ ! -w work_dir ]; then echo work_dir not writable; fi;
+ if [ -L work_dir ]; then echo work_dir is a symlink; fi;
+ if [ ! -w work_dir/a ]; then echo work_dir/a not writable; fi;
+ if [ -L work_dir/a ]; then echo work_dir/a is a symlink; fi;
+ if [ ! -w work_dir/c ]; then echo work_dir/c not writable; fi;
+ if [ -L work_dir/c ]; then echo work_dir/c is a symlink; fi;
+ if [ ! -w work_dir/c/d ]; then echo work_dir/c/d not writable; fi;
+ if [ -L work_dir/c/d ]; then echo work_dir/c/d is a symlink; fi;
+ if [ ! -w work_dir/e ]; then echo work_dir/e not writable; fi;
+ if [ -L work_dir/e ]; then echo work_dir/e is a symlink ; fi;
+inputs:
+ input_dir: Directory
+outputs:
+ output_dir:
+ type: Directory
+ outputBinding:
+ glob: work_dir
+ test_result:
+ type: stdout
diff --git a/v1.0/v1.0/recursive-input-directory.yml b/v1.0/v1.0/recursive-input-directory.yml
new file mode 100644
index 000000000..7c83f3f26
--- /dev/null
+++ b/v1.0/v1.0/recursive-input-directory.yml
@@ -0,0 +1,3 @@
+input_dir:
+ class: Directory
+ location: testdir
\ No newline at end of file
diff --git a/v1.0/v1.0/revsort-packed.cwl b/v1.0/v1.0/revsort-packed.cwl
new file mode 100644
index 000000000..05fcb667e
--- /dev/null
+++ b/v1.0/v1.0/revsort-packed.cwl
@@ -0,0 +1,125 @@
+{
+ "cwlVersion": "v1.0",
+ "$graph": [
+ {
+ "class": "Workflow",
+ "doc": "Reverse the lines in a document, then sort those lines.",
+ "hints": [
+ {
+ "class": "DockerRequirement",
+ "dockerPull": "debian:stretch-slim"
+ }
+ ],
+ "inputs": [
+ {
+ "type": "File",
+ "doc": "The input file to be processed.",
+ "id": "#main/input"
+ },
+ {
+ "type": "boolean",
+ "default": true,
+ "doc": "If true, reverse (decending) sort",
+ "id": "#main/reverse_sort"
+ }
+ ],
+ "outputs": [
+ {
+ "type": "File",
+ "outputSource": "#main/sorted/output",
+ "doc": "The output with the lines reversed and sorted.",
+ "id": "#main/output"
+ }
+ ],
+ "steps": [
+ {
+ "in": [
+ {
+ "source": "#main/input",
+ "id": "#main/rev/input"
+ }
+ ],
+ "out": [
+ "#main/rev/output"
+ ],
+ "run": "#revtool.cwl",
+ "id": "#main/rev"
+ },
+ {
+ "in": [
+ {
+ "source": "#main/rev/output",
+ "id": "#main/sorted/input"
+ },
+ {
+ "source": "#main/reverse_sort",
+ "id": "#main/sorted/reverse"
+ }
+ ],
+ "out": [
+ "#main/sorted/output"
+ ],
+ "run": "#sorttool.cwl",
+ "id": "#main/sorted"
+ }
+ ],
+ "id": "#main"
+ },
+ {
+ "class": "CommandLineTool",
+ "doc": "Reverse each line using the `rev` command",
+ "inputs": [
+ {
+ "type": "File",
+ "inputBinding": {},
+ "id": "#revtool.cwl/input"
+ }
+ ],
+ "outputs": [
+ {
+ "type": "File",
+ "outputBinding": {
+ "glob": "output.txt"
+ },
+ "id": "#revtool.cwl/output"
+ }
+ ],
+ "baseCommand": "rev",
+ "stdout": "output.txt",
+ "id": "#revtool.cwl"
+ },
+ {
+ "class": "CommandLineTool",
+ "doc": "Sort lines using the `sort` command",
+ "inputs": [
+ {
+ "id": "#sorttool.cwl/reverse",
+ "type": "boolean",
+ "inputBinding": {
+ "position": 1,
+ "prefix": "--reverse"
+ }
+ },
+ {
+ "id": "#sorttool.cwl/input",
+ "type": "File",
+ "inputBinding": {
+ "position": 2
+ }
+ }
+ ],
+ "outputs": [
+ {
+ "id": "#sorttool.cwl/output",
+ "type": "File",
+ "outputBinding": {
+ "glob": "output.txt"
+ }
+ }
+ ],
+ "baseCommand": "sort",
+ "stdout": "output.txt",
+ "id": "#sorttool.cwl"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/v1.0/v1.0/revsort.cwl b/v1.0/v1.0/revsort.cwl
index a6b2774ad..ba8fe5958 100644
--- a/v1.0/v1.0/revsort.cwl
+++ b/v1.0/v1.0/revsort.cwl
@@ -10,7 +10,7 @@ cwlVersion: v1.0
# in which the command line tools will execute.
hints:
- class: DockerRequirement
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
# The inputs array defines the structure of the input object that describes
diff --git a/v1.0/v1.0/scatter-empty-job1.json b/v1.0/v1.0/scatter-empty-job1.json
new file mode 100644
index 000000000..dfd8a4904
--- /dev/null
+++ b/v1.0/v1.0/scatter-empty-job1.json
@@ -0,0 +1,3 @@
+{
+ "inp": []
+}
diff --git a/v1.0/v1.0/scatter-empty-job2.json b/v1.0/v1.0/scatter-empty-job2.json
new file mode 100644
index 000000000..a73a3785f
--- /dev/null
+++ b/v1.0/v1.0/scatter-empty-job2.json
@@ -0,0 +1,4 @@
+{
+ "inp1": ["one", "two"],
+ "inp2": []
+}
diff --git a/v1.0/v1.0/scatter-empty-job3.json b/v1.0/v1.0/scatter-empty-job3.json
new file mode 100644
index 000000000..6aa7b6d5e
--- /dev/null
+++ b/v1.0/v1.0/scatter-empty-job3.json
@@ -0,0 +1,4 @@
+{
+ "inp1": [],
+ "inp2": ["one", "two"]
+}
diff --git a/v1.0/v1.0/scatter-empty-job4.json b/v1.0/v1.0/scatter-empty-job4.json
new file mode 100644
index 000000000..e90c2665a
--- /dev/null
+++ b/v1.0/v1.0/scatter-empty-job4.json
@@ -0,0 +1,4 @@
+{
+ "inp1": [],
+ "inp2": []
+}
diff --git a/v1.0/v1.0/scatter-valueFrom-tool.cwl b/v1.0/v1.0/scatter-valueFrom-tool.cwl
new file mode 100644
index 000000000..a406b8c6b
--- /dev/null
+++ b/v1.0/v1.0/scatter-valueFrom-tool.cwl
@@ -0,0 +1,15 @@
+cwlVersion: v1.0
+class: CommandLineTool
+inputs:
+ scattered_message:
+ type: string
+ inputBinding:
+ position: 2
+ message:
+ type: string
+ inputBinding:
+ position: 1
+outputs:
+ out_message:
+ type: stdout
+baseCommand: echo
diff --git a/v1.0/v1.0/scatter-valuefrom-job3.json b/v1.0/v1.0/scatter-valuefrom-job3.json
new file mode 100644
index 000000000..23b6e38c4
--- /dev/null
+++ b/v1.0/v1.0/scatter-valuefrom-job3.json
@@ -0,0 +1 @@
+{"scattered_messages": ["message a", "message b"]}
diff --git a/v1.0/v1.0/scatter-valuefrom-wf6.cwl b/v1.0/v1.0/scatter-valuefrom-wf6.cwl
new file mode 100644
index 000000000..ed60c7684
--- /dev/null
+++ b/v1.0/v1.0/scatter-valuefrom-wf6.cwl
@@ -0,0 +1,21 @@
+cwlVersion: v1.0
+class: Workflow
+requirements:
+ - class: ScatterFeatureRequirement
+ - class: StepInputExpressionRequirement
+inputs:
+ scattered_messages: string[]
+outputs:
+ out_message:
+ type: File[]
+ outputSource: step1/out_message
+steps:
+ step1:
+ run: scatter-valueFrom-tool.cwl
+ scatter: [scattered_message]
+ scatterMethod: dotproduct
+ in:
+ scattered_message: scattered_messages
+ message:
+ valueFrom: "Hello"
+ out: [out_message]
diff --git a/v1.0/v1.0/search-job.json b/v1.0/v1.0/search-job.json
index b5a1b61e5..a5f9d9805 100644
--- a/v1.0/v1.0/search-job.json
+++ b/v1.0/v1.0/search-job.json
@@ -3,5 +3,9 @@
"class": "File",
"location": "whale.txt"
},
+ "secondfile": {
+ "class": "File",
+ "location": "hello.txt"
+ },
"term": "find"
}
diff --git a/v1.0/v1.0/search.cwl b/v1.0/v1.0/search.cwl
index 1c976e048..e65ab69ee 100644
--- a/v1.0/v1.0/search.cwl
+++ b/v1.0/v1.0/search.cwl
@@ -2,7 +2,7 @@ cwlVersion: v1.0
$graph:
- id: index
class: CommandLineTool
- baseCommand: python2
+ baseCommand: python
arguments:
- valueFrom: input.txt
position: 1
@@ -12,9 +12,13 @@ $graph:
- entryname: input.txt
entry: $(inputs.file)
- class: InlineJavascriptRequirement
+ hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
inputs:
file: File
+ secondfile: File
index.py:
type: File
default:
@@ -30,15 +34,21 @@ $graph:
secondaryFiles:
- ".idx1"
- "^.idx2"
- - '$(self.location+".idx3")'
- - '$({"location": self.location+".idx4", "class": "File"})'
- - '${ return self.location+".idx5"; }'
+ - '$(self.basename).idx3'
+ - '${ return self.basename+".idx4"; }'
+ - '$({"path": self.path+".idx5", "class": "File"})'
+ - '$(self.nameroot).idx6$(self.nameext)'
+ - '${ return [self.basename+".idx7", inputs.secondfile]; }'
+ - "_idx8"
- id: search
class: CommandLineTool
- baseCommand: python2
+ baseCommand: python
requirements:
- class: InlineJavascriptRequirement
+ hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
inputs:
file:
type: File
@@ -47,9 +57,11 @@ $graph:
secondaryFiles:
- ".idx1"
- "^.idx2"
- - '$(self.location+".idx3")'
- - '$({"location": self.location+".idx4", "class": "File"})'
- - '${ return self.location+".idx5"; }'
+ - '$(self.basename).idx3'
+ - '${ return self.basename+".idx4"; }'
+ - '$(self.nameroot).idx6$(self.nameext)'
+ - '${ return [self.basename+".idx7"]; }'
+ - "_idx8"
search.py:
type: File
default:
@@ -72,6 +84,7 @@ $graph:
class: Workflow
inputs:
infile: File
+ secondfile: File
term: string
outputs:
outfile:
@@ -85,12 +98,13 @@ $graph:
index:
run: "#index"
in:
- file: "#main/infile"
+ file: infile
+ secondfile: secondfile
out: [result]
search:
run: "#search"
in:
- file: "#main/index/result"
- term: "#main/term"
+ file: index/result
+ term: term
out: [result]
diff --git a/v1.0/v1.0/shellchar.cwl b/v1.0/v1.0/shellchar.cwl
new file mode 100644
index 000000000..fc32afd1d
--- /dev/null
+++ b/v1.0/v1.0/shellchar.cwl
@@ -0,0 +1,13 @@
+#!/usr/bin/env cwl-runner
+class: CommandLineTool
+cwlVersion: v1.0
+doc: |
+ Ensure that arguments containing shell directives are not interpreted and
+ that `shellQuote: false` has no effect when ShellCommandRequirement is not in
+ effect.
+inputs: []
+outputs:
+ stdout_file: stdout
+ stderr_file: stderr
+baseCommand: echo
+arguments: [{valueFrom: "foo 1>&2", shellQuote: false}]
diff --git a/v1.0/v1.0/shellchar2.cwl b/v1.0/v1.0/shellchar2.cwl
new file mode 100644
index 000000000..cb0b1ce78
--- /dev/null
+++ b/v1.0/v1.0/shellchar2.cwl
@@ -0,0 +1,14 @@
+#!/usr/bin/env cwl-runner
+class: CommandLineTool
+cwlVersion: v1.0
+doc: |
+ Ensure that `shellQuote: true` is the default behavior when
+ ShellCommandRequirement is in effect.
+requirements:
+ ShellCommandRequirement: {}
+inputs: []
+outputs:
+ stdout_file: stdout
+ stderr_file: stderr
+baseCommand: echo
+arguments: ["foo 1>&2"]
diff --git a/v1.0/v1.0/shelltest.cwl b/v1.0/v1.0/shelltest.cwl
deleted file mode 100644
index 2360a4f61..000000000
--- a/v1.0/v1.0/shelltest.cwl
+++ /dev/null
@@ -1,20 +0,0 @@
-class: CommandLineTool
-cwlVersion: v1.0
-doc: "Reverse each line using the `rev` command then sort."
-requirements:
- - class: ShellCommandRequirement
-inputs:
- - id: input
- type: File
-outputs:
- - id: output
- type: File
- outputBinding:
- glob: output.txt
-
-arguments:
- - rev
- - {valueFrom: $(inputs.input)}
- - {valueFrom: " | ", shellQuote: false}
- - sort
- - {valueFrom: "> output.txt", shellQuote: false}
diff --git a/v1.0/v1.0/special_file b/v1.0/v1.0/special_file
new file mode 100644
index 000000000..f70d7bba4
--- /dev/null
+++ b/v1.0/v1.0/special_file
@@ -0,0 +1 @@
+42
\ No newline at end of file
diff --git a/v1.0/v1.0/stage-unprovided-file.cwl b/v1.0/v1.0/stage-unprovided-file.cwl
new file mode 100644
index 000000000..e82784ee7
--- /dev/null
+++ b/v1.0/v1.0/stage-unprovided-file.cwl
@@ -0,0 +1,25 @@
+cwlVersion: v1.0
+class: CommandLineTool
+
+requirements: { InlineJavascriptRequirement: {} }
+
+inputs:
+ - id: infile
+ type: File?
+ inputBinding:
+ prefix: -cfg
+ valueFrom: $(self.basename)
+ - id: "args.py"
+ type: File
+ default:
+ class: File
+ location: args.py
+ inputBinding:
+ position: -1
+
+baseCommand: python
+
+outputs:
+- id: args
+ type: string[]
+
diff --git a/v1.0/v1.0/stagefile.cwl b/v1.0/v1.0/stagefile.cwl
index 242f10fe6..ec0825a3f 100644
--- a/v1.0/v1.0/stagefile.cwl
+++ b/v1.0/v1.0/stagefile.cwl
@@ -1,5 +1,8 @@
class: CommandLineTool
cwlVersion: v1.0
+hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
requirements:
InitialWorkDirRequirement:
listing:
diff --git a/v1.0/v1.0/steplevel-resreq.cwl b/v1.0/v1.0/steplevel-resreq.cwl
new file mode 100644
index 000000000..b2b24c5fe
--- /dev/null
+++ b/v1.0/v1.0/steplevel-resreq.cwl
@@ -0,0 +1,33 @@
+#!/usr/bin/env cwl-runner
+class: Workflow
+cwlVersion: v1.0
+
+requirements:
+ ResourceRequirement:
+ coresMin: 4
+ coresMax: 4
+
+inputs: []
+
+steps:
+ step1:
+ requirements:
+ ResourceRequirement:
+ coresMin: 1
+ coresMax: 1
+ run:
+ class: CommandLineTool
+ inputs: []
+ outputs:
+ output:
+ type: stdout
+ baseCommand: echo
+ stdout: cores.txt
+ arguments: [ $(runtime.cores) ]
+ in: []
+ out: [output]
+
+outputs:
+ out:
+ type: File
+ outputSource: step1/output
diff --git a/v1.0/v1.0/string-job.json b/v1.0/v1.0/string-job.json
new file mode 100644
index 000000000..ff89222db
--- /dev/null
+++ b/v1.0/v1.0/string-job.json
@@ -0,0 +1,3 @@
+{
+ "message": "hello"
+}
\ No newline at end of file
diff --git a/v1.0/v1.0/sum-job.json b/v1.0/v1.0/sum-job.json
new file mode 100644
index 000000000..64e29bfa1
--- /dev/null
+++ b/v1.0/v1.0/sum-job.json
@@ -0,0 +1,4 @@
+{
+ "int_1": 5,
+ "int_2": 7
+}
\ No newline at end of file
diff --git a/v1.0/v1.0/sum-wf.cwl b/v1.0/v1.0/sum-wf.cwl
new file mode 100644
index 000000000..566957b64
--- /dev/null
+++ b/v1.0/v1.0/sum-wf.cwl
@@ -0,0 +1,49 @@
+cwlVersion: v1.0
+class: Workflow
+
+requirements:
+ - class: StepInputExpressionRequirement
+ - class: MultipleInputFeatureRequirement
+ - class: InlineJavascriptRequirement
+
+inputs:
+ int_1:
+ type:
+ - int
+ - string
+ int_2:
+ type:
+ - int
+ - string
+
+outputs:
+ result:
+ type: int
+ outputSource: sum/result
+
+steps:
+ sum:
+ in:
+ data:
+ source: [int_1, int_2]
+ valueFrom: |
+ ${
+ var sum = 0;
+ for (var i = 0; i < self.length; i++){
+ sum += self[i];
+ };
+ return sum;
+ }
+ out:
+ - result
+ run:
+ class: ExpressionTool
+ inputs:
+ data:
+ type: int
+ outputs:
+ result: int
+ expression: |
+ ${
+ return {"result": inputs.data};
+ }
diff --git a/v1.0/v1.0/template-tool.cwl b/v1.0/v1.0/template-tool.cwl
index 28816ee8a..d70085fd3 100755
--- a/v1.0/v1.0/template-tool.cwl
+++ b/v1.0/v1.0/template-tool.cwl
@@ -5,14 +5,14 @@ requirements:
- class: InlineJavascriptRequirement
expressionLib:
- { $include: underscore.js }
- - "var t = function(s) { return _.template(s)({'inputs': inputs}); };"
+ - "var t = function(s) { return _.template(s, {variable: 'data'})({'inputs': inputs}); };"
- class: InitialWorkDirRequirement
listing:
- entryname: foo.txt
- entry: $(t("The file is <%= inputs.file1.path.split('/').slice(-1)[0] %>\n"))
+ entry: $(t("The file is <%= data.inputs.file1.path.split('/').slice(-1)[0] %>\n"))
hints:
DockerRequirement:
- dockerPull: "debian:8"
+ dockerPull: "debian:stretch-slim"
inputs:
- id: file1
type: File
diff --git a/v1.0/v1.0/test-cwl-out.cwl b/v1.0/v1.0/test-cwl-out.cwl
index b79b648ad..c84adb771 100644
--- a/v1.0/v1.0/test-cwl-out.cwl
+++ b/v1.0/v1.0/test-cwl-out.cwl
@@ -4,7 +4,7 @@ requirements:
- class: ShellCommandRequirement
hints:
DockerRequirement:
- dockerPull: "debian:wheezy"
+ dockerPull: "debian:stretch-slim"
inputs: []
diff --git a/v1.0/v1.0/test-cwl-out2.cwl b/v1.0/v1.0/test-cwl-out2.cwl
index 049fee029..23724a3cf 100644
--- a/v1.0/v1.0/test-cwl-out2.cwl
+++ b/v1.0/v1.0/test-cwl-out2.cwl
@@ -4,7 +4,7 @@ requirements:
- class: ShellCommandRequirement
hints:
DockerRequirement:
- dockerPull: "debian:wheezy"
+ dockerPull: "debian:stretch-slim"
inputs: []
diff --git a/v1.0/v1.0/tmap-tool.cwl b/v1.0/v1.0/tmap-tool.cwl
index 505d60036..734487a76 100755
--- a/v1.0/v1.0/tmap-tool.cwl
+++ b/v1.0/v1.0/tmap-tool.cwl
@@ -3,7 +3,12 @@
"cwlVersion": "v1.0",
"class": "CommandLineTool",
-
+ "hints": [
+ {
+ "class": "DockerRequirement",
+ "dockerPull": "python:2-slim"
+ }
+ ],
"inputs": [
{
"id": "reads",
diff --git a/v1.0/v1.0/valueFrom-constant.cwl b/v1.0/v1.0/valueFrom-constant.cwl
new file mode 100644
index 000000000..9d17a5492
--- /dev/null
+++ b/v1.0/v1.0/valueFrom-constant.cwl
@@ -0,0 +1,31 @@
+class: CommandLineTool
+cwlVersion: v1.0
+
+requirements:
+ - class: InlineJavascriptRequirement
+
+hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
+
+inputs:
+ - id: array_input
+ type:
+ - type: array
+ items: File
+ inputBinding:
+ valueFrom: replacementValue
+
+ - id: args.py
+ type: File
+ default:
+ class: File
+ location: args.py
+ inputBinding:
+ position: -1
+
+outputs:
+ - id: args
+ type: string[]
+
+baseCommand: python
diff --git a/v1.0/v1.0/vf-concat.cwl b/v1.0/v1.0/vf-concat.cwl
new file mode 100644
index 000000000..5cd99d203
--- /dev/null
+++ b/v1.0/v1.0/vf-concat.cwl
@@ -0,0 +1,19 @@
+cwlVersion: v1.0
+class: CommandLineTool
+requirements:
+ - class: InlineJavascriptRequirement
+
+baseCommand: echo
+inputs:
+ - id: parameter
+ type: string?
+ inputBinding:
+ valueFrom: $("a ")$("string")
+outputs:
+ out:
+ type: string
+ outputBinding:
+ glob: output.txt
+ loadContents: true
+ outputEval: $(self[0].contents)
+stdout: output.txt
diff --git a/v1.0/v1.0/writable-dir-docker.cwl b/v1.0/v1.0/writable-dir-docker.cwl
new file mode 100644
index 000000000..a60851c27
--- /dev/null
+++ b/v1.0/v1.0/writable-dir-docker.cwl
@@ -0,0 +1,23 @@
+cwlVersion: v1.0
+
+class: CommandLineTool
+
+requirements:
+ - class: InlineJavascriptRequirement
+ - class: InitialWorkDirRequirement
+ listing:
+ - entryname: emptyWritableDir
+ entry: "$({class: 'Directory', listing: []})"
+ writable: true
+
+hints:
+ - class: DockerRequirement
+ dockerPull: alpine
+
+inputs: []
+outputs:
+ out:
+ type: Directory
+ outputBinding:
+ glob: emptyWritableDir
+arguments: [touch, emptyWritableDir/blurg]
diff --git a/v1.0/v1.0/writable-dir.cwl b/v1.0/v1.0/writable-dir.cwl
new file mode 100644
index 000000000..52cf488a4
--- /dev/null
+++ b/v1.0/v1.0/writable-dir.cwl
@@ -0,0 +1,16 @@
+cwlVersion: v1.0
+class: CommandLineTool
+requirements:
+ InlineJavascriptRequirement: {}
+ InitialWorkDirRequirement:
+ listing:
+ - entryname: emptyWritableDir
+ writable: true
+ entry: "$({class: 'Directory', listing: []})"
+inputs: []
+outputs:
+ out:
+ type: Directory
+ outputBinding:
+ glob: emptyWritableDir
+arguments: [touch, emptyWritableDir/blurg]
diff --git a/v1.1.0-dev1/CommandLineTool.yml b/v1.1.0-dev1/CommandLineTool.yml
index bbf0d33e5..bb3ace891 100644
--- a/v1.1.0-dev1/CommandLineTool.yml
+++ b/v1.1.0-dev1/CommandLineTool.yml
@@ -36,57 +36,40 @@ $graph:
- {$include: intro.md}
- |
- ## Introduction to v1.1.0-dev1
-
- This is the in progress first development version of the first
- maintenance release of the CWL CommandLineTool specification.
- Version 1.1 introduces the followings additions:
-
- * Addition of `stdin` type shortcut for `CommandInputParamater`s.
-
- ## Introduction to v1.0
-
- This specification represents the first full release from the CWL group.
- Since draft-3, version 1.0 introduces the following changes and additions:
-
- * The [Directory](#Directory) type.
- * Syntax simplifcations: denoted by the `map<>` syntax. Example: inputs
- contains a list of items, each with an id. Now one can specify
- a mapping of that identifier to the corresponding
- `CommandInputParamater`.
- ```
- inputs:
- - id: one
- type: string
- doc: First input parameter
- - id: two
- type: int
- doc: Second input parameter
- ```
- can be
- ```
- inputs:
- one:
- type: string
- doc: First input parameter
- two:
- type: int
- doc: Second input parameter
- ```
- * [InitialWorkDirRequirement](#InitialWorkDirRequirement): list of
- files and subdirectories to be present in the output directory prior
- to execution.
- * Shortcuts for specifying the standard [output](#stdout) and/or
- [error](#stderr) streams as a (streamable) File output.
- * [SoftwareRequirement](#SoftwareRequirement) for describing software
- dependencies of a tool.
- * The common `description` field has been renamed to `doc`.
-
- ## Errata
-
- Post v1.0 release changes to the spec.
-
- * 13 July 2016: Mark `baseCommand` as optional and update descriptive text.
+ ## Introduction to CWL Command Line Tool standard v1.1.0-dev1
+
+ This specification represents the latest development release from the
+ CWL group. Since the v1.0 release, v1.1 introduces the
+ following updates to the CWL Command Line Tool standard.
+ Documents should to use `cwlVersion: v1.1.0-dev1` to make use of new
+ syntax and features introduced in v1.1.0-dev1. Existing v1.0 documents
+ should be trivially updatav1.1.0-dev1ble by changing `cwlVersion`, however
+ CWL documents that relied on previously undefined or
+ underspecified behavior may have slightly different behavior in
+ v1.1.0-dev1.
+
+ ## Changelog
+
+ * Default values for some fields are now expressed in the schema.
+ * When defining record types with `CommandInputRecordSchema`, fields of
+ type `File` may now include `format`, `loadContents`,
+ `secondaryFiles` and `streamable`.
+ * `CommandInputRecordSchema`, `CommandInputEnumSchema`, and
+ `CommandInputArraySchema` now have an optional `doc` field.
+ * `inputBinding` has been added as an optional field for
+ `CommandInputRecordSchema` (was previously in CWL `draft-3` but
+ disappeared in `v1.0`)
+ * Any `doc` field may be an array of strings in addition to the
+ previously allowed single string.
+ * Addition of `stdin` type shortcut for `CommandInputParameter`.
+ * Clarify that the designated output directory should be empty
+ except for files or directories specified using
+ [InitialWorkDirRequirement](#InitialWorkDirRequirement).
+ * Clarify semantics of `shellQuote`
+ * Expressions are now allowed to evaluate to `null` or `Dirent` in
+ [InitialWorkDirRequirement.listing](#InitialWorkDirRequirement).
+
+ See also the [CWL Workflow Description, v1.1.0-dev1 changelog](Workflow.html#Changelog).
## Purpose
@@ -140,6 +123,7 @@ $graph:
- type: record
name: CommandLineBinding
extends: InputBinding
+ docParent: "#CommandInputParameter"
doc: |
When listed under `inputBinding` in the input schema, the term
@@ -182,6 +166,7 @@ $graph:
doc: "Command line prefix to add before the value."
- name: separate
type: boolean?
+ default: true
doc: |
If true (default), then the prefix and value must be added as separate
command line arguments; if false, prefix and value must be concatenated
@@ -204,31 +189,40 @@ $graph:
If `valueFrom` is an expression, evaluate the expression to yield the
actual value to use to build the command line and apply the binding
rules above. If the inputBinding is associated with an input
- parameter, the value of `self` in the expression will be the value of the
- input parameter.
+ parameter, the value of `self` in the expression will be the value of
+ the input parameter. Input parameter defaults (as specified by the
+ `InputParameter.default` field) must be applied before evaluating the
+ expression.
When a binding is part of the `CommandLineTool.arguments` field,
the `valueFrom` field is required.
+
- name: shellQuote
type: boolean?
+ default: true
doc: |
If `ShellCommandRequirement` is in the requirements for the current command,
this controls whether the value is quoted on the command line (default is true).
Use `shellQuote: false` to inject metacharacters for operations such as pipes.
+ If `shellQuote` is true or null, the implementation must not
+ permit interpretation of any shell metacharacters or directives.
+
+
- type: record
name: CommandOutputBinding
- extends: OutputBinding
+ extends: LoadContents
doc: |
Describes how to generate an output parameter based on the files produced
by a CommandLineTool.
- The output parameter is generated by applying these operations in
- the following order:
+ The output parameter value is generated by applying these operations in the
+ following order:
- glob
- loadContents
- outputEval
+ - secondaryFiles
fields:
- name: glob
type:
@@ -244,15 +238,6 @@ $graph:
return a string or an array of strings, which will then be evaluated as
one or more glob patterns. Must only match and return files which
actually exist.
- - name: loadContents
- type:
- - "null"
- - boolean
- jsonldPredicate: "cwl:loadContents"
- doc: |
- For each file matched in `glob`, read up to
- the first 64 KiB of text from the file and place it in the `contents`
- field of the file object for manipulation by `outputEval`.
- name: outputEval
type:
- "null"
@@ -267,10 +252,17 @@ $graph:
the File objects must include up to the first 64 KiB of file contents
in the `contents` field.
+- name: CommandLineBindable
+ type: record
+ fields:
+ inputBinding:
+ type: CommandLineBinding?
+ jsonldPredicate: "cwl:inputBinding"
+ doc: Describes how to turn this object into command line arguments.
- name: CommandInputRecordField
type: record
- extends: InputRecordField
+ extends: [InputRecordField, CommandLineBindable]
specialize:
- specializeFrom: InputRecordSchema
specializeTo: CommandInputRecordSchema
@@ -284,15 +276,17 @@ $graph:
- name: CommandInputRecordSchema
type: record
- extends: InputRecordSchema
+ extends: [InputRecordSchema, CommandInputSchema, CommandLineBindable]
specialize:
- specializeFrom: InputRecordField
specializeTo: CommandInputRecordField
+ - specializeFrom: InputBinding
+ specializeTo: CommandLineBinding
- name: CommandInputEnumSchema
type: record
- extends: InputEnumSchema
+ extends: [InputEnumSchema, CommandInputSchema, CommandLineBindable]
specialize:
- specializeFrom: InputBinding
specializeTo: CommandLineBinding
@@ -300,7 +294,7 @@ $graph:
- name: CommandInputArraySchema
type: record
- extends: InputArraySchema
+ extends: [InputArraySchema, CommandInputSchema, CommandLineBindable]
specialize:
- specializeFrom: InputRecordSchema
specializeTo: CommandInputRecordSchema
@@ -322,8 +316,13 @@ $graph:
specializeTo: CommandOutputEnumSchema
- specializeFrom: OutputArraySchema
specializeTo: CommandOutputArraySchema
- - specializeFrom: OutputBinding
- specializeTo: CommandOutputBinding
+ fields:
+ - name: outputBinding
+ type: CommandOutputBinding?
+ jsonldPredicate: "cwl:outputBinding"
+ doc: |
+ Describes how to generate this output object based on the files
+ produced by a CommandLineTool
- name: CommandOutputRecordSchema
@@ -344,8 +343,6 @@ $graph:
specializeTo: CommandOutputEnumSchema
- specializeFrom: OutputArraySchema
specializeTo: CommandOutputArraySchema
- - specializeFrom: OutputBinding
- specializeTo: CommandOutputBinding
- name: CommandOutputArraySchema
@@ -358,17 +355,12 @@ $graph:
specializeTo: CommandOutputEnumSchema
- specializeFrom: OutputArraySchema
specializeTo: CommandOutputArraySchema
- - specializeFrom: OutputBinding
- specializeTo: CommandOutputBinding
- type: record
name: CommandInputParameter
extends: InputParameter
doc: An input parameter for a CommandLineTool.
- specialize:
- - specializeFrom: InputBinding
- specializeTo: CommandLineBinding
fields:
- name: type
type:
@@ -393,20 +385,22 @@ $graph:
typeDSL: True
doc: |
Specify valid types of data that may be assigned to this parameter.
+ - name: inputBinding
+ type: CommandLineBinding?
+ doc: |
+ Describes how to turns the input parameters of a process into
+ command line arguments.
+ jsonldPredicate: "cwl:inputBinding"
- type: record
name: CommandOutputParameter
extends: OutputParameter
doc: An output parameter for a CommandLineTool.
- specialize:
- - specializeFrom: OutputBinding
- specializeTo: CommandOutputBinding
fields:
- name: type
type:
- "null"
- CWLType
- - stdin
- stdout
- stderr
- CommandOutputRecordSchema
@@ -427,6 +421,11 @@ $graph:
typeDSL: True
doc: |
Specify valid types of data that may be assigned to this parameter.
+ - name: outputBinding
+ type: CommandOutputBinding?
+ jsonldPredicate: "cwl:outputBinding"
+ doc: Describes how to generate this output object based on the files
+ produced by a CommandLineTool
- name: stdin
type: enum
@@ -464,8 +463,8 @@ $graph:
The following
```
outputs:
- an_output_name:
- type: stdout
+ an_output_name:
+ type: stdout
stdout: a_stdout_file
```
@@ -746,26 +745,60 @@ $graph:
fields:
- name: package
type: string
- doc: "The common name of the software to be configured."
+ doc: |
+ The name of the software to be made available. If the name is
+ common, inconsistent, or otherwise ambiguous it should be combined with
+ one or more identifiers in the `specs` field.
- name: version
type: string[]?
- doc: "The (optional) version of the software to configured."
+ doc: |
+ The (optional) versions of the software that are known to be
+ compatible.
- name: specs
type: string[]?
+ jsonldPredicate: {_type: "@id", noLinkCheck: true}
doc: |
- Must be one or more IRIs identifying resources for installing or
- enabling the software. Implementations may provide resolvers which map
- well-known software spec IRIs to some configuration action.
-
- For example, an IRI `https://packages.debian.org/jessie/bowtie` could
- be resolved with `apt-get install bowtie`. An IRI
- `https://anaconda.org/bioconda/bowtie` could be resolved with `conda
+ One or more [IRI](https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier)s
+ identifying resources for installing or enabling the software named in
+ the `package` field. Implementations may provide resolvers which map
+ these software identifer IRIs to some configuration action; or they can
+ use only the name from the `package` field on a best effort basis.
+
+ For example, the IRI https://packages.debian.org/bowtie could
+ be resolved with `apt-get install bowtie`. The IRI
+ https://anaconda.org/bioconda/bowtie could be resolved with `conda
install -c bioconda bowtie`.
- Tools may also provide IRIs to index entries such as
- [RRID](http://www.identifiers.org/rrid/), such as
- `http://identifiers.org/rrid/RRID:SCR_005476`
-
+ IRIs can also be system independent and used to map to a specific
+ software installation or selection mechanism.
+ Using [RRID](https://www.identifiers.org/rrid/) as an example:
+ https://identifiers.org/rrid/RRID:SCR_005476
+ could be fulfilled using the above mentioned Debian or bioconda
+ package, a local installation managed by [Environement Modules](http://modules.sourceforge.net/),
+ or any other mechanism the platform chooses. IRIs can also be from
+ identifer sources that are discipline specific yet still system
+ independent. As an example, the equivalent [ELIXIR Tools and Data
+ Service Registry](https://bio.tools) IRI to the previous RRID example is
+ https://bio.tools/tool/bowtie2/version/2.2.8.
+ If supported by a given registry, implementations are encouraged to
+ query these system independent sofware identifier IRIs directly for
+ links to packaging systems.
+
+ A site specific IRI can be listed as well. For example, an academic
+ computing cluster using Environement Modules could list the IRI
+ `https://hpc.example.edu/modules/bowtie-tbb/1.22` to indicate that
+ `module load bowtie-tbb/1.1.2` should be executed to make available
+ `bowtie` version 1.1.2 compiled with the TBB library prior to running
+ the accompanying Workflow or CommandLineTool. Note that the example IRI
+ is specific to a particular institution and computing environment as
+ the Environment Modules system does not have a common namespace or
+ standardized naming convention.
+
+ This last example is the least portable and should only be used if
+ mechanisms based off of the `package` field or more generic IRIs are
+ unavailable or unsuitable. While harmless to other sites, site specific
+ software IRIs should be left out of shared CWL descriptions to avoid
+ clutter.
- name: Dirent
type: record
@@ -781,7 +814,8 @@ $graph:
_id: cwl:entryname
doc: |
The name of the file or subdirectory to create in the output directory.
- If `entry` is a File or Directory, this overrides `basename`. Optional.
+ If `entry` is a File or Directory, the `entryname` field overrides the value
+ of `basename` of the File or Directory object. Optional.
- name: entry
type: [string, Expression]
jsonldPredicate:
@@ -803,6 +837,7 @@ $graph:
file.
- name: writable
type: boolean?
+ default: false
doc: |
If true, the file or directory must be writable by the tool. Changes
to the file or directory must be isolated and not visible by any other
@@ -810,6 +845,9 @@ $graph:
the original file or directory. Default false (files and directories
read-only by default).
+ A directory marked as `writable: true` implies that all files and
+ subdirectories are recursively writable as well.
+
- name: InitialWorkDirRequirement
type: record
@@ -828,7 +866,7 @@ $graph:
- name: listing
type:
- type: array
- items: [File, Directory, Dirent, string, Expression]
+ items: ["null", File, Directory, Dirent, string, Expression]
- string
- Expression
jsonldPredicate:
@@ -838,7 +876,14 @@ $graph:
designated output directory prior to executing the command line tool.
May be an expression. If so, the expression return value must validate
- as `{type: array, items: [File, Directory]}`.
+ as `{type: array, items: ["null", File, Directory, Dirent]}`.
+
+ Files or Directories which are listed in the input parameters and
+ appear in the `InitialWorkDirRequirement` listing must have their
+ `path` set to their staged location in the designated output directory.
+ If the same File or Directory appears more than once in the
+ `InitialWorkDirRequirement` listing, the implementation must choose
+ exactly one value for `path`; how this value is chosen is undefined.
- name: EnvVarRequirement
@@ -945,3 +990,84 @@ $graph:
- name: outdirMax
type: ["null", long, string, Expression]
doc: Maximum reserved filesystem based storage for the designated output directory, in mebibytes (2**20)
+
+
+- type: record
+ name: TimeLimit
+ extends: ProcessRequirement
+ doc: |
+ Set an upper limit on the execution time of a CommandLineTool or
+ ExpressionTool. A tool execution which exceeds the time limit may
+ be preemptively terminated and considered failed. May also be
+ used by batch systems to make scheduling decisions.
+ mapSubject: timeLimit
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'TimeLimit'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: timeLimit
+ type: [long, string, Expression]
+ doc: |
+ The time limit, in seconds. A time limit of zero means no
+ time limit. Negative time limits are an error.
+
+
+- type: record
+ name: WorkReuse
+ extends: ProcessRequirement
+ doc: |
+ For implementations that support reusing output from past work (on
+ the assumption that same code and same input produce same
+ results), control whether to enable or disable the reuse behavior
+ for a particular tool or step (to accomodate situations where that
+ assumption is incorrect). A reused step is not executed but
+ instead returns the same output as the original execution.
+
+ If `enableReuse` is not specified, correct tools should assume it
+ is enabled by default.
+ mapSubject: enableReuse
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'WorkReuse'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: enableReuse
+ type: [boolean, string, Expression]
+ default: true
+
+
+- type: record
+ name: NetworkAccess
+ extends: ProcessRequirement
+ doc: |
+ Indicate whether a process requires outgoing IPv4/IPv6 network
+ access. Choice of IPv4 or IPv6 is implementation and site
+ specific, correct tools must support both.
+
+ If `networkAccess` is false or not specified, tools must not
+ assume network access, except for localhost (the loopback device).
+
+ If `networkAccess` is true, the tool must be able to make outgoing
+ connections to network resources. Resources may be on a private
+ subnet or the public Internet. However, implementations and sites
+ may apply their own security policies to restrict what is
+ accessible by the tool.
+
+ Enabling network access does not imply a publically routable IP
+ address or the ability to accept inbound connections.
+
+ mapSubject: networkAccess
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'NetworkAccess'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: networkAccess
+ type: [boolean, string, Expression]
diff --git a/v1.1.0-dev1/Process.yml b/v1.1.0-dev1/Process.yml
index 4208b1430..4d2aafb39 100644
--- a/v1.1.0-dev1/Process.yml
+++ b/v1.1.0-dev1/Process.yml
@@ -54,9 +54,73 @@ $graph:
type: record
docParent: "#CWLType"
doc: |
- Represents a file (or group of files if `secondaryFiles` is specified) that
- must be accessible by tools using standard POSIX file system call API such as
+ Represents a file (or group of files when `secondaryFiles` is provided) that
+ will be accessible by tools using standard POSIX file system call API such as
open(2) and read(2).
+
+ Files are represented as objects with `class` of `File`. File objects have
+ a number of properties that provide metadata about the file.
+
+ The `location` property of a File is a URI that uniquely identifies the
+ file. Implementations must support the file:// URI scheme and may support
+ other schemes such as http://. The value of `location` may also be a
+ relative reference, in which case it must be resolved relative to the URI
+ of the document it appears in. Alternately to `location`, implementations
+ must also accept the `path` property on File, which must be a filesystem
+ path available on the same host as the CWL runner (for inputs) or the
+ runtime environment of a command line tool execution (for command line tool
+ outputs).
+
+ If no `location` or `path` is specified, a file object must specify
+ `contents` with the UTF-8 text content of the file. This is a "file
+ literal". File literals do not correspond to external resources, but are
+ created on disk with `contents` with when needed for a executing a tool.
+ Where appropriate, expressions can return file literals to define new files
+ on a runtime. The maximum size of `contents` is 64 kilobytes.
+
+ The `basename` property defines the filename on disk where the file is
+ staged. This may differ from the resource name. If not provided,
+ `basename` must be computed from the last path part of `location` and made
+ available to expressions.
+
+ The `secondaryFiles` property is a list of File or Directory objects that
+ must be staged in the same directory as the primary file. It is an error
+ for file names to be duplicated in `secondaryFiles`.
+
+ The `size` property is the size in bytes of the File. It must be computed
+ from the resource and made available to expressions. The `checksum` field
+ contains a cryptographic hash of the file content for use it verifying file
+ contents. Implementations may, at user option, enable or disable
+ computation of the `checksum` field for performance or other reasons.
+ However, the ability to compute output checksums is required to pass the
+ CWL conformance test suite.
+
+ When executing a CommandLineTool, the files and secondary files may be
+ staged to an arbitrary directory, but must use the value of `basename` for
+ the filename. The `path` property must be file path in the context of the
+ tool execution runtime (local to the compute node, or within the executing
+ container). All computed properties should be available to expressions.
+ File literals also must be staged and `path` must be set.
+
+ When collecting CommandLineTool outputs, `glob` matching returns file paths
+ (with the `path` property) and the derived properties. This can all be
+ modified by `outputEval`. Alternately, if the file `cwl.output.json` is
+ present in the output, `outputBinding` is ignored.
+
+ File objects in the output must provide either a `location` URI or a `path`
+ property in the context of the tool execution runtime (local to the compute
+ node, or within the executing container).
+
+ When evaluating an ExpressionTool, file objects must be referenced via
+ `location` (the expression tool does not have access to files on disk so
+ `path` is meaningless) or as file literals. It is legal to return a file
+ object with an existing `location` but a different `basename`. The
+ `loadContents` field of ExpressionTool inputs behaves the same as on
+ CommandLineTool inputs, however it is not meaningful on the outputs.
+
+ An ExpressionTool may forward file references from input to output by using
+ the same value for `location`.
+
fields:
- name: class
type:
@@ -175,12 +239,12 @@ $graph:
items: [File, Directory]
jsonldPredicate: "cwl:secondaryFiles"
doc: |
- A list of additional files that are associated with the primary file
- and must be transferred alongside the primary file. Examples include
- indexes of the primary file, or external references which must be
- included when loading primary document. A file object listed in
- `secondaryFiles` may itself include `secondaryFiles` for which the same
- rules apply.
+ A list of additional files or directories that are associated with the
+ primary file and must be transferred alongside the primary file.
+ Examples include indexes of the primary file, or external references
+ which must be included when loading primary document. A file object
+ listed in `secondaryFiles` may itself include `secondaryFiles` for
+ which the same rules apply.
- name: format
type: string?
jsonldPredicate:
@@ -222,6 +286,49 @@ $graph:
docAfter: "#File"
doc: |
Represents a directory to present to a command line tool.
+
+ Directories are represented as objects with `class` of `Directory`. Directory objects have
+ a number of properties that provide metadata about the directory.
+
+ The `location` property of a Directory is a URI that uniquely identifies
+ the directory. Implementations must support the file:// URI scheme and may
+ support other schemes such as http://. Alternately to `location`,
+ implementations must also accept the `path` property on Direcotry, which
+ must be a filesystem path available on the same host as the CWL runner (for
+ inputs) or the runtime environment of a command line tool execution (for
+ command line tool outputs).
+
+ A Directory object may have a `listing` field. This is a list of File and
+ Directory objects that are contained in the Directory. For each entry in
+ `listing`, the `basename` property defines the name of the File or
+ Subdirectory when staged to disk. If `listing` is not provided, the
+ implementation must have some way of fetching the Directory listing at
+ runtime based on the `location` field.
+
+ If a Directory does not have `location`, it is a Directory literal. A
+ Directory literal must provide `listing`. Directory literals must be
+ created on disk at runtime as needed.
+
+ The resources in a Directory literal do not need to have any implied
+ relationship in their `location`. For example, a Directory listing may
+ contain two files located on different hosts. It is the responsibility of
+ the runtime to ensure that those files are staged to disk appropriately.
+ Secondary files associated with files in `listing` must also be staged to
+ the same Directory.
+
+ When executing a CommandLineTool, Directories must be recursively staged
+ first and have local values of `path` assigend.
+
+ Directory objects in CommandLineTool output must provide either a
+ `location` URI or a `path` property in the context of the tool execution
+ runtime (local to the compute node, or within the executing container).
+
+ An ExpressionTool may forward file references from input to output by using
+ the same value for `location`.
+
+ Name conflicts (the same `basename` appearing multiple times in `listing`
+ or in any entry in `secondaryFiles` in the listing) is a fatal error.
+
fields:
- name: class
type:
@@ -305,7 +412,7 @@ $graph:
jsonldPredicate:
_id: "cwl:listing"
-- name: SchemaBase
+- name: Labeled
type: record
abstract: true
fields:
@@ -317,15 +424,38 @@ $graph:
doc: "A short, human-readable label of this object."
-- name: Parameter
+- name: Identified
type: record
- extends: SchemaBase
abstract: true
- doc: |
- Define an input or output parameter to a process.
+ fields:
+ - name: id
+ type: string?
+ jsonldPredicate: "@id"
+ doc: "The unique identifier for this object."
+
+
+- name: LoadContents
+ type: record
+ abstract: true
+ fields:
+ - name: loadContents
+ type:
+ - "null"
+ - boolean
+ jsonldPredicate: "cwl:loadContents"
+ doc: |
+ Only valid when `type: File` or is an array of `items: File`.
+
+ Read up to the first 64 KiB of text from the file and place it in the
+ "contents" field of the file object for use by expressions.
+
+- name: FieldBase
+ type: record
+ extends: Labeled
+ abstract: true
fields:
- - name: secondaryFiles
+ secondaryFiles:
type:
- "null"
- string
@@ -336,13 +466,29 @@ $graph:
doc: |
Only valid when `type: File` or is an array of `items: File`.
- Describes files that must be included alongside the primary file(s).
+ Provides a pattern or expression specifying files or directories that
+ must be included alongside the primary file. All listed secondary
+ files must be present. An implementation may fail workflow execution
+ if an expected secondary file does not exist.
If the value is an expression, the value of `self` in the expression
- must be the primary input or output File to which this binding applies.
-
- If the value is a string, it specifies that the following pattern
- should be applied to the primary file:
+ must be the primary input or output File object to which this binding
+ applies. The `basename`, `nameroot` and `nameext` fields must be
+ present in `self`. For `CommandLineTool` outputs the `path` field must
+ also be present. The expression must return a filename string relative
+ to the path to the primary File, a File or Directory object with either
+ `path` or `location` and `basename` fields set, or an array consisting
+ of strings or File or Directory objects. It is legal to reference an
+ unchanged File or Directory object taken from input as a secondaryFile.
+
+ To work on non-filename-preserving storage systems, portable tool
+ descriptions should avoid constructing new values from `location`, but
+ should construct relative references using `basename` or `nameroot`
+ instead.
+
+ If a value in `secondaryFiles` is a string that is not an expression,
+ it specifies that the following pattern should be applied to the path
+ of the primary file to yield a filename relative to the primary File:
1. If string begins with one or more caret `^` characters, for each
caret, remove the last file extension from the path (the last
@@ -350,7 +496,22 @@ $graph:
extensions, the path is unchanged.
2. Append the remainder of the string to the end of the file path.
- - name: format
+ streamable:
+ type: boolean?
+ doc: |
+ Only valid when `type: File` or is an array of `items: File`.
+
+ A value of `true` indicates that the file is read or written
+ sequentially without seeking. An implementation may use this flag to
+ indicate whether it is valid to stream file contents using a named
+ pipe. Default: `false`.
+
+
+- name: InputFormat
+ type: record
+ abstract: true
+ fields:
+ format:
type:
- "null"
- string
@@ -364,30 +525,38 @@ $graph:
doc: |
Only valid when `type: File` or is an array of `items: File`.
- For input parameters, this must be one or more IRIs of concept nodes
+ This must be one or more IRIs of concept nodes
that represents file formats which are allowed as input to this
parameter, preferrably defined within an ontology. If no ontology is
available, file formats may be tested by exact match.
- For output parameters, this is the file format that will be assigned to
- the output parameter.
- - name: streamable
- type: boolean?
+- name: OutputFormat
+ type: record
+ abstract: true
+ fields:
+ format:
+ type:
+ - "null"
+ - string
+ - Expression
+ jsonldPredicate:
+ _id: cwl:format
+ _type: "@id"
+ identity: true
doc: |
Only valid when `type: File` or is an array of `items: File`.
- A value of `true` indicates that the file is read or written
- sequentially without seeking. An implementation may use this flag to
- indicate whether it is valid to stream file contents using a named
- pipe. Default: `false`.
+ This is the file format that will be assigned to the output
+ File object.
- - name: doc
- type:
- - string?
- - string[]?
- doc: "A documentation string for this type, or an array of strings which should be concatenated."
- jsonldPredicate: "rdfs:comment"
+
+- name: Parameter
+ type: record
+ extends: [FieldBase, sld:Documented, Identified]
+ abstract: true
+ doc: |
+ Define an input or output parameter to a process.
- type: enum
@@ -403,7 +572,6 @@ $graph:
- name: InputBinding
type: record
- abstract: true
fields:
- name: loadContents
type:
@@ -411,32 +579,36 @@ $graph:
- boolean
jsonldPredicate: "cwl:loadContents"
doc: |
+ Use of `loadContents` in `InputBinding` is deprecated.
+ Preserved for v1.0 backwards compatability. Will be removed in
+ CWL v2.0. Use `InputParameter.loadContents` instead.
+
Only valid when `type: File` or is an array of `items: File`.
Read up to the first 64 KiB of text from the file and place it in the
"contents" field of the file object for use by expressions.
-- name: OutputBinding
- type: record
- abstract: true
-
-
- name: InputSchema
- extends: SchemaBase
+ extends: [Labeled, sld:Documented]
type: record
abstract: true
+ fields:
+ - name: name
+ type: string?
+ jsonldPredicate: "@id"
+ doc: "The identifier for this type"
- name: OutputSchema
- extends: SchemaBase
+ extends: [Labeled]
type: record
abstract: true
- name: InputRecordField
type: record
- extends: "sld:RecordField"
+ extends: [sld:RecordField, FieldBase, InputFormat, LoadContents]
specialize:
- specializeFrom: "sld:RecordSchema"
specializeTo: InputRecordSchema
@@ -446,14 +618,6 @@ $graph:
specializeTo: InputArraySchema
- specializeFrom: "sld:PrimitiveType"
specializeTo: CWLType
- fields:
- - name: inputBinding
- type: InputBinding?
- jsonldPredicate: "cwl:inputBinding"
- - name: label
- type: string?
- jsonldPredicate: "rdfs:label"
- doc: "A short, human-readable label of this process object."
- name: InputRecordSchema
@@ -467,10 +631,6 @@ $graph:
- name: InputEnumSchema
type: record
extends: ["sld:EnumSchema", InputSchema]
- fields:
- - name: inputBinding
- type: InputBinding?
- jsonldPredicate: "cwl:inputBinding"
- name: InputArraySchema
@@ -485,15 +645,11 @@ $graph:
specializeTo: InputArraySchema
- specializeFrom: "sld:PrimitiveType"
specializeTo: CWLType
- fields:
- - name: inputBinding
- type: InputBinding?
- jsonldPredicate: "cwl:inputBinding"
- name: OutputRecordField
type: record
- extends: "sld:RecordField"
+ extends: [sld:RecordField, FieldBase, OutputFormat]
specialize:
- specializeFrom: "sld:RecordSchema"
specializeTo: OutputRecordSchema
@@ -503,10 +659,6 @@ $graph:
specializeTo: OutputArraySchema
- specializeFrom: "sld:PrimitiveType"
specializeTo: CWLType
- fields:
- - name: outputBinding
- type: OutputBinding?
- jsonldPredicate: "cwl:outputBinding"
- name: OutputRecordSchema
@@ -522,10 +674,7 @@ $graph:
type: record
extends: ["sld:EnumSchema", OutputSchema]
docParent: "#OutputParameter"
- fields:
- - name: outputBinding
- type: OutputBinding?
- jsonldPredicate: "cwl:outputBinding"
+
- name: OutputArraySchema
type: record
@@ -540,77 +689,29 @@ $graph:
specializeTo: OutputArraySchema
- specializeFrom: "sld:PrimitiveType"
specializeTo: CWLType
- fields:
- - name: outputBinding
- type: OutputBinding?
- jsonldPredicate: "cwl:outputBinding"
- name: InputParameter
type: record
- extends: Parameter
+ abstract: true
+ extends: [Parameter, InputFormat, LoadContents]
fields:
- - name: id
- type: string
- jsonldPredicate: "@id"
- doc: "The unique identifier for this parameter object."
-
- - name: inputBinding
- type: InputBinding?
- jsonldPredicate: "cwl:inputBinding"
- doc: |
- Describes how to handle the inputs of a process and convert them
- into a concrete form for execution, such as command line parameters.
-
- name: default
type: Any?
jsonldPredicate:
- _id: cwl:default
+ _id: sld:default
noLinkCheck: true
doc: |
- The default value for this parameter if not provided in the input
- object.
+ The default value to use for this parameter if the parameter is missing
+ from the input object, or if the value of the parameter in the input
+ object is `null`. Default values are applied before evaluating expressions
+ (e.g. dependent `valueFrom` fields).
-- name: RegularInputParameter
- type: record
- extends: InputParameter
- fields:
- - name: type
- type:
- - "null"
- - CWLType
- - InputRecordSchema
- - InputEnumSchema
- - InputArraySchema
- - string
- - type: array
- items:
- - CWLType
- - InputRecordSchema
- - InputEnumSchema
- - InputArraySchema
- - string
- jsonldPredicate:
- "_id": "sld:type"
- "_type": "@vocab"
- refScope: 2
- typeDSL: True
- doc: |
- Specify valid types of data that may be assigned to this parameter.
- name: OutputParameter
type: record
- extends: Parameter
- fields:
- - name: id
- type: string
- jsonldPredicate: "@id"
- doc: "The unique identifier for this parameter object."
- - name: outputBinding
- type: OutputBinding?
- jsonldPredicate: "cwl:outputBinding"
- doc: |
- Describes how to handle the outputs of a process.
+ extends: [Parameter, OutputFormat]
+ abstract: true
- type: record
@@ -627,6 +728,7 @@ $graph:
- type: record
name: Process
+ extends: [Identified, Labeled, sld:Documented]
abstract: true
doc: |
@@ -635,10 +737,6 @@ $graph:
directly executed.
fields:
- - name: id
- type: string?
- jsonldPredicate: "@id"
- doc: "The unique identifier for this process object."
- name: inputs
type:
type: array
@@ -653,6 +751,13 @@ $graph:
values. Input parameters include a schema for each parameter which is
used to validate the input object. It may also be used to build a user
interface for constructing the input object.
+
+ When accepting an input object, all input parameters must have a value.
+ If an input parameter is missing from the input object, it must be
+ assigned a value of `null` (or the value of `default` for that
+ parameter, if provided) for the purposes of validation and evaluation
+ of expressions.
+
- name: outputs
type:
type: array
@@ -687,14 +792,6 @@ $graph:
_id: cwl:hints
noLinkCheck: true
mapSubject: class
- - name: label
- type: string?
- jsonldPredicate: "rdfs:label"
- doc: "A short, human-readable label of this process object."
- - name: doc
- type: string?
- jsonldPredicate: "rdfs:comment"
- doc: "A long, human-readable description of this process object."
- name: cwlVersion
type: CWLVersion?
doc: |
@@ -726,6 +823,10 @@ $graph:
be called from CWL expressions.
+- name: CommandInputSchema
+ type: record
+ abstract: true
+
- name: SchemaDefRequirement
type: record
extends: ProcessRequirement
@@ -747,5 +848,5 @@ $graph:
- name: types
type:
type: array
- items: InputSchema
+ items: CommandInputSchema
doc: The list of type definitions.
diff --git a/v1.1.0-dev1/README.md b/v1.1.0-dev1/README.md
index 2c5f66e7e..d457c1709 100644
--- a/v1.1.0-dev1/README.md
+++ b/v1.1.0-dev1/README.md
@@ -2,8 +2,8 @@
The CWL specifications are divided up into several documents.
-The [User Guide](UserGuide.html) provides a gentle introduction to writing CWL
-command line tools and workflows.
+The [User Guide](http://www.commonwl.org/user_guide/) provides a gentle
+introduction to writing CWL command line tools and workflows.
The [Command Line Tool Description Specification](CommandLineTool.html)
specifies the document schema and execution semantics for wrapping and
diff --git a/v1.1.0-dev1/UserGuide.yml b/v1.1.0-dev1/UserGuide.yml
index 95688ce51..0fd2a061e 100644
--- a/v1.1.0-dev1/UserGuide.yml
+++ b/v1.1.0-dev1/UserGuide.yml
@@ -3,877 +3,3 @@
doc:
- $include: userguide-intro.md
- - |
- # Wrapping Command Line Tools
-
- - |
- ## First example
-
- The simplest "hello world" program. This accepts one input parameter,
- writes a message to the terminal or job log, and produces no permanent
- output. CWL documents are written in [JSON](http://json.org) or
- [YAML](http://yaml.org), or a mix of the two.
-
- *1st-tool.cwl*
- ```
- - $include: examples/1st-tool.cwl
- - |
- ```
-
- Use a YAML object in a separate file to describe the input of a run:
-
- *echo-job.yml*
- ```
- - $include: examples/echo-job.yml
- - |
- ```
-
- Now invoke `cwl-runner` with the tool wrapper and the input object on the
- command line:
-
- ```
- $ cwl-runner 1st-tool.cwl echo-job.yml
- [job 140199012414352] $ echo 'Hello world!'
- Hello world!
- Final process status is success
- ```
-
- What's going on here? Let's break it down:
-
- ```
- cwlVersion: v1.1.0-dev1
- class: CommandLineTool
- ```
-
- The `cwlVersion` field indicates the version of the CWL spec used by the
- document. The `class` field indicates this document describes a command
- line tool.
-
- ```
- baseCommand: echo
- ```
-
- The `baseCommand` provides the name of program that will actually run
- (echo)
-
- ```
- inputs:
- message:
- type: string
- inputBinding:
- position: 1
- ```
-
- The `inputs` section describes the inputs of the tool. This is a list of input
- parameters and each parameter includes an identifier, a data type, and
- optionally an `inputBinding` which describes how this input parameter
- should appear on the command line. In this example, the `position` field
- indicates where it should appear on the command line.
-
- ```
- outputs: []
- ```
-
- This tool has no formal output, so the `outputs` section is an empty list.
-
- - |
- ## Essential input parameters
-
- The `inputs` of a tool is a list of input parameters that control how to
- run the tool. Each parameter has an `id` for the name of parameter, and
- `type` describing what types of values are valid for that parameter.
-
- Available primitive types are *string*, *int*, *long*, *float*, *double*,
- and *null*; complex types are *array* and *record*; in addition there are
- special types *File*, *Directory* and *Any*.
-
- The following example demonstrates some input parameters with different
- types and appearing on the command line in different ways:
-
-
- *inp.cwl*
- ```
- - $include: examples/inp.cwl
- - |
- ```
-
- *inp-job.yml*
- ```
- - $include: examples/inp-job.yml
- - |
- ```
-
- Notice that "example_file", as a `File` type, must be provided as an
- object with the fields `class: File` and `path`.
-
- Next, create a whale.txt and invoke `cwl-runner` with the tool wrapper and the input object on the
- command line:
-
- ```
- $ touch whale.txt
- $ cwl-runner inp.cwl inp-job.yml
- [job 140020149614160] /home/example$ echo -f -i42 --example-string hello --file=/home/example/whale.txt
- -f -i42 --example-string hello --file=/home/example/whale.txt
- Final process status is success
- ```
-
- The field `inputBinding` is optional and indicates whether and how the
- input parameter should be appear on the tool's command line. If
- `inputBinding` is missing, the parameter does not appear on the command
- line. Let's look at each example in detail.
-
- ```
- example_flag:
- type: boolean
- inputBinding:
- position: 1
- prefix: -f
- ```
-
- Boolean types are treated as a flag. If the input parameter
- "example_flag" is "true", then `prefix` will be added to the
- command line. If false, no flag is added.
-
- ```
- example_string:
- type: string
- inputBinding:
- position: 3
- prefix: --example-string
- ```
-
- String types appear on the command line as literal values. The `prefix`
- is optional, if provided, it appears as a separate argument on the
- command line before the parameter . In the example above, this is
- rendered as `--example-string hello`.
-
- ```
- example_int:
- type: int
- inputBinding:
- position: 2
- prefix: -i
- separate: false
- ```
-
- Integer (and floating point) types appear on the command line with
- decimal text representation. When the option `separate` is false (the
- default value is true), the prefix and value are combined into a single
- argument. In the example above, this is rendered as `-i42`.
-
-
- ```
- example_file:
- type: File?
- inputBinding:
- prefix: --file=
- separate: false
- position: 4
- ```
-
- File types appear on the command line as the path to the file. When the
- parameter type ends with a question mark `?` it indicates that the
- parameter is optional. In the example above, this is rendered as
- `--file=/home/example/whale.txt`. However, if the "example_file"
- parameter were not provided in the input, nothing would appear on the
- command line.
-
- Input files are read-only. If you wish to update an input file, you must
- first copy it to the output directory.
-
- The value of `position` is used to determine where parameter should
- appear on the command line. Positions are relative to one another, not
- absolute. As a result, positions do not have to be sequential, three
- parameters with positions `[1, 3, 5]` will result in the same command
- line as `[1, 2, 3]`. More than one parameter can have the same position
- (ties are broken using the parameter name), and the position field itself
- is optional. the default position is 0.
-
- The `baseCommand` field always comes before parameters.
-
- - |
- ## Returning output files
-
- The `outputs` of a tool is a list of output parameters that should be
- returned after running the tool. Each parameter has an `id` for the name
- of parameter, and `type` describing what types of values are valid for
- that parameter.
-
- When a tool runs under CWL, the starting working directory is the
- designated output directory. The underlying tool or script must record
- its results in the form of files created in the output directory. The
- output parameters returned by the CWL tool are either the output files
- themselves, or come from examining the content of those files.
-
- *tar.cwl*
- ```
- - $include: examples/tar.cwl
- - |
- ```
-
- *tar-job.yml*
- ```
- - $include: examples/tar-job.yml
- - |
- ```
-
- Next, create a tar file for the example and invoke `cwl-runner` with the tool wrapper and the input object on the
- command line:
- ```
- $ touch hello.txt && tar -cvf hello.tar hello.txt
- $ cwl-runner tar.cwl tar-job.yml
- [job 139868145165200] $ tar xf /home/example/hello.tar
- Final process status is success
- {
- "example_out": {
- "location": "hello.txt",
- "size": 13,
- "class": "File",
- "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b"
- }
- }
- ```
-
- The field `outputBinding` describes how to to set the value of each
- output parameter.
-
- ```
- outputs:
- example_out:
- type: File
- outputBinding:
- glob: hello.txt
- ```
-
- The `glob` field consists of the name of a file in the output directory.
- If you don't know name of the file in advance, you can use a wildcard
- pattern.
-
- - |
- ## Capturing a tool's standard output stream
-
- To capture a tool's standard output stream, add the `stdout` field with
- the name of the file where the output stream should go. Then add `type:
- stdout` on the corresponding output parameter.
-
- *stdout.cwl*
- ```
- - $include: examples/stdout.cwl
- - |
- ```
-
- *echo-job.yml*
- ```
- - $include: examples/echo-job.yml
- - |
- ```
-
- Now invoke `cwl-runner` providing the tool wrapper and the input object
- on the command line:
-
- ```
- $ cwl-runner stdout.cwl echo-job.yml
- [job 140199012414352] $ echo 'Hello world!' > output.txt
- Final process status is success
- {
- "output": {
- "location": "output.txt",
- "size": 13,
- "class": "File",
- "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b"
- }
- }
- $ cat output.txt
- Hello world!
- ```
-
- - |
- ## Parameter references
-
- In a previous example, we extracted a file using the "tar" program.
- However, that example was very limited because it assumed that the file
- we were interested in was called "hello.txt". In this example, you will
- see how to reference the value of input parameters dynamically from other
- fields.
-
- *tar-param.cwl*
- ```
- - $include: examples/tar-param.cwl
- - |
- ```
-
- *tar-param-job.yml*
- ```
- - $include: examples/tar-param-job.yml
- - |
- ```
-
- Create your input files and invoke `cwl-runner` with the tool wrapper and the input object on the
- command line:
- ```
- $ rm hello.tar || true && touch goodbye.txt && tar -cvf hello.tar goodbye.txt
- $ cwl-runner tar-param.cwl tar-param-job.yml
- [job 139868145165200] $ tar xf /home/example/hello.tar goodbye.txt
- Final process status is success
- {
- "example_out": {
- "location": "goodbye.txt",
- "size": 24,
- "class": "File",
- "checksum": "sha1$dd0a4c4c49ba43004d6611771972b6cf969c1c01"
- }
- }
- ```
-
- Certain fields permit parameter references which are enclosed in `$(...)`.
- These are evaluated and replaced with value being referenced.
-
- ```
- outputs:
- example_out:
- type: File
- outputBinding:
- glob: $(inputs.extractfile)
- ```
-
- References are written using a subset of Javascript syntax. In this
- example, `$(inputs.extractfile)`, `$(inputs["extractfile"])`, and
- `$(inputs['extractfile'])` are equivalent.
-
- The value of the "inputs" variable is the input object provided when the
- CWL tool was invoked.
-
- Note that because File parameters are objects, to get the path to an
- input file you must reference the path field on a file object; to
- reference the path to the tar file in the above example you would write
- `$(inputs.tarfile.path)`.
-
- - |
- ## Running tools inside Docker
-
- [Docker](http://docker.io) containers simplify software installation by providing a complete
- known-good runtime for software and its dependencies. However,
- containers are also purposefully isolated from the host system, so in
- order to run a tool inside a Docker container there is additional work to
- ensure that input files are available inside the container and output
- files can be recovered from the container. CWL can perform this work
- automatically, allowing you to use Docker to simplify your software
- management while avoiding the complexity of invoking and managing Docker
- containers.
-
- This example runs a simple Node.js script inside a Docker container.
-
- *docker.cwl*
- ```
- - $include: examples/docker.cwl
- - |
- ```
-
- *docker-job.yml*
- ```
- - $include: examples/docker-job.yml
- - |
- ```
-
- Provide a hello.js and invoke `cwl-runner` providing the tool wrapper and the input object
- on the command line:
-
- ```
- $ echo "console.log(\"Hello World\");" > hello.js
- $ cwl-runner docker.cwl docker-job.yml
- [job 140259721854416] /home/example$ docker run -i --volume=/home/example/hello.js:/var/lib/cwl/job369354770_examples/hello.js:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpDLs5hm:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp node:slim node /var/lib/cwl/job369354770_examples/hello.js
- Hello world!
- Final process status is success
- ```
-
- Notice the CWL runner has constructed a Docker command line to run the
- script. One of the responsibilies of the CWL runner is to the paths of
- input files to reflect the location where they appear inside the
- container. In this example, the path to the script `hello.js` is
- `/home/example/hello.js` outside the container but
- `/var/lib/cwl/job369354770_examples/hello.js` inside the container, as
- reflected in the invocation of the `node` command.
-
- - |
- ## Additional command line arguments and runtime parameters
-
- Sometimes tools require additional command line options that don't
- correspond exactly to input parameters.
-
- In this example, we will wrap the Java compiler to compile a java source
- file to a class file. By default, `javac` will create the class files in
- the same directory as the source file. However, CWL input files (and the
- directories in which they appear) may be read-only, so we need to
- instruct javac to write the class file to the designated output directory
- instead.
-
- *arguments.cwl*
- ```
- - $include: examples/arguments.cwl
- - |
- ```
-
- *arguments-job.yml*
- ```
- - $include: examples/arguments-job.yml
- - |
- ```
-
- Now create a sample Java file and invoke `cwl-runner` providing the tool wrapper and the input object
- on the command line:
-
- ```
- $ echo "public class Hello {}" > Hello.java
- $ cwl-runner arguments.cwl arguments-job.yml
- [job arguments.cwl] /tmp/tmpwYALo1$ docker \
- run \
- -i \
- --volume=/home/peter/work/common-workflow-language/v1.1.0-dev1/examples/Hello.java:/var/lib/cwl/stg8939ac04-7443-4990-a518-1855b2322141/Hello.java:ro \
- --volume=/tmp/tmpwYALo1:/var/spool/cwl:rw \
- --volume=/tmp/tmpptIAJ8:/tmp:rw \
- --workdir=/var/spool/cwl \
- --read-only=true \
- --user=1001 \
- --rm \
- --env=TMPDIR=/tmp \
- --env=HOME=/var/spool/cwl \
- java:7 \
- javac \
- -d \
- /var/spool/cwl \
- /var/lib/cwl/stg8939ac04-7443-4990-a518-1855b2322141/Hello.java
- Final process status is success
- {
- "classfile": {
- "size": 416,
- "location": "/home/example/Hello.class",
- "checksum": "sha1$2f7ac33c1f3aac3f1fec7b936b6562422c85b38a",
- "class": "File"
- }
- }
-
- ```
-
- Here we use the `arguments` field to add an additional argument to the
- command line that isn't tied to a specific input parameter.
-
- ```
- arguments: ["-d", $(runtime.outdir)]
- ```
-
- This example references a runtime parameter. Runtime parameters
- provide information about the hardware or software environment when the
- tool is actually executed. The `$(runtime.outdir)` parameter is the path
- to the designated output directory. Other parameters include
- `$(runtime.tmpdir)`, `$(runtime.ram)`, `$(runtime.cores)`,
- `$(runtime.outdirSize)`, and `$(runtime.tmpdirSize)`. See
- the [Runtime Environment](CommandLineTool.html#Runtime_environment)
- section of the CWL specification for details.
-
- - |
- ## Array inputs
-
- It is easy to add arrays of input parameters represented to the command
- line. To specify an array parameter, the array definition is nested
- under the `type` field with `type: array` and `items` defining the valid
- data types that may appear in the array.
-
- *array-inputs.cwl*
- ```
- - $include: examples/array-inputs.cwl
- - |
- ```
-
- *array-inputs-job.yml*
- ```
- - $include: examples/array-inputs-job.yml
- - |
- ```
-
- Now invoke `cwl-runner` providing the tool wrapper and the input object
- on the command line:
-
- ```
- $ cwl-runner array-inputs.cwl array-inputs-job.yml
- [job 140334923640912] /home/example$ echo -A one two three -B=four -B=five -B=six -C=seven,eight,nine
- -A one two three -B=four -B=five -B=six -C=seven,eight,nine
- Final process status is success
- {}
- ```
-
- The `inputBinding` can appear either on the outer array parameter
- definition or the inner array element definition, and these produce
- different behavior when constructing the command line, as shown above.
- In addition, the `itemSeperator` field, if provided, specifies that array
- values should be concatenated into a single argument separated by the
- item separator string.
-
- You can specify arrays of arrays, arrays of records, and other complex
- types.
-
- - |
- ## Array outputs
-
- You can also capture multiple output files into an array of files using `glob`.
-
- *array-outputs.cwl*
- ```
- - $include: examples/array-outputs.cwl
- - |
- ```
-
- *array-outputs-job.yml*
- ```
- - $include: examples/array-outputs-job.yml
- - |
- ```
-
- Now invoke `cwl-runner` providing the tool wrapper and the input object
- on the command line:
-
- ```
- $ cwl-runner array-outputs.cwl array-outputs-job.yml
- [job 140190876078160] /home/example$ touch foo.txt bar.dat baz.txt
- Final process status is success
- {
- "output": [
- {
- "size": 0,
- "location": "/home/peter/work/common-workflow-language/draft-3/examples/foo.txt",
- "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
- "class": "File"
- },
- {
- "size": 0,
- "location": "/home/peter/work/common-workflow-language/draft-3/examples/baz.txt",
- "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
- "class": "File"
- }
- ]
- }
- ```
-
- - |
- ## Record inputs, dependent and mutually exclusive parameters
-
- Sometimes an underlying tool has several arguments that must be provided
- together (they are dependent) or several arguments that cannot be
- provided together (they are exclusive). You can use records and type
- unions to group parameters together to describe these two conditions.
-
- *record.cwl*
- ```
- - $include: examples/record.cwl
- - |
- ```
-
- *record-job1.yml*
- ```
- - $include: examples/record-job1.yml
- - |
- ```
-
- ```
- $ cwl-runner record.cwl record-job1.yml
- Workflow error:
- Error validating input record, could not validate field `dependent_parameters` because
- missing required field `itemB`
- ```
-
- In the first example, you can't provide `itemA` without also providing `itemB`.
-
- *record-job2.yml*
- ```
- - $include: examples/record-job2.yml
- - |
- ```
-
- ```
- $ cwl-runner record.cwl record-job2.yml
- [job 140566927111376] /home/example$ echo -A one -B two -C three
- -A one -B two -C three
- Final process status is success
- {}
- ```
-
- In the second example, `itemC` and `itemD` are exclusive, so only `itemC`
- is added to the command line and `itemD` is ignored.
-
- *record-job3.yml*
- ```
- - $include: examples/record-job3.yml
- - |
- ```
-
- ```
- $ cwl-runner record.cwl record-job3.yml
- [job 140606932172880] /home/example$ echo -A one -B two -D four
- -A one -B two -D four
- Final process status is success
- {}
- ```
-
- In the third example, only `itemD` is provided, so it appears on the
- command line.
-
- - |
- ## Environment variables
-
- Tools run in a restricted environment and do not inherit most environment
- variables from the parent process. You can set environment variables for
- the tool using `EnvVarRequirement`.
-
- *env.cwl*
- ```
- - $include: examples/env.cwl
- - |
- ```
-
- *echo-job.yml*
- ```
- - $include: examples/echo-job.yml
- - |
- ```
-
- Now invoke `cwl-runner` with the tool wrapper and the input object on the
- command line:
-
- ```
- $ cwl-runner env.cwl echo-job.yml
- [job 140710387785808] /home/example$ env
- PATH=/bin:/usr/bin:/usr/local/bin
- HELLO=Hello world!
- TMPDIR=/tmp/tmp63Obpk
- Final process status is success
- {}
- ```
-
- - |
- ## Javascript expressions
-
- If you need to manipulate input parameters, include the requirement
- `InlineJavascriptRequirement` and then anywhere a parameter reference is
- legal you can provide a fragment of Javascript that will be evaluated by
- the CWL runner.
-
- *expression.cwl*
- ```
- - $include: examples/expression.cwl
- - |
- ```
-
- As this tool does not require any `inputs` we can run it with an (almost) empty job file:
-
- *empty.yml*
- ```
- {}
- |
- ```
-
- We can then run `expression.cwl`:
-
- ```
- $ cwl-runner expression.cwl empty.yml
- [job 140000594593168] /home/example$ echo -A 2 -B baz -C 10 9 8 7 6 5 4 3 2 1
- -A 2 -B baz -C 10 9 8 7 6 5 4 3 2 1
- Final process status is success
- {}
- ```
-
- You can only use expressions in certain fields. These are: `filename`,
- `fileContent`, `envValue`, `valueFrom`, `glob`, `outputEval`, `stdin`,
- `stdout`, `coresMin`, `coresMax`, `ramMin`, `ramMax`, `tmpdirMin`,
- `tmpdirMax`, `outdirMin`, and `outdirMax`.
-
- - |
- ## Creating files at runtime
-
- Sometimes you need to create a file on the fly from input parameters,
- such as tools which expect to read their input configuration from a file
- rather than the command line parameters. To do this, use
- `InitialWorkDirRequirement`.
-
- *createfile.cwl*
- ```
- - $include: examples/createfile.cwl
- - |
- ```
-
- *echo-job.yml*
- ```
- - $include: examples/echo-job.yml
- - |
- ```
-
- Now invoke `cwl-runner` with the tool wrapper and the input object on the
- command line:
-
- ```
- $ cwltool createfile.cwl echo-job.yml
- [job 140528604979344] /home/example$ cat example.conf
- CONFIGVAR=Hello world!
- Final process status is success
- {}
- ```
-
- - |
- ## Staging input files in the output directory
-
- Normally, input files are located in a read-only directory separate from
- the output directory. This causes problems if the underlying tool
- expects to write its output files alongside the input file in the same
- directory. You use `InitialWorkDirRequirement` to stage input files into the
- output directory. In this example, we use a Javascript expression to
- extract the base name of the input file from its leading directory path.
-
- *linkfile.cwl*
- ```
- - $include: examples/linkfile.cwl
- - |
- ```
-
- *arguments-job.yml*
- ```
- - $include: examples/arguments-job.yml
- - |
- ```
-
- Now invoke `cwl-runner` with the tool wrapper and the input object on the
- command line:
-
- ```
- $ cwl-runner linkfile.cwl arguments-job.yml
- [job 139928309171664] /home/example$ docker run -i --volume=/home/example/Hello.java:/var/lib/cwl/job557617295_examples/Hello.java:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpmNbApw:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp java:7 javac Hello.java
- Final process status is success
- {
- "classfile": {
- "size": 416,
- "location": "/home/example/Hello.class",
- "checksum": "sha1$2f7ac33c1f3aac3f1fec7b936b6562422c85b38a",
- "class": "File"
- }
- }
- ```
-
- - |
- # Writing Workflows
-
- ## First workflow
-
- This workflow extracts a java source file from a tar file and then
- compiles it.
-
- *1st-workflow.cwl*
- ```
- - $include: examples/1st-workflow.cwl
- - |
- ```
-
- Use a JSON object in a separate file to describe the input of a run:
-
- *1st-workflow-job.yml*
- ```
- - $include: examples/1st-workflow-job.yml
- - |
- ```
-
- Now invoke `cwl-runner` with the tool wrapper and the input object on the
- command line:
-
- ```
- $ echo "public class Hello {}" > Hello.java && tar -cvf hello.tar Hello.java
- $ cwl-runner 1st-workflow.cwl 1st-workflow-job.yml
- [job untar] /tmp/tmp94qFiM$ tar xf /home/example/hello.tar Hello.java
- [step untar] completion status is success
- [job compile] /tmp/tmpu1iaKL$ docker run -i --volume=/tmp/tmp94qFiM/Hello.java:/var/lib/cwl/job301600808_tmp94qFiM/Hello.java:ro --volume=/tmp/tmpu1iaKL:/var/spool/cwl:rw --volume=/tmp/tmpfZnNdR:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp java:7 javac -d /var/spool/cwl /var/lib/cwl/job301600808_tmp94qFiM/Hello.java
- [step compile] completion status is success
- [workflow 1st-workflow.cwl] outdir is /home/example
- Final process status is success
- {
- "classout": {
- "location": "/home/example/Hello.class",
- "checksum": "sha1$e68df795c0686e9aa1a1195536bd900f5f417b18",
- "class": "File",
- "size": 416
- }
- }
- ```
-
- What's going on here? Let's break it down:
-
- ```
- cwlVersion: v1.1.0-dev1
- class: Workflow
- ```
-
- The `cwlVersion` field indicates the version of the CWL spec used by the
- document. The `class` field indicates this document describes a workflow.
-
-
- ```
- inputs:
- inp: File
- ex: string
- ```
-
- The `inputs` section describes the inputs of the workflow. This is a
- list of input parameters where each parameter consists of an identifier
- and a data type. These parameters can be used as sources for input to
- specific workflows steps.
-
- ```
- outputs:
- classout:
- type: File
- outputSource: compile/classfile
- ```
-
- The `outputs` section describes the outputs of the workflow. This is a
- list of output parameters where each parameter consists of an identifier
- and a data type. The `outputSource` connects the output parameter `classfile`
- of the `compile` step to the workflow output parameter `classout`.
-
- ```
- steps:
- untar:
- run: tar-param.cwl
- in:
- tarfile: inp
- extractfile: ex
- outputs: [example_out]
- ```
-
- The `steps` section describes the actual steps of the workflow. In this
- example, the first step extracts a file from a tar file, and the second
- step compiles the file from the first step using the java compiler.
- Workflow steps are not necessarily run in the order they are listed,
- instead the order is determined by the dependencies between steps (using
- `source`). In addition, workflow steps which do not depend on one
- another may run in parallel.
-
- The first step, `untar` runs `tar-param.cwl` (described previously in
- [Parameter references](#Parameter_references)). This tool has two input
- parameters, `tarfile` and `extractfile` and one output parameter
- `example_out`.
-
- The `inputs` section of the workflow step connects these two input
- parameters to the inputs of the workflow, `inp` and `ex` using
- `source`. This means that when the workflow step is executed, the values
- assigned to `inp` and `ex` will be used for the parameters `tarfile`
- and `extractfile` in order to run the tool.
-
- The `outputs` section of the workflow step lists the output parameters
- that are expected from the tool.
-
- ```
- compile:
- run: arguments.cwl
- in:
- src: untar/example_out
- outputs: [classfile]
- ```
-
- The second step `compile` depends on the results from the first step by
- connecting the input parameter `src` to the output parameter of `untar`
- using `untar/example_out`. The output of this step `classfile` is
- connected to the `outputs` section for the Workflow, described above.
diff --git a/v1.1.0-dev1/Workflow.yml b/v1.1.0-dev1/Workflow.yml
index 265da83e4..fcc4688dc 100644
--- a/v1.1.0-dev1/Workflow.yml
+++ b/v1.1.0-dev1/Workflow.yml
@@ -34,41 +34,40 @@ $graph:
- |
- ## Introduction to v1.1.0-dev1
-
- This is the first development release of the 1.1.0 version of the CWL
- Workflow specification.
-
- ## Introduction to v1.0
-
- This specification represents the first full release from the CWL group.
- Since draft-3, this draft introduces the following changes and additions:
-
- * The `inputs` and `outputs` fields have been renamed `in` and `out`.
- * Syntax simplifcations: denoted by the `map<>` syntax. Example: `in`
- contains a list of items, each with an id. Now one can specify
- a mapping of that identifier to the corresponding
- `InputParameter`.
- ```
- in:
- - id: one
- type: string
- doc: First input parameter
- - id: two
- type: int
- doc: Second input parameter
- ```
- can be
- ```
- in:
- one:
- type: string
- doc: First input parameter
- two:
- type: int
- doc: Second input parameter
- ```
- * The common field `description` has been renamed to `doc`.
+ ## Introduction to CWL Workflow standard v1.1.0-dev1
+
+ This specification represents the latest stable release from the
+ CWL group. Since the v1.0 release, v1.1.0-dev1 introduces the
+ following updates to the CWL Workflow standard.
+ Documents should to use `cwlVersion: v1.1.0-dev1` to make use of new
+ syntax and features introduced in v1.1.0-dev1. Existing v1.0 documents
+ should be trivially updatable by changing `cwlVersion`, however
+ CWL documents that relied on previously undefined or
+ underspecified behavior may have slightly different behavior in
+ v1.1.0-dev1.
+
+ ## Changelog
+
+ * Default values for some fields are now expressed in the schema
+ * When defining record types, fields of type `File` may now include
+ `format`, `secondaryFiles`, and `streamable`.
+ * The Input schemas (Record, Array, Enum), now permits a `doc` field
+ * Any `doc` field may be an array of strings in addition to the
+ previously allowed single string.
+ * The `run` field now introduces a subscope so that embedded
+ definitions do not have identifier conflicts with the
+ outer workflow.
+ * `Workflow.inputs` and `ExpressionTool.inputs` now refer to the type
+ `WorkflowInputParameter` instead of `InputParameter`; the only syntax
+ difference from v1.0 is that `inputBinding` is now deprecated (as
+ there is no command line to bind input values to) and `inputBinding`
+ in a `Workflow` context will be removed in v2.0. Use the new
+ `WorkflowInputParameter.loadContents` instead.
+ * The presence of `OutputRecordField.outputBinding` in v1.0 was due to
+ a technical error and as it makes no semantic sense it was removed.
+
+
+ See also the [CWL Command Line Tool Description, v1.1.0-dev1 changelog](CommandLineTool.html#Changelog).
## Purpose
@@ -108,12 +107,48 @@ $graph:
doc: |
Specify valid types of data that may be assigned to this parameter.
+
+- name: WorkflowInputParameter
+ type: record
+ extends: InputParameter
+ docParent: "#Workflow"
+ fields:
+ - name: type
+ type:
+ - "null"
+ - CWLType
+ - InputRecordSchema
+ - InputEnumSchema
+ - InputArraySchema
+ - string
+ - type: array
+ items:
+ - CWLType
+ - InputRecordSchema
+ - InputEnumSchema
+ - InputArraySchema
+ - string
+ jsonldPredicate:
+ "_id": "sld:type"
+ "_type": "@vocab"
+ refScope: 2
+ typeDSL: True
+ doc: |
+ Specify valid types of data that may be assigned to this parameter.
+ - name: inputBinding
+ type: InputBinding?
+ doc: |
+ Deprecated. Preserved for v1.0 backwards compatability. Will be removed in
+ CWL v2.0. Use `WorkflowInputParameter.loadContents` instead.
+ jsonldPredicate: "cwl:inputBinding"
+
+
- type: record
name: ExpressionTool
extends: Process
specialize:
- specializeFrom: InputParameter
- specializeTo: RegularInputParameter
+ specializeTo: WorkflowInputParameter
- specializeFrom: OutputParameter
specializeTo: ExpressionToolOutputParameter
documentRoot: true
@@ -163,6 +198,7 @@ $graph:
- name: linkMerge
type: ["null", LinkMergeMethod]
jsonldPredicate: "cwl:linkMerge"
+ default: merge_nested
doc: |
The method to use to merge multiple sources into a single array.
If not specified, the default method is "merge_nested".
@@ -208,6 +244,7 @@ $graph:
- name: linkMerge
type: LinkMergeMethod?
jsonldPredicate: "cwl:linkMerge"
+ default: merge_nested
doc: |
The method to use to merge multiple inbound links into a single array.
If not specified, the default method is "merge_nested".
@@ -215,7 +252,7 @@ $graph:
- type: record
name: WorkflowStepInput
- extends: Sink
+ extends: [Identified, Sink, LoadContents]
docParent: "#WorkflowStep"
doc: |
The input of a workflow step connects an upstream parameter (from the
@@ -225,10 +262,11 @@ $graph:
## Input object
A WorkflowStepInput object must contain an `id` field in the form
- `#fieldname` or `#stepname.fieldname`. When the `id` field contains a
- period `.` the field name consists of the characters following the final
- period. This defines a field of the workflow step input object with the
- value of the `source` parameter(s).
+ `#fieldname` or `#prefix/fieldname`. When the `id` field contains a slash
+ `/` the field name consists of the characters following the final slash
+ (the prefix portion may contain one or more slashes to indicate scope).
+ This defines a field of the workflow step input object with the value of
+ the `source` parameter(s).
## Merging
@@ -258,17 +296,14 @@ $graph:
single elements.
fields:
- - name: id
- type: string
- jsonldPredicate: "@id"
- doc: "A unique identifier for this workflow input parameter."
- name: default
type: ["null", Any]
doc: |
- The default value for this parameter if there is no `source`
- field.
+ The default value for this parameter to use if either there is no
+ `source` field, or the value produced by the `source` is `null`. The
+ default must be applied prior to scattering or evaluating `valueFrom`.
jsonldPredicate:
- _id: "cwl:default"
+ _id: "sld:default"
noLinkCheck: true
- name: valueFrom
type:
@@ -292,28 +327,25 @@ $graph:
The value of `inputs` in the parameter reference or expression must be
the input object to the workflow step after assigning the `source`
- values and then scattering. The order of evaluating `valueFrom` among
- step input parameters is undefined and the result of evaluating
- `valueFrom` on a parameter must not be visible to evaluation of
- `valueFrom` on other parameters.
+ values, applying `default`, and then scattering. The order of
+ evaluating `valueFrom` among step input parameters is undefined and the
+ result of evaluating `valueFrom` on a parameter must not be visible to
+ evaluation of `valueFrom` on other parameters.
- type: record
name: WorkflowStepOutput
docParent: "#WorkflowStep"
+ extends: Identified
doc: |
Associate an output parameter of the underlying process with a workflow
parameter. The workflow parameter (given in the `id` field) be may be used
as a `source` to connect with input parameters of other workflow steps, or
with an output parameter of the process.
- fields:
- - name: id
- type: string
- jsonldPredicate: "@id"
- doc: |
- A unique identifier for this workflow output parameter. This is the
- identifier to use in the `source` field of `WorkflowStepInput` to
- connect the output value to downstream parameters.
+
+ A unique identifier for this workflow output parameter. This is
+ the identifier to use in the `source` field of `WorkflowStepInput`
+ to connect the output value to downstream parameters.
- name: ScatterMethod
@@ -328,6 +360,7 @@ $graph:
- name: WorkflowStep
type: record
+ extends: [Identified, Labeled, sld:Documented]
docParent: "#Workflow"
doc: |
A workflow step is an executable element of a workflow. It specifies the
@@ -348,13 +381,18 @@ $graph:
The `scatter` field specifies one or more input parameters which will be
scattered. An input parameter may be listed more than once. The declared
- type of each input parameter is implicitly wrapped in an array for each
- time it appears in the `scatter` field. As a result, upstream parameters
- which are connected to scattered parameters may be arrays.
+ type of each input parameter is implicitly becomes an array of items of the
+ input parameter type. If a parameter is listed more than once, it becomes
+ a nested array. As a result, upstream parameters which are connected to
+ scattered parameters must be arrays.
All output parameter types are also implicitly wrapped in arrays. Each job
in the scatter results in an entry in the output array.
+ If any scattered parameter runtime value is an empty array, all outputs are
+ set to empty arrays and no work is done for the step, according to
+ applicable scattering rules.
+
If `scatter` declares more than one input parameter, `scatterMethod`
describes how to decompose the input into a discrete set of jobs.
@@ -378,11 +416,10 @@ $graph:
[SubworkflowFeatureRequirement](#SubworkflowFeatureRequirement) must be
specified in the workflow or workflow step requirements.
+ It is a fatal error if a workflow directly or indirectly invokes itself as
+ a subworkflow (recursive workflows are not allowed).
+
fields:
- - name: id
- type: string
- jsonldPredicate: "@id"
- doc: "The unique identifier for this workflow step."
- name: in
type: WorkflowStepInput[]
jsonldPredicate:
@@ -429,19 +466,12 @@ $graph:
workflow engine that may be helpful in executing this workflow step. It is
not an error if an implementation cannot satisfy all hints, however
the implementation may report a warning.
- - name: label
- type: string?
- jsonldPredicate: "rdfs:label"
- doc: "A short, human-readable label of this process object."
- - name: doc
- type: string?
- jsonldPredicate: "rdfs:comment"
- doc: "A long, human-readable description of this process object."
- name: run
type: [string, Process]
jsonldPredicate:
- "_id": "cwl:run"
- "_type": "@id"
+ _id: "cwl:run"
+ _type: "@id"
+ subscope: run
doc: |
Specifies the process to run.
- name: scatter
@@ -468,7 +498,7 @@ $graph:
documentRoot: true
specialize:
- specializeFrom: InputParameter
- specializeTo: RegularInputParameter
+ specializeTo: WorkflowInputParameter
- specializeFrom: OutputParameter
specializeTo: WorkflowOutputParameter
doc: |
diff --git a/v1.1.0-dev1/concepts.md b/v1.1.0-dev1/concepts.md
index c1a9ced8d..6dea53464 100644
--- a/v1.1.0-dev1/concepts.md
+++ b/v1.1.0-dev1/concepts.md
@@ -6,7 +6,7 @@
**YAML**: http://yaml.org
-**Avro**: https://avro.apache.org/docs/current/spec.html
+**Avro**: https://avro.apache.org/docs/1.8.1/spec.html
**Uniform Resource Identifier (URI) Generic Syntax**: https://tools.ietf.org/html/rfc3986)
@@ -323,7 +323,7 @@ Expressions are denoted by the syntax `$(...)` or `${...}`. A code
fragment wrapped in the `$(...)` syntax must be evaluated as a
[ECMAScript expression](http://www.ecma-international.org/ecma-262/5.1/#sec-11). A
code fragment wrapped in the `${...}` syntax must be evaluated as a
-[EMACScript function body](http://www.ecma-international.org/ecma-262/5.1/#sec-13)
+[ECMAScript function body](http://www.ecma-international.org/ecma-262/5.1/#sec-13)
for an anonymous, zero-argument function. Expressions must return a valid JSON
data type: one of null, string, number, boolean, array, object. Other return
values must result in a `permanentFailure`. Implementations must permit any
@@ -375,6 +375,9 @@ must include the field `cwl:tool` supplying an IRI to the default CWL
document that should be executed using the fields of the input object as
input parameters.
+The `cwl-runner` interface is required for conformance testing and is
+documented in [cwl-runner.cwl](cwl-runner.cwl).
+
## Discovering CWL documents on a local filesystem
To discover CWL documents look in the following locations:
@@ -385,5 +388,5 @@ To discover CWL documents look in the following locations:
`$XDG_DATA_HOME/commonwl/` (usually `$HOME/.local/share/commonwl`)
-`$XDF_DATA_HOME` is from the [XDG Base Directory
+`$XDG_DATA_HOME` is from the [XDG Base Directory
Specification](http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html)
diff --git a/v1.1.0-dev1/conformance_test_v1.1.0-dev1.yaml b/v1.1.0-dev1/conformance_test_v1.1.0-dev1.yaml
index f25e3505a..27c9dfa06 100644
--- a/v1.1.0-dev1/conformance_test_v1.1.0-dev1.yaml
+++ b/v1.1.0-dev1/conformance_test_v1.1.0-dev1.yaml
@@ -1,11 +1,12 @@
- job: v1.1.0-dev1/bwa-mem-job.json
tool: v1.1.0-dev1/bwa-mem-tool.cwl
output:
- args: [bwa, mem, -t, '4', -I, '1,2,3,4', -m, '3',
+ args: [bwa, mem, -t, '2', -I, '1,2,3,4', -m, '3',
chr20.fa,
example_human_Illumina.pe_1.fastq,
example_human_Illumina.pe_2.fastq]
doc: General test of command line generation
+ tags: [ required ]
- output:
args: [bwa, mem, chr20.fa,
@@ -15,6 +16,7 @@
job: v1.1.0-dev1/bwa-mem-job.json
tool: v1.1.0-dev1/binding-test.cwl
doc: Test nested prefixes with arrays
+ tags: [ required ]
- output:
args: [tmap, mapall, stage1, map1, --min-seq-length, '20', map2, --min-seq-length,
@@ -23,18 +25,21 @@
job: v1.1.0-dev1/tmap-job.json
tool: v1.1.0-dev1/tmap-tool.cwl
doc: Test nested command line bindings
+ tags: [ schema_def ]
- output:
args: [cat, hello.txt]
job: v1.1.0-dev1/cat-job.json
tool: v1.1.0-dev1/cat1-testcli.cwl
doc: Test command line with optional input (missing)
+ tags: [ required ]
- output:
args: [cat, -n, hello.txt]
job: v1.1.0-dev1/cat-n-job.json
tool: v1.1.0-dev1/cat1-testcli.cwl
doc: Test command line with optional input (provided)
+ tags: [ required ]
- output:
"foo": {
@@ -46,6 +51,7 @@
job: v1.1.0-dev1/cat-job.json
tool: v1.1.0-dev1/template-tool.cwl
doc: Test InitialWorkDirRequirement ExpressionEngineRequirement.engineConfig feature
+ tags: [ initial_work_dir, inline_javascript ]
- job: v1.1.0-dev1/cat-job.json
output:
@@ -56,6 +62,7 @@
size: 13
tool: v1.1.0-dev1/cat3-tool.cwl
doc: Test command execution in Docker with stdout redirection
+ tags: [ required ]
- job: v1.1.0-dev1/cat-job.json
tool: v1.1.0-dev1/cat3-tool-shortcut.cwl
@@ -66,6 +73,7 @@
location: Any
size: 13
doc: Test command execution in Docker with simplified syntax stdout redirection
+ tags: [ required ]
- job: v1.1.0-dev1/cat-job.json
output:
@@ -76,6 +84,7 @@
size: 13
tool: v1.1.0-dev1/cat3-tool-mediumcut.cwl
doc: Test command execution in Docker with stdout redirection
+ tags: [ required ]
- job: v1.1.0-dev1/empty.json
tool: v1.1.0-dev1/stderr.cwl
@@ -86,6 +95,7 @@
checksum: sha1$f1d2d2f924e986ac86fdf7b36c94bcdf32beec15
size: 4
location: error.txt
+ tags: [ shell_command ]
- job: v1.1.0-dev1/empty.json
tool: v1.1.0-dev1/stderr-shortcut.cwl
@@ -96,6 +106,7 @@
checksum: sha1$f1d2d2f924e986ac86fdf7b36c94bcdf32beec15
size: 4
location: Any
+ tags: [ shell_command ]
- output:
output_file:
@@ -106,6 +117,7 @@
job: v1.1.0-dev1/empty.json
tool: v1.1.0-dev1/stderr-mediumcut.cwl
doc: Test command line with stderr redirection, named brief syntax
+ tags: [ shell_command ]
- job: v1.1.0-dev1/cat-job.json
output:
@@ -116,88 +128,109 @@
size: 13
tool: v1.1.0-dev1/cat4-tool.cwl
doc: Test command execution in Docker with stdin and stdout redirection
+ tags: [ required ]
-- job: v1.1.0-dev1/cat-job.json
+- job: v1.1.0-dev1/wc-job.json
output:
- output_txt:
+ output:
class: File
- checksum: sha1$47a013e660d408619d894b20806b1d5086aab03b
- location: output.txt
- size: 13
- tool: v1.1.0-dev1/cat4b-tool.cwl
- doc: Test command execution in Docker with stdin and stdout redirection
- with stdin shortcut
+ checksum: sha1$631bfbac524e2d04cdcc5ec33ade827fc10b06ae
+ location: output
+ size: 15
+ tool: v1.1.0-dev1/wc-tool-shortcut.cwl
+ doc: Test command execution in with stdin and stdout redirection using stdin shortcut
+ tags: [ required ]
- job: v1.1.0-dev1/empty.json
tool: v1.1.0-dev1/null-expression1-tool.cwl
output:
output: 1
doc: Test default usage of Any in expressions.
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/null-expression1-job.json
tool: v1.1.0-dev1/null-expression1-tool.cwl
output:
- output: 2
+ output: 1
doc: Test explicitly passing null to Any type inputs with default values.
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/null-expression2-job.json
tool: v1.1.0-dev1/null-expression1-tool.cwl
output:
output: 2
doc: Testing the string 'null' does not trip up an Any with a default value.
+ tags: [ inline_javascript ]
-## TODO: Upgrade framework to allow specifying negative tests.
-# - job: v1.1.0-dev1/empty.json
-# tool: v1.1.0-dev1/null-expression2-tool.cwl
-# failure: true
-# doc: Test Any without defaults can be unspecified.
+- job: v1.1.0-dev1/empty.json
+ tool: v1.1.0-dev1/null-expression2-tool.cwl
+ should_fail: true
+ doc: Test Any without defaults can be unspecified.
+ tags: [ inline_javascript ]
-# - job: v1.1.0-dev1/null-expression1-job.json
-# tool: v1.1.0-dev1/null-expression2-tool.cwl
-# failure: true
-# doc: Test explicitly passing null to Any type without a default value.
+- job: v1.1.0-dev1/null-expression1-job.json
+ tool: v1.1.0-dev1/null-expression2-tool.cwl
+ should_fail: true
+ doc: Test explicitly passing null to Any type without a default value.
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/null-expression2-job.json
tool: v1.1.0-dev1/null-expression2-tool.cwl
output:
output: 2
doc: Testing the string 'null' does not trip up an Any without a default value.
+ tags: [ inline_javascript ]
-- job: v1.1.0-dev1/wc-job.json
+- job: v1.1.0-dev1/any-type-job.json
+ tool: v1.1.0-dev1/any-type-compat.cwl
+ output:
+ output1: ["hello", "world"]
+ output2: ["foo", "bar"]
+ output3: hello
+ doc: Testing Any type compatibility in outputSource
+ tags: [ required ]
+
+- job: v1.1.0-dev1/cat-job.json
output:
output:
- checksum: sha1$3596ea087bfdaf52380eae441077572ed289d657
+ checksum: sha1$47a013e660d408619d894b20806b1d5086aab03b
class: File
location: output
- size: 3
- tool: v1.1.0-dev1/wc-tool.cwl
+ size: 13
+ tool: v1.1.0-dev1/cat-tool.cwl
doc: Test command execution in with stdin and stdout redirection
+ tags: [ required ]
- job: v1.1.0-dev1/parseInt-job.json
output: {output: 42}
tool: v1.1.0-dev1/parseInt-tool.cwl
doc: Test ExpressionTool with Docker-based expression engine
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/wc-job.json
output: {output: 16}
tool: v1.1.0-dev1/wc2-tool.cwl
doc: Test outputEval to transform output
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/wc-job.json
output: {count_output: 16}
tool: v1.1.0-dev1/count-lines1-wf.cwl
doc: Test two step workflow with imported tools
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/wc-job.json
output: {count_output: 16}
tool: v1.1.0-dev1/count-lines2-wf.cwl
doc: Test two step workflow with inline tools
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/count-lines3-job.json
output:
count_output: [16, 1]
tool: v1.1.0-dev1/count-lines3-wf.cwl
doc: Test single step workflow with Scatter step
+ tags: [ scatter, inline_javascript ]
- job: v1.1.0-dev1/count-lines4-job.json
output:
@@ -206,6 +239,7 @@
doc: |
Test single step workflow with Scatter step and two data links connected to
same input, default merge behavior
+ tags: [ scatter, multiple_input, inline_javascript ]
- job: v1.1.0-dev1/count-lines6-job.json
output:
@@ -214,6 +248,7 @@
doc: |
Test single step workflow with Scatter step and two data links connected to
same input, nested merge behavior
+ tags: [ scatter, multiple_input, inline_javascript ]
- job: v1.1.0-dev1/count-lines6-job.json
output:
@@ -222,16 +257,34 @@
doc: |
Test single step workflow with Scatter step and two data links connected to
same input, flattened merge behavior
+ tags: [ multiple_input, inline_javascript ]
+
+- job: v1.1.0-dev1/count-lines6-job.json
+ output:
+ count_output: 32
+ tool: v1.1.0-dev1/count-lines13-wf.cwl
+ doc: |
+ Test that no MultipleInputFeatureRequirement is necessary when
+ workflow step source is a single-item list
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/empty.json
output: {count_output: 1}
tool: v1.1.0-dev1/count-lines5-wf.cwl
doc: Test workflow with default value for input parameter (missing)
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/wc-job.json
output: {count_output: 16}
tool: v1.1.0-dev1/count-lines5-wf.cwl
doc: Test workflow with default value for input parameter (provided)
+ tags: [ inline_javacscript ]
+
+- job: v1.1.0-dev1/empty.json
+ output: {default_output: workflow_default}
+ tool: v1.1.0-dev1/echo-wf-default.cwl
+ doc: Test that workflow defaults override tool defaults
+ tags: [ required ]
- job: v1.1.0-dev1/env-job.json
output:
@@ -242,41 +295,83 @@
size: 15
tool: v1.1.0-dev1/env-tool1.cwl
doc: Test EnvVarRequirement
+ tags: [ env_var ]
- job: v1.1.0-dev1/scatter-job1.json
output:
out: ["foo one", "foo two", "foo three", "foo four"]
tool: v1.1.0-dev1/scatter-wf1.cwl
doc: Test workflow scatter with single scatter parameter
+ tags: [ scatter ]
- job: v1.1.0-dev1/scatter-job2.json
output:
out: [["foo one three", "foo one four"], ["foo two three", "foo two four"]]
tool: v1.1.0-dev1/scatter-wf2.cwl
doc: Test workflow scatter with two scatter parameters and nested_crossproduct join method
+ tags: [ scatter ]
- job: v1.1.0-dev1/scatter-job2.json
output:
out: ["foo one three", "foo one four", "foo two three", "foo two four"]
tool: "v1.1.0-dev1/scatter-wf3.cwl#main"
doc: Test workflow scatter with two scatter parameters and flat_crossproduct join method
+ tags: [ scatter ]
- job: v1.1.0-dev1/scatter-job2.json
output:
out: ["foo one three", "foo two four"]
tool: "v1.1.0-dev1/scatter-wf4.cwl#main"
doc: Test workflow scatter with two scatter parameters and dotproduct join method
+ tags: [ scatter ]
+
+- job: v1.1.0-dev1/scatter-empty-job1.json
+ output:
+ out: []
+ tool: v1.1.0-dev1/scatter-wf1.cwl
+ doc: Test workflow scatter with single empty list parameter
+ tags: [ scatter ]
+
+- job: v1.1.0-dev1/scatter-empty-job2.json
+ output:
+ out: [[], []]
+ tool: v1.1.0-dev1/scatter-wf2.cwl
+ doc: Test workflow scatter with two scatter parameters and nested_crossproduct join method with second list empty
+ tags: [ scatter ]
+
+- job: v1.1.0-dev1/scatter-empty-job3.json
+ output:
+ out: []
+ tool: "v1.1.0-dev1/scatter-wf3.cwl#main"
+ doc: Test workflow scatter with two scatter parameters and nested_crossproduct join method with first list empty
+ tags: [ scatter ]
+
+- job: v1.1.0-dev1/scatter-empty-job2.json
+ output:
+ out: []
+ tool: "v1.1.0-dev1/scatter-wf3.cwl#main"
+ doc: Test workflow scatter with two scatter parameters, one of which is empty and flat_crossproduct join method
+ tags: [ scatter ]
+
+- job: v1.1.0-dev1/scatter-empty-job4.json
+ output:
+ out: []
+ tool: "v1.1.0-dev1/scatter-wf4.cwl#main"
+ doc: Test workflow scatter with two empty scatter parameters and dotproduct join method
+ tags: [ scatter ]
- tool: v1.1.0-dev1/echo-tool.cwl
job: v1.1.0-dev1/env-job.json
output:
{"out": "hello test env\n"}
doc: Test Any type input parameter
+ tags: [ required ]
- job: v1.1.0-dev1/wc-job.json
output: {count_output: 16}
tool: v1.1.0-dev1/count-lines8-wf.cwl
doc: Test nested workflow
+ tags: [ subworkflow ]
- job: v1.1.0-dev1/env-job.json
output:
@@ -287,6 +382,7 @@
size: 15
tool: v1.1.0-dev1/env-wf1.cwl
doc: Test requirement priority
+ tags: [ env_var ]
- job: v1.1.0-dev1/env-job.json
output:
@@ -297,6 +393,7 @@
size: 9
tool: v1.1.0-dev1/env-wf2.cwl
doc: Test requirements override hints
+ tags: [ env_var ]
- job: v1.1.0-dev1/env-job.json
output:
@@ -307,11 +404,31 @@
size: 9
tool: v1.1.0-dev1/env-wf3.cwl
doc: Test requirements on workflow steps
+ tags: [ env_var ]
- job: v1.1.0-dev1/empty.json
output: {count_output: 16}
tool: v1.1.0-dev1/count-lines9-wf.cwl
- doc: Test default value on input parameter
+ doc: Test default value on step input parameter
+ tags: [ inline_javascript ]
+
+- job: v1.1.0-dev1/empty.json
+ output: {count_output: 16}
+ tool: v1.1.0-dev1/count-lines11-wf.cwl
+ doc: Test use default value on step input parameter with empty source
+ tags: [ inline_javascript ]
+
+- job: v1.1.0-dev1/file1-null.json
+ output: {count_output: 16}
+ tool: v1.1.0-dev1/count-lines11-wf.cwl
+ doc: Test use default value on step input parameter with null source
+ tags: [ inline_javascript ]
+
+- job: v1.1.0-dev1/cat-job.json
+ output: {count_output: 1}
+ tool: v1.1.0-dev1/count-lines11-wf.cwl
+ doc: Test default value on step input parameter overridden by provided source
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/revsort-job.json
output:
@@ -321,7 +438,8 @@
location: output.txt
size: 1111
tool: v1.1.0-dev1/revsort.cwl
- doc: Test sample workflows from the specification
+ doc: Test simple workflow
+ tags: [ required ]
- job: v1.1.0-dev1/cat-job.json
output:
@@ -332,6 +450,7 @@
size: 13
tool: v1.1.0-dev1/cat5-tool.cwl
doc: Test unknown hints are ignored.
+ tags: [ required ]
- job: v1.1.0-dev1/search-job.json
output:
@@ -374,7 +493,36 @@
"class": "File",
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"size": 0
- }
+ },
+ {
+ "location": "input.idx6.txt",
+ "class": "File",
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "size": 0
+ },
+ {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "class": "File",
+ "location": "input.txt.idx7",
+ "size": 0
+ },
+ {
+ "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
+ "class": "File",
+ "location": "hello.txt",
+ "size": 13
+ },
+ {
+ "class": "Directory",
+ "listing": [{
+ "basename": "index",
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "class": "File",
+ "location": "index",
+ "size": 0
+ }],
+ "location": "input.txt_idx8",
+ }
],
"size": 1111
}
@@ -382,6 +530,7 @@
doc: |
Test InitialWorkDirRequirement linking input files and capturing secondaryFiles
on input and output.
+ tags: [ initial_work_dir, inline_javascript ]
- job: v1.1.0-dev1/rename-job.json
output:
@@ -393,6 +542,7 @@
tool: v1.1.0-dev1/rename.cwl
doc: |
Test InitialWorkDirRequirement with expression in filename.
+ tags: [ initial_work_dir ]
- job: v1.1.0-dev1/wc-job.json
output:
@@ -400,6 +550,7 @@
tool: v1.1.0-dev1/wc4-tool.cwl
doc: |
Test inline expressions
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/schemadef-job.json
output:
@@ -411,6 +562,7 @@
tool: v1.1.0-dev1/schemadef-tool.cwl
doc: |
Test SchemaDefRequirement definition used in tool parameter
+ tags: [ schema_def, inline_javascript ]
- job: v1.1.0-dev1/schemadef-job.json
output:
@@ -422,6 +574,7 @@
tool: v1.1.0-dev1/schemadef-wf.cwl
doc: |
Test SchemaDefRequirement definition used in workflow parameter
+ tags: [ schema_def ]
- job: v1.1.0-dev1/empty.json
output: {
@@ -492,11 +645,14 @@
"t6": "zab1",
"t7": "zab1",
"t8": "zab1",
- "t9": 2
+ "t9": 2,
+ "t27": null,
+ "t28": 3
}
tool: v1.1.0-dev1/params.cwl
doc: |
Test parameter evaluation, no support for JS expressions
+ tags: [ required ]
- job: v1.1.0-dev1/empty.json
output: {
@@ -567,16 +723,20 @@
"t6": "zab1",
"t7": "zab1",
"t8": "zab1",
- "t9": 2
+ "t9": 2,
+ "t27": null,
+ "t28": 3
}
tool: v1.1.0-dev1/params2.cwl
doc: |
Test parameter evaluation, with support for JS expressions
+ tags: [ inline_javascript ]
- output: {}
job: v1.1.0-dev1/cat-job.json
tool: v1.1.0-dev1/metadata.cwl
doc: Test metadata
+ tags: [ required ]
- job: v1.1.0-dev1/formattest-job.json
output:
@@ -589,6 +749,7 @@
tool: v1.1.0-dev1/formattest.cwl
doc: |
Test simple format checking.
+ tags: [ required ]
- job: v1.1.0-dev1/formattest2-job.json
output:
@@ -601,6 +762,7 @@
tool: v1.1.0-dev1/formattest2.cwl
doc: |
Test format checking against ontology using subclassOf.
+ tags: [ required ]
- job: v1.1.0-dev1/formattest2-job.json
output:
@@ -613,6 +775,7 @@
tool: v1.1.0-dev1/formattest3.cwl
doc: |
Test format checking against ontology using equivalentClass.
+ tags: [ required ]
- tool: v1.1.0-dev1/optional-output.cwl
job: v1.1.0-dev1/cat-job.json
@@ -625,21 +788,34 @@
checksum: "sha1$47a013e660d408619d894b20806b1d5086aab03b"
doc: |
Test optional output file and optional secondaryFile on output.
+ tags: [ docker ]
+
+
+- job: v1.1.0-dev1/empty.json
+ output: {
+ "out": "a string\n"
+ }
+ tool: v1.1.0-dev1/vf-concat.cwl
+ doc: Test that second expression in concatenated valueFrom is not ignored
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/step-valuefrom-wf.json
output: {count_output: 16}
tool: v1.1.0-dev1/step-valuefrom-wf.cwl
doc: Test valueFrom on workflow step.
+ tags: [ step_input, inline_javascript ]
- job: v1.1.0-dev1/step-valuefrom-job.json
output: {val: "3\n"}
tool: v1.1.0-dev1/step-valuefrom2-wf.cwl
doc: Test valueFrom on workflow step with multiple sources
+ tags: [ step_input, inline_javascript, multiple_input ]
- job: v1.1.0-dev1/step-valuefrom-job.json
output: {val: "3\n"}
tool: v1.1.0-dev1/step-valuefrom3-wf.cwl
doc: Test valueFrom on workflow step referencing other inputs
+ tags: [ step_input, inline_javascript ]
- job: v1.1.0-dev1/record-output-job.json
output:
@@ -659,6 +835,7 @@
}
tool: v1.1.0-dev1/record-output.cwl
doc: Test record type output binding.
+ tags: [ shell_command ]
- job: v1.1.0-dev1/empty.json
output: {
@@ -673,6 +850,7 @@
doc: |
Test support for reading cwl.output.json when running in a Docker container
and just 'path' is provided.
+ tags: [ shell_command ]
- job: v1.1.0-dev1/empty.json
output: {
@@ -687,6 +865,7 @@
doc: |
Test support for reading cwl.output.json when running in a Docker container
and just 'location' is provided.
+ tags: [ shell_command ]
- job: v1.1.0-dev1/abc.json
output:
@@ -710,36 +889,62 @@
}]
tool: v1.1.0-dev1/glob-expr-list.cwl
doc: Test support for returning multiple glob patterns from expression
+ tags: [ required ]
- job: v1.1.0-dev1/scatter-valuefrom-job1.json
output:
out: ["foo one one", "foo one two", "foo one three", "foo one four"]
tool: v1.1.0-dev1/scatter-valuefrom-wf1.cwl
doc: Test workflow scatter with single scatter parameter and valueFrom on step input
+ tags: [ scatter, step_input ]
- job: v1.1.0-dev1/scatter-valuefrom-job2.json
output:
out: [["foo one one three", "foo one one four"], ["foo one two three", "foo one two four"]]
tool: v1.1.0-dev1/scatter-valuefrom-wf2.cwl
doc: Test workflow scatter with two scatter parameters and nested_crossproduct join method and valueFrom on step input
+ tags: [ scatter, step_input ]
- job: v1.1.0-dev1/scatter-valuefrom-job2.json
output:
out: ["foo one one three", "foo one one four", "foo one two three", "foo one two four"]
tool: "v1.1.0-dev1/scatter-valuefrom-wf3.cwl#main"
doc: Test workflow scatter with two scatter parameters and flat_crossproduct join method and valueFrom on step input
+ tags: [ scatter, step_input ]
- job: v1.1.0-dev1/scatter-valuefrom-job2.json
output:
out: ["foo one one three", "foo one two four"]
tool: "v1.1.0-dev1/scatter-valuefrom-wf4.cwl#main"
doc: Test workflow scatter with two scatter parameters and dotproduct join method and valueFrom on step input
+ tags: [ scatter, step_input ]
- job: v1.1.0-dev1/scatter-valuefrom-job1.json
output:
out: ["foo one one", "foo two two", "foo three three", "foo four four"]
tool: v1.1.0-dev1/scatter-valuefrom-wf5.cwl
doc: Test workflow scatter with single scatter parameter and valueFrom on step input
+ tags: [ scatter, step_input ]
+
+- job: v1.1.0-dev1/scatter-valuefrom-job3.json
+ tool: v1.1.0-dev1/scatter-valuefrom-wf6.cwl
+ doc: Test valueFrom eval on scattered input parameter
+ output:
+ out_message: [
+ {
+ "checksum": "sha1$98030575f6fc40e5021be5a8803a6bef94aee11f",
+ "location": Any,
+ "class": "File",
+ "size": 16
+ },
+ {
+ "checksum": "sha1$edcacd50778d98ae113015406b3195c165059dd8",
+ "location": Any,
+ "class": "File",
+ "size": 16
+ }
+ ]
+ tags: [ scatter, step_input ]
- job: v1.1.0-dev1/conflict-job.json
output: {
@@ -752,6 +957,7 @@
}
tool: "v1.1.0-dev1/conflict-wf.cwl#collision"
doc: Test workflow two input files with same name.
+ tags: [ required ]
- job: v1.1.0-dev1/dir-job.yml
output:
@@ -763,6 +969,7 @@
}
tool: v1.1.0-dev1/dir.cwl
doc: Test directory input with parameter reference
+ tags: [ shell_command ]
- job: v1.1.0-dev1/dir-job.yml
output:
@@ -774,28 +981,30 @@
}
tool: v1.1.0-dev1/dir2.cwl
doc: Test directory input in Docker
+ tags: [ required ]
- job: v1.1.0-dev1/dir3-job.yml
output:
"outdir": {
"class": "Directory",
"listing": [
- {
- "class": "File",
- "location": "hello.txt",
- "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
- "size": 13
- },
{
"class": "File",
"location": "goodbye.txt",
"checksum": "sha1$dd0a4c4c49ba43004d6611771972b6cf969c1c01",
"size": 24
+ },
+ {
+ "class": "File",
+ "location": "hello.txt",
+ "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
+ "size": 13
}
],
}
tool: v1.1.0-dev1/dir3.cwl
- doc: Test directory input in Docker
+ doc: Test directory output
+ tags: [ required ]
- job: v1.1.0-dev1/dir4-job.yml
output: {
@@ -808,18 +1017,20 @@
}
tool: v1.1.0-dev1/dir4.cwl
doc: Test directories in secondaryFiles
+ tags: [ shell_command ]
- job: v1.1.0-dev1/dir-job.yml
output: {
"outlist": {
- "checksum": "sha1$907a866a3e0b7f1fc5a2222531c5fb9063704438",
- "size": 33,
+ "checksum": "sha1$13cda8661796ae241da3a18668fb552161a72592",
+ "size": 20,
"location": "output.txt",
"class": "File"
}
}
tool: v1.1.0-dev1/dir5.cwl
doc: Test dynamic initial work dir
+ tags: [ shell_command, initial_work_dir ]
- job: v1.1.0-dev1/stagefile-job.yml
output: {
@@ -832,6 +1043,7 @@
}
tool: v1.1.0-dev1/stagefile.cwl
doc: Test writable staged files.
+ tags: [ initial_work_dir ]
- job: v1.1.0-dev1/file-literal.yml
output:
@@ -842,16 +1054,18 @@
size: 18
tool: v1.1.0-dev1/cat3-tool.cwl
doc: Test file literal as input
+ tags: [ required ]
-- job: examples/arguments-job.yml
+- job: v1.1.0-dev1/arguments-job.yml
output:
classfile:
checksum: sha1$e68df795c0686e9aa1a1195536bd900f5f417b18
location: Hello.class
class: File
size: 184
- tool: examples/linkfile.cwl
+ tool: v1.1.0-dev1/linkfile.cwl
doc: Test expression in InitialWorkDir listing
+ tags: [ inline_javascript, initial_work_dir ]
- job: v1.1.0-dev1/wc-job.json
output:
@@ -862,6 +1076,7 @@
size: 21
tool: v1.1.0-dev1/nameroot.cwl
doc: Test nameroot/nameext expression in arguments, stdout
+ tags: [ required ]
- job: v1.1.0-dev1/dir-job.yml
output:
@@ -873,6 +1088,7 @@
}
tool: v1.1.0-dev1/dir6.cwl
doc: Test directory input with inputBinding
+ tags: [ shell_command ]
- job: v1.1.0-dev1/nested-array-job.yml
output:
@@ -883,16 +1099,19 @@
size: 2
tool: v1.1.0-dev1/nested-array.cwl
doc: Test command line generation of array-of-arrays
+ tags: [ required ]
- job: v1.1.0-dev1/empty.json
output: {}
tool: v1.1.0-dev1/envvar.cwl
doc: Test $HOME and $TMPDIR are set correctly
+ tags: [ shell_command ]
- job: v1.1.0-dev1/empty.json
output: {}
tool: v1.1.0-dev1/envvar2.cwl
doc: Test $HOME and $TMPDIR are set correctly in Docker
+ tags: [ shell_command ]
- job: v1.1.0-dev1/empty.json
output:
@@ -904,6 +1123,7 @@
}
tool: "v1.1.0-dev1/js-expr-req-wf.cwl#wf"
doc: Test that expressionLib requirement of individual tool step overrides expressionLib of workflow.
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/initialworkdirrequirement-docker-out-job.json
output:
@@ -920,11 +1140,13 @@
"size": 12010
tool: v1.1.0-dev1/initialworkdirrequirement-docker-out.cwl
doc: Test output of InitialWorkDir
+ tags: [ docker, initial_work_dir ]
- job: v1.1.0-dev1/wc-job.json
output: {count_output: 16}
tool: v1.1.0-dev1/count-lines10-wf.cwl
doc: Test embedded subworkflow
+ tags: [ subworkflow ]
- job: v1.1.0-dev1/docker-array-secondaryfiles-job.json
output: {
@@ -937,6 +1159,7 @@
}
tool: v1.1.0-dev1/docker-array-secondaryfiles.cwl
doc: Test secondaryFiles on array of files.
+ tags: [ docker, inline_javascript, shell_command ]
- job: v1.1.0-dev1/dir7.yml
output: {
@@ -961,6 +1184,7 @@
}
tool: v1.1.0-dev1/dir7.cwl
doc: Test directory literal output created by ExpressionTool
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/empty.json
output:
@@ -971,6 +1195,7 @@
size: 19
tool: v1.1.0-dev1/file-literal-ex.cwl
doc: Test file literal output created by ExpressionTool
+ tags: [ inline_javascript ]
- job: v1.1.0-dev1/empty.json
output:
@@ -982,6 +1207,7 @@
}
tool: v1.1.0-dev1/docker-output-dir.cwl
doc: Test dockerOutputDirectory
+ tags: [ docker ]
- job: v1.1.0-dev1/empty.json
output:
@@ -992,8 +1218,503 @@
size: 15
tool: v1.1.0-dev1/imported-hint.cwl
doc: Test hints with $import
+ tags: [ required ]
- output: {}
- job: v1.0/default_path_job.yml
- tool: v1.0/default_path.cwl
+ job: v1.1.0-dev1/default_path_job.yml
+ tool: v1.1.0-dev1/default_path.cwl
doc: Test warning instead of error when default path is not found
+ tags: [ required ]
+
+- output:
+ args: [-A,'2',-B,baz,-C,'10','9','8','7','6','5','4','3','2','1']
+ job: v1.1.0-dev1/empty.json
+ tool: v1.1.0-dev1/inline-js.cwl
+ doc: Test InlineJavascriptRequirement with multiple expressions in the same tool
+ tags: [ inline_javascript ]
+
+- job: v1.1.0-dev1/recursive-input-directory.yml
+ output:
+ output_dir: {
+ "basename": "work_dir",
+ "class": "Directory",
+ "listing": [
+ {
+ "basename": "a",
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "class": "File",
+ "location": "work_dir/a",
+ "size": 0
+ },
+ {
+ "basename": "b",
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "class": "File",
+ "location": "work_dir/b",
+ "size": 0
+ },
+ {
+ "basename": "c",
+ "class": "Directory",
+ "listing": [
+ {
+ "basename": "d",
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "class": "File",
+ "location": "work_dir/c/d",
+ "size": 0
+ }
+ ],
+ "location": "work_dir/c",
+ },
+ {
+ "basename": "e",
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "class": "File",
+ "location": "work_dir/e",
+ "size": 0
+ },
+ ],
+ "location": "work_dir",
+ }
+ test_result: {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "class": "File",
+ "location": "output.txt",
+ "size": 0
+ }
+ tool: v1.1.0-dev1/recursive-input-directory.cwl
+ doc: Test if a writable input directory is recursivly copied and writable
+ tags: [ inline_javascript, initial_work_dir, shell_command ]
+
+- output:
+ out: "t\n"
+ job: v1.1.0-dev1/empty.json
+ tool: v1.1.0-dev1/null-defined.cwl
+ doc: Test that missing parameters are null (not undefined) in expression
+ tags: [ inline_javascript ]
+
+- output:
+ out: "f\n"
+ job: v1.1.0-dev1/cat-job.json
+ tool: v1.1.0-dev1/null-defined.cwl
+ doc: Test that provided parameter is not null in expression
+ tags: [ inline_javascript ]
+
+- job: v1.1.0-dev1/revsort-job.json
+ output:
+ output:
+ class: File
+ checksum: sha1$b9214658cc453331b62c2282b772a5c063dbd284
+ location: output.txt
+ size: 1111
+ tool: v1.1.0-dev1/revsort-packed.cwl#main
+ doc: Test compound workflow document
+ tags: [ required ]
+
+- job: v1.1.0-dev1/basename-fields-job.yml
+ output:
+ extFile:
+ checksum: sha1$301a72c82a835e1737caf30f94d0eec210c4d9f1
+ class: File
+ size: 5
+ location: Any
+ path: Any
+ rootFile:
+ checksum: sha1$b4a583c391e234cf210e1d576f68f674c8ad7ecd
+ class: File
+ size: 10
+ location: Any
+ path: Any
+ tool: v1.1.0-dev1/basename-fields-test.cwl
+ doc: Test that nameroot and nameext are generated from basename at execution time by the runner
+ tags: [ step_input_expression ]
+
+- job: v1.1.0-dev1/wc-job.json
+ output: {}
+ tool: v1.1.0-dev1/initialwork-path.cwl
+ doc: Test that file path in $(inputs) for initialworkdir is in $(outdir).
+ tags: [ initial_work_dir ]
+
+- job: v1.1.0-dev1/count-lines6-job.json
+ output:
+ count_output: 34
+ tool: v1.1.0-dev1/count-lines12-wf.cwl
+ doc: |
+ Test single step workflow with Scatter step and two data links connected to
+ same input, flattened merge behavior. Workflow inputs are set as list
+ tags: [ multiple_input, inline_javascript ]
+
+- job: v1.1.0-dev1/sum-job.json
+ output:
+ result: 12
+ tool: v1.1.0-dev1/sum-wf.cwl
+ doc: Test step input with multiple sources with multiple types
+ tags: [ step_input, inline_javascript, multiple_input ]
+
+- job: v1.1.0-dev1/empty.json
+ output: {
+ "stderr_file": {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "location": Any,
+ "class": "File",
+ "size": 0
+ },
+ "stdout_file": {
+ "checksum": "sha1$1555252d52d4ec3262538a4426a83a99cfff4402",
+ "location": Any,
+ "class": "File",
+ "size": 9
+ }
+ }
+ tool: v1.1.0-dev1/shellchar.cwl
+ doc: "Test that shell directives are not interpreted."
+ tags: [ required ]
+
+- job: v1.1.0-dev1/empty.json
+ output: {
+ "stderr_file": {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "location": Any,
+ "class": "File",
+ "size": 0
+ },
+ "stdout_file": {
+ "checksum": "sha1$1555252d52d4ec3262538a4426a83a99cfff4402",
+ "location": Any,
+ "class": "File",
+ "size": 9
+ }
+ }
+ tool: v1.1.0-dev1/shellchar2.cwl
+ doc: "Test that shell directives are quoted."
+ tags: [ shell_command ]
+
+- job: v1.1.0-dev1/empty.json
+ output:
+ out: {
+ "basename": "emptyWritableDir",
+ "listing": [
+ {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "basename": "blurg",
+ "location": "blurg",
+ "class": "File",
+ "size": 0
+ }
+ ],
+ "location": "emptyWritableDir",
+ "class": "Directory"
+ }
+ tool: v1.1.0-dev1/writable-dir.cwl
+ doc: Test empty writable dir with InitialWorkDirRequirement
+ tags: [ inline_javascript, initial_work_dir ]
+
+- job: v1.1.0-dev1/empty.json
+ output:
+ out: {
+ "basename": "emptyWritableDir",
+ "listing": [
+ {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "basename": "blurg",
+ "location": "blurg",
+ "class": "File",
+ "size": 0
+ }
+ ],
+ "location": "emptyWritableDir",
+ "class": "Directory"
+ }
+ tool: v1.1.0-dev1/writable-dir-docker.cwl
+ doc: Test empty writable dir with InitialWorkDirRequirement inside Docker
+ tags: [ inline_javascript, initial_work_dir ]
+
+- job: v1.1.0-dev1/dynresreq-job.json
+ tool: v1.1.0-dev1/dynresreq.cwl
+ doc: Test dynamic resource reqs referencing inputs
+ output:
+ output: {
+ "location": "cores.txt",
+ "size": 2,
+ "class": "File",
+ "checksum": "sha1$7448d8798a4380162d4b56f9b452e2f6f9e24e7a"
+ }
+ tags: [ resource ]
+
+- job: v1.1.0-dev1/file-literal.yml
+ output:
+ output_file:
+ class: File
+ checksum: sha1$d0e04ff6c413c7d57f9a0ca0a33cd3ab52e2dd9c
+ location: output.txt
+ size: 18
+ tool: v1.1.0-dev1/cat3-nodocker.cwl
+ doc: Test file literal as input without Docker
+ tags: [ required ]
+
+- doc: Test that OutputBinding.glob is sorted as specified by POSIX
+ job: v1.1.0-dev1/empty.json
+ tool: v1.1.0-dev1/glob_test.cwl
+ output:
+ letters:
+ - checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
+ location: a
+ class: File
+ size: 0
+ - checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
+ location: b
+ class: File
+ size: 0
+ - checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
+ location: c
+ class: File
+ size: 0
+ - checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
+ location: w
+ class: File
+ size: 0
+ - checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
+ location: x
+ class: File
+ size: 0
+ - checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
+ location: y
+ class: File
+ size: 0
+ - checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
+ location: z
+ class: File
+ size: 0
+ tags: [ required ]
+
+- doc: Test InitialWorkDirRequirement with a nested directory structure from another step
+ job: v1.1.0-dev1/empty.json
+ output:
+ ya_empty:
+ class: File
+ checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
+ location: ya
+ size: 0
+ tool: v1.1.0-dev1/iwdr_with_nested_dirs.cwl
+ tags: [ initial_work_dir ]
+
+- job: v1.1.0-dev1/bool-empty-inputbinding-job.json
+ output: {
+ "args": [
+ ]
+ }
+ tool: v1.1.0-dev1/bool-empty-inputbinding.cwl
+ doc: "Test that boolean flags do not appear on command line if inputBinding is empty and not null"
+ tags: [ required ]
+
+- job: v1.1.0-dev1/empty.json
+ output: {
+ "args": []
+ }
+ tool: v1.1.0-dev1/stage-unprovided-file.cwl
+ doc: Test that expression engine does not fail to evaluate reference to self
+ with unprovided input
+ tags: [ inline_javascript ]
+
+- tool: v1.1.0-dev1/exit-success.cwl
+ doc: Test successCodes
+ job: v1.1.0-dev1/empty.json
+ output: {}
+ tags: [ required ]
+
+- job: v1.1.0-dev1/dynresreq-job.json
+ doc: Test simple workflow with a dynamic resource requirement
+ tool: v1.1.0-dev1/dynresreq-workflow.cwl
+ output:
+ cores: {
+ "location": "output",
+ "size": 2,
+ "class": "File",
+ "checksum": "sha1$7448d8798a4380162d4b56f9b452e2f6f9e24e7a"
+ }
+ tags: [ resource ]
+
+- job: v1.1.0-dev1/empty-array-job.json
+ output: {
+ "args": []
+ }
+ tool: v1.1.0-dev1/empty-array-input.cwl
+ doc: "Test that empty array input does not add anything to command line"
+ tags: [ required ]
+
+- job: v1.1.0-dev1/empty.json
+ tool: v1.1.0-dev1/steplevel-resreq.cwl
+ doc: Test that ResourceRequirement on a step level redefines requirement on the workflow level
+ output:
+ out: {
+ "location": "cores.txt",
+ "size": 2,
+ "class": "File",
+ "checksum": "sha1$e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e"
+ }
+ tags: [ resource ]
+
+- job: v1.1.0-dev1/array-of-strings-job.yml
+ output: {
+ "args": ["replacementValue"]
+ }
+ tool: v1.1.0-dev1/valueFrom-constant.cwl
+ doc: Test valueFrom with constant value overriding provided array inputs
+ tags: [ inline_javascript ]
+
+- job: v1.1.0-dev1/record-secondaryFiles-job.yml
+ output: {}
+ tool: v1.1.0-dev1/record-in-secondaryFiles.cwl
+ doc: Test secondaryFiles on anonymous record fields
+ tags: [ required ]
+
+- job: v1.1.0-dev1/record-secondaryFiles-job.yml
+ output: {}
+ tool: v1.1.0-dev1/record-sd-secondaryFiles.cwl
+ doc: Test secondaryFiles on SchemaDefRequirement record fields
+ tags: [ required ]
+
+- job: v1.1.0-dev1/empty.json
+ output: {
+ "record_output": {
+ "f1": {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "location": "A",
+ "secondaryFiles": [
+ {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "location": "A.s2",
+ "class": "File",
+ "size": 0
+ }
+ ],
+ "class": "File",
+ "size": 0
+ },
+ "f2": [
+ {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "location": "B",
+ "secondaryFiles": [
+ {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "location": "B.s3",
+ "class": "File",
+ "size": 0
+ }
+ ],
+ "class": "File",
+ "size": 0
+ },
+ {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "location": "C",
+ "secondaryFiles": [
+ {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "location": "C.s3",
+ "class": "File",
+ "size": 0
+ }
+ ],
+ "class": "File",
+ "size": 0
+ }
+ ]
+ }
+ }
+ tool: v1.1.0-dev1/record-out-secondaryFiles.cwl
+ doc: Test secondaryFiles on output record fields
+ tags: [ required ]
+
+- job: v1.1.0-dev1/record-secondaryFiles-job.yml
+ output: {}
+ tool: v1.1.0-dev1/record-in-secondaryFiles-wf.cwl
+ doc: Test secondaryFiles propagation through workflow
+ tags: [ required ]
+
+- job: v1.1.0-dev1/record-secondaryFiles-job.yml
+ should_fail: true
+ tool: v1.1.0-dev1/record-in-secondaryFiles-missing-wf.cwl
+ doc: Test checking when secondaryFiles are missing
+ tags: [ required ]
+
+- job: v1.1.0-dev1/record-format-job.yml
+ output: {}
+ tool: v1.1.0-dev1/record-in-format.cwl
+ doc: Test format on anonymous record fields
+ tags: [ required ]
+
+- job: v1.1.0-dev1/record-format-job2.yml
+ should_fail: true
+ tool: v1.1.0-dev1/record-in-format.cwl
+ doc: Test file format checking on parameter
+ tags: [ format_checking ]
+
+- job: v1.1.0-dev1/record-format-job3.yml
+ should_fail: true
+ tool: v1.1.0-dev1/record-in-format.cwl
+ doc: Test file format checking on record field
+ tags: [ format_checking ]
+
+- job: v1.1.0-dev1/record-format-job4.yml
+ should_fail: true
+ tool: v1.1.0-dev1/record-in-format.cwl
+ doc: Test file format checking on array item
+ tags: [ format_checking ]
+
+- job: v1.1.0-dev1/record-secondaryFiles-job.yml
+ output: {
+ "f1out": {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "class": "File",
+ "location": "A",
+ "size": 0,
+ "format": "http://example.com/format1"
+ },
+ "record_output": {
+ "f2out": {
+ "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ "class": "File",
+ "location": "B",
+ "size": 0,
+ "format": "http://example.com/format2"
+ }
+ }
+ }
+ tool: v1.1.0-dev1/record-out-format.cwl
+ doc: Test format on output record fields
+ tags: [ required ]
+
+- job: v1.1.0-dev1/wf-loadContents-job.yml
+ output: {
+ "my_int": 42
+ }
+ tool: v1.1.0-dev1/wf-loadContents.cwl
+ doc: Test WorkflowInputParameter.inputBinding.loadContents
+ tags: [ required ]
+
+- job: v1.1.0-dev1/wf-loadContents-job.yml
+ output: {
+ "my_int": 42
+ }
+ tool: v1.1.0-dev1/wf-loadContents2.cwl
+ doc: Test WorkflowInputParameter.loadContents
+ tags: [ required ]
+
+- job: v1.1.0-dev1/wf-loadContents-job.yml
+ output: {
+ "my_int": 42
+ }
+ tool: v1.1.0-dev1/wf-loadContents3.cwl
+ doc: Test loadContents on InputParameter.loadContents (expression)
+ tags: [ required ]
+
+- job: v1.1.0-dev1/wf-loadContents-job.yml
+ output: {
+ "my_int": 42
+ }
+ tool: v1.1.0-dev1/wf-loadContents4.cwl
+ doc: Test WorkflowStepInput.loadContents
+ tags: [ required ]
diff --git a/v1.1.0-dev1/contrib.md b/v1.1.0-dev1/contrib.md
index af6f6e8da..4e999eb9d 100644
--- a/v1.1.0-dev1/contrib.md
+++ b/v1.1.0-dev1/contrib.md
@@ -1,7 +1,7 @@
Authors:
-* Peter Amstutz , Arvados Project, Curoverse
-* Michael R. Crusoe , Common Workflow Language
+* Peter Amstutz , Arvados Project, Veritas Genetics
+* Michael R. Crusoe , Common Workflow Language
project
* Nebojša Tijanić , Seven Bridges Genomics
@@ -9,11 +9,11 @@ Contributors:
* Brad Chapman , Harvard Chan School of Public Health
* John Chilton , Galaxy Project, Pennsylvania State University
-* Michael Heuer ,UC Berkeley AMPLab
+* Michael Heuer , UC Berkeley AMPLab
* Andrey Kartashov , Cincinnati Children's Hospital
* Dan Leehr , Duke University
* Hervé Ménager , Institut Pasteur
* Maya Nedeljkovich , Seven Bridges Genomics
* Matt Scales , Institute of Cancer Research, London
-* Stian Soiland-Reyes [soiland-reyes@cs.manchester.ac.uk](mailto:soiland-reyes@cs.manchester.ac.uk), University of Manchester
+* Stian Soiland-Reyes , University of Manchester
* Luka Stojanovic , Seven Bridges Genomics
diff --git a/v1.1.0-dev1/cwl-runner.cwl b/v1.1.0-dev1/cwl-runner.cwl
new file mode 100644
index 000000000..e87064e04
--- /dev/null
+++ b/v1.1.0-dev1/cwl-runner.cwl
@@ -0,0 +1,58 @@
+#!/usr/bin/env cwl-runner
+class: CommandLineTool
+cwlVersion: v1.1.0-dev1
+
+doc: |
+ Generic interface to run a Common Workflow Language tool or workflow from the
+ command line. To be implemented by each CWL compliant execution platform for
+ testing conformance to the standard and optionally for use by users.
+
+inputs:
+ outdir:
+ type: string?
+ doc: |
+ Output directory, defaults to the current directory
+ inputBinding:
+ prefix: "--outdir"
+
+ quiet:
+ type: boolean?
+ doc: no diagnostic output
+ inputBinding:
+ prefix: "--quiet"
+
+ version:
+ type: boolean?
+ doc: report the name & version, then quit without further processing
+ inputBinding:
+ prefix: "--version"
+
+ processfile:
+ type: File?
+ doc: |
+ The CommandLineTool, ExpressionTool, or Workflow description to run.
+ Optional if the jobfile has a `cwl:tool` field to indicate which process
+ description to run.
+ inputBinding:
+ position: 1
+
+ jobfile:
+ type: File
+ doc: The input job document.
+ inputBinding:
+ position: 2
+
+baseCommand: cwl-runner
+
+outputs:
+ log: stderr
+ output_object_document: stdout # in the CWL Output Object Document format
+ format: iana:application/json
+
+successCodes:
+ - 0 # success
+
+permanentFailCodes:
+ - 33 # failure due to unimplemented feature
+
+$namespaces: { iana: https://www.iana.org/assignments/media-types/ }
diff --git a/v1.1.0-dev1/invocation.md b/v1.1.0-dev1/invocation.md
index ce7524cc9..6850ef7a5 100644
--- a/v1.1.0-dev1/invocation.md
+++ b/v1.1.0-dev1/invocation.md
@@ -54,9 +54,12 @@ system or transfer files via explicit download to the host. Implementations
may choose not to provide access to files not explicitly specified in the input
object or process requirements.
-Output files produced by tool execution must be written to the **designated
-output directory**. The initial current working directory when executing
-the tool must be the designated output directory.
+Output files produced by tool execution must be written to the
+**designated output directory**. The initial current working
+directory when executing the tool must be the designated output
+directory. The designated output directory should be empty, except
+for files or directories specified using
+[InitialWorkDirRequirement](InitialWorkDirRequirement).
Files may also be written to the **designated temporary directory**. This
directory must be isolated and not shared with other processes. Any files
diff --git a/v1.1.0-dev1/salad/.gitignore b/v1.1.0-dev1/salad/.gitignore
index 85014c5d4..3d7187d48 100644
--- a/v1.1.0-dev1/salad/.gitignore
+++ b/v1.1.0-dev1/salad/.gitignore
@@ -2,3 +2,11 @@
.eggs
*.egg-info/
*pyc
+
+build/
+dist/
+ruamel
+typeshed/2and3/ruamel/yaml
+# virtualenv
+venv/
+.cache/
diff --git a/v1.1.0-dev1/salad/.travis.yml b/v1.1.0-dev1/salad/.travis.yml
index 869d06fbe..f8be49a56 100644
--- a/v1.1.0-dev1/salad/.travis.yml
+++ b/v1.1.0-dev1/salad/.travis.yml
@@ -1,13 +1,19 @@
sudo: false
language: python
+cache: pip
python:
- 2.7
+ - 3.4
- 3.5
+ - 3.6
os:
- linux
-
+jobs:
+ include:
+ - stage: release-test
+ script: RELEASE_SKIP=head PYVER= ./release-test.sh
+script: tox
install:
- pip install tox-travis
-script: tox
notifications:
email: false
diff --git a/v1.1.0-dev1/salad/MANIFEST.in b/v1.1.0-dev1/salad/MANIFEST.in
index bf8066c1f..661c696f5 100644
--- a/v1.1.0-dev1/salad/MANIFEST.in
+++ b/v1.1.0-dev1/salad/MANIFEST.in
@@ -1,4 +1,9 @@
include gittaggers.py Makefile
-include schema_salad/tests/*.py schema_salad/tests/*.yml schema_salad/tests/*.owl
+include schema_salad/tests/*
+include schema_salad/tests/test_schema/*.md
+include schema_salad/tests/test_schema/*.yml
+include schema_salad/tests/test_schema/*.cwl
include schema_salad/metaschema/*
+include schema_salad/tests/docimp/*
+global-exclude *~
global-exclude *.pyc
diff --git a/v1.1.0-dev1/salad/Makefile b/v1.1.0-dev1/salad/Makefile
index ba220d3c5..dec3bfaa8 100644
--- a/v1.1.0-dev1/salad/Makefile
+++ b/v1.1.0-dev1/salad/Makefile
@@ -22,16 +22,23 @@
# make coverage-report to check coverage of the python scripts by the tests
MODULE=schema_salad
+PACKAGE=schema-salad
-# `SHELL=bash` Will break Titus's laptop, so don't use BASH-isms like
+# `SHELL=bash` doesn't work for some, so don't use BASH-isms like
# `[[` conditional expressions.
PYSOURCES=$(wildcard ${MODULE}/**.py tests/*.py) setup.py
-DEVPKGS=pep8 diff_cover autopep8 pylint coverage pep257 pytest
+DEVPKGS=pep8 diff_cover autopep8 pylint coverage pep257 pytest flake8
+COVBASE=coverage run --branch --append --source=${MODULE} \
+ --omit=schema_salad/tests/*
-VERSION=$(shell git describe --tags --dirty | sed s/v//)
+# Updating the Major & Minor version below?
+# Don't forget to update setup.py as well
+VERSION=2.7.$(shell date +%Y%m%d%H%M%S --utc --date=`git log --first-parent \
+ --max-count=1 --format=format:%cI`)
## all : default task
-all: ./setup.py develop
+all:
+ pip install -e .
## help : print this help message and exit
help: Makefile
@@ -52,7 +59,7 @@ install: FORCE
dist: dist/${MODULE}-$(VERSION).tar.gz
dist/${MODULE}-$(VERSION).tar.gz: $(SOURCES)
- ./setup.py sdist
+ ./setup.py sdist bdist_wheel
## clean : clean up all temporary / machine-generated files
clean: FORCE
@@ -102,38 +109,32 @@ pylint_report.txt: ${PYSOURCES}
diff_pylint_report: pylint_report.txt
diff-quality --violations=pylint pylint_report.txt
-.coverage: $(PYSOURCES)
- coverage run --branch --source=${MODULE} setup.py test
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+.coverage: $(PYSOURCES) all
+ rm -f .coverage
+ $(COVBASE) setup.py test
+ $(COVBASE) -m schema_salad.main \
--print-jsonld-context schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-rdfs schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-avro schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-rdf schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-pre schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-index schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.main \
+ $(COVBASE) -m schema_salad.main \
--print-metadata schema_salad/metaschema/metaschema.yml \
> /dev/null
- coverage run --append --branch --source=${MODULE} \
- -m schema_salad.makedoc schema_salad/metaschema/metaschema.yml \
+ $(COVBASE) -m schema_salad.makedoc \
+ schema_salad/metaschema/metaschema.yml \
> /dev/null
coverage.xml: .coverage
@@ -169,15 +170,23 @@ list-author-emails:
@echo 'name, E-Mail Address'
@git log --format='%aN,%aE' | sort -u | grep -v 'root'
-mypy: ${PYSOURCES}
- rm -Rf typeshed/2.7/ruamel/yaml
+mypy2: ${PYSOURCES}
+ rm -Rf typeshed/2and3/ruamel/yaml
ln -s $(shell python -c 'from __future__ import print_function; import ruamel.yaml; import os.path; print(os.path.dirname(ruamel.yaml.__file__))') \
- typeshed/2.7/ruamel/
- MYPYPATH=typeshed/2.7 mypy --py2 --disallow-untyped-calls \
- --fast-parser --warn-redundant-casts --warn-unused-ignores \
+ typeshed/2and3/ruamel/
+ MYPYPATH=$MYPYPATH:typeshed/2.7:typeshed/2and3 mypy --py2 --disallow-untyped-calls \
+ --warn-redundant-casts \
schema_salad
-jenkins:
+mypy3: ${PYSOURCES}
+ rm -Rf typeshed/2and3/ruamel/yaml
+ ln -s $(shell python -c 'from __future__ import print_function; import ruamel.yaml; import os.path; print(os.path.dirname(ruamel.yaml.__file__))') \
+ typeshed/2and3/ruamel/
+ MYPYPATH=$MYPYPATH:typeshed/3:typeshed/2and3 mypy --disallow-untyped-calls \
+ --warn-redundant-casts \
+ schema_salad
+
+jenkins: FORCE
rm -Rf env && virtualenv env
. env/bin/activate ; \
pip install -U setuptools pip wheel ; \
@@ -187,6 +196,21 @@ jenkins:
. env3/bin/activate ; \
pip install -U setuptools pip wheel ; \
${MAKE} install-dep ; \
- pip install -U mypy-lang typed-ast ; ${MAKE} mypy
+ pip install -U -r mypy_requirements.txt ; ${MAKE} mypy2
+ # pip install -U -r mypy_requirements.txt ; ${MAKE} mypy3
+
+release-test: FORCE
+ git diff-index --quiet HEAD -- || ( echo You have uncommited changes, please commit them and try again; false )
+ PYVER=2.7 ./release-test.sh
+ PYVER=3 ./release-test.sh
+
+release: release-test
+ . testenv2.7_2/bin/activate && \
+ testenv2.7_2/src/${PACKAGE}/setup.py sdist bdist_wheel
+ . testenv2.7_2/bin/activate && \
+ pip install twine && \
+ twine upload testenv2.7_2/src/${PACKAGE}/dist/* && \
+ git tag ${VERSION} && git push --tags
+
FORCE:
diff --git a/v1.1.0-dev1/salad/README.rst b/v1.1.0-dev1/salad/README.rst
index 11d1b06b6..7901a7624 100644
--- a/v1.1.0-dev1/salad/README.rst
+++ b/v1.1.0-dev1/salad/README.rst
@@ -1,3 +1,10 @@
+|Build Status| |Build status|
+
+.. |Build Status| image:: https://img.shields.io/travis/common-workflow-language/schema_salad/master.svg?label=unix%20build
+ :target: https://travis-ci.org/common-workflow-language/schema_salad
+.. |Build status| image:: https://img.shields.io/appveyor/ci/mr-c/schema-salad/master.svg?label=windows%20build
+ :target: https://ci.appveyor.com/project/mr-c/schema-salad/branch/master
+
Schema Salad
------------
@@ -27,6 +34,13 @@ Usage
$ python
>>> import schema_salad
+To install from source::
+
+ git clone https://github.com/common-workflow-language/schema_salad
+ cd schema_salad
+ python setup.py install
+
+
Documentation
-------------
@@ -70,6 +84,6 @@ provides for robust support of inline documentation.
.. _JSON-LD: http://json-ld.org
.. _Avro: http://avro.apache.org
.. _metaschema: https://github.com/common-workflow-language/schema_salad/blob/master/schema_salad/metaschema/metaschema.yml
-.. _specification: http://www.commonwl.org/draft-3/SchemaSalad.html
-.. _Language: https://github.com/common-workflow-language/common-workflow-language/blob/master/draft-3/CommandLineTool.yml
+.. _specification: http://www.commonwl.org/v1.0/SchemaSalad.html
+.. _Language: https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/CommandLineTool.yml
.. _RDF: https://www.w3.org/RDF/
diff --git a/v1.1.0-dev1/salad/appveyor.yml b/v1.1.0-dev1/salad/appveyor.yml
new file mode 100644
index 000000000..b7fe5d54f
--- /dev/null
+++ b/v1.1.0-dev1/salad/appveyor.yml
@@ -0,0 +1,34 @@
+version: .{build}-{branch}
+
+environment:
+
+ matrix:
+ - PYTHON: "C:\\Python27"
+ PYTHON_VERSION: "2.7.x"
+ PYTHON_ARCH: "32"
+
+ - PYTHON: "C:\\Python27-x64"
+ PYTHON_VERSION: "2.7.x"
+ PYTHON_ARCH: "64"
+
+ - PYTHON: "C:\\Python34-x64"
+ PYTHON_VERSION: "3.4.x"
+ PYTHON_ARCH: "64"
+
+ - PYTHON: "C:\\Python35-x64"
+ PYTHON_VERSION: "3.5.x"
+ PYTHON_ARCH: "64"
+
+install:
+ - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
+ - "python --version"
+ - "pip install --disable-pip-version-check --user --upgrade pip"
+ - "pip install --disable-pip-version-check --user --upgrade setuptools"
+
+build_script:
+ - "%CMD_IN_ENV% pip install ."
+
+
+test_script:
+ - "%CMD_IN_ENV% python setup.py test"
+
diff --git a/v1.1.0-dev1/salad/mypy.ini b/v1.1.0-dev1/salad/mypy.ini
new file mode 100644
index 000000000..3dccfb5bc
--- /dev/null
+++ b/v1.1.0-dev1/salad/mypy.ini
@@ -0,0 +1,2 @@
+[mypy-ruamel.*]
+ignore_errors = True
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/mypy_requirements.txt b/v1.1.0-dev1/salad/mypy_requirements.txt
new file mode 100644
index 000000000..3fc6a3cad
--- /dev/null
+++ b/v1.1.0-dev1/salad/mypy_requirements.txt
@@ -0,0 +1 @@
+mypy==0.520
diff --git a/v1.1.0-dev1/salad/release-test.sh b/v1.1.0-dev1/salad/release-test.sh
index 0873214f0..c77057182 100755
--- a/v1.1.0-dev1/salad/release-test.sh
+++ b/v1.1.0-dev1/salad/release-test.sh
@@ -5,63 +5,104 @@ set -x
package=schema-salad
module=schema_salad
-repo=https://github.com/common-workflow-language/schema_salad.git
-run_tests="py.test --pyarg ${module}"
-pipver=6.0 # minimum required version of pip
+slug=${TRAVIS_PULL_REQUEST_SLUG:=common-workflow-language/schema_salad}
+repo=https://github.com/${slug}.git
+run_tests="bin/py.test --pyarg ${module}"
+pipver=8.0.1 # minimum required version of pip
+setupver=20.10.1 # minimum required version of setuptools
+PYVER=${PYVER:=2.7}
-rm -Rf testenv? || /bin/true
+rm -Rf "testenv${PYVER}_"? || /bin/true
-export HEAD=`git rev-parse HEAD`
-virtualenv testenv1
-virtualenv testenv2
-virtualenv testenv3
-virtualenv testenv4
+export HEAD=${TRAVIS_PULL_REQUEST_SHA:-$(git rev-parse HEAD)}
-# First we test the head
-source testenv1/bin/activate
-rm testenv1/lib/python-wheels/setuptools* \
- && pip install --force-reinstall -U pip==${pipver} \
- && pip install setuptools==20.10.1
-make install-dependencies
-make test
-pip uninstall -y ${package} || true; pip uninstall -y ${package} || true; make install
-mkdir testenv1/not-${module}
-# if there is a subdir named '${module}' py.test will execute tests
-# there instead of the installed module's tests
-pushd testenv1/not-${module}; ../bin/${run_tests}; popd
+if [ "${RELEASE_SKIP}" != "head" ]
+then
+ virtualenv "testenv${PYVER}_1" -p "python${PYVER}"
+ # First we test the head
+ # shellcheck source=/dev/null
+ source "testenv${PYVER}_1/bin/activate"
+ rm "testenv${PYVER}_1/lib/python-wheels/setuptools"* \
+ && pip install --force-reinstall -U pip==${pipver} \
+ && pip install setuptools==${setupver} wheel
+ make install-dependencies
+ make test
+ pip uninstall -y ${package} || true; pip uninstall -y ${package} \
+ || true; make install
+ mkdir "testenv${PYVER}_1/not-${module}"
+ # if there is a subdir named '${module}' py.test will execute tests
+ # there instead of the installed module's tests
+
+ pushd "testenv${PYVER}_1/not-${module}"
+ # shellcheck disable=SC2086
+ ../${run_tests}; popd
+fi
+
+
+virtualenv "testenv${PYVER}_2" -p "python${PYVER}"
+virtualenv "testenv${PYVER}_3" -p "python${PYVER}"
+virtualenv "testenv${PYVER}_4" -p "python${PYVER}"
+virtualenv "testenv${PYVER}_5" -p "python${PYVER}"
# Secondly we test via pip
-cd testenv2
+pushd "testenv${PYVER}_2"
+# shellcheck source=/dev/null
source bin/activate
rm lib/python-wheels/setuptools* \
&& pip install --force-reinstall -U pip==${pipver} \
- && pip install setuptools==20.10.1
-pip install -e git+${repo}@${HEAD}#egg=${package}
-cd src/${package}
+ && pip install setuptools==${setupver} wheel
+# The following can fail if you haven't pushed your commits to ${repo}
+pip install -e "git+${repo}@${HEAD}#egg=${package}"
+pushd src/${package}
make install-dependencies
make dist
make test
-cp dist/${package}*tar.gz ../../../testenv3/
+cp dist/${package}*tar.gz "../../../testenv${PYVER}_3/"
+cp dist/${module}*whl "../../../testenv${PYVER}_4/"
pip uninstall -y ${package} || true; pip uninstall -y ${package} || true; make install
-cd ../.. # no subdir named ${proj} here, safe for py.testing the installed module
-bin/${run_tests}
+popd # ../.. no subdir named ${proj} here, safe for py.testing the installed module
+# shellcheck disable=SC2086
+${run_tests}
+popd
-# Is the distribution in testenv2 complete enough to build another
-# functional distribution?
+# Is the source distribution in testenv${PYVER}_2 complete enough to build
+# another functional distribution?
-cd ../testenv3/
+pushd "testenv${PYVER}_3/"
+# shellcheck source=/dev/null
source bin/activate
rm lib/python-wheels/setuptools* \
&& pip install --force-reinstall -U pip==${pipver} \
- && pip install setuptools==20.10.1
+ && pip install setuptools==${setupver} wheel
pip install ${package}*tar.gz
pip install pytest
-tar xzf ${package}*tar.gz
-cd ${package}*
+mkdir out
+tar --extract --directory=out -z -f ${package}*.tar.gz
+pushd out/${package}*
make dist
make test
pip uninstall -y ${package} || true; pip uninstall -y ${package} || true; make install
mkdir ../not-${module}
-pushd ../not-${module} ; ../bin/${run_tests}; popd
+pushd ../not-${module}
+# shellcheck disable=SC2086
+../../${run_tests}; popd
+popd
+popd
+
+# Is the wheel in testenv${PYVER}_2 installable and will it pass the tests
+
+pushd "testenv${PYVER}_4/"
+# shellcheck source=/dev/null
+source bin/activate
+rm lib/python-wheels/setuptools* \
+ && pip install --force-reinstall -U pip==${pipver} \
+ && pip install setuptools==${setupver} wheel
+pip install ${module}*.whl
+pip install pytest
+mkdir not-${module}
+pushd not-${module}
+# shellcheck disable=SC2086
+../${run_tests}; popd
+popd
diff --git a/v1.1.0-dev1/salad/requirements.txt b/v1.1.0-dev1/salad/requirements.txt
index e30575aee..456835f38 100644
--- a/v1.1.0-dev1/salad/requirements.txt
+++ b/v1.1.0-dev1/salad/requirements.txt
@@ -1,10 +1,10 @@
-requests
-ruamel.yaml==0.12.4
-rdflib>=4.1.
-rdflib-jsonld>=0.3.0
-mistune
-typing>=3.5.2 ; python_version>="2.7"
-avro ; python_version<"3"
-avro-python3 ; python_version>="3"
-CacheControl
-lockfile
+typing==3.5.3
+avro-cwl==1.8.4 ; python_version>="3"
+future ; python_version>="3"
+avro==1.8.1 ; python_version<"3"
+ruamel.yaml>=0.12.4, <0.15
+rdflib==4.2.2
+rdflib-jsonld==0.4.0
+mistune==0.7.3
+CacheControl==0.11.7
+lockfile==0.12.2
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/__init__.py b/v1.1.0-dev1/salad/schema_salad/__init__.py
index 381ec7664..b2fc02cbc 100644
--- a/v1.1.0-dev1/salad/schema_salad/__init__.py
+++ b/v1.1.0-dev1/salad/schema_salad/__init__.py
@@ -1,9 +1,32 @@
+from __future__ import absolute_import
import logging
+import os
import sys
import typing
+import six
+
+from .utils import onWindows
__author__ = 'peter.amstutz@curoverse.com'
_logger = logging.getLogger("salad")
_logger.addHandler(logging.StreamHandler())
_logger.setLevel(logging.INFO)
+
+if six.PY3:
+
+ if onWindows:
+ # create '/tmp' folder if not present
+ # required by autotranslate module
+ # TODO: remove when https://github.com/PythonCharmers/python-future/issues/295
+ # is fixed
+ if not os.path.exists("/tmp"):
+ try:
+ os.makedirs("/tmp")
+ except OSError as exception:
+ _logger.error(u"Cannot create '\\tmp' folder in root needed for",
+ "'cwltool' Python 3 installation.")
+ exit(1)
+
+ from past import autotranslate # type: ignore
+ autotranslate(['avro', 'avro.schema'])
diff --git a/v1.1.0-dev1/salad/schema_salad/__main__.py b/v1.1.0-dev1/salad/schema_salad/__main__.py
index 4bf3d7eae..5890f6f28 100644
--- a/v1.1.0-dev1/salad/schema_salad/__main__.py
+++ b/v1.1.0-dev1/salad/schema_salad/__main__.py
@@ -1,3 +1,4 @@
+from __future__ import absolute_import
from . import main
import sys
import typing
diff --git a/v1.1.0-dev1/salad/schema_salad/add_dictlist.py b/v1.1.0-dev1/salad/schema_salad/add_dictlist.py
deleted file mode 100644
index 53bd4d48d..000000000
--- a/v1.1.0-dev1/salad/schema_salad/add_dictlist.py
+++ /dev/null
@@ -1,7 +0,0 @@
-import sys
-from typing import Any, Dict
-
-def add_dictlist(di, key, val): # type: (Dict, Any, Any) -> None
- if key not in di:
- di[key] = []
- di[key].append(val)
diff --git a/v1.1.0-dev1/salad/schema_salad/aslist.py b/v1.1.0-dev1/salad/schema_salad/aslist.py
deleted file mode 100644
index 0332a2be5..000000000
--- a/v1.1.0-dev1/salad/schema_salad/aslist.py
+++ /dev/null
@@ -1,10 +0,0 @@
-import sys
-from typing import Any, List
-
-def aslist(l): # type: (Any) -> List
- """Convenience function to wrap single items and lists, and return lists unchanged."""
-
- if isinstance(l, list):
- return l
- else:
- return [l]
diff --git a/v1.1.0-dev1/salad/schema_salad/codegen.py b/v1.1.0-dev1/salad/schema_salad/codegen.py
new file mode 100644
index 000000000..cac082ac5
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/codegen.py
@@ -0,0 +1,100 @@
+import json
+import sys
+from six.moves import urllib, cStringIO
+import collections
+import logging
+from pkg_resources import resource_stream
+from .utils import aslist, flatten
+from . import schema
+from .codegen_base import shortname, CodeGenBase
+from .python_codegen import PythonCodeGen
+from .java_codegen import JavaCodeGen
+from .ref_resolver import Loader
+from typing import List, Dict, Text, Any, Union, Text
+from ruamel.yaml.comments import CommentedSeq, CommentedMap
+
+class GoCodeGen(object):
+ pass
+
+
+def codegen(lang, # type: str
+ i, # type: List[Dict[Text, Any]]
+ schema_metadata, # type: Dict[Text, Any]
+ loader # type: Loader
+ ):
+ # type: (...) -> None
+
+ j = schema.extend_and_specialize(i, loader)
+
+ cg = None # type: CodeGenBase
+ if lang == "python":
+ cg = PythonCodeGen(sys.stdout)
+ elif lang == "java":
+ cg = JavaCodeGen(schema_metadata.get("$base", schema_metadata.get("id")))
+ else:
+ raise Exception("Unsupported code generation language '%s'" % lang)
+
+ cg.prologue()
+
+ documentRoots = []
+
+ for rec in j:
+ if rec["type"] in ("enum", "record"):
+ cg.type_loader(rec)
+ cg.add_vocab(shortname(rec["name"]), rec["name"])
+
+ for rec in j:
+ if rec["type"] == "enum":
+ for s in rec["symbols"]:
+ cg.add_vocab(shortname(s), s)
+
+ if rec["type"] == "record":
+ if rec.get("documentRoot"):
+ documentRoots.append(rec["name"])
+ cg.begin_class(rec["name"], aslist(rec.get("extends", [])), rec.get("doc"),
+ rec.get("abstract"))
+ cg.add_vocab(shortname(rec["name"]), rec["name"])
+
+ for f in rec.get("fields", []):
+ if f.get("jsonldPredicate") == "@id":
+ fieldpred = f["name"]
+ tl = cg.uri_loader(cg.type_loader(f["type"]), True, False, None)
+ cg.declare_id_field(fieldpred, tl, f.get("doc"))
+ break
+
+ for f in rec.get("fields", []):
+ optional = bool("https://w3id.org/cwl/salad#null" in f["type"])
+ tl = cg.type_loader(f["type"])
+ jld = f.get("jsonldPredicate")
+ fieldpred = f["name"]
+ if isinstance(jld, dict):
+ refScope = jld.get("refScope")
+
+ if jld.get("typeDSL"):
+ tl = cg.typedsl_loader(tl, refScope)
+ elif jld.get("_type") == "@id":
+ tl = cg.uri_loader(tl, jld.get("identity"), False, refScope)
+ elif jld.get("_type") == "@vocab":
+ tl = cg.uri_loader(tl, False, True, refScope)
+
+ mapSubject = jld.get("mapSubject")
+ if mapSubject:
+ tl = cg.idmap_loader(f["name"], tl, mapSubject, jld.get("mapPredicate"))
+
+ if "_id" in jld and jld["_id"][0] != "@":
+ fieldpred = jld["_id"]
+
+ if jld == "@id":
+ continue
+
+ cg.declare_field(fieldpred, tl, f.get("doc"), optional)
+
+ cg.end_class(rec["name"])
+
+ rootType = list(documentRoots)
+ rootType.append({
+ "type": "array",
+ "items": documentRoots
+ })
+
+ cg.epilogue(cg.type_loader(rootType))
diff --git a/v1.1.0-dev1/salad/schema_salad/codegen_base.py b/v1.1.0-dev1/salad/schema_salad/codegen_base.py
new file mode 100644
index 000000000..d1fe282a9
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/codegen_base.py
@@ -0,0 +1,71 @@
+import collections
+from six.moves import urllib
+from typing import List, Text, Dict, Union, Any
+from . import schema
+from .schema import shortname
+
+class TypeDef(object):
+ def __init__(self, name, init):
+ # type: (Text, Text) -> None
+ self.name = name
+ self.init = init
+
+class CodeGenBase(object):
+ def __init__(self):
+ # type: () -> None
+ self.collected_types = collections.OrderedDict() # type: collections.OrderedDict[Text, TypeDef]
+ self.vocab = {} # type: Dict[Text, Text]
+
+ def declare_type(self, t):
+ # type: (TypeDef) -> TypeDef
+ if t not in self.collected_types:
+ self.collected_types[t.name] = t
+ return t
+
+ def add_vocab(self, name, uri):
+ # type: (Text, Text) -> None
+ self.vocab[name] = uri
+
+ def prologue(self):
+ # type: () -> None
+ raise NotImplementedError()
+
+ def safe_name(self, n):
+ # type: (Text) -> Text
+ return schema.avro_name(n)
+
+ def begin_class(self, classname, extends, doc, abstract):
+ # type: (Text, List[Text], Text, bool) -> None
+ raise NotImplementedError()
+
+ def end_class(self, classname):
+ # type: (Text) -> None
+ raise NotImplementedError()
+
+ def type_loader(self, t):
+ # type: (Union[List[Any], Dict[Text, Any]]) -> TypeDef
+ raise NotImplementedError()
+
+ def declare_field(self, name, typedef, doc, optional):
+ # type: (Text, TypeDef, Text, bool) -> None
+ raise NotImplementedError()
+
+ def declare_id_field(self, name, typedef, doc):
+ # type: (Text, TypeDef, Text) -> None
+ raise NotImplementedError()
+
+ def uri_loader(self, inner, scoped_id, vocab_term, refScope):
+ # type: (TypeDef, bool, bool, Union[int, None]) -> TypeDef
+ raise NotImplementedError()
+
+ def idmap_loader(self, field, inner, mapSubject, mapPredicate):
+ # type: (Text, TypeDef, Text, Union[Text, None]) -> TypeDef
+ raise NotImplementedError()
+
+ def typedsl_loader(self, inner, refScope):
+ # type: (TypeDef, Union[int, None]) -> TypeDef
+ raise NotImplementedError()
+
+ def epilogue(self, rootLoader):
+ # type: (TypeDef) -> None
+ raise NotImplementedError()
diff --git a/v1.1.0-dev1/salad/schema_salad/flatten.py b/v1.1.0-dev1/salad/schema_salad/flatten.py
deleted file mode 100644
index 90c93d280..000000000
--- a/v1.1.0-dev1/salad/schema_salad/flatten.py
+++ /dev/null
@@ -1,24 +0,0 @@
-import sys
-from typing import Any, Tuple
-
-# http://rightfootin.blogspot.com/2006/09/more-on-python-flatten.html
-def flatten(l, ltypes=(list, tuple)):
- # type: (Any, Any) -> Any
- if l is None:
- return []
- if not isinstance(l, ltypes):
- return [l]
-
- ltype = type(l)
- l = list(l)
- i = 0
- while i < len(l):
- while isinstance(l[i], ltypes):
- if not l[i]:
- l.pop(i)
- i -= 1
- break
- else:
- l[i:i + 1] = l[i]
- i += 1
- return ltype(l)
diff --git a/v1.1.0-dev1/salad/schema_salad/java_codegen.py b/v1.1.0-dev1/salad/schema_salad/java_codegen.py
new file mode 100644
index 000000000..57cb7fc0a
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/java_codegen.py
@@ -0,0 +1,152 @@
+import json
+import sys
+import six
+from six.moves import urllib, cStringIO
+import collections
+import logging
+from pkg_resources import resource_stream
+from .utils import aslist, flatten
+from . import schema
+from .codegen_base import TypeDef, CodeGenBase, shortname
+from typing import Text
+import os
+
+class JavaCodeGen(CodeGenBase):
+ def __init__(self, base):
+ # type: (Text) -> None
+
+ super(JavaCodeGen, self).__init__()
+ sp = urllib.parse.urlsplit(base)
+ self.package = ".".join(list(reversed(sp.netloc.split("."))) + sp.path.strip("/").split("/"))
+ self.outdir = self.package.replace(".", "/")
+
+ def prologue(self):
+ if not os.path.exists(self.outdir):
+ os.makedirs(self.outdir)
+
+ def safe_name(self, n):
+ avn = schema.avro_name(n)
+ if avn in ("class", "extends", "abstract"):
+ # reserved words
+ avn = avn+"_"
+ return avn
+
+ def interface_name(self, n):
+ return self.safe_name(n)
+
+ def begin_class(self, classname, extends, doc, abstract):
+ cls = self.interface_name(classname)
+ self.current_class = cls
+ self.current_class_is_abstract = abstract
+ self.current_loader = cStringIO()
+ self.current_fields = cStringIO()
+ with open(os.path.join(self.outdir, "%s.java" % cls), "w") as f:
+ if extends:
+ ext = "extends " + ", ".join(self.interface_name(e) for e in extends)
+ else:
+ ext = ""
+ f.write("""package {package};
+
+public interface {cls} {ext} {{
+""".
+ format(package=self.package,
+ cls=cls,
+ ext=ext))
+
+ if self.current_class_is_abstract:
+ return
+
+ with open(os.path.join(self.outdir, "%sImpl.java" % cls), "w") as f:
+ f.write("""package {package};
+
+public class {cls}Impl implements {cls} {{
+""".
+ format(package=self.package,
+ cls=cls,
+ ext=ext))
+ self.current_loader.write("""
+ void Load() {
+""")
+
+ def end_class(self, classname):
+ with open(os.path.join(self.outdir, "%s.java" % self.current_class), "a") as f:
+ f.write("""
+}
+""")
+ if self.current_class_is_abstract:
+ return
+
+ self.current_loader.write("""
+ }
+""")
+
+ with open(os.path.join(self.outdir, "%sImpl.java" % self.current_class), "a") as f:
+ f.write(self.current_fields.getvalue())
+ f.write(self.current_loader.getvalue())
+ f.write("""
+}
+""")
+
+ prims = {
+ u"http://www.w3.org/2001/XMLSchema#string": TypeDef("String", "Support.StringLoader()"),
+ u"http://www.w3.org/2001/XMLSchema#int": TypeDef("Integer", "Support.IntLoader()"),
+ u"http://www.w3.org/2001/XMLSchema#long": TypeDef("Long", "Support.LongLoader()"),
+ u"http://www.w3.org/2001/XMLSchema#float": TypeDef("Float", "Support.FloatLoader()"),
+ u"http://www.w3.org/2001/XMLSchema#double": TypeDef("Double", "Support.DoubleLoader()"),
+ u"http://www.w3.org/2001/XMLSchema#boolean": TypeDef("Boolean", "Support.BoolLoader()"),
+ u"https://w3id.org/cwl/salad#null": TypeDef("null_type", "Support.NullLoader()"),
+ u"https://w3id.org/cwl/salad#Any": TypeDef("Any_type", "Support.AnyLoader()")
+ }
+
+ def type_loader(self, t):
+ if isinstance(t, list) and len(t) == 2:
+ if t[0] == "https://w3id.org/cwl/salad#null":
+ t = t[1]
+ if isinstance(t, basestring):
+ if t in self.prims:
+ return self.prims[t]
+ return TypeDef("Object", "")
+
+ def declare_field(self, name, typedef, doc, optional):
+ fieldname = self.safe_name(name)
+ with open(os.path.join(self.outdir, "%s.java" % self.current_class), "a") as f:
+ f.write("""
+ {type} get{capfieldname}();
+""".
+ format(fieldname=fieldname,
+ capfieldname=fieldname[0].upper() + fieldname[1:],
+ type=typedef.name))
+
+ if self.current_class_is_abstract:
+ return
+
+ self.current_fields.write("""
+ private {type} {fieldname};
+ public {type} get{capfieldname}() {{
+ return this.{fieldname};
+ }}
+""".
+ format(fieldname=fieldname,
+ capfieldname=fieldname[0].upper() + fieldname[1:],
+ type=typedef.name))
+
+ self.current_loader.write("""
+ this.{fieldname} = null; // TODO: loaders
+ """.
+ format(fieldname=fieldname))
+
+
+ def declare_id_field(self, name, typedef, doc):
+ pass
+
+ def uri_loader(self, inner, scoped_id, vocab_term, refScope):
+ return inner
+
+ def idmap_loader(self, field, inner, mapSubject, mapPredicate):
+ return inner
+
+ def typedsl_loader(self, inner, refScope):
+ return inner
+
+ def epilogue(self, rootLoader):
+ pass
diff --git a/v1.1.0-dev1/salad/schema_salad/jsonld_context.py b/v1.1.0-dev1/salad/schema_salad/jsonld_context.py
index d4d203fbc..b28127f59 100755
--- a/v1.1.0-dev1/salad/schema_salad/jsonld_context.py
+++ b/v1.1.0-dev1/salad/schema_salad/jsonld_context.py
@@ -1,6 +1,11 @@
+from __future__ import absolute_import
import collections
import shutil
import json
+
+import six
+from six.moves import urllib
+
import ruamel.yaml as yaml
try:
from ruamel.yaml import CSafeLoader as SafeLoader
@@ -16,32 +21,38 @@
from rdflib import Graph, URIRef
import rdflib.namespace
from rdflib.namespace import RDF, RDFS
-import urlparse
import logging
-from .aslist import aslist
-from typing import Any, cast, Dict, Iterable, Tuple, Union
-from .ref_resolver import Loader
+from schema_salad.utils import aslist
+from typing import (cast, Any, Dict, Iterable, List, Optional, Text, Tuple,
+ Union)
+from .ref_resolver import Loader, ContextType
_logger = logging.getLogger("salad")
-def pred(datatype, field, name, context, defaultBase, namespaces):
- # type: (Dict[str, Union[Dict, str]], Dict, str, Loader.ContextType, str, Dict[str, rdflib.namespace.Namespace]) -> Union[Dict, str]
- split = urlparse.urlsplit(name)
+def pred(datatype, # type: Dict[str, Union[Dict, str]]
+ field, # type: Optional[Dict]
+ name, # type: str
+ context, # type: ContextType
+ defaultBase, # type: str
+ namespaces # type: Dict[str, rdflib.namespace.Namespace]
+ ):
+ # type: (...) -> Union[Dict, Text]
+ split = urllib.parse.urlsplit(name)
- vee = None # type: Union[str, unicode]
+ vee = None # type: Optional[Text]
- if split.scheme:
+ if split.scheme != '':
vee = name
- (ns, ln) = rdflib.namespace.split_uri(unicode(vee))
+ (ns, ln) = rdflib.namespace.split_uri(six.text_type(vee))
name = ln
if ns[0:-1] in namespaces:
- vee = unicode(namespaces[ns[0:-1]][ln])
+ vee = six.text_type(namespaces[ns[0:-1]][ln])
_logger.debug("name, v %s %s", name, vee)
- v = None # type: Any
+ v = None # type: Optional[Dict]
- if field and "jsonldPredicate" in field:
+ if field is not None and "jsonldPredicate" in field:
if isinstance(field["jsonldPredicate"], dict):
v = {}
for k, val in field["jsonldPredicate"].items():
@@ -62,11 +73,6 @@ def pred(datatype, field, name, context, defaultBase, namespaces):
"Dictionaries")
else:
raise Exception("jsonldPredicate must be a List of Dictionaries.")
- # if not v:
- # if field and "jsonldPrefix" in field:
- # defaultBase = field["jsonldPrefix"]
- # elif "jsonldPrefix" in datatype:
- # defaultBase = datatype["jsonldPrefix"]
ret = v or vee
@@ -84,25 +90,34 @@ def pred(datatype, field, name, context, defaultBase, namespaces):
return ret
-def process_type(t, g, context, defaultBase, namespaces, defaultPrefix):
- # type: (Dict[str, Any], Graph, Loader.ContextType, str, Dict[str, rdflib.namespace.Namespace], str) -> None
- if t["type"] == "record":
+def process_type(t, # type: Dict[str, Any]
+ g, # type: Graph
+ context, # type: ContextType
+ defaultBase, # type: str
+ namespaces, # type: Dict[str, rdflib.namespace.Namespace]
+ defaultPrefix # type: str
+ ):
+ # type: (...) -> None
+ if t["type"] not in ("record", "enum"):
+ return
+
+ if "name" in t:
recordname = t["name"]
- _logger.debug("Processing record %s\n", t)
+ _logger.debug("Processing %s %s\n", t.get("type"), t)
classnode = URIRef(recordname)
g.add((classnode, RDF.type, RDFS.Class))
- split = urlparse.urlsplit(recordname)
- if "jsonldPrefix" in t:
- predicate = "%s:%s" % (t["jsonldPrefix"], recordname)
- elif split.scheme:
- (ns, ln) = rdflib.namespace.split_uri(unicode(recordname))
- predicate = recordname
- recordname = ln
- else:
- predicate = "%s:%s" % (defaultPrefix, recordname)
+ split = urllib.parse.urlsplit(recordname)
+ predicate = recordname
+ if t.get("inVocab", True):
+ if split.scheme:
+ (ns, ln) = rdflib.namespace.split_uri(six.text_type(recordname))
+ predicate = recordname
+ recordname = ln
+ else:
+ predicate = "%s:%s" % (defaultPrefix, recordname)
if context.get(recordname, predicate) != predicate:
raise Exception("Predicate collision on '%s', '%s' != '%s'" % (
@@ -115,20 +130,22 @@ def process_type(t, g, context, defaultBase, namespaces, defaultPrefix):
recordname, predicate, type(predicate))
context[recordname] = predicate
+ if t["type"] == "record":
for i in t.get("fields", []):
fieldname = i["name"]
_logger.debug("Processing field %s", i)
- v = pred(t, i, fieldname, context, defaultPrefix, namespaces)
+ v = pred(t, i, fieldname, context, defaultPrefix,
+ namespaces) # type: Union[Dict[Any, Any], Text, None]
- if isinstance(v, basestring):
+ if isinstance(v, six.string_types):
v = v if v[0] != "@" else None
- else:
+ elif v is not None:
v = v["_@id"] if v.get("_@id", "@")[0] != "@" else None
- if v:
- (ns, ln) = rdflib.namespace.split_uri(unicode(v))
+ if bool(v):
+ (ns, ln) = rdflib.namespace.split_uri(six.text_type(v))
if ns[0:-1] in namespaces:
propnode = namespaces[ns[0:-1]][ln]
else:
@@ -139,7 +156,7 @@ def process_type(t, g, context, defaultBase, namespaces, defaultPrefix):
# TODO generate range from datatype.
- if isinstance(i["type"], dict) and "name" in i["type"]:
+ if isinstance(i["type"], dict):
process_type(i["type"], g, context, defaultBase,
namespaces, defaultPrefix)
@@ -147,15 +164,15 @@ def process_type(t, g, context, defaultBase, namespaces, defaultPrefix):
for e in aslist(t["extends"]):
g.add((classnode, RDFS.subClassOf, URIRef(e)))
elif t["type"] == "enum":
- _logger.debug("Processing enum %s", t["name"])
+ _logger.debug("Processing enum %s", t.get("name"))
for i in t["symbols"]:
pred(t, None, i, context, defaultBase, namespaces)
def salad_to_jsonld_context(j, schema_ctx):
- # type: (Iterable, Dict[str, Any]) -> Tuple[Loader.ContextType, Graph]
- context = {} # type: Loader.ContextType
+ # type: (Iterable, Dict[str, Any]) -> Tuple[ContextType, Graph]
+ context = {} # type: ContextType
namespaces = {}
g = Graph()
defaultPrefix = ""
@@ -178,8 +195,11 @@ def salad_to_jsonld_context(j, schema_ctx):
return (context, g)
-def fix_jsonld_ids(obj, ids):
- # type: (Union[Dict[unicode, Any], List[Dict[unicode, Any]]], List[unicode]) -> None
+
+def fix_jsonld_ids(obj, # type: Union[Dict[Text, Any], List[Dict[Text, Any]]]
+ ids # type: List[Text]
+ ):
+ # type: (...) -> None
if isinstance(obj, dict):
for i in ids:
if i in obj:
@@ -190,18 +210,23 @@ def fix_jsonld_ids(obj, ids):
for entry in obj:
fix_jsonld_ids(entry, ids)
-def makerdf(workflow, wf, ctx, graph=None):
- # type: (Union[str, unicode], Union[List[Dict[unicode, Any]], Dict[unicode, Any]], Loader.ContextType, Graph) -> Graph
+
+def makerdf(workflow, # type: Text
+ wf, # type: Union[List[Dict[Text, Any]], Dict[Text, Any]]
+ ctx, # type: ContextType
+ graph=None # type: Graph
+ ):
+ # type: (...) -> Graph
prefixes = {}
idfields = []
- for k, v in ctx.iteritems():
+ for k, v in six.iteritems(ctx):
if isinstance(v, dict):
url = v["@id"]
else:
url = v
if url == "@id":
idfields.append(k)
- doc_url, frg = urlparse.urldefrag(url)
+ doc_url, frg = urllib.parse.urldefrag(url)
if "/" in frg:
p = frg.split("/")[0]
prefixes[p] = u"%s#%s/" % (doc_url, p)
@@ -216,16 +241,16 @@ def makerdf(workflow, wf, ctx, graph=None):
if isinstance(wf, list):
for w in wf:
w["@context"] = ctx
- g.parse(data=json.dumps(w), format='json-ld', location=workflow)
+ g.parse(data=json.dumps(w), format='json-ld', publicID=str(workflow))
else:
wf["@context"] = ctx
- g.parse(data=json.dumps(wf), format='json-ld', location=workflow)
+ g.parse(data=json.dumps(wf), format='json-ld', publicID=str(workflow))
# Bug in json-ld loader causes @id fields to be added to the graph
for sub, pred, obj in g.triples((None, URIRef("@id"), None)):
g.remove((sub, pred, obj))
- for k2, v2 in prefixes.iteritems():
+ for k2, v2 in six.iteritems(prefixes):
g.namespace_manager.bind(k2, v2)
return g
diff --git a/v1.1.0-dev1/salad/schema_salad/main.py b/v1.1.0-dev1/salad/schema_salad/main.py
index 1896e8fd0..540078277 100644
--- a/v1.1.0-dev1/salad/schema_salad/main.py
+++ b/v1.1.0-dev1/salad/schema_salad/main.py
@@ -1,32 +1,120 @@
from __future__ import print_function
+from __future__ import absolute_import
import argparse
import logging
import sys
import traceback
-import pkg_resources # part of setuptools
-from . import schema
-from . import jsonld_context
-from . import makedoc
import json
+import os
+import re
+import itertools
+
+import six
+from six.moves import urllib
+
+import pkg_resources # part of setuptools
+
+from typing import Any, Dict, List, Union, Pattern, Text, Tuple, cast
+
from rdflib import Graph, plugin
from rdflib.serializer import Serializer
-import os
-import urlparse
-from .ref_resolver import Loader
+from . import schema
+from . import jsonld_context
+from . import makedoc
from . import validate
-from typing import Any, Dict, List, Union
-
+from . import codegen
+from .sourceline import strip_dup_lineno
+from .ref_resolver import Loader, file_uri
_logger = logging.getLogger("salad")
from rdflib.plugin import register, Parser
register('json-ld', Parser, 'rdflib_jsonld.parser', 'JsonLDParser')
-def printrdf(workflow, wf, ctx, sr):
- # type: (str, Union[List[Dict[unicode, Any]], Dict[unicode, Any]], Dict[unicode, Any], str) -> None
+def printrdf(workflow, # type: str
+ wf, # type: Union[List[Dict[Text, Any]], Dict[Text, Any]]
+ ctx, # type: Dict[Text, Any]
+ sr # type: str
+ ):
+ # type: (...) -> None
g = jsonld_context.makerdf(workflow, wf, ctx)
- print(g.serialize(format=sr))
+ print(g.serialize(format=sr, encoding='utf-8').decode('utf-8'))
+
+def regex_chunk(lines, regex):
+ # type: (List[str], Pattern[str]) -> List[List[str]]
+ lst = list(itertools.dropwhile(lambda x: not regex.match(x), lines))
+ arr = []
+ while lst:
+ ret = [lst[0]]+list(itertools.takewhile(lambda x: not regex.match(x),
+ lst[1:]))
+ arr.append(ret)
+ lst = list(itertools.dropwhile(lambda x: not regex.match(x),
+ lst[1:]))
+ return arr
+
+
+def chunk_messages(message): # type: (str) -> List[Tuple[int, str]]
+ file_regex = re.compile(r'^(.+:\d+:\d+:)(\s+)(.+)$')
+ item_regex = re.compile(r'^\s*\*\s+')
+ arr = []
+ for chun in regex_chunk(message.splitlines(), file_regex):
+ fst = chun[0]
+ mat = file_regex.match(fst)
+ place = mat.group(1)
+ indent = len(mat.group(2))
+
+ lst = [mat.group(3)]+chun[1:]
+ if [x for x in lst if item_regex.match(x)]:
+ for item in regex_chunk(lst, item_regex):
+ msg = re.sub(item_regex, '', "\n".join(item))
+ arr.append((indent, place+' '+re.sub(r'[\n\s]+',
+ ' ',
+ msg)))
+ else:
+ msg = re.sub(item_regex, '', "\n".join(lst))
+ arr.append((indent, place+' '+re.sub(r'[\n\s]+',
+ ' ',
+ msg)))
+ return arr
+
+
+def to_one_line_messages(message): # type: (str) -> str
+ ret = []
+ max_elem = (0, '')
+ for (indent, msg) in chunk_messages(message):
+ if indent > max_elem[0]:
+ max_elem = (indent, msg)
+ else:
+ ret.append(max_elem[1])
+ max_elem = (indent, msg)
+ ret.append(max_elem[1])
+ return "\n".join(ret)
+
+
+def reformat_yaml_exception_message(message): # type: (str) -> str
+ line_regex = re.compile(r'^\s+in "(.+)", line (\d+), column (\d+)$')
+ fname_regex = re.compile(r'^file://'+os.getcwd()+'/')
+ msgs = message.splitlines()
+ ret = []
+
+ if len(msgs) == 3:
+ msgs = msgs[1:]
+ nblanks = 0
+ elif len(msgs) == 4:
+ c_msg = msgs[0]
+ c_file, c_line, c_column = line_regex.match(msgs[1]).groups()
+ c_file = re.sub(fname_regex, '', c_file)
+ ret.append("%s:%s:%s: %s" % (c_file, c_line, c_column, c_msg))
+
+ msgs = msgs[2:]
+ nblanks = 2
+
+ p_msg = msgs[0]
+ p_file, p_line, p_column = line_regex.match(msgs[1]).groups()
+ p_file = re.sub(fname_regex, '', p_file)
+ ret.append("%s:%s:%s:%s %s" % (p_file, p_line, p_column, ' '*nblanks, p_msg))
+ return "\n".join(ret)
def main(argsl=None): # type: (List[str]) -> int
@@ -54,8 +142,15 @@ def main(argsl=None): # type: (List[str]) -> int
"--print-index", action="store_true", help="Print node index")
exgroup.add_argument("--print-metadata",
action="store_true", help="Print document metadata")
- exgroup.add_argument("--version", action="store_true",
- help="Print version")
+ exgroup.add_argument("--print-inheritance-dot",
+ action="store_true", help="Print graphviz file of inheritance")
+ exgroup.add_argument("--print-fieldrefs-dot",
+ action="store_true", help="Print graphviz file of field refs")
+
+ exgroup.add_argument("--codegen", type=str, metavar="language", help="Generate classes in target language, currently supported: python")
+
+ exgroup.add_argument("--print-oneline", action="store_true",
+ help="Print each error message in oneline")
exgroup = parser.add_mutually_exclusive_group()
exgroup.add_argument("--strict", action="store_true", help="Strict validation (unrecognized or out of place fields are error)",
@@ -71,11 +166,18 @@ def main(argsl=None): # type: (List[str]) -> int
exgroup.add_argument("--debug", action="store_true",
help="Print even more logging")
- parser.add_argument("schema", type=str)
+ parser.add_argument("schema", type=str, nargs="?", default=None)
parser.add_argument("document", type=str, nargs="?", default=None)
+ parser.add_argument("--version", "-v", action="store_true",
+ help="Print version", default=None)
+
args = parser.parse_args(argsl)
+ if args.version is None and args.schema is None:
+ print('%s: error: too few arguments' % sys.argv[0])
+ return 1
+
if args.quiet:
_logger.setLevel(logging.WARN)
if args.debug:
@@ -84,10 +186,10 @@ def main(argsl=None): # type: (List[str]) -> int
pkg = pkg_resources.require("schema_salad")
if pkg:
if args.version:
- print("%s %s" % (sys.argv[0], pkg[0].version))
+ print("%s Current version: %s" % (sys.argv[0], pkg[0].version))
return 0
else:
- _logger.info("%s %s", sys.argv[0], pkg[0].version)
+ _logger.info("%s Current version: %s", sys.argv[0], pkg[0].version)
# Get the metaschema to validate the schema
metaschema_names, metaschema_doc, metaschema_loader = schema.get_metaschema()
@@ -95,8 +197,8 @@ def main(argsl=None): # type: (List[str]) -> int
# Load schema document and resolve refs
schema_uri = args.schema
- if not urlparse.urlparse(schema_uri)[0]:
- schema_uri = "file://" + os.path.abspath(schema_uri)
+ if not (urllib.parse.urlparse(schema_uri)[0] and urllib.parse.urlparse(schema_uri)[0] in [u'http', u'https', u'file']):
+ schema_uri = file_uri(os.path.abspath(schema_uri))
schema_raw_doc = metaschema_loader.fetch(schema_uri)
try:
@@ -104,9 +206,13 @@ def main(argsl=None): # type: (List[str]) -> int
schema_raw_doc, schema_uri)
except (validate.ValidationException) as e:
_logger.error("Schema `%s` failed link checking:\n%s",
- args.schema, e, exc_info=(True if args.debug else False))
- _logger.debug("Index is %s", metaschema_loader.idx.keys())
- _logger.debug("Vocabulary is %s", metaschema_loader.vocab.keys())
+ args.schema, e, exc_info=(True if args.debug else False))
+ _logger.debug("Index is %s", list(metaschema_loader.idx.keys()))
+ _logger.debug("Vocabulary is %s", list(metaschema_loader.vocab.keys()))
+ return 1
+ except (RuntimeError) as e:
+ _logger.error("Schema `%s` read error:\n%s",
+ args.schema, e, exc_info=(True if args.debug else False))
return 1
# Optionally print the schema after ref resolution
@@ -115,13 +221,14 @@ def main(argsl=None): # type: (List[str]) -> int
return 0
if not args.document and args.print_index:
- print(json.dumps(metaschema_loader.idx.keys(), indent=4))
+ print(json.dumps(list(metaschema_loader.idx.keys()), indent=4))
return 0
# Validate the schema document against the metaschema
try:
schema.validate_doc(metaschema_names, schema_doc,
- metaschema_loader, args.strict)
+ metaschema_loader, args.strict,
+ source=schema_metadata.get("name"))
except validate.ValidationException as e:
_logger.error("While validating schema `%s`:\n%s" %
(args.schema, str(e)))
@@ -133,11 +240,20 @@ def main(argsl=None): # type: (List[str]) -> int
metactx = schema_raw_doc.get("$namespaces", {})
if "$base" in schema_raw_doc:
metactx["@base"] = schema_raw_doc["$base"]
- (schema_ctx, rdfs) = jsonld_context.salad_to_jsonld_context(schema_doc, metactx)
+ if schema_doc is not None:
+ (schema_ctx, rdfs) = jsonld_context.salad_to_jsonld_context(
+ schema_doc, metactx)
+ else:
+ raise Exception("schema_doc is None??")
# Create the loader that will be used to load the target document.
document_loader = Loader(schema_ctx)
+ if args.codegen:
+ codegen.codegen(args.codegen, cast(List[Dict[Text, Any]], schema_doc),
+ schema_metadata, document_loader)
+ return 0
+
# Make the Avro validation that will be used to validate the target
# document
if isinstance(schema_doc, list):
@@ -149,8 +265,8 @@ def main(argsl=None): # type: (List[str]) -> int
if isinstance(avsc_names, Exception):
_logger.error("Schema `%s` error:\n%s", args.schema,
- avsc_names, exc_info=((type(avsc_names), avsc_names,
- None) if args.debug else None))
+ avsc_names, exc_info=((type(avsc_names), avsc_names,
+ None) if args.debug else None))
if args.print_avro:
print(json.dumps(avsc_obj, indent=4))
return 1
@@ -175,6 +291,14 @@ def main(argsl=None): # type: (List[str]) -> int
print(json.dumps(schema_metadata, indent=4))
return 0
+ if args.print_inheritance_dot:
+ schema.print_inheritance(schema_doc, sys.stdout)
+ return 0
+
+ if args.print_fieldrefs_dot:
+ schema.print_fieldrefs(schema_doc, document_loader, sys.stdout)
+ return 0
+
# If no document specified, all done.
if not args.document:
print("Schema `%s` is valid" % args.schema)
@@ -183,12 +307,21 @@ def main(argsl=None): # type: (List[str]) -> int
# Load target document and resolve refs
try:
uri = args.document
- if not urlparse.urlparse(uri)[0]:
+ if not urllib.parse.urlparse(uri)[0]:
doc = "file://" + os.path.abspath(uri)
document, doc_metadata = document_loader.resolve_ref(uri)
- except (validate.ValidationException, RuntimeError) as e:
+ except validate.ValidationException as e:
+ msg = strip_dup_lineno(six.text_type(e))
+ msg = to_one_line_messages(str(msg)) if args.print_oneline else msg
+ _logger.error("Document `%s` failed validation:\n%s",
+ args.document, msg, exc_info=args.debug)
+ return 1
+ except RuntimeError as e:
+ msg = strip_dup_lineno(six.text_type(e))
+ msg = reformat_yaml_exception_message(str(msg))
+ msg = to_one_line_messages(msg) if args.print_oneline else msg
_logger.error("Document `%s` failed validation:\n%s",
- args.document, e, exc_info=args.debug)
+ args.document, msg, exc_info=args.debug)
return 1
# Optionally print the document after ref resolution
@@ -197,7 +330,7 @@ def main(argsl=None): # type: (List[str]) -> int
return 0
if args.print_index:
- print(json.dumps(document_loader.idx.keys(), indent=4))
+ print(json.dumps(list(document_loader.idx.keys()), indent=4))
return 0
# Validate the schema document against the metaschema
@@ -205,8 +338,9 @@ def main(argsl=None): # type: (List[str]) -> int
schema.validate_doc(avsc_names, document,
document_loader, args.strict)
except validate.ValidationException as e:
+ msg = to_one_line_messages(str(e)) if args.print_oneline else str(e)
_logger.error("While validating document `%s`:\n%s" %
- (args.document, str(e)))
+ (args.document, msg))
return 1
# Optionally convert the document to RDF
diff --git a/v1.1.0-dev1/salad/schema_salad/makedoc.py b/v1.1.0-dev1/salad/schema_salad/makedoc.py
index 7cdc1162d..45bcab231 100644
--- a/v1.1.0-dev1/salad/schema_salad/makedoc.py
+++ b/v1.1.0-dev1/salad/schema_salad/makedoc.py
@@ -1,23 +1,28 @@
+from __future__ import absolute_import
+
import mistune
-from . import schema
+import argparse
import json
import os
import copy
import re
import sys
-from StringIO import StringIO
import logging
-import urlparse
-from .aslist import aslist
-from .add_dictlist import add_dictlist
-import re
-import argparse
-from typing import Any, IO, Union
+from io import open
+
+from . import schema
+from .utils import add_dictlist, aslist
+
+import six
+from six.moves import range
+from six.moves import urllib
+from six import StringIO
+from typing import cast, Any, Dict, IO, List, Optional, Set, Text, Union
_logger = logging.getLogger("salad")
-def has_types(items): # type: (Any) -> List[basestring]
+def has_types(items): # type: (Any) -> List[Text]
r = [] # type: List
if isinstance(items, dict):
if items["type"] == "https://w3id.org/cwl/salad#record":
@@ -30,13 +35,13 @@ def has_types(items): # type: (Any) -> List[basestring]
for i in items:
r.extend(has_types(i))
return r
- if isinstance(items, basestring):
+ if isinstance(items, six.string_types):
return [items]
return []
-def linkto(item):
- _, frg = urlparse.urldefrag(item)
+def linkto(item): # type: (Text) -> Text
+ _, frg = urllib.parse.urldefrag(item)
return "[%s](#%s)" % (frg, to_id(frg))
@@ -46,11 +51,17 @@ def __init__(self): # type: () -> None
super(mistune.Renderer, self).__init__()
self.options = {}
- def header(self, text, level, raw=None):
+ def header(self, text, level, raw=None): # type: (Text, int, Any) -> Text
return """%s """ % (level, to_id(text), text, level)
+ def table(self, header, body): # type: (Text, Text) -> Text
+ return (
+ '\n'
+ ) % (header, body)
+
-def to_id(text): # type: (Union[str, unicode]) -> Union[str, unicode]
+def to_id(text): # type: (Text) -> Text
textid = text
if text[0] in ("0", "1", "2", "3", "4", "5", "6", "7", "8", "9"):
try:
@@ -104,6 +115,7 @@ def contents(self, idn): # type: (str) -> str
c += """"""
return c
+
basicTypes = ("https://w3id.org/cwl/salad#null",
"http://www.w3.org/2001/XMLSchema#boolean",
"http://www.w3.org/2001/XMLSchema#int",
@@ -129,7 +141,7 @@ def number_headings(toc, maindoc): # type: (ToC, str) -> str
if not skip:
m = re.match(r'^(#+) (.*)', line)
- if m:
+ if m is not None:
num = toc.add_entry(len(m.group(1)), m.group(2))
line = "%s %s %s" % (m.group(1), num, m.group(2))
line = re.sub(r'^(https?://\S+)', r'[\1](\1)', line)
@@ -151,8 +163,8 @@ def fix_doc(doc): # type: (Union[List[str], str]) -> str
class RenderType(object):
- def __init__(self, toc, j, renderlist, redirects):
- # type: (ToC, List[Dict], str, Dict) -> None
+ def __init__(self, toc, j, renderlist, redirects, primitiveType):
+ # type: (ToC, List[Dict], str, Dict, str) -> None
self.typedoc = StringIO()
self.toc = toc
self.subs = {} # type: Dict[str, str]
@@ -160,7 +172,8 @@ def __init__(self, toc, j, renderlist, redirects):
self.docAfter = {} # type: Dict[str, List]
self.rendered = set() # type: Set[str]
self.redirects = redirects
- self.title = None # type: str
+ self.title = None # type: Optional[str]
+ self.primitiveType = primitiveType
for t in j:
if "extends" in t:
@@ -196,8 +209,8 @@ def __init__(self, toc, j, renderlist, redirects):
if tp not in self.uses:
self.uses[tp] = []
if (t["name"], f["name"]) not in self.uses[tp]:
- _, frg1 = urlparse.urldefrag(t["name"])
- _, frg2 = urlparse.urldefrag(f["name"])
+ _, frg1 = urllib.parse.urldefrag(t["name"])
+ _, frg2 = urllib.parse.urldefrag(f["name"])
self.uses[tp].append((frg1, frg2))
if tp not in basicTypes and tp not in self.record_refs[t["name"]]:
self.record_refs[t["name"]].append(tp)
@@ -213,9 +226,13 @@ def __init__(self, toc, j, renderlist, redirects):
("docAfter" not in f))):
self.render_type(f, 1)
- def typefmt(self, tp, redirects, nbsp=False, jsonldPredicate=None):
- # type: (Any, Dict[str, str], bool, Dict[str, str]) -> Union[str, unicode]
- global primitiveType
+ def typefmt(self,
+ tp, # type: Any
+ redirects, # type: Dict[str, str]
+ nbsp=False, # type: bool
+ jsonldPredicate=None # type: Optional[Dict[str, str]]
+ ):
+ # type: (...) -> Text
if isinstance(tp, list):
if nbsp and len(tp) <= 3:
return " | ".join([self.typefmt(n, redirects, jsonldPredicate=jsonldPredicate) for n in tp])
@@ -223,19 +240,23 @@ def typefmt(self, tp, redirects, nbsp=False, jsonldPredicate=None):
return " | ".join([self.typefmt(n, redirects) for n in tp])
if isinstance(tp, dict):
if tp["type"] == "https://w3id.org/cwl/salad#array":
- ar = "array<%s>" % (self.typefmt(tp["items"], redirects, nbsp=True))
- if jsonldPredicate and "mapSubject" in jsonldPredicate:
+ ar = "array<%s>" % (self.typefmt(
+ tp["items"], redirects, nbsp=True))
+ if jsonldPredicate is not None and "mapSubject" in jsonldPredicate:
if "mapPredicate" in jsonldPredicate:
ar += " | map<%s.%s, %s.%s>" % (self.typefmt(tp["items"], redirects),
- jsonldPredicate["mapSubject"],
- self.typefmt(tp["items"], redirects),
- jsonldPredicate["mapPredicate"])
+ jsonldPredicate[
+ "mapSubject"],
+ self.typefmt(
+ tp["items"], redirects),
+ jsonldPredicate["mapPredicate"])
ar += " | map<%s.%s, %s>" % (self.typefmt(tp["items"], redirects),
- jsonldPredicate["mapSubject"],
- self.typefmt(tp["items"], redirects))
+ jsonldPredicate[
+ "mapSubject"],
+ self.typefmt(tp["items"], redirects))
return ar
if tp["type"] in ("https://w3id.org/cwl/salad#record", "https://w3id.org/cwl/salad#enum"):
- frg = schema.avro_name(tp["name"])
+ frg = cast(Text, schema.avro_name(tp["name"]))
if tp["name"] in redirects:
return """%s """ % (redirects[tp["name"]], frg)
elif tp["name"] in self.typemap:
@@ -248,18 +269,22 @@ def typefmt(self, tp, redirects, nbsp=False, jsonldPredicate=None):
if str(tp) in redirects:
return """%s """ % (redirects[tp], redirects[tp])
elif str(tp) in basicTypes:
- return """%s """ % (primitiveType, schema.avro_name(str(tp)))
+ return """%s """ % (self.primitiveType, schema.avro_name(str(tp)))
else:
- _, frg = urlparse.urldefrag(tp)
- if frg:
+ _, frg = urllib.parse.urldefrag(tp)
+ if frg is not '':
tp = frg
return """%s """ % (to_id(tp), tp)
+ raise Exception("We should not be here!")
def render_type(self, f, depth): # type: (Dict[str, Any], int) -> None
if f["name"] in self.rendered or f["name"] in self.redirects:
return
self.rendered.add(f["name"])
+ if f.get("abstract"):
+ return
+
if "doc" not in f:
f["doc"] = ""
@@ -299,22 +324,24 @@ def extendsfrom(item, ex):
if f["type"] != "documentation":
lines = []
- for l in f["doc"].splitlines():
- if len(l) > 0 and l[0] == "#":
- l = ("#" * depth) + l
- lines.append(l)
+ for line in f["doc"].splitlines():
+ if len(line) > 0 and line[0] == "#":
+ line = ("#" * depth) + line
+ lines.append(line)
f["doc"] = "\n".join(lines)
- _, frg = urlparse.urldefrag(f["name"])
+ _, frg = urllib.parse.urldefrag(f["name"])
num = self.toc.add_entry(depth, frg)
- doc = "## %s %s\n" % (num, frg)
+ doc = u"%s %s %s\n" % (("#" * depth), num, frg)
else:
- doc = ""
+ doc = u""
if self.title is None and f["doc"]:
- self.title = f["doc"][0:f["doc"].index("\n")]
- if self.title.startswith('# '):
- self.title = self.title[2:]
+ title = f["doc"][0:f["doc"].index("\n")]
+ if title.startswith('# '):
+ self.title = title[2:]
+ else:
+ self.title = title
if f["type"] == "documentation":
f["doc"] = number_headings(self.toc, f["doc"])
@@ -394,19 +421,20 @@ def extendsfrom(item, ex):
self.render_type(self.typemap[s], depth)
-def avrold_doc(j, outdoc, renderlist, redirects, brand, brandlink):
- # type: (List[Dict[unicode, Any]], IO[Any], str, Dict, str, str) -> None
+def avrold_doc(j, outdoc, renderlist, redirects, brand, brandlink, primtype):
+ # type: (List[Dict[Text, Any]], IO[Any], str, Dict, str, str, str) -> None
toc = ToC()
toc.start_numbering = False
- rt = RenderType(toc, j, renderlist, redirects)
- content = rt.typedoc.getvalue() # type: unicode
+ rt = RenderType(toc, j, renderlist, redirects, primtype)
+ content = rt.typedoc.getvalue() # type: Text
outdoc.write("""
`, ``, and ``) or characters\n[not allowed](http://www.iana.org/assignments/idna-tables-6.3.0/idna-tables-6.3.0.xhtml)\nfor [Internationalized Domain Names for Applications](https://tools.ietf.org/html/rfc6452)\nthen implementations may terminate the process with a\n`permanentFailure`.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#path",
+ "_type": "@id"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#File/basename",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "The base name of the file, that is, the name of the file without any\nleading directory path. The base name must not contain a slash `/`.\n\nIf not provided, the implementation must set this field based on the\n`location` field by taking the final path component after parsing\n`location` as an IRI. If `basename` is provided, it is not required to\nmatch the value from `location`.\n\nWhen this file is made available to a CommandLineTool, it must be named\nwith `basename`, i.e. the final component of the `path` field must match\n`basename`.\n",
+ "jsonldPredicate": "cwl:basename"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#File/dirname",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "The name of the directory containing file, that is, the path leading up\nto the final slash in the path such that `dirname + '/' + basename ==\npath`.\n\nThe implementation must set this field based on the value of `path`\nprior to evaluating parameter references or expressions in a\nCommandLineTool document. This field must not be used in any other\ncontext.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#File/nameroot",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "The basename root such that `nameroot + nameext == basename`, and\n`nameext` is empty or begins with a period and contains at most one\nperiod. For the purposess of path splitting leading periods on the\nbasename are ignored; a basename of `.cshrc` will have a nameroot of\n`.cshrc`.\n\nThe implementation must set this field automatically based on the value\nof `basename` prior to evaluating parameter references or expressions.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#File/nameext",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "The basename extension such that `nameroot + nameext == basename`, and\n`nameext` is empty or begins with a period and contains at most one\nperiod. Leading periods on the basename are ignored; a basename of\n`.cshrc` will have an empty `nameext`.\n\nThe implementation must set this field automatically based on the value\nof `basename` prior to evaluating parameter references or expressions.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#File/checksum",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Optional hash code for validating file integrity. Currently must be in the form\n\"sha1$ + hexadecimal string\" using the SHA-1 algorithm.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#File/size",
+ "type": [
+ "null",
+ "long"
+ ],
+ "doc": "Optional file size"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#File/secondaryFiles",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": [
+ "https://w3id.org/cwl/cwl#File",
+ "https://w3id.org/cwl/cwl#Directory"
+ ]
+ }
+ ],
+ "jsonldPredicate": "cwl:secondaryFiles",
+ "doc": "A list of additional files that are associated with the primary file\nand must be transferred alongside the primary file. Examples include\nindexes of the primary file, or external references which must be\nincluded when loading primary document. A file object listed in\n`secondaryFiles` may itself include `secondaryFiles` for which the same\nrules apply.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#File/format",
+ "type": [
+ "null",
+ "string"
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#format",
+ "_type": "@id",
+ "identity": true
+ },
+ "doc": "The format of the file: this must be an IRI of a concept node that\nrepresents the file format, preferrably defined within an ontology.\nIf no ontology is available, file formats may be tested by exact match.\n\nReasoning about format compatability must be done by checking that an\ninput file format is the same, `owl:equivalentClass` or\n`rdfs:subClassOf` the format required by the input parameter.\n`owl:equivalentClass` is transitive with `rdfs:subClassOf`, e.g. if\n` owl:equivalentClass ` and ` owl:subclassOf ` then infer\n` owl:subclassOf `.\n\nFile format ontologies may be provided in the \"$schema\" metadata at the\nroot of the document. If no ontologies are specified in `$schema`, the\nruntime may perform exact file format matches.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#File/contents",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "File contents literal. Maximum of 64 KiB.\n\nIf neither `location` nor `path` is provided, `contents` must be\nnon-null. The implementation must assign a unique identifier for the\n`location` field. When the file is staged as input to CommandLineTool,\nthe value of `contents` must be written to a file.\n\nIf `loadContents` of `inputBinding` or `outputBinding` is true and\n`location` is valid, the implementation must read up to the first 64\nKiB of text from the file and place it in the \"contents\" field.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Directory",
+ "type": "record",
+ "docAfter": "https://w3id.org/cwl/cwl#File",
+ "doc": "Represents a directory to present to a command line tool.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#Directory/class",
+ "type": {
+ "type": "enum",
+ "symbols": [
+ "https://w3id.org/cwl/cwl#Directory"
+ ]
+ },
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ },
+ "doc": "Must be `Directory` to indicate this object describes a Directory."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Directory/location",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "An IRI that identifies the directory resource. This may be a relative\nreference, in which case it must be resolved using the base IRI of the\ndocument. The location may refer to a local or remote resource. If\nthe `listing` field is not set, the implementation must use the\nlocation IRI to retrieve directory listing. If an implementation is\nunable to retrieve the directory listing stored at a remote resource (due to\nunsupported protocol, access denied, or other issue) it must signal an\nerror.\n\nIf the `location` field is not provided, the `listing` field must be\nprovided. The implementation must assign a unique identifier for\nthe `location` field.\n\nIf the `path` field is provided but the `location` field is not, an\nimplementation may assign the value of the `path` field to `location`,\nthen follow the rules above.\n",
+ "jsonldPredicate": {
+ "_id": "@id",
+ "_type": "@id"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Directory/path",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "The local path where the Directory is made available prior to executing a\nCommandLineTool. This must be set by the implementation. This field\nmust not be used in any other context. The command line tool being\nexecuted must be able to to access the directory at `path` using the POSIX\n`opendir(2)` syscall.\n\nIf the `path` contains [POSIX shell metacharacters](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02)\n(`|`,`&`, `;`, `<`, `>`, `(`,`)`, `$`,`` ` ``, `\\`, `\"`, `'`,\n``, ``, and ``) or characters\n[not allowed](http://www.iana.org/assignments/idna-tables-6.3.0/idna-tables-6.3.0.xhtml)\nfor [Internationalized Domain Names for Applications](https://tools.ietf.org/html/rfc6452)\nthen implementations may terminate the process with a\n`permanentFailure`.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#path",
+ "_type": "@id"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Directory/basename",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "The base name of the directory, that is, the name of the file without any\nleading directory path. The base name must not contain a slash `/`.\n\nIf not provided, the implementation must set this field based on the\n`location` field by taking the final path component after parsing\n`location` as an IRI. If `basename` is provided, it is not required to\nmatch the value from `location`.\n\nWhen this file is made available to a CommandLineTool, it must be named\nwith `basename`, i.e. the final component of the `path` field must match\n`basename`.\n",
+ "jsonldPredicate": "cwl:basename"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Directory/listing",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": [
+ "https://w3id.org/cwl/cwl#File",
+ "https://w3id.org/cwl/cwl#Directory"
+ ]
+ }
+ ],
+ "doc": "List of files or subdirectories contained in this directory. The name\nof each file or subdirectory is determined by the `basename` field of\neach `File` or `Directory` object. It is an error if a `File` shares a\n`basename` with any other entry in `listing`. If two or more\n`Directory` object share the same `basename`, this must be treated as\nequivalent to a single subdirectory with the listings recursively\nmerged.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#listing"
+ }
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#SchemaBase",
+ "type": "record",
+ "abstract": true,
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#SchemaBase/label",
+ "type": [
+ "null",
+ "string"
+ ],
+ "jsonldPredicate": "rdfs:label",
+ "doc": "A short, human-readable label of this object."
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Parameter",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#SchemaBase",
+ "abstract": true,
+ "doc": "Define an input or output parameter to a process.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#Parameter/secondaryFiles",
+ "type": [
+ "null",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression",
+ {
+ "type": "array",
+ "items": [
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ]
+ }
+ ],
+ "jsonldPredicate": "cwl:secondaryFiles",
+ "doc": "Only valid when `type: File` or is an array of `items: File`.\n\nDescribes files that must be included alongside the primary file(s).\n\nIf the value is an expression, the value of `self` in the expression\nmust be the primary input or output File to which this binding applies.\n\nIf the value is a string, it specifies that the following pattern\nshould be applied to the primary file:\n\n 1. If string begins with one or more caret `^` characters, for each\n caret, remove the last file extension from the path (the last\n period `.` and all following characters). If there are no file\n extensions, the path is unchanged.\n 2. Append the remainder of the string to the end of the file path.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Parameter/format",
+ "type": [
+ "null",
+ "string",
+ {
+ "type": "array",
+ "items": "string"
+ },
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#format",
+ "_type": "@id",
+ "identity": true
+ },
+ "doc": "Only valid when `type: File` or is an array of `items: File`.\n\nFor input parameters, this must be one or more IRIs of concept nodes\nthat represents file formats which are allowed as input to this\nparameter, preferrably defined within an ontology. If no ontology is\navailable, file formats may be tested by exact match.\n\nFor output parameters, this is the file format that will be assigned to\nthe output parameter.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Parameter/streamable",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "doc": "Only valid when `type: File` or is an array of `items: File`.\n\nA value of `true` indicates that the file is read or written\nsequentially without seeking. An implementation may use this flag to\nindicate whether it is valid to stream file contents using a named\npipe. Default: `false`.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Parameter/doc",
+ "type": [
+ "null",
+ "string",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "doc": "A documentation string for this type, or an array of strings which should be concatenated.",
+ "jsonldPredicate": "rdfs:comment"
+ }
+ ]
+ },
+ {
+ "type": "enum",
+ "name": "https://w3id.org/cwl/cwl#Expression",
+ "doc": "'Expression' is not a real type. It indicates that a field must allow\nruntime parameter references. If [InlineJavascriptRequirement](#InlineJavascriptRequirement)\nis declared and supported by the platform, the field must also allow\nJavascript expressions.\n",
+ "symbols": [
+ "https://w3id.org/cwl/cwl#ExpressionPlaceholder"
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputBinding",
+ "type": "record",
+ "abstract": true,
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#InputBinding/loadContents",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "jsonldPredicate": "cwl:loadContents",
+ "doc": "Only valid when `type: File` or is an array of `items: File`.\n\nRead up to the first 64 KiB of text from the file and place it in the\n\"contents\" field of the file object for use by expressions.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputBinding",
+ "type": "record",
+ "abstract": true
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputSchema",
+ "extends": "https://w3id.org/cwl/cwl#SchemaBase",
+ "type": "record",
+ "abstract": true
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputSchema",
+ "extends": "https://w3id.org/cwl/cwl#SchemaBase",
+ "type": "record",
+ "abstract": true
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputRecordField",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/salad#RecordField",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#RecordSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#InputRecordSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#EnumSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#InputEnumSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#ArraySchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#InputArraySchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#PrimitiveType",
+ "specializeTo": "https://w3id.org/cwl/cwl#CWLType"
+ }
+ ],
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#InputRecordField/inputBinding",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#InputBinding"
+ ],
+ "jsonldPredicate": "cwl:inputBinding"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputRecordField/label",
+ "type": [
+ "null",
+ "string"
+ ],
+ "jsonldPredicate": "rdfs:label",
+ "doc": "A short, human-readable label of this process object."
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputRecordSchema",
+ "type": "record",
+ "extends": [
+ "https://w3id.org/cwl/salad#RecordSchema",
+ "https://w3id.org/cwl/cwl#InputSchema"
+ ],
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#RecordField",
+ "specializeTo": "https://w3id.org/cwl/cwl#InputRecordField"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputEnumSchema",
+ "type": "record",
+ "extends": [
+ "https://w3id.org/cwl/salad#EnumSchema",
+ "https://w3id.org/cwl/cwl#InputSchema"
+ ],
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#InputEnumSchema/inputBinding",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#InputBinding"
+ ],
+ "jsonldPredicate": "cwl:inputBinding"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputArraySchema",
+ "type": "record",
+ "extends": [
+ "https://w3id.org/cwl/salad#ArraySchema",
+ "https://w3id.org/cwl/cwl#InputSchema"
+ ],
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#RecordSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#InputRecordSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#EnumSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#InputEnumSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#ArraySchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#InputArraySchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#PrimitiveType",
+ "specializeTo": "https://w3id.org/cwl/cwl#CWLType"
+ }
+ ],
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#InputArraySchema/inputBinding",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#InputBinding"
+ ],
+ "jsonldPredicate": "cwl:inputBinding"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputRecordField",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/salad#RecordField",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#RecordSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#OutputRecordSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#EnumSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#OutputEnumSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#ArraySchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#OutputArraySchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#PrimitiveType",
+ "specializeTo": "https://w3id.org/cwl/cwl#CWLType"
+ }
+ ],
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputRecordField/outputBinding",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#OutputBinding"
+ ],
+ "jsonldPredicate": "cwl:outputBinding"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputRecordSchema",
+ "type": "record",
+ "extends": [
+ "https://w3id.org/cwl/salad#RecordSchema",
+ "https://w3id.org/cwl/cwl#OutputSchema"
+ ],
+ "docParent": "https://w3id.org/cwl/cwl#OutputParameter",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#RecordField",
+ "specializeTo": "https://w3id.org/cwl/cwl#OutputRecordField"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputEnumSchema",
+ "type": "record",
+ "extends": [
+ "https://w3id.org/cwl/salad#EnumSchema",
+ "https://w3id.org/cwl/cwl#OutputSchema"
+ ],
+ "docParent": "https://w3id.org/cwl/cwl#OutputParameter",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputEnumSchema/outputBinding",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#OutputBinding"
+ ],
+ "jsonldPredicate": "cwl:outputBinding"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputArraySchema",
+ "type": "record",
+ "extends": [
+ "https://w3id.org/cwl/salad#ArraySchema",
+ "https://w3id.org/cwl/cwl#OutputSchema"
+ ],
+ "docParent": "https://w3id.org/cwl/cwl#OutputParameter",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#RecordSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#OutputRecordSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#EnumSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#OutputEnumSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#ArraySchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#OutputArraySchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/salad#PrimitiveType",
+ "specializeTo": "https://w3id.org/cwl/cwl#CWLType"
+ }
+ ],
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputArraySchema/outputBinding",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#OutputBinding"
+ ],
+ "jsonldPredicate": "cwl:outputBinding"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputParameter",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#Parameter",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#InputParameter/id",
+ "type": "string",
+ "jsonldPredicate": "@id",
+ "doc": "The unique identifier for this parameter object."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputParameter/inputBinding",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#InputBinding"
+ ],
+ "jsonldPredicate": "cwl:inputBinding",
+ "doc": "Describes how to handle the inputs of a process and convert them\ninto a concrete form for execution, such as command line parameters.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputParameter/default",
+ "type": [
+ "null",
+ "Any"
+ ],
+ "jsonldPredicate": "cwl:default",
+ "doc": "The default value for this parameter if not provided in the input\nobject.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InputParameter/type",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#CWLType",
+ "https://w3id.org/cwl/cwl#InputRecordSchema",
+ "https://w3id.org/cwl/cwl#InputEnumSchema",
+ "https://w3id.org/cwl/cwl#InputArraySchema",
+ "string",
+ {
+ "type": "array",
+ "items": [
+ "https://w3id.org/cwl/cwl#CWLType",
+ "https://w3id.org/cwl/cwl#InputRecordSchema",
+ "https://w3id.org/cwl/cwl#InputEnumSchema",
+ "https://w3id.org/cwl/cwl#InputArraySchema",
+ "string"
+ ]
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#type",
+ "_type": "@vocab",
+ "refScope": 2,
+ "typeDSL": true
+ },
+ "doc": "Specify valid types of data that may be assigned to this parameter.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputParameter",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#Parameter",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputParameter/id",
+ "type": "string",
+ "jsonldPredicate": "@id",
+ "doc": "The unique identifier for this parameter object."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#OutputParameter/outputBinding",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#OutputBinding"
+ ],
+ "jsonldPredicate": "cwl:outputBinding",
+ "doc": "Describes how to handle the outputs of a process.\n"
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "abstract": true,
+ "doc": "A process requirement declares a prerequisite that may or must be fulfilled\nbefore executing a process. See [`Process.hints`](#process) and\n[`Process.requirements`](#process).\n\nProcess requirements are the primary mechanism for specifying extensions to\nthe CWL core specification.\n"
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#Process",
+ "abstract": true,
+ "doc": "\nThe base executable type in CWL is the `Process` object defined by the\ndocument. Note that the `Process` object is abstract and cannot be\ndirectly executed.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#Process/id",
+ "type": [
+ "null",
+ "string"
+ ],
+ "jsonldPredicate": "@id",
+ "doc": "The unique identifier for this process object."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Process/inputs",
+ "type": {
+ "type": "array",
+ "items": "https://w3id.org/cwl/cwl#InputParameter"
+ },
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#inputs",
+ "mapSubject": "id",
+ "mapPredicate": "type"
+ },
+ "doc": "Defines the input parameters of the process. The process is ready to\nrun when all required input parameters are associated with concrete\nvalues. Input parameters include a schema for each parameter which is\nused to validate the input object. It may also be used to build a user\ninterface for constructing the input object.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Process/outputs",
+ "type": {
+ "type": "array",
+ "items": "https://w3id.org/cwl/cwl#OutputParameter"
+ },
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#outputs",
+ "mapSubject": "id",
+ "mapPredicate": "type"
+ },
+ "doc": "Defines the parameters representing the output of the process. May be\nused to generate and/or validate the output object.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Process/requirements",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "https://w3id.org/cwl/cwl#ProcessRequirement"
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#requirements",
+ "mapSubject": "class"
+ },
+ "doc": "Declares requirements that apply to either the runtime environment or the\nworkflow engine that must be met in order to execute this process. If\nan implementation cannot satisfy all requirements, or a requirement is\nlisted which is not recognized by the implementation, it is a fatal\nerror and the implementation must not attempt to run the process,\nunless overridden at user option.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Process/hints",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "Any"
+ }
+ ],
+ "doc": "Declares hints applying to either the runtime environment or the\nworkflow engine that may be helpful in executing this process. It is\nnot an error if an implementation cannot satisfy all hints, however\nthe implementation may report a warning.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#hints",
+ "noLinkCheck": true,
+ "mapSubject": "class"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Process/label",
+ "type": [
+ "null",
+ "string"
+ ],
+ "jsonldPredicate": "rdfs:label",
+ "doc": "A short, human-readable label of this process object."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Process/doc",
+ "type": [
+ "null",
+ "string"
+ ],
+ "jsonldPredicate": "rdfs:comment",
+ "doc": "A long, human-readable description of this process object."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Process/cwlVersion",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#CWLVersion"
+ ],
+ "doc": "CWL document version. Always required at the document root. Not\nrequired for a Process embedded inside another Process.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#cwlVersion",
+ "_type": "@vocab"
+ }
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InlineJavascriptRequirement",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "Indicates that the workflow platform must support inline Javascript expressions.\nIf this requirement is not present, the workflow platform must not perform expression\ninterpolatation.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#InlineJavascriptRequirement/class",
+ "type": "string",
+ "doc": "Always 'InlineJavascriptRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InlineJavascriptRequirement/expressionLib",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "doc": "Additional code fragments that will also be inserted\nbefore executing the expression code. Allows for function definitions that may\nbe called from CWL expressions.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#SchemaDefRequirement",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "This field consists of an array of type definitions which must be used when\ninterpreting the `inputs` and `outputs` fields. When a `type` field\ncontain a IRI, the implementation must check if the type is defined in\n`schemaDefs` and use that definition. If the type is not found in\n`schemaDefs`, it is an error. The entries in `schemaDefs` must be\nprocessed in the order listed such that later schema definitions may refer\nto earlier schema definitions.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#SchemaDefRequirement/class",
+ "type": "string",
+ "doc": "Always 'SchemaDefRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#SchemaDefRequirement/types",
+ "type": {
+ "type": "array",
+ "items": "https://w3id.org/cwl/cwl#InputSchema"
+ },
+ "doc": "The list of type definitions."
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineToolDoc",
+ "type": "documentation",
+ "doc": [
+ "# Common Workflow Language (CWL) Command Line Tool Description, v1.0\n\nThis version:\n * https://w3id.org/cwl/v1.0/\n\nCurrent version:\n * https://w3id.org/cwl/\n",
+ "\n\n",
+ "\n",
+ "\n\n",
+ "# Abstract\n\nA Command Line Tool is a non-interactive executable program that reads\nsome input, performs a computation, and terminates after producing some\noutput. Command line programs are a flexible unit of code sharing and\nreuse, unfortunately the syntax and input/output semantics among command\nline programs is extremely heterogeneous. A common layer for describing\nthe syntax and semantics of programs can reduce this incidental\ncomplexity by providing a consistent way to connect programs together.\nThis specification defines the Common Workflow Language (CWL) Command\nLine Tool Description, a vendor-neutral standard for describing the\nsyntax and input/output semantics of command line programs.\n",
+ "\n",
+ "## Introduction to v1.0\n\nThis specification represents the first full release from the CWL group.\nSince draft-3, version 1.0 introduces the following changes and additions:\n\n * The [Directory](#Directory) type.\n * Syntax simplifcations: denoted by the `map<>` syntax. Example: inputs\n contains a list of items, each with an id. Now one can specify\n a mapping of that identifier to the corresponding\n `CommandInputParamater`.\n ```\n inputs:\n - id: one\n type: string\n doc: First input parameter\n - id: two\n type: int\n doc: Second input parameter\n ```\n can be\n ```\n inputs:\n one:\n type: string\n doc: First input parameter\n two:\n type: int\n doc: Second input parameter\n ```\n * [InitialWorkDirRequirement](#InitialWorkDirRequirement): list of\n files and subdirectories to be present in the output directory prior\n to execution.\n * Shortcuts for specifying the standard [output](#stdout) and/or\n [error](#stderr) streams as a (streamable) File output.\n * [SoftwareRequirement](#SoftwareRequirement) for describing software\n dependencies of a tool.\n * The common `description` field has been renamed to `doc`.\n\n## Errata\n\nPost v1.0 release changes to the spec.\n\n * 13 July 2016: Mark `baseCommand` as optional and update descriptive text.\n\n## Purpose\n\nStandalone programs are a flexible and interoperable form of code reuse.\nUnlike monolithic applications, applications and analysis workflows which\nare composed of multiple separate programs can be written in multiple\nlanguages and execute concurrently on multiple hosts. However, POSIX\ndoes not dictate computer-readable grammar or semantics for program input\nand output, resulting in extremely heterogeneous command line grammar and\ninput/output semantics among program. This is a particular problem in\ndistributed computing (multi-node compute clusters) and virtualized\nenvironments (such as Docker containers) where it is often necessary to\nprovision resources such as input files before executing the program.\n\nOften this gap is filled by hard coding program invocation and\nimplicitly assuming requirements will be met, or abstracting program\ninvocation with wrapper scripts or descriptor documents. Unfortunately,\nwhere these approaches are application or platform specific it creates a\nsignificant barrier to reproducibility and portability, as methods\ndeveloped for one platform must be manually ported to be used on new\nplatforms. Similarly it creates redundant work, as wrappers for popular\ntools must be rewritten for each application or platform in use.\n\nThe Common Workflow Language Command Line Tool Description is designed to\nprovide a common standard description of grammar and semantics for\ninvoking programs used in data-intensive fields such as Bioinformatics,\nChemistry, Physics, Astronomy, and Statistics. This specification\ndefines a precise data and execution model for Command Line Tools that\ncan be implemented on a variety of computing platforms, ranging from a\nsingle workstation to cluster, grid, cloud, and high performance\ncomputing platforms.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#EnvironmentDef",
+ "doc": "Define an environment variable that will be set in the runtime environment\nby the workflow platform when executing the command line tool. May be the\nresult of executing an expression, such as getting a parameter from input.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#EnvironmentDef/envName",
+ "type": "string",
+ "doc": "The environment variable name"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#EnvironmentDef/envValue",
+ "type": [
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "The environment variable value"
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#CommandLineBinding",
+ "extends": "https://w3id.org/cwl/cwl#InputBinding",
+ "doc": "\nWhen listed under `inputBinding` in the input schema, the term\n\"value\" refers to the the corresponding value in the input object. For\nbinding objects listed in `CommandLineTool.arguments`, the term \"value\"\nrefers to the effective value after evaluating `valueFrom`.\n\nThe binding behavior when building the command line depends on the data\ntype of the value. If there is a mismatch between the type described by\nthe input schema and the effective value, such as resulting from an\nexpression evaluation, an implementation must use the data type of the\neffective value.\n\n - **string**: Add `prefix` and the string to the command line.\n\n - **number**: Add `prefix` and decimal representation to command line.\n\n - **boolean**: If true, add `prefix` to the command line. If false, add\n nothing.\n\n - **File**: Add `prefix` and the value of\n [`File.path`](#File) to the command line.\n\n - **array**: If `itemSeparator` is specified, add `prefix` and the join\n the array into a single string with `itemSeparator` separating the\n items. Otherwise first add `prefix`, then recursively process\n individual elements.\n\n - **object**: Add `prefix` only, and recursively add object fields for\n which `inputBinding` is specified.\n\n - **null**: Add nothing.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineBinding/position",
+ "type": [
+ "null",
+ "int"
+ ],
+ "doc": "The sorting key. Default position is 0."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineBinding/prefix",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Command line prefix to add before the value."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineBinding/separate",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "doc": "If true (default), then the prefix and value must be added as separate\ncommand line arguments; if false, prefix and value must be concatenated\ninto a single command line argument.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineBinding/itemSeparator",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Join the array elements into a single string with the elements\nseparated by by `itemSeparator`.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineBinding/valueFrom",
+ "type": [
+ "null",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "jsonldPredicate": "cwl:valueFrom",
+ "doc": "If `valueFrom` is a constant string value, use this as the value and\napply the binding rules above.\n\nIf `valueFrom` is an expression, evaluate the expression to yield the\nactual value to use to build the command line and apply the binding\nrules above. If the inputBinding is associated with an input\nparameter, the value of `self` in the expression will be the value of the\ninput parameter.\n\nWhen a binding is part of the `CommandLineTool.arguments` field,\nthe `valueFrom` field is required.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineBinding/shellQuote",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "doc": "If `ShellCommandRequirement` is in the requirements for the current command,\nthis controls whether the value is quoted on the command line (default is true).\nUse `shellQuote: false` to inject metacharacters for operations such as pipes.\n"
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#CommandOutputBinding",
+ "extends": "https://w3id.org/cwl/cwl#OutputBinding",
+ "doc": "Describes how to generate an output parameter based on the files produced\nby a CommandLineTool.\n\nThe output parameter is generated by applying these operations in\nthe following order:\n\n - glob\n - loadContents\n - outputEval\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandOutputBinding/glob",
+ "type": [
+ "null",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "doc": "Find files relative to the output directory, using POSIX glob(3)\npathname matching. If an array is provided, find files that match any\npattern in the array. If an expression is provided, the expression must\nreturn a string or an array of strings, which will then be evaluated as\none or more glob patterns. Must only match and return files which\nactually exist.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandOutputBinding/loadContents",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "jsonldPredicate": "cwl:loadContents",
+ "doc": "For each file matched in `glob`, read up to\nthe first 64 KiB of text from the file and place it in the `contents`\nfield of the file object for manipulation by `outputEval`.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandOutputBinding/outputEval",
+ "type": [
+ "null",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "Evaluate an expression to generate the output value. If `glob` was\nspecified, the value of `self` must be an array containing file objects\nthat were matched. If no files were matched, `self` must be a zero\nlength array; if a single file was matched, the value of `self` is an\narray of a single element. Additionally, if `loadContents` is `true`,\nthe File objects must include up to the first 64 KiB of file contents\nin the `contents` field.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandInputRecordField",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#InputRecordField",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputRecordSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputRecordSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputEnumSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputEnumSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputArraySchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputArraySchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputBinding",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandLineBinding"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandInputRecordSchema",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#InputRecordSchema",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputRecordField",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputRecordField"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandInputEnumSchema",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#InputEnumSchema",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputBinding",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandLineBinding"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandInputArraySchema",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#InputArraySchema",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputRecordSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputRecordSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputEnumSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputEnumSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputArraySchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputArraySchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputBinding",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandLineBinding"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandOutputRecordField",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#OutputRecordField",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputRecordSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputRecordSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputEnumSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputEnumSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputArraySchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputArraySchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputBinding",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputBinding"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandOutputRecordSchema",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#OutputRecordSchema",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputRecordField",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputRecordField"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandOutputEnumSchema",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#OutputEnumSchema",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputRecordSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputRecordSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputEnumSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputEnumSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputArraySchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputArraySchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputBinding",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputBinding"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandOutputArraySchema",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#OutputArraySchema",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputRecordSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputRecordSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputEnumSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputEnumSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputArraySchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputArraySchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputBinding",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputBinding"
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#CommandInputParameter",
+ "extends": "https://w3id.org/cwl/cwl#InputParameter",
+ "doc": "An input parameter for a CommandLineTool.",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputRecordSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputRecordSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputEnumSchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputEnumSchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputArraySchema",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputArraySchema"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputBinding",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandLineBinding"
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#CommandOutputParameter",
+ "extends": "https://w3id.org/cwl/cwl#OutputParameter",
+ "doc": "An output parameter for a CommandLineTool.",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputBinding",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputBinding"
+ }
+ ],
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandOutputParameter/type",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#CWLType",
+ "https://w3id.org/cwl/cwl#stdout",
+ "https://w3id.org/cwl/cwl#stderr",
+ "https://w3id.org/cwl/cwl#CommandOutputRecordSchema",
+ "https://w3id.org/cwl/cwl#CommandOutputEnumSchema",
+ "https://w3id.org/cwl/cwl#CommandOutputArraySchema",
+ "string",
+ {
+ "type": "array",
+ "items": [
+ "https://w3id.org/cwl/cwl#CWLType",
+ "https://w3id.org/cwl/cwl#CommandOutputRecordSchema",
+ "https://w3id.org/cwl/cwl#CommandOutputEnumSchema",
+ "https://w3id.org/cwl/cwl#CommandOutputArraySchema",
+ "string"
+ ]
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#type",
+ "_type": "@vocab",
+ "refScope": 2,
+ "typeDSL": true
+ },
+ "doc": "Specify valid types of data that may be assigned to this parameter.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#stdout",
+ "type": "enum",
+ "symbols": [
+ "https://w3id.org/cwl/cwl#stdout"
+ ],
+ "docParent": "https://w3id.org/cwl/cwl#CommandOutputParameter",
+ "doc": "Only valid as a `type` for a `CommandLineTool` output with no\n`outputBinding` set.\n\nThe following\n```\noutputs:\n an_output_name:\n type: stdout\n\nstdout: a_stdout_file\n```\nis equivalent to\n```\noutputs:\n an_output_name:\n type: File\n streamable: true\n outputBinding:\n glob: a_stdout_file\n\nstdout: a_stdout_file\n```\n\nIf there is no `stdout` name provided, a random filename will be created.\nFor example, the following\n```\noutputs:\n an_output_name:\n type: stdout\n```\nis equivalent to\n```\noutputs:\n an_output_name:\n type: File\n streamable: true\n outputBinding:\n glob: random_stdout_filenameABCDEFG\n\nstdout: random_stdout_filenameABCDEFG\n```\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#stderr",
+ "type": "enum",
+ "symbols": [
+ "https://w3id.org/cwl/cwl#stderr"
+ ],
+ "docParent": "https://w3id.org/cwl/cwl#CommandOutputParameter",
+ "doc": "Only valid as a `type` for a `CommandLineTool` output with no\n`outputBinding` set.\n\nThe following\n```\noutputs:\n an_output_name:\n type: stderr\n\nstderr: a_stderr_file\n```\nis equivalent to\n```\noutputs:\n an_output_name:\n type: File\n streamable: true\n outputBinding:\n glob: a_stderr_file\n\nstderr: a_stderr_file\n```\n\nIf there is no `stderr` name provided, a random filename will be created.\nFor example, the following\n```\noutputs:\n an_output_name:\n type: stderr\n```\nis equivalent to\n```\noutputs:\n an_output_name:\n type: File\n streamable: true\n outputBinding:\n glob: random_stderr_filenameABCDEFG\n\nstderr: random_stderr_filenameABCDEFG\n```\n"
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#CommandLineTool",
+ "extends": "https://w3id.org/cwl/cwl#Process",
+ "documentRoot": true,
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#InputParameter",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandInputParameter"
+ },
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputParameter",
+ "specializeTo": "https://w3id.org/cwl/cwl#CommandOutputParameter"
+ }
+ ],
+ "doc": "This defines the schema of the CWL Command Line Tool Description document.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineTool/class",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ },
+ "type": "string"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineTool/baseCommand",
+ "doc": "Specifies the program to execute. If an array, the first element of\nthe array is the command to execute, and subsequent elements are\nmandatory command line arguments. The elements in `baseCommand` must\nappear before any command line bindings from `inputBinding` or\n`arguments`.\n\nIf `baseCommand` is not provided or is an empty array, the first\nelement of the command line produced after processing `inputBinding` or\n`arguments` must be used as the program to execute.\n\nIf the program includes a path separator character it must\nbe an absolute path, otherwise it is an error. If the program does not\ninclude a path separator, search the `$PATH` variable in the runtime\nenvironment of the workflow runner find the absolute path of the\nexecutable.\n",
+ "type": [
+ "null",
+ "string",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#baseCommand",
+ "_container": "@list"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineTool/arguments",
+ "doc": "Command line bindings which are not directly associated with input parameters.\n",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": [
+ "string",
+ "https://w3id.org/cwl/cwl#Expression",
+ "https://w3id.org/cwl/cwl#CommandLineBinding"
+ ]
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#arguments",
+ "_container": "@list"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineTool/stdin",
+ "type": [
+ "null",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "A path to a file whose contents must be piped into the command's\nstandard input stream.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineTool/stderr",
+ "type": [
+ "null",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "jsonldPredicate": "https://w3id.org/cwl/cwl#stderr",
+ "doc": "Capture the command's standard error stream to a file written to\nthe designated output directory.\n\nIf `stderr` is a string, it specifies the file name to use.\n\nIf `stderr` is an expression, the expression is evaluated and must\nreturn a string with the file name to use to capture stderr. If the\nreturn value is not a string, or the resulting path contains illegal\ncharacters (such as the path separator `/`) it is an error.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineTool/stdout",
+ "type": [
+ "null",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "jsonldPredicate": "https://w3id.org/cwl/cwl#stdout",
+ "doc": "Capture the command's standard output stream to a file written to\nthe designated output directory.\n\nIf `stdout` is a string, it specifies the file name to use.\n\nIf `stdout` is an expression, the expression is evaluated and must\nreturn a string with the file name to use to capture stdout. If the\nreturn value is not a string, or the resulting path contains illegal\ncharacters (such as the path separator `/`) it is an error.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineTool/successCodes",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "int"
+ }
+ ],
+ "doc": "Exit codes that indicate the process completed successfully.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineTool/temporaryFailCodes",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "int"
+ }
+ ],
+ "doc": "Exit codes that indicate the process failed due to a possibly\ntemporary condition, where executing the process with the same\nruntime environment and inputs may produce different results.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#CommandLineTool/permanentFailCodes",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "int"
+ }
+ ],
+ "doc": "Exit codes that indicate the process failed due to a permanent logic error, where executing the process with the same runtime environment and same inputs is expected to always fail."
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#DockerRequirement",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "Indicates that a workflow component should be run in a\n[Docker](http://docker.com) container, and specifies how to fetch or build\nthe image.\n\nIf a CommandLineTool lists `DockerRequirement` under\n`hints` (or `requirements`), it may (or must) be run in the specified Docker\ncontainer.\n\nThe platform must first acquire or install the correct Docker image as\nspecified by `dockerPull`, `dockerImport`, `dockerLoad` or `dockerFile`.\n\nThe platform must execute the tool in the container using `docker run` with\nthe appropriate Docker image and tool command line.\n\nThe workflow platform may provide input files and the designated output\ndirectory through the use of volume bind mounts. The platform may rewrite\nfile paths in the input object to correspond to the Docker bind mounted\nlocations.\n\nWhen running a tool contained in Docker, the workflow platform must not\nassume anything about the contents of the Docker container, such as the\npresence or absence of specific software, except to assume that the\ngenerated command line represents a valid command within the runtime\nenvironment of the container.\n\n## Interaction with other requirements\n\nIf [EnvVarRequirement](#EnvVarRequirement) is specified alongside a\nDockerRequirement, the environment variables must be provided to Docker\nusing `--env` or `--env-file` and interact with the container's preexisting\nenvironment as defined by Docker.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#DockerRequirement/class",
+ "type": "string",
+ "doc": "Always 'DockerRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#DockerRequirement/dockerPull",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Specify a Docker image to retrieve using `docker pull`."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#DockerRequirement/dockerLoad",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Specify a HTTP URL from which to download a Docker image using `docker load`."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#DockerRequirement/dockerFile",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Supply the contents of a Dockerfile which will be built using `docker build`."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#DockerRequirement/dockerImport",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Provide HTTP URL to download and gunzip a Docker images using `docker import."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#DockerRequirement/dockerImageId",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "The image id that will be used for `docker run`. May be a\nhuman-readable image name or the image identifier hash. May be skipped\nif `dockerPull` is specified, in which case the `dockerPull` image id\nmust be used.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#DockerRequirement/dockerOutputDirectory",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Set the designated output directory to a specific location inside the\nDocker container.\n"
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#SoftwareRequirement",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "A list of software packages that should be configured in the environment of\nthe defined process.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#SoftwareRequirement/class",
+ "type": "string",
+ "doc": "Always 'SoftwareRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#SoftwareRequirement/packages",
+ "type": {
+ "type": "array",
+ "items": "https://w3id.org/cwl/cwl#SoftwarePackage"
+ },
+ "doc": "The list of software to be configured.",
+ "jsonldPredicate": {
+ "mapSubject": "package",
+ "mapPredicate": "specs"
+ }
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#SoftwarePackage",
+ "type": "record",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#SoftwarePackage/package",
+ "type": "string",
+ "doc": "The common name of the software to be configured."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#SoftwarePackage/version",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "doc": "The (optional) version of the software to configured."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#SoftwarePackage/specs",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "doc": "Must be one or more IRIs identifying resources for installing or\nenabling the software. Implementations may provide resolvers which map\nwell-known software spec IRIs to some configuration action.\n\nFor example, an IRI `https://packages.debian.org/jessie/bowtie` could\nbe resolved with `apt-get install bowtie`. An IRI\n`https://anaconda.org/bioconda/bowtie` could be resolved with `conda\ninstall -c bioconda bowtie`.\n\nTools may also provide IRIs to index entries such as\n[RRID](http://www.identifiers.org/rrid/), such as\n`http://identifiers.org/rrid/RRID:SCR_005476`\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Dirent",
+ "type": "record",
+ "doc": "Define a file or subdirectory that must be placed in the designated output\ndirectory prior to executing the command line tool. May be the result of\nexecuting an expression, such as building a configuration file from a\ntemplate.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#Dirent/entryname",
+ "type": [
+ "null",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#entryname"
+ },
+ "doc": "The name of the file or subdirectory to create in the output directory.\nIf `entry` is a File or Directory, this overrides `basename`. Optional.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Dirent/entry",
+ "type": [
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#entry"
+ },
+ "doc": "If the value is a string literal or an expression which evaluates to a\nstring, a new file must be created with the string as the file contents.\n\nIf the value is an expression that evaluates to a `File` object, this\nindicates the referenced file should be added to the designated output\ndirectory prior to executing the tool.\n\nIf the value is an expression that evaluates to a `Dirent` object, this\nindicates that the File or Directory in `entry` should be added to the\ndesignated output directory with the name in `entryname`.\n\nIf `writable` is false, the file may be made available using a bind\nmount or file system link to avoid unnecessary copying of the input\nfile.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Dirent/writable",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "doc": "If true, the file or directory must be writable by the tool. Changes\nto the file or directory must be isolated and not visible by any other\nCommandLineTool process. This may be implemented by making a copy of\nthe original file or directory. Default false (files and directories\nread-only by default).\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InitialWorkDirRequirement",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "Define a list of files and subdirectories that must be created by the workflow platform in the designated output directory prior to executing the command line tool.",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#InitialWorkDirRequirement/class",
+ "type": "string",
+ "doc": "InitialWorkDirRequirement",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#InitialWorkDirRequirement/listing",
+ "type": [
+ {
+ "type": "array",
+ "items": [
+ "https://w3id.org/cwl/cwl#File",
+ "https://w3id.org/cwl/cwl#Directory",
+ "https://w3id.org/cwl/cwl#Dirent",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ]
+ },
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#listing"
+ },
+ "doc": "The list of files or subdirectories that must be placed in the\ndesignated output directory prior to executing the command line tool.\n\nMay be an expression. If so, the expression return value must validate\nas `{type: array, items: [File, Directory]}`.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#EnvVarRequirement",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "Define a list of environment variables which will be set in the\nexecution environment of the tool. See `EnvironmentDef` for details.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#EnvVarRequirement/class",
+ "type": "string",
+ "doc": "Always 'EnvVarRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#EnvVarRequirement/envDef",
+ "type": {
+ "type": "array",
+ "items": "https://w3id.org/cwl/cwl#EnvironmentDef"
+ },
+ "doc": "The list of environment variables.",
+ "jsonldPredicate": {
+ "mapSubject": "envName",
+ "mapPredicate": "envValue"
+ }
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#ShellCommandRequirement",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "Modify the behavior of CommandLineTool to generate a single string\ncontaining a shell command line. Each item in the argument list must be\njoined into a string separated by single spaces and quoted to prevent\nintepretation by the shell, unless `CommandLineBinding` for that argument\ncontains `shellQuote: false`. If `shellQuote: false` is specified, the\nargument is joined into the command string without quoting, which allows\nthe use of shell metacharacters such as `|` for pipes.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#ShellCommandRequirement/class",
+ "type": "string",
+ "doc": "Always 'ShellCommandRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#ResourceRequirement",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "Specify basic hardware resource requirements.\n\n\"min\" is the minimum amount of a resource that must be reserved to schedule\na job. If \"min\" cannot be satisfied, the job should not be run.\n\n\"max\" is the maximum amount of a resource that the job shall be permitted\nto use. If a node has sufficient resources, multiple jobs may be scheduled\non a single node provided each job's \"max\" resource requirements are\nmet. If a job attempts to exceed its \"max\" resource allocation, an\nimplementation may deny additional resources, which may result in job\nfailure.\n\nIf \"min\" is specified but \"max\" is not, then \"max\" == \"min\"\nIf \"max\" is specified by \"min\" is not, then \"min\" == \"max\".\n\nIt is an error if max < min.\n\nIt is an error if the value of any of these fields is negative.\n\nIf neither \"min\" nor \"max\" is specified for a resource, an implementation may provide a default.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#ResourceRequirement/class",
+ "type": "string",
+ "doc": "Always 'ResourceRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ResourceRequirement/coresMin",
+ "type": [
+ "null",
+ "long",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "Minimum reserved number of CPU cores"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ResourceRequirement/coresMax",
+ "type": [
+ "null",
+ "int",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "Maximum reserved number of CPU cores"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ResourceRequirement/ramMin",
+ "type": [
+ "null",
+ "long",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "Minimum reserved RAM in mebibytes (2**20)"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ResourceRequirement/ramMax",
+ "type": [
+ "null",
+ "long",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "Maximum reserved RAM in mebibytes (2**20)"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ResourceRequirement/tmpdirMin",
+ "type": [
+ "null",
+ "long",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "Minimum reserved filesystem based storage for the designated temporary directory, in mebibytes (2**20)"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ResourceRequirement/tmpdirMax",
+ "type": [
+ "null",
+ "long",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "Maximum reserved filesystem based storage for the designated temporary directory, in mebibytes (2**20)"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ResourceRequirement/outdirMin",
+ "type": [
+ "null",
+ "long",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "Minimum reserved filesystem based storage for the designated output directory, in mebibytes (2**20)"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ResourceRequirement/outdirMax",
+ "type": [
+ "null",
+ "long",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "Maximum reserved filesystem based storage for the designated output directory, in mebibytes (2**20)"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowDoc",
+ "type": "documentation",
+ "doc": [
+ "# Common Workflow Language (CWL) Workflow Description, v1.0\n\nThis version:\n * https://w3id.org/cwl/v1.0/\n\nCurrent version:\n * https://w3id.org/cwl/\n",
+ "\n\n",
+ "\n",
+ "\n\n",
+ "# Abstract\n\nOne way to define a workflow is: an analysis task represented by a\ndirected graph describing a sequence of operations that transform an\ninput data set to output. This specification defines the Common Workflow\nLanguage (CWL) Workflow description, a vendor-neutral standard for\nrepresenting workflows intended to be portable across a variety of\ncomputing platforms.\n",
+ "\n",
+ "\n## Introduction to v1.0\n\nThis specification represents the first full release from the CWL group.\nSince draft-3, this draft introduces the following changes and additions:\n\n * The `inputs` and `outputs` fields have been renamed `in` and `out`.\n * Syntax simplifcations: denoted by the `map<>` syntax. Example: `in`\n contains a list of items, each with an id. Now one can specify\n a mapping of that identifier to the corresponding\n `InputParameter`.\n ```\n in:\n - id: one\n type: string\n doc: First input parameter\n - id: two\n type: int\n doc: Second input parameter\n ```\n can be\n ```\n in:\n one:\n type: string\n doc: First input parameter\n two:\n type: int\n doc: Second input parameter\n ```\n * The common field `description` has been renamed to `doc`.\n\n## Purpose\n\nThe Common Workflow Language Command Line Tool Description express\nworkflows for data-intensive science, such as Bioinformatics, Chemistry,\nPhysics, and Astronomy. This specification is intended to define a data\nand execution model for Workflows that can be implemented on top of a\nvariety of computing platforms, ranging from an individual workstation to\ncluster, grid, cloud, and high performance computing systems.\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ExpressionToolOutputParameter",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#OutputParameter",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#ExpressionToolOutputParameter/type",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#CWLType",
+ "https://w3id.org/cwl/cwl#OutputRecordSchema",
+ "https://w3id.org/cwl/cwl#OutputEnumSchema",
+ "https://w3id.org/cwl/cwl#OutputArraySchema",
+ "string",
+ {
+ "type": "array",
+ "items": [
+ "https://w3id.org/cwl/cwl#CWLType",
+ "https://w3id.org/cwl/cwl#OutputRecordSchema",
+ "https://w3id.org/cwl/cwl#OutputEnumSchema",
+ "https://w3id.org/cwl/cwl#OutputArraySchema",
+ "string"
+ ]
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#type",
+ "_type": "@vocab",
+ "refScope": 2,
+ "typeDSL": true
+ },
+ "doc": "Specify valid types of data that may be assigned to this parameter.\n"
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#ExpressionTool",
+ "extends": "https://w3id.org/cwl/cwl#Process",
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputParameter",
+ "specializeTo": "https://w3id.org/cwl/cwl#ExpressionToolOutputParameter"
+ }
+ ],
+ "documentRoot": true,
+ "doc": "Execute an expression as a Workflow step.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#ExpressionTool/class",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ },
+ "type": "string"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ExpressionTool/expression",
+ "type": [
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "doc": "The expression to execute. The expression must return a JSON object which\nmatches the output parameters of the ExpressionTool.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#LinkMergeMethod",
+ "type": "enum",
+ "docParent": "https://w3id.org/cwl/cwl#WorkflowStepInput",
+ "doc": "The input link merge method, described in [WorkflowStepInput](#WorkflowStepInput).",
+ "symbols": [
+ "https://w3id.org/cwl/cwl#LinkMergeMethod/merge_nested",
+ "https://w3id.org/cwl/cwl#LinkMergeMethod/merge_flattened"
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowOutputParameter",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#OutputParameter",
+ "docParent": "https://w3id.org/cwl/cwl#Workflow",
+ "doc": "Describe an output parameter of a workflow. The parameter must be\nconnected to one or more parameters defined in the workflow that will\nprovide the value of the output parameter.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowOutputParameter/outputSource",
+ "doc": "Specifies one or more workflow parameters that supply the value of to\nthe output parameter.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#outputSource",
+ "_type": "@id",
+ "refScope": 0
+ },
+ "type": [
+ "null",
+ "string",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowOutputParameter/linkMerge",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#LinkMergeMethod"
+ ],
+ "jsonldPredicate": "cwl:linkMerge",
+ "doc": "The method to use to merge multiple sources into a single array.\nIf not specified, the default method is \"merge_nested\".\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowOutputParameter/type",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#CWLType",
+ "https://w3id.org/cwl/cwl#OutputRecordSchema",
+ "https://w3id.org/cwl/cwl#OutputEnumSchema",
+ "https://w3id.org/cwl/cwl#OutputArraySchema",
+ "string",
+ {
+ "type": "array",
+ "items": [
+ "https://w3id.org/cwl/cwl#CWLType",
+ "https://w3id.org/cwl/cwl#OutputRecordSchema",
+ "https://w3id.org/cwl/cwl#OutputEnumSchema",
+ "https://w3id.org/cwl/cwl#OutputArraySchema",
+ "string"
+ ]
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#type",
+ "_type": "@vocab",
+ "refScope": 2,
+ "typeDSL": true
+ },
+ "doc": "Specify valid types of data that may be assigned to this parameter.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Sink",
+ "type": "record",
+ "abstract": true,
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#Sink/source",
+ "doc": "Specifies one or more workflow parameters that will provide input to\nthe underlying step parameter.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#source",
+ "_type": "@id",
+ "refScope": 2
+ },
+ "type": [
+ "null",
+ "string",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Sink/linkMerge",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#LinkMergeMethod"
+ ],
+ "jsonldPredicate": "cwl:linkMerge",
+ "doc": "The method to use to merge multiple inbound links into a single array.\nIf not specified, the default method is \"merge_nested\".\n"
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#WorkflowStepInput",
+ "extends": "https://w3id.org/cwl/cwl#Sink",
+ "docParent": "https://w3id.org/cwl/cwl#WorkflowStep",
+ "doc": "The input of a workflow step connects an upstream parameter (from the\nworkflow inputs, or the outputs of other workflows steps) with the input\nparameters of the underlying step.\n\n## Input object\n\nA WorkflowStepInput object must contain an `id` field in the form\n`#fieldname` or `#stepname.fieldname`. When the `id` field contains a\nperiod `.` the field name consists of the characters following the final\nperiod. This defines a field of the workflow step input object with the\nvalue of the `source` parameter(s).\n\n## Merging\n\nTo merge multiple inbound data links,\n[MultipleInputFeatureRequirement](#MultipleInputFeatureRequirement) must be specified\nin the workflow or workflow step requirements.\n\nIf the sink parameter is an array, or named in a [workflow\nscatter](#WorkflowStep) operation, there may be multiple inbound data links\nlisted in the `source` field. The values from the input links are merged\ndepending on the method specified in the `linkMerge` field. If not\nspecified, the default method is \"merge_nested\".\n\n* **merge_nested**\n\n The input must be an array consisting of exactly one entry for each\n input link. If \"merge_nested\" is specified with a single link, the value\n from the link must be wrapped in a single-item list.\n\n* **merge_flattened**\n\n 1. The source and sink parameters must be compatible types, or the source\n type must be compatible with single element from the \"items\" type of\n the destination array parameter.\n 2. Source parameters which are arrays are concatenated.\n Source parameters which are single element types are appended as\n single elements.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStepInput/id",
+ "type": "string",
+ "jsonldPredicate": "@id",
+ "doc": "A unique identifier for this workflow input parameter."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStepInput/default",
+ "type": [
+ "null",
+ "Any"
+ ],
+ "doc": "The default value for this parameter if there is no `source`\nfield.\n",
+ "jsonldPredicate": "cwl:default"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStepInput/valueFrom",
+ "type": [
+ "null",
+ "string",
+ "https://w3id.org/cwl/cwl#Expression"
+ ],
+ "jsonldPredicate": "cwl:valueFrom",
+ "doc": "To use valueFrom, [StepInputExpressionRequirement](#StepInputExpressionRequirement) must\nbe specified in the workflow or workflow step requirements.\n\nIf `valueFrom` is a constant string value, use this as the value for\nthis input parameter.\n\nIf `valueFrom` is a parameter reference or expression, it must be\nevaluated to yield the actual value to be assiged to the input field.\n\nThe `self` value of in the parameter reference or expression must be\nthe value of the parameter(s) specified in the `source` field, or\nnull if there is no `source` field.\n\nThe value of `inputs` in the parameter reference or expression must be\nthe input object to the workflow step after assigning the `source`\nvalues and then scattering. The order of evaluating `valueFrom` among\nstep input parameters is undefined and the result of evaluating\n`valueFrom` on a parameter must not be visible to evaluation of\n`valueFrom` on other parameters.\n"
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#WorkflowStepOutput",
+ "docParent": "https://w3id.org/cwl/cwl#WorkflowStep",
+ "doc": "Associate an output parameter of the underlying process with a workflow\nparameter. The workflow parameter (given in the `id` field) be may be used\nas a `source` to connect with input parameters of other workflow steps, or\nwith an output parameter of the process.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStepOutput/id",
+ "type": "string",
+ "jsonldPredicate": "@id",
+ "doc": "A unique identifier for this workflow output parameter. This is the\nidentifier to use in the `source` field of `WorkflowStepInput` to\nconnect the output value to downstream parameters.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ScatterMethod",
+ "type": "enum",
+ "docParent": "https://w3id.org/cwl/cwl#WorkflowStep",
+ "doc": "The scatter method, as described in [workflow step scatter](#WorkflowStep).",
+ "symbols": [
+ "https://w3id.org/cwl/cwl#ScatterMethod/dotproduct",
+ "https://w3id.org/cwl/cwl#ScatterMethod/nested_crossproduct",
+ "https://w3id.org/cwl/cwl#ScatterMethod/flat_crossproduct"
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep",
+ "type": "record",
+ "docParent": "https://w3id.org/cwl/cwl#Workflow",
+ "doc": "A workflow step is an executable element of a workflow. It specifies the\nunderlying process implementation (such as `CommandLineTool` or another\n`Workflow`) in the `run` field and connects the input and output parameters\nof the underlying process to workflow parameters.\n\n# Scatter/gather\n\nTo use scatter/gather,\n[ScatterFeatureRequirement](#ScatterFeatureRequirement) must be specified\nin the workflow or workflow step requirements.\n\nA \"scatter\" operation specifies that the associated workflow step or\nsubworkflow should execute separately over a list of input elements. Each\njob making up a scatter operation is independent and may be executed\nconcurrently.\n\nThe `scatter` field specifies one or more input parameters which will be\nscattered. An input parameter may be listed more than once. The declared\ntype of each input parameter is implicitly wrapped in an array for each\ntime it appears in the `scatter` field. As a result, upstream parameters\nwhich are connected to scattered parameters may be arrays.\n\nAll output parameter types are also implicitly wrapped in arrays. Each job\nin the scatter results in an entry in the output array.\n\nIf `scatter` declares more than one input parameter, `scatterMethod`\ndescribes how to decompose the input into a discrete set of jobs.\n\n * **dotproduct** specifies that each of the input arrays are aligned and one\n element taken from each array to construct each job. It is an error\n if all input arrays are not the same length.\n\n * **nested_crossproduct** specifies the Cartesian product of the inputs,\n producing a job for every combination of the scattered inputs. The\n output must be nested arrays for each level of scattering, in the\n order that the input arrays are listed in the `scatter` field.\n\n * **flat_crossproduct** specifies the Cartesian product of the inputs,\n producing a job for every combination of the scattered inputs. The\n output arrays must be flattened to a single level, but otherwise listed in the\n order that the input arrays are listed in the `scatter` field.\n\n# Subworkflows\n\nTo specify a nested workflow as part of a workflow step,\n[SubworkflowFeatureRequirement](#SubworkflowFeatureRequirement) must be\nspecified in the workflow or workflow step requirements.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep/id",
+ "type": "string",
+ "jsonldPredicate": "@id",
+ "doc": "The unique identifier for this workflow step."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep/in",
+ "type": {
+ "type": "array",
+ "items": "https://w3id.org/cwl/cwl#WorkflowStepInput"
+ },
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#in",
+ "mapSubject": "id",
+ "mapPredicate": "source"
+ },
+ "doc": "Defines the input parameters of the workflow step. The process is ready to\nrun when all required input parameters are associated with concrete\nvalues. Input parameters include a schema for each parameter which is\nused to validate the input object. It may also be used build a user\ninterface for constructing the input object.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep/out",
+ "type": [
+ {
+ "type": "array",
+ "items": [
+ "string",
+ "https://w3id.org/cwl/cwl#WorkflowStepOutput"
+ ]
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#out",
+ "_type": "@id",
+ "identity": true
+ },
+ "doc": "Defines the parameters representing the output of the process. May be\nused to generate and/or validate the output object.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep/requirements",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "https://w3id.org/cwl/cwl#ProcessRequirement"
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#requirements",
+ "mapSubject": "class"
+ },
+ "doc": "Declares requirements that apply to either the runtime environment or the\nworkflow engine that must be met in order to execute this workflow step. If\nan implementation cannot satisfy all requirements, or a requirement is\nlisted which is not recognized by the implementation, it is a fatal\nerror and the implementation must not attempt to run the process,\nunless overridden at user option.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep/hints",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "Any"
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#hints",
+ "noLinkCheck": true,
+ "mapSubject": "class"
+ },
+ "doc": "Declares hints applying to either the runtime environment or the\nworkflow engine that may be helpful in executing this workflow step. It is\nnot an error if an implementation cannot satisfy all hints, however\nthe implementation may report a warning.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep/label",
+ "type": [
+ "null",
+ "string"
+ ],
+ "jsonldPredicate": "rdfs:label",
+ "doc": "A short, human-readable label of this process object."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep/doc",
+ "type": [
+ "null",
+ "string"
+ ],
+ "jsonldPredicate": "rdfs:comment",
+ "doc": "A long, human-readable description of this process object."
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep/run",
+ "type": [
+ "string",
+ "https://w3id.org/cwl/cwl#Process"
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#run",
+ "_type": "@id"
+ },
+ "doc": "Specifies the process to run.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep/scatter",
+ "type": [
+ "null",
+ "string",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#scatter",
+ "_type": "@id",
+ "_container": "@list",
+ "refScope": 0
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#WorkflowStep/scatterMethod",
+ "doc": "Required if `scatter` is an array of more than one element.\n",
+ "type": [
+ "null",
+ "https://w3id.org/cwl/cwl#ScatterMethod"
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/cwl#scatterMethod",
+ "_type": "@vocab"
+ }
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Workflow",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#Process",
+ "documentRoot": true,
+ "specialize": [
+ {
+ "specializeFrom": "https://w3id.org/cwl/cwl#OutputParameter",
+ "specializeTo": "https://w3id.org/cwl/cwl#WorkflowOutputParameter"
+ }
+ ],
+ "doc": "A workflow describes a set of **steps** and the **dependencies** between\nthose steps. When a step produces output that will be consumed by a\nsecond step, the first step is a dependency of the second step.\n\nWhen there is a dependency, the workflow engine must execute the preceeding\nstep and wait for it to successfully produce output before executing the\ndependent step. If two steps are defined in the workflow graph that\nare not directly or indirectly dependent, these steps are **independent**,\nand may execute in any order or execute concurrently. A workflow is\ncomplete when all steps have been executed.\n\nDependencies between parameters are expressed using the `source` field on\n[workflow step input parameters](#WorkflowStepInput) and [workflow output\nparameters](#WorkflowOutputParameter).\n\nThe `source` field expresses the dependency of one parameter on another\nsuch that when a value is associated with the parameter specified by\n`source`, that value is propagated to the destination parameter. When all\ndata links inbound to a given step are fufilled, the step is ready to\nexecute.\n\n## Workflow success and failure\n\nA completed step must result in one of `success`, `temporaryFailure` or\n`permanentFailure` states. An implementation may choose to retry a step\nexecution which resulted in `temporaryFailure`. An implementation may\nchoose to either continue running other steps of a workflow, or terminate\nimmediately upon `permanentFailure`.\n\n* If any step of a workflow execution results in `permanentFailure`, then\nthe workflow status is `permanentFailure`.\n\n* If one or more steps result in `temporaryFailure` and all other steps\ncomplete `success` or are not executed, then the workflow status is\n`temporaryFailure`.\n\n* If all workflow steps are executed and complete with `success`, then the\nworkflow status is `success`.\n\n# Extensions\n\n[ScatterFeatureRequirement](#ScatterFeatureRequirement) and\n[SubworkflowFeatureRequirement](#SubworkflowFeatureRequirement) are\navailable as standard [extensions](#Extensions_and_Metadata) to core\nworkflow semantics.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#Workflow/class",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ },
+ "type": "string"
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#Workflow/steps",
+ "doc": "The individual steps that make up the workflow. Each step is executed when all of its\ninput data links are fufilled. An implementation may choose to execute\nthe steps in a different order than listed and/or execute steps\nconcurrently, provided that dependencies between steps are met.\n",
+ "type": [
+ {
+ "type": "array",
+ "items": "https://w3id.org/cwl/cwl#WorkflowStep"
+ }
+ ],
+ "jsonldPredicate": {
+ "mapSubject": "id"
+ }
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#SubworkflowFeatureRequirement",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "Indicates that the workflow platform must support nested workflows in\nthe `run` field of [WorkflowStep](#WorkflowStep).\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#SubworkflowFeatureRequirement/class",
+ "type": "string",
+ "doc": "Always 'SubworkflowFeatureRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#ScatterFeatureRequirement",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "Indicates that the workflow platform must support the `scatter` and\n`scatterMethod` fields of [WorkflowStep](#WorkflowStep).\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#ScatterFeatureRequirement/class",
+ "type": "string",
+ "doc": "Always 'ScatterFeatureRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/cwl#MultipleInputFeatureRequirement",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "Indicates that the workflow platform must support multiple inbound data links\nlisted in the `source` field of [WorkflowStepInput](#WorkflowStepInput).\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#MultipleInputFeatureRequirement/class",
+ "type": "string",
+ "doc": "Always 'MultipleInputFeatureRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ }
+ ]
+ },
+ {
+ "type": "record",
+ "name": "https://w3id.org/cwl/cwl#StepInputExpressionRequirement",
+ "extends": "https://w3id.org/cwl/cwl#ProcessRequirement",
+ "doc": "Indicate that the workflow platform must support the `valueFrom` field\nof [WorkflowStepInput](#WorkflowStepInput).\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/cwl#StepInputExpressionRequirement/class",
+ "type": "string",
+ "doc": "Always 'StepInputExpressionRequirement'",
+ "jsonldPredicate": {
+ "_id": "@type",
+ "_type": "@vocab"
+ }
+ }
+ ]
+ }
+]
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/docimp/d1.yml b/v1.1.0-dev1/salad/schema_salad/tests/docimp/d1.yml
new file mode 100644
index 000000000..92cb8150b
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/docimp/d1.yml
@@ -0,0 +1,7 @@
+$graph:
+- name: "Semantic_Annotations_for_Linked_Avro_Data"
+ type: documentation
+ doc:
+ - $include: d2.md
+ - $import: d3.yml
+ - $import: d4.yml
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/docimp/d2.md b/v1.1.0-dev1/salad/schema_salad/tests/docimp/d2.md
new file mode 100644
index 000000000..2032c4a77
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/docimp/d2.md
@@ -0,0 +1 @@
+*Hello*
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/docimp/d3.yml b/v1.1.0-dev1/salad/schema_salad/tests/docimp/d3.yml
new file mode 100644
index 000000000..99211ecd2
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/docimp/d3.yml
@@ -0,0 +1,3 @@
+- "hello 2"
+- $include: d5.md
+- "hello 3"
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/docimp/d4.yml b/v1.1.0-dev1/salad/schema_salad/tests/docimp/d4.yml
new file mode 100644
index 000000000..1587596e2
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/docimp/d4.yml
@@ -0,0 +1,3 @@
+- "hello 4"
+- $include: d5.md
+- "hello 5"
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/docimp/d5.md b/v1.1.0-dev1/salad/schema_salad/tests/docimp/d5.md
new file mode 100644
index 000000000..a8eba23e8
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/docimp/d5.md
@@ -0,0 +1 @@
+*dee dee dee five*
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/docimp/dpre.json b/v1.1.0-dev1/salad/schema_salad/tests/docimp/dpre.json
new file mode 100644
index 000000000..c23da589e
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/docimp/dpre.json
@@ -0,0 +1,13 @@
+[
+ {
+ "name": "file:///home/peter/work/salad/schema_salad/tests/docimp/d1.yml#Semantic_Annotations_for_Linked_Avro_Data",
+ "type": "documentation",
+ "doc": [
+ "*Hello*",
+ "hello 2",
+ "hello 3",
+ "hello 4",
+ "hello 5"
+ ]
+ }
+]
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/frag.yml b/v1.1.0-dev1/salad/schema_salad/tests/frag.yml
new file mode 100644
index 000000000..7e8818d39
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/frag.yml
@@ -0,0 +1,4 @@
+- id: foo1
+ bar: b1
+- id: foo2
+ bar: b2
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/hello.txt b/v1.1.0-dev1/salad/schema_salad/tests/hello.txt
new file mode 100644
index 000000000..a04238969
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/hello.txt
@@ -0,0 +1 @@
+hello world!
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/hellofield.yml b/v1.1.0-dev1/salad/schema_salad/tests/hellofield.yml
new file mode 100644
index 000000000..11bc98f0d
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/hellofield.yml
@@ -0,0 +1,5 @@
+{
+"name": "hello",
+"doc": {"$include": "hello.txt"},
+"type": "string"
+}
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/matcher.py b/v1.1.0-dev1/salad/schema_salad/tests/matcher.py
new file mode 100644
index 000000000..50c0c60b1
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/matcher.py
@@ -0,0 +1,32 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import difflib
+import json
+import re
+
+
+class JsonDiffMatcher(object):
+ """Raise AssertionError with a readable JSON diff when not __eq__().
+
+ Used with assert_called_with() so it's possible for a human to see
+ the differences between expected and actual call arguments that
+ include non-trivial data structures.
+ """
+ def __init__(self, expected):
+ self.expected = expected
+
+ def __eq__(self, actual):
+ expected_json = json.dumps(self.expected, sort_keys=True, indent=2)
+ actual_json = json.dumps(actual, sort_keys=True, indent=2)
+ if expected_json != actual_json:
+ raise AssertionError("".join(difflib.context_diff(
+ expected_json.splitlines(1),
+ actual_json.splitlines(1),
+ fromfile="Expected", tofile="Actual")))
+ return True
+
+
+def StripYAMLComments(yml):
+ return re.sub(r'(?ms)^(#.*?\n)*\n*', '', yml)
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/metaschema-pre.yml b/v1.1.0-dev1/salad/schema_salad/tests/metaschema-pre.yml
new file mode 100644
index 000000000..284ea6d28
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/metaschema-pre.yml
@@ -0,0 +1,649 @@
+[
+ {
+ "name": "https://w3id.org/cwl/salad#Semantic_Annotations_for_Linked_Avro_Data",
+ "type": "documentation",
+ "doc": [
+ "# Semantic Annotations for Linked Avro Data (SALAD)\n\nAuthor:\n\n* Peter Amstutz , Curoverse\n\nContributors:\n\n* The developers of Apache Avro\n* The developers of JSON-LD\n* Neboj\u0161a Tijani\u0107 , Seven Bridges Genomics\n\n# Abstract\n\nSalad is a schema language for describing structured linked data documents\nin JSON or YAML documents. A Salad schema provides rules for\npreprocessing, structural validation, and link checking for documents\ndescribed by a Salad schema. Salad builds on JSON-LD and the Apache Avro\ndata serialization system, and extends Avro with features for rich data\nmodeling such as inheritance, template specialization, object identifiers,\nand object references. Salad was developed to provide a bridge between the\nrecord oriented data modeling supported by Apache Avro and the Semantic\nWeb.\n\n# Status of This Document\n\nThis document is the product of the [Common Workflow Language working\ngroup](https://groups.google.com/forum/#!forum/common-workflow-language). The\nlatest version of this document is available in the \"schema_salad\" repository at\n\nhttps://github.com/common-workflow-language/schema_salad\n\nThe products of the CWL working group (including this document) are made available\nunder the terms of the Apache License, version 2.0.\n\n\n\n# Introduction\n\nThe JSON data model is an extremely popular way to represent structured\ndata. It is attractive because of its relative simplicity and is a\nnatural fit with the standard types of many programming languages.\nHowever, this simplicity means that basic JSON lacks expressive features\nuseful for working with complex data structures and document formats, such\nas schemas, object references, and namespaces.\n\nJSON-LD is a W3C standard providing a way to describe how to interpret a\nJSON document as Linked Data by means of a \"context\". JSON-LD provides a\npowerful solution for representing object references and namespaces in JSON\nbased on standard web URIs, but is not itself a schema language. Without a\nschema providing a well defined structure, it is difficult to process an\narbitrary JSON-LD document as idiomatic JSON because there are many ways to\nexpress the same data that are logically equivalent but structurally\ndistinct.\n\nSeveral schema languages exist for describing and validating JSON data,\nsuch as the Apache Avro data serialization system, however none understand\nlinked data. As a result, to fully take advantage of JSON-LD to build the\nnext generation of linked data applications, one must maintain separate\nJSON schema, JSON-LD context, RDF schema, and human documentation, despite\nsignificant overlap of content and obvious need for these documents to stay\nsynchronized.\n\nSchema Salad is designed to address this gap. It provides a schema\nlanguage and processing rules for describing structured JSON content\npermitting URI resolution and strict document validation. The schema\nlanguage supports linked data through annotations that describe the linked\ndata interpretation of the content, enables generation of JSON-LD context\nand RDF schema, and production of RDF triples by applying the JSON-LD\ncontext. The schema language also provides for robust support of inline\ndocumentation.\n\n## Introduction to v1.0\n\nThis is the second version of of the Schema Salad specification. It is\ndeveloped concurrently with v1.0 of the Common Workflow Language for use in\nspecifying the Common Workflow Language, however Schema Salad is intended to be\nuseful to a broader audience. Compared to the draft-1 schema salad\nspecification, the following changes have been made:\n\n* Use of [mapSubject and mapPredicate](#Identifier_maps) to transform maps to lists of records.\n* Resolution of the [domain Specific Language for types](#Domain_Specific_Language_for_types)\n* Consolidation of the formal [schema into section 5](#Schema).\n\n## References to Other Specifications\n\n**Javascript Object Notation (JSON)**: http://json.org\n\n**JSON Linked Data (JSON-LD)**: http://json-ld.org\n\n**YAML**: http://yaml.org\n\n**Avro**: https://avro.apache.org/docs/current/spec.html\n\n**Uniform Resource Identifier (URI) Generic Syntax**: https://tools.ietf.org/html/rfc3986)\n\n**Resource Description Framework (RDF)**: http://www.w3.org/RDF/\n\n**UTF-8**: https://www.ietf.org/rfc/rfc2279.txt)\n\n## Scope\n\nThis document describes the syntax, data model, algorithms, and schema\nlanguage for working with Salad documents. It is not intended to document\na specific implementation of Salad, however it may serve as a reference for\nthe behavior of conforming implementations.\n\n## Terminology\n\nThe terminology used to describe Salad documents is defined in the Concepts\nsection of the specification. The terms defined in the following list are\nused in building those definitions and in describing the actions of an\nSalad implementation:\n\n**may**: Conforming Salad documents and Salad implementations are permitted but\nnot required to be interpreted as described.\n\n**must**: Conforming Salad documents and Salad implementations are required\nto be interpreted as described; otherwise they are in error.\n\n**error**: A violation of the rules of this specification; results are\nundefined. Conforming implementations may detect and report an error and may\nrecover from it.\n\n**fatal error**: A violation of the rules of this specification; results\nare undefined. Conforming implementations must not continue to process the\ndocument and may report an error.\n\n**at user option**: Conforming software may or must (depending on the modal verb in\nthe sentence) behave as described; if it does, it must provide users a means to\nenable or disable the behavior described.\n\n# Document model\n\n## Data concepts\n\nAn **object** is a data structure equivalent to the \"object\" type in JSON,\nconsisting of a unordered set of name/value pairs (referred to here as\n**fields**) and where the name is a string and the value is a string, number,\nboolean, array, or object.\n\nA **document** is a file containing a serialized object, or an array of\nobjects.\n\nA **document type** is a class of files that share a common structure and\nsemantics.\n\nA **document schema** is a formal description of the grammar of a document type.\n\nA **base URI** is a context-dependent URI used to resolve relative references.\n\nAn **identifier** is a URI that designates a single document or single\nobject within a document.\n\nA **vocabulary** is the set of symbolic field names and enumerated symbols defined\nby a document schema, where each term maps to absolute URI.\n\n## Syntax\n\nConforming Salad documents are serialized and loaded using YAML syntax and\nUTF-8 text encoding. Salad documents are written using the JSON-compatible\nsubset of YAML. Features of YAML such as headers and type tags that are\nnot found in the standard JSON data model must not be used in conforming\nSalad documents. It is a fatal error if the document is not valid YAML.\n\nA Salad document must consist only of either a single root object or an\narray of objects.\n\n## Document context\n\n### Implied context\n\nThe implicit context consists of the vocabulary defined by the schema and\nthe base URI. By default, the base URI must be the URI that was used to\nload the document. It may be overridden by an explicit context.\n\n### Explicit context\n\nIf a document consists of a root object, this object may contain the\nfields `$base`, `$namespaces`, `$schemas`, and `$graph`:\n\n * `$base`: Must be a string. Set the base URI for the document used to\n resolve relative references.\n\n * `$namespaces`: Must be an object with strings as values. The keys of\n the object are namespace prefixes used in the document; the values of\n the object are the prefix expansions.\n\n * `$schemas`: Must be an array of strings. This field may list URI\n references to documents in RDF-XML format which will be queried for RDF\n schema data. The subjects and predicates described by the RDF schema\n may provide additional semantic context for the document, and may be\n used for validation of prefixed extension fields found in the document.\n\nOther directives beginning with `$` must be ignored.\n\n## Document graph\n\nIf a document consists of a single root object, this object may contain the\nfield `$graph`. This field must be an array of objects. If present, this\nfield holds the primary content of the document. A document that consists\nof array of objects at the root is an implicit graph.\n\n## Document metadata\n\nIf a document consists of a single root object, metadata about the\ndocument, such as authorship, may be declared in the root object.\n\n## Document schema\n\nDocument preprocessing, link validation and schema validation require a\ndocument schema. A schema may consist of:\n\n * At least one record definition object which defines valid fields that\n make up a record type. Record field definitions include the valid types\n that may be assigned to each field and annotations to indicate fields\n that represent identifiers and links, described below in \"Semantic\n Annotations\".\n\n * Any number of enumerated type objects which define a set of finite set of symbols that are\n valid value of the type.\n\n * Any number of documentation objects which allow in-line documentation of the schema.\n\nThe schema for defining a salad schema (the metaschema) is described in\ndetail in \"Schema validation\".\n\n### Record field annotations\n\nIn a document schema, record field definitions may include the field\n`jsonldPredicate`, which may be either a string or object. Implementations\nmust use the following document preprocessing of fields by the following\nrules:\n\n * If the value of `jsonldPredicate` is `@id`, the field is an identifier\n field.\n\n * If the value of `jsonldPredicate` is an object, and contains that\n object contains the field `_type` with the value `@id`, the field is a\n link field.\n\n * If the value of `jsonldPredicate` is an object, and contains that\n object contains the field `_type` with the value `@vocab`, the field is a\n vocabulary field, which is a subtype of link field.\n\n## Document traversal\n\nTo perform document document preprocessing, link validation and schema\nvalidation, the document must be traversed starting from the fields or\narray items of the root object or array and recursively visiting each child\nitem which contains an object or arrays.\n\n# Document preprocessing\n\nAfter processing the explicit context (if any), document preprocessing\nbegins. Starting from the document root, object fields values or array\nitems which contain objects or arrays are recursively traversed\ndepth-first. For each visited object, field names, identifier fields, link\nfields, vocabulary fields, and `$import` and `$include` directives must be\nprocessed as described in this section. The order of traversal of child\nnodes within a parent node is undefined.\n",
+ "## Field name resolution\n\nThe document schema declares the vocabulary of known field names. During\npreprocessing traversal, field name in the document which are not part of\nthe schema vocabulary must be resolved to absolute URIs. Under \"strict\"\nvalidation, it is an error for a document to include fields which are not\npart of the vocabulary and not resolvable to absolute URIs. Fields names\nwhich are not part of the vocabulary are resolved using the following\nrules:\n\n* If an field name URI begins with a namespace prefix declared in the\ndocument context (`@context`) followed by a colon `:`, the prefix and\ncolon must be replaced by the namespace declared in `@context`.\n\n* If there is a vocabulary term which maps to the URI of a resolved\nfield, the field name must be replace with the vocabulary term.\n\n* If a field name URI is an absolute URI consisting of a scheme and path\nand is not part of the vocabulary, no processing occurs.\n\nField name resolution is not relative. It must not be affected by the\nbase URI.\n\n### Field name resolution example\n\nGiven the following schema:\n\n```\n",
+ "{\n \"$namespaces\": {\n \"acid\": \"http://example.com/acid#\"\n },\n \"$graph\": [{\n \"name\": \"ExampleType\",\n \"type\": \"record\",\n \"fields\": [{\n \"name\": \"base\",\n \"type\": \"string\",\n \"jsonldPredicate\": \"http://example.com/base\"\n }]\n }]\n}\n",
+ "```\n\nProcess the following example:\n\n```\n",
+ " {\n \"base\": \"one\",\n \"form\": {\n \"http://example.com/base\": \"two\",\n \"http://example.com/three\": \"three\",\n },\n \"acid:four\": \"four\"\n }\n",
+ "```\n\nThis becomes:\n\n```\n",
+ " {\n \"base\": \"one\",\n \"form\": {\n \"base\": \"two\",\n \"http://example.com/three\": \"three\",\n },\n \"http://example.com/acid#four\": \"four\"\n }\n",
+ "```\n",
+ "## Identifier resolution\n\nThe schema may designate one or more fields as identifier fields to identify\nspecific objects. Processing must resolve relative identifiers to absolute\nidentifiers using the following rules:\n\n * If an identifier URI is prefixed with `#` it is a URI relative\n fragment identifier. It is resolved relative to the base URI by setting\n or replacing the fragment portion of the base URI.\n\n * If an identifier URI does not contain a scheme and is not prefixed `#` it\n is a parent relative fragment identifier. It is resolved relative to the\n base URI by the following rule: if the base URI does not contain a\n document fragment, set the fragment portion of the base URI. If the base\n URI does contain a document fragment, append a slash `/` followed by the\n identifier field to the fragment portion of the base URI.\n\n * If an identifier URI begins with a namespace prefix declared in\n `$namespaces` followed by a colon `:`, the prefix and colon must be\n replaced by the namespace declared in `$namespaces`.\n\n * If an identifier URI is an absolute URI consisting of a scheme and path,\n no processing occurs.\n\nWhen preprocessing visits a node containing an identifier, that identifier\nmust be used as the base URI to process child nodes.\n\nIt is an error for more than one object in a document to have the same\nabsolute URI.\n\n### Identifier resolution example\n\nGiven the following schema:\n\n```\n",
+ "{\n \"$namespaces\": {\n \"acid\": \"http://example.com/acid#\"\n },\n \"$graph\": [{\n \"name\": \"ExampleType\",\n \"type\": \"record\",\n \"fields\": [{\n \"name\": \"id\",\n \"type\": \"string\",\n \"jsonldPredicate\": \"@id\"\n }]\n }]\n}\n",
+ "```\n\nProcess the following example:\n\n```\n",
+ " {\n \"id\": \"http://example.com/base\",\n \"form\": {\n \"id\": \"one\",\n \"things\": [\n {\n \"id\": \"two\"\n },\n {\n \"id\": \"#three\",\n },\n {\n \"id\": \"four#five\",\n },\n {\n \"id\": \"acid:six\",\n }\n ]\n }\n }\n",
+ "```\n\nThis becomes:\n\n```\n",
+ "{\n \"id\": \"http://example.com/base\",\n \"form\": {\n \"id\": \"http://example.com/base#one\",\n \"things\": [\n {\n \"id\": \"http://example.com/base#one/two\"\n },\n {\n \"id\": \"http://example.com/base#three\"\n },\n {\n \"id\": \"http://example.com/four#five\",\n },\n {\n \"id\": \"http://example.com/acid#six\",\n }\n ]\n }\n}\n",
+ "```\n",
+ "## Link resolution\n\nThe schema may designate one or more fields as link fields reference other\nobjects. Processing must resolve links to either absolute URIs using the\nfollowing rules:\n\n* If a reference URI is prefixed with `#` it is a relative\nfragment identifier. It is resolved relative to the base URI by setting\nor replacing the fragment portion of the base URI.\n\n* If a reference URI does not contain a scheme and is not prefixed with `#`\nit is a path relative reference. If the reference URI contains `#` in any\nposition other than the first character, the reference URI must be divided\ninto a path portion and a fragment portion split on the first instance of\n`#`. The path portion is resolved relative to the base URI by the following\nrule: if the path portion of the base URI ends in a slash `/`, append the\npath portion of the reference URI to the path portion of the base URI. If\nthe path portion of the base URI does not end in a slash, replace the final\npath segment with the path portion of the reference URI. Replace the\nfragment portion of the base URI with the fragment portion of the reference\nURI.\n\n* If a reference URI begins with a namespace prefix declared in `$namespaces`\nfollowed by a colon `:`, the prefix and colon must be replaced by the\nnamespace declared in `$namespaces`.\n\n* If a reference URI is an absolute URI consisting of a scheme and path,\nno processing occurs.\n\nLink resolution must not affect the base URI used to resolve identifiers\nand other links.\n\n### Link resolution example\n\nGiven the following schema:\n\n```\n",
+ "{\n \"$namespaces\": {\n \"acid\": \"http://example.com/acid#\"\n },\n \"$graph\": [{\n \"name\": \"ExampleType\",\n \"type\": \"record\",\n \"fields\": [{\n \"name\": \"link\",\n \"type\": \"string\",\n \"jsonldPredicate\": {\n \"_type\": \"@id\"\n }\n }]\n }]\n}\n",
+ "```\n\nProcess the following example:\n\n```\n",
+ "{\n \"$base\": \"http://example.com/base\",\n \"link\": \"http://example.com/base/zero\",\n \"form\": {\n \"link\": \"one\",\n \"things\": [\n {\n \"link\": \"two\"\n },\n {\n \"link\": \"#three\",\n },\n {\n \"link\": \"four#five\",\n },\n {\n \"link\": \"acid:six\",\n }\n ]\n }\n}\n",
+ "```\n\nThis becomes:\n\n```\n",
+ "{\n \"$base\": \"http://example.com/base\",\n \"link\": \"http://example.com/base/zero\",\n \"form\": {\n \"link\": \"http://example.com/one\",\n \"things\": [\n {\n \"link\": \"http://example.com/two\"\n },\n {\n \"link\": \"http://example.com/base#three\"\n },\n {\n \"link\": \"http://example.com/four#five\",\n },\n {\n \"link\": \"http://example.com/acid#six\",\n }\n ]\n }\n}\n",
+ "```\n",
+ "## Vocabulary resolution\n\n The schema may designate one or more vocabulary fields which use terms\n defined in the vocabulary. Processing must resolve vocabulary fields to\n either vocabulary terms or absolute URIs by first applying the link\n resolution rules defined above, then applying the following additional\n rule:\n\n * If a reference URI is a vocabulary field, and there is a vocabulary\n term which maps to the resolved URI, the reference must be replace with\n the vocabulary term.\n\n### Vocabulary resolution example\n\nGiven the following schema:\n\n```\n",
+ "{\n \"$namespaces\": {\n \"acid\": \"http://example.com/acid#\"\n },\n \"$graph\": [{\n \"name\": \"Colors\",\n \"type\": \"enum\",\n \"symbols\": [\"acid:red\"]\n },\n {\n \"name\": \"ExampleType\",\n \"type\": \"record\",\n \"fields\": [{\n \"name\": \"voc\",\n \"type\": \"string\",\n \"jsonldPredicate\": {\n \"_type\": \"@vocab\"\n }\n }]\n }]\n}\n",
+ "```\n\nProcess the following example:\n\n```\n",
+ " {\n \"form\": {\n \"things\": [\n {\n \"voc\": \"red\",\n },\n {\n \"voc\": \"http://example.com/acid#red\",\n },\n {\n \"voc\": \"http://example.com/acid#blue\",\n }\n ]\n }\n }\n",
+ "```\n\nThis becomes:\n\n```\n",
+ " {\n \"form\": {\n \"things\": [\n {\n \"voc\": \"red\",\n },\n {\n \"voc\": \"red\",\n },\n {\n \"voc\": \"http://example.com/acid#blue\",\n }\n ]\n }\n }\n",
+ "```\n",
+ "## Import\n\nDuring preprocessing traversal, an implementation must resolve `$import`\ndirectives. An `$import` directive is an object consisting of exactly one\nfield `$import` specifying resource by URI string. It is an error if there\nare additional fields in the `$import` object, such additional fields must\nbe ignored.\n\nThe URI string must be resolved to an absolute URI using the link\nresolution rules described previously. Implementations must support\nloading from `file`, `http` and `https` resources. The URI referenced by\n`$import` must be loaded and recursively preprocessed as a Salad document.\nThe external imported document does not inherit the context of the\nimporting document, and the default base URI for processing the imported\ndocument must be the URI used to retrieve the imported document. If the\n`$import` URI includes a document fragment, the fragment must be excluded\nfrom the base URI used to preprocess the imported document.\n\nOnce loaded and processed, the `$import` node is replaced in the document\nstructure by the object or array yielded from the import operation.\n\nURIs may reference document fragments which refer to specific an object in\nthe target document. This indicates that the `$import` node must be\nreplaced by only the object with the appropriate fragment identifier.\n\nIt is a fatal error if an import directive refers to an external resource\nor resource fragment which does not exist or is not accessible.\n\n### Import example\n\nimport.yml:\n```\n{\n \"hello\": \"world\"\n}\n\n```\n\nparent.yml:\n```\n{\n \"form\": {\n \"bar\": {\n \"$import\": \"import.yml\"\n }\n }\n}\n\n```\n\nThis becomes:\n\n```\n{\n \"form\": {\n \"bar\": {\n \"hello\": \"world\"\n }\n }\n}\n```\n\n## Include\n\nDuring preprocessing traversal, an implementation must resolve `$include`\ndirectives. An `$include` directive is an object consisting of exactly one\nfield `$include` specifying a URI string. It is an error if there are\nadditional fields in the `$include` object, such additional fields must be\nignored.\n\nThe URI string must be resolved to an absolute URI using the link\nresolution rules described previously. The URI referenced by `$include` must\nbe loaded as a text data. Implementations must support loading from\n`file`, `http` and `https` resources. Implementations may transcode the\ncharacter encoding of the text data to match that of the parent document,\nbut must not interpret or parse the text document in any other way.\n\nOnce loaded, the `$include` node is replaced in the document structure by a\nstring containing the text data loaded from the resource.\n\nIt is a fatal error if an import directive refers to an external resource\nwhich does not exist or is not accessible.\n\n### Include example\n\nparent.yml:\n```\n{\n \"form\": {\n \"bar\": {\n \"$include\": \"include.txt\"\n }\n }\n}\n\n```\n\ninclude.txt:\n```\nhello world\n\n```\n\nThis becomes:\n\n```\n{\n \"form\": {\n \"bar\": \"hello world\"\n }\n}\n```\n\n\n## Mixin\n\nDuring preprocessing traversal, an implementation must resolve `$mixin`\ndirectives. An `$mixin` directive is an object consisting of the field\n`$mixin` specifying resource by URI string. If there are additional fields in\nthe `$mixin` object, these fields override fields in the object which is loaded\nfrom the `$mixin` URI.\n\nThe URI string must be resolved to an absolute URI using the link resolution\nrules described previously. Implementations must support loading from `file`,\n`http` and `https` resources. The URI referenced by `$mixin` must be loaded\nand recursively preprocessed as a Salad document. The external imported\ndocument must inherit the context of the importing document, however the file\nURI for processing the imported document must be the URI used to retrieve the\nimported document. The `$mixin` URI must not include a document fragment.\n\nOnce loaded and processed, the `$mixin` node is replaced in the document\nstructure by the object or array yielded from the import operation.\n\nURIs may reference document fragments which refer to specific an object in\nthe target document. This indicates that the `$mixin` node must be\nreplaced by only the object with the appropriate fragment identifier.\n\nIt is a fatal error if an import directive refers to an external resource\nor resource fragment which does not exist or is not accessible.\n\n### Mixin example\n\nmixin.yml:\n```\n{\n \"hello\": \"world\",\n \"carrot\": \"orange\"\n}\n\n```\n\nparent.yml:\n```\n{\n \"form\": {\n \"bar\": {\n \"$mixin\": \"mixin.yml\"\n \"carrot\": \"cake\"\n }\n }\n}\n\n```\n\nThis becomes:\n\n```\n{\n \"form\": {\n \"bar\": {\n \"hello\": \"world\",\n \"carrot\": \"cake\"\n }\n }\n}\n```\n",
+ "## Identifier maps\n\nThe schema may designate certain fields as having a `mapSubject`. If the\nvalue of the field is a JSON object, it must be transformed into an array of\nJSON objects. Each key-value pair from the source JSON object is a list\nitem, each list item must be a JSON objects, and the value of the key is\nassigned to the field specified by `mapSubject`.\n\nFields which have `mapSubject` specified may also supply a `mapPredicate`.\nIf the value of a map item is not a JSON object, the item is transformed to a\nJSON object with the key assigned to the field specified by `mapSubject` and\nthe value assigned to the field specified by `mapPredicate`.\n\n### Identifier map example\n\nGiven the following schema:\n\n```\n",
+ "{\n \"$graph\": [{\n \"name\": \"MappedType\",\n \"type\": \"record\",\n \"documentRoot\": true,\n \"fields\": [{\n \"name\": \"mapped\",\n \"type\": {\n \"type\": \"array\",\n \"items\": \"ExampleRecord\"\n },\n \"jsonldPredicate\": {\n \"mapSubject\": \"key\",\n \"mapPredicate\": \"value\"\n }\n }],\n },\n {\n \"name\": \"ExampleRecord\",\n \"type\": \"record\",\n \"fields\": [{\n \"name\": \"key\",\n \"type\": \"string\"\n }, {\n \"name\": \"value\",\n \"type\": \"string\"\n }\n ]\n }]\n}\n",
+ "```\n\nProcess the following example:\n\n```\n",
+ "{\n \"mapped\": {\n \"shaggy\": {\n \"value\": \"scooby\"\n },\n \"fred\": \"daphne\"\n }\n}",
+ "```\n\nThis becomes:\n\n```\n",
+ "{\n \"mapped\": [\n {\n \"value\": \"daphne\",\n \"key\": \"fred\"\n },\n {\n \"value\": \"scooby\",\n \"key\": \"shaggy\"\n }\n ]\n}",
+ "```\n",
+ "## Domain Specific Language for types\n\nFields may be tagged `typeDSL: true`. If so, the field is expanded using the\nfollowing micro-DSL for schema salad types:\n\n* If the type ends with a question mark `?` it is expanded to a union with `null`\n* If the type ends with square brackets `[]` it is expanded to an array with items of the preceeding type symbol\n* The type may end with both `[]?` to indicate it is an optional array.\n* Identifier resolution is applied after type DSL expansion.\n\n### Type DSL example\n\nGiven the following schema:\n\n```\n",
+ "{\n \"$graph\": [\n {\"$import\": \"metaschema_base.yml\"},\n {\n \"name\": \"TypeDSLExample\",\n \"type\": \"record\",\n \"documentRoot\": true,\n \"fields\": [{\n \"name\": \"extype\",\n \"type\": \"string\",\n \"jsonldPredicate\": {\n _type: \"@vocab\",\n \"typeDSL\": true\n }\n }]\n }]\n}\n",
+ "```\n\nProcess the following example:\n\n```\n",
+ "[{\n \"extype\": \"string\"\n}, {\n \"extype\": \"string?\"\n}, {\n \"extype\": \"string[]\"\n}, {\n \"extype\": \"string[]?\"\n}]\n",
+ "```\n\nThis becomes:\n\n```\n",
+ "[\n {\n \"extype\": \"string\"\n }, \n {\n \"extype\": [\n \"null\", \n \"string\"\n ]\n }, \n {\n \"extype\": {\n \"type\": \"array\", \n \"items\": \"string\"\n }\n }, \n {\n \"extype\": [\n \"null\", \n {\n \"type\": \"array\", \n \"items\": \"string\"\n }\n ]\n }\n]\n",
+ "```\n"
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#Link_Validation",
+ "type": "documentation",
+ "doc": "# Link validation\n\nOnce a document has been preprocessed, an implementation may validate\nlinks. The link validation traversal may visit fields which the schema\ndesignates as link fields and check that each URI references an existing\nobject in the current document, an imported document, file system, or\nnetwork resource. Failure to validate links may be a fatal error. Link\nvalidation behavior for individual fields may be modified by `identity` and\n`noLinkCheck` in the `jsonldPredicate` section of the field schema.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#Schema_validation",
+ "type": "documentation",
+ "doc": ""
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#Schema",
+ "type": "documentation",
+ "doc": "# Schema\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#Documented",
+ "type": "record",
+ "abstract": true,
+ "docParent": "https://w3id.org/cwl/salad#Schema",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#Documented/doc",
+ "type": [
+ "null",
+ "string",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "doc": "A documentation string for this object, or an array of strings which should be concatenated.",
+ "jsonldPredicate": "rdfs:comment"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#PrimitiveType",
+ "type": "enum",
+ "symbols": [
+ "https://w3id.org/cwl/salad#null",
+ "http://www.w3.org/2001/XMLSchema#boolean",
+ "http://www.w3.org/2001/XMLSchema#int",
+ "http://www.w3.org/2001/XMLSchema#long",
+ "http://www.w3.org/2001/XMLSchema#float",
+ "http://www.w3.org/2001/XMLSchema#double",
+ "http://www.w3.org/2001/XMLSchema#string"
+ ],
+ "doc": [
+ "Salad data types are based on Avro schema declarations. Refer to the\n[Avro schema declaration documentation](https://avro.apache.org/docs/current/spec.html#schemas) for\ndetailed information.\n",
+ "null: no value",
+ "boolean: a binary value",
+ "int: 32-bit signed integer",
+ "long: 64-bit signed integer",
+ "float: single precision (32-bit) IEEE 754 floating-point number",
+ "double: double precision (64-bit) IEEE 754 floating-point number",
+ "string: Unicode character sequence"
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#Any",
+ "type": "enum",
+ "symbols": [
+ "https://w3id.org/cwl/salad#Any"
+ ],
+ "docAfter": "https://w3id.org/cwl/salad#PrimitiveType",
+ "doc": "The **Any** type validates for any non-null value.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#RecordField",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/salad#Documented",
+ "doc": "A field of a record.",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#RecordField/name",
+ "type": "string",
+ "jsonldPredicate": "@id",
+ "doc": "The name of the field\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#RecordField/type",
+ "type": [
+ "PrimitiveType",
+ "RecordSchema",
+ "EnumSchema",
+ "ArraySchema",
+ "string",
+ {
+ "type": "array",
+ "items": [
+ "PrimitiveType",
+ "RecordSchema",
+ "EnumSchema",
+ "ArraySchema",
+ "string"
+ ]
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#type",
+ "_type": "@vocab",
+ "typeDSL": true,
+ "refScope": 2
+ },
+ "doc": "The field type\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#RecordSchema",
+ "type": "record",
+ "fields": [
+ {
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "RecordField"
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#fields",
+ "mapSubject": "name",
+ "mapPredicate": "type"
+ },
+ "doc": "Defines the fields of the record.",
+ "name": "https://w3id.org/cwl/salad#RecordSchema/fields"
+ },
+ {
+ "doc": "Must be `record`",
+ "type": {
+ "type": "enum",
+ "symbols": [
+ "https://w3id.org/cwl/salad#record"
+ ]
+ },
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#type",
+ "_type": "@vocab",
+ "typeDSL": true,
+ "refScope": 2
+ },
+ "name": "https://w3id.org/cwl/salad#RecordSchema/type"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#EnumSchema",
+ "type": "record",
+ "doc": "Define an enumerated type.\n",
+ "fields": [
+ {
+ "type": {
+ "type": "array",
+ "items": "string"
+ },
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#symbols",
+ "_type": "@id",
+ "identity": true
+ },
+ "doc": "Defines the set of valid symbols.",
+ "name": "https://w3id.org/cwl/salad#EnumSchema/symbols"
+ },
+ {
+ "doc": "Must be `enum`",
+ "type": {
+ "type": "enum",
+ "symbols": [
+ "https://w3id.org/cwl/salad#enum"
+ ]
+ },
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#type",
+ "_type": "@vocab",
+ "typeDSL": true,
+ "refScope": 2
+ },
+ "name": "https://w3id.org/cwl/salad#EnumSchema/type"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#ArraySchema",
+ "type": "record",
+ "fields": [
+ {
+ "type": [
+ "PrimitiveType",
+ "RecordSchema",
+ "EnumSchema",
+ "ArraySchema",
+ "string",
+ {
+ "type": "array",
+ "items": [
+ "PrimitiveType",
+ "RecordSchema",
+ "EnumSchema",
+ "ArraySchema",
+ "string"
+ ]
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#items",
+ "_type": "@vocab",
+ "refScope": 2
+ },
+ "doc": "Defines the type of the array elements.",
+ "name": "https://w3id.org/cwl/salad#ArraySchema/items"
+ },
+ {
+ "doc": "Must be `array`",
+ "type": {
+ "type": "enum",
+ "symbols": [
+ "https://w3id.org/cwl/salad#array"
+ ]
+ },
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#type",
+ "_type": "@vocab",
+ "typeDSL": true,
+ "refScope": 2
+ },
+ "name": "https://w3id.org/cwl/salad#ArraySchema/type"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate",
+ "type": "record",
+ "doc": "Attached to a record field to define how the parent record field is handled for\nURI resolution and JSON-LD context generation.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate/_id",
+ "type": [
+ "null",
+ "string"
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#_id",
+ "_type": "@id",
+ "identity": true
+ },
+ "doc": "The predicate URI that this field corresponds to.\nCorresponds to JSON-LD `@id` directive.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate/_type",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "The context type hint, corresponds to JSON-LD `@type` directive.\n\n* If the value of this field is `@id` and `identity` is false or\nunspecified, the parent field must be resolved using the link\nresolution rules. If `identity` is true, the parent field must be\nresolved using the identifier expansion rules.\n\n* If the value of this field is `@vocab`, the parent field must be\n resolved using the vocabulary resolution rules.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate/_container",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Structure hint, corresponds to JSON-LD `@container` directive.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate/identity",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "doc": "If true and `_type` is `@id` this indicates that the parent field must\nbe resolved according to identity resolution rules instead of link\nresolution rules. In addition, the field value is considered an\nassertion that the linked value exists; absence of an object in the loaded document\nwith the URI is not an error.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate/noLinkCheck",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "doc": "If true, this indicates that link validation traversal must stop at\nthis field. This field (it is is a URI) or any fields under it (if it\nis an object or array) are not subject to link checking.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate/mapSubject",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "If the value of the field is a JSON object, it must be transformed\ninto an array of JSON objects, where each key-value pair from the\nsource JSON object is a list item, the list items must be JSON objects,\nand the key is assigned to the field specified by `mapSubject`.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate/mapPredicate",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Only applies if `mapSubject` is also provided. If the value of the\nfield is a JSON object, it is transformed as described in `mapSubject`,\nwith the addition that when the value of a map item is not an object,\nthe item is transformed to a JSON object with the key assigned to the\nfield specified by `mapSubject` and the value assigned to the field\nspecified by `mapPredicate`.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate/refScope",
+ "type": [
+ "null",
+ "int"
+ ],
+ "doc": "If the field contains a relative reference, it must be resolved by\nsearching for valid document references in each successive parent scope\nin the document fragment. For example, a reference of `foo` in the\ncontext `#foo/bar/baz` will first check for the existence of\n`#foo/bar/baz/foo`, followed by `#foo/bar/foo`, then `#foo/foo` and\nthen finally `#foo`. The first valid URI in the search order shall be\nused as the fully resolved value of the identifier. The value of the\nrefScope field is the specified number of levels from the containing\nidentifer scope before starting the search, so if `refScope: 2` then\n\"baz\" and \"bar\" must be stripped to get the base `#foo` and search\n`#foo/foo` and the `#foo`. The last scope searched must be the top\nlevel scope before determining if the identifier cannot be resolved.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate/typeDSL",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "doc": "Field must be expanded based on the the Schema Salad type DSL.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#JsonldPredicate/subscope",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Append the subscope to the current scope when performing\nidentifier resolution to objects under this field.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#SpecializeDef",
+ "type": "record",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#SpecializeDef/specializeFrom",
+ "type": "string",
+ "doc": "The data type to be replaced",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#specializeFrom",
+ "_type": "@id",
+ "refScope": 1
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#SpecializeDef/specializeTo",
+ "type": "string",
+ "doc": "The new data type to replace with",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#specializeTo",
+ "_type": "@id",
+ "refScope": 1
+ }
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#NamedType",
+ "type": "record",
+ "abstract": true,
+ "docParent": "https://w3id.org/cwl/salad#Schema",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#NamedType/name",
+ "type": "string",
+ "jsonldPredicate": "@id",
+ "doc": "The identifier for this type"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#NamedType/inVocab",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "doc": "By default or if \"true\", include the short name of this type in the\nvocabulary (the keys of the JSON-LD context). If false, do not include\nthe short name in the vocabulary.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#DocType",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/salad#Documented",
+ "abstract": true,
+ "docParent": "https://w3id.org/cwl/salad#Schema",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#DocType/docParent",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Hint to indicate that during documentation generation, documentation\nfor this type should appear in a subsection under `docParent`.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#docParent",
+ "_type": "@id"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#DocType/docChild",
+ "type": [
+ "null",
+ "string",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "doc": "Hint to indicate that during documentation generation, documentation\nfor `docChild` should appear in a subsection under this type.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#docChild",
+ "_type": "@id"
+ }
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#DocType/docAfter",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Hint to indicate that during documentation generation, documentation\nfor this type should appear after the `docAfter` section at the same\nlevel.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#docAfter",
+ "_type": "@id"
+ }
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#SchemaDefinedType",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/salad#DocType",
+ "doc": "Abstract base for schema-defined types.\n",
+ "abstract": true,
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#SchemaDefinedType/jsonldPredicate",
+ "type": [
+ "null",
+ "string",
+ "JsonldPredicate"
+ ],
+ "doc": "Annotate this type with linked data context.\n",
+ "jsonldPredicate": "sld:jsonldPredicate"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#SchemaDefinedType/documentRoot",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "doc": "If true, indicates that the type is a valid at the document root. At\nleast one type in a schema must be tagged with `documentRoot: true`.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#SaladRecordField",
+ "type": "record",
+ "extends": "https://w3id.org/cwl/salad#RecordField",
+ "doc": "A field of a record.",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#SaladRecordField/jsonldPredicate",
+ "type": [
+ "null",
+ "string",
+ "JsonldPredicate"
+ ],
+ "doc": "Annotate this type with linked data context.\n",
+ "jsonldPredicate": "sld:jsonldPredicate"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#SaladRecordField/default",
+ "type": [
+ "null",
+ "Any"
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#default",
+ "noLinkCheck": true
+ },
+ "doc": "The default value to use for this field if the field is missing or \"null\".\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#SaladRecordSchema",
+ "docParent": "https://w3id.org/cwl/salad#Schema",
+ "type": "record",
+ "extends": [
+ "https://w3id.org/cwl/salad#NamedType",
+ "https://w3id.org/cwl/salad#RecordSchema",
+ "https://w3id.org/cwl/salad#SchemaDefinedType"
+ ],
+ "documentRoot": true,
+ "specialize": [
+ {
+ "specializeTo": "https://w3id.org/cwl/salad#SaladRecordField",
+ "specializeFrom": "https://w3id.org/cwl/salad#RecordField"
+ }
+ ],
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#SaladRecordSchema/abstract",
+ "type": [
+ "null",
+ "boolean"
+ ],
+ "doc": "If true, this record is abstract and may be used as a base for other\nrecords, but is not valid on its own.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#SaladRecordSchema/extends",
+ "type": [
+ "null",
+ "string",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#extends",
+ "_type": "@id",
+ "refScope": 1
+ },
+ "doc": "Indicates that this record inherits fields from one or more base records.\n"
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#SaladRecordSchema/specialize",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": "SpecializeDef"
+ }
+ ],
+ "doc": "Only applies if `extends` is declared. Apply type specialization using the\nbase record as a template. For each field inherited from the base\nrecord, replace any instance of the type `specializeFrom` with\n`specializeTo`.\n",
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#specialize",
+ "mapSubject": "specializeFrom",
+ "mapPredicate": "specializeTo"
+ }
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#SaladEnumSchema",
+ "docParent": "https://w3id.org/cwl/salad#Schema",
+ "type": "record",
+ "extends": [
+ "https://w3id.org/cwl/salad#NamedType",
+ "https://w3id.org/cwl/salad#EnumSchema",
+ "https://w3id.org/cwl/salad#SchemaDefinedType"
+ ],
+ "documentRoot": true,
+ "doc": "Define an enumerated type.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#SaladEnumSchema/extends",
+ "type": [
+ "null",
+ "string",
+ {
+ "type": "array",
+ "items": "string"
+ }
+ ],
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#extends",
+ "_type": "@id",
+ "refScope": 1
+ },
+ "doc": "Indicates that this enum inherits symbols from a base enum.\n"
+ }
+ ]
+ },
+ {
+ "name": "https://w3id.org/cwl/salad#Documentation",
+ "type": "record",
+ "docParent": "https://w3id.org/cwl/salad#Schema",
+ "extends": [
+ "https://w3id.org/cwl/salad#NamedType",
+ "https://w3id.org/cwl/salad#DocType"
+ ],
+ "documentRoot": true,
+ "doc": "A documentation section. This type exists to facilitate self-documenting\nschemas but has no role in formal validation.\n",
+ "fields": [
+ {
+ "name": "https://w3id.org/cwl/salad#Documentation/type",
+ "doc": "Must be `documentation`",
+ "type": {
+ "type": "enum",
+ "symbols": [
+ "https://w3id.org/cwl/salad#documentation"
+ ]
+ },
+ "jsonldPredicate": {
+ "_id": "https://w3id.org/cwl/salad#type",
+ "_type": "@vocab",
+ "typeDSL": true,
+ "refScope": 2
+ }
+ }
+ ]
+ }
+]
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/pt.yml b/v1.1.0-dev1/salad/schema_salad/tests/pt.yml
new file mode 100644
index 000000000..5f2377fd2
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/pt.yml
@@ -0,0 +1,28 @@
+$namespaces:
+ sld: "https://w3id.org/cwl/salad#"
+ dct: "http://purl.org/dc/terms/"
+ rdf: "http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+ rdfs: "http://www.w3.org/2000/01/rdf-schema#"
+ xsd: "http://www.w3.org/2001/XMLSchema#"
+name: PrimitiveType
+type: enum
+symbols:
+ - "sld:null"
+ - "xsd:boolean"
+ - "xsd:int"
+ - "xsd:long"
+ - "xsd:float"
+ - "xsd:double"
+ - "xsd:string"
+doc:
+ - |
+ Salad data types are based on Avro schema declarations. Refer to the
+ [Avro schema declaration documentation](https://avro.apache.org/docs/current/spec.html#schemas) for
+ detailed information.
+ - "null: no value"
+ - "boolean: a binary value"
+ - "int: 32-bit signed integer"
+ - "long: 64-bit signed integer"
+ - "float: single precision (32-bit) IEEE 754 floating-point number"
+ - "double: double precision (64-bit) IEEE 754 floating-point number"
+ - "string: Unicode character sequence"
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_cg.py b/v1.1.0-dev1/salad/schema_salad/tests/test_cg.py
new file mode 100644
index 000000000..f353c169a
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_cg.py
@@ -0,0 +1,177 @@
+import schema_salad.metaschema as cg_metaschema
+import unittest
+import logging
+import os
+import json
+from schema_salad.ref_resolver import file_uri
+
+from .matcher import JsonDiffMatcher
+from .util import get_data
+
+
+class TestGeneratedMetaschema(unittest.TestCase):
+ def test_load(self):
+ doc = {
+ "type": "record",
+ "fields": [{
+ "name": "hello",
+ "doc": "Hello test case",
+ "type": "string"
+ }]
+ }
+ rs = cg_metaschema.RecordSchema(doc, "http://example.com/", cg_metaschema.LoadingOptions())
+ self.assertEqual("record", rs.type)
+ self.assertEqual("http://example.com/#hello", rs.fields[0].name)
+ self.assertEqual("Hello test case", rs.fields[0].doc)
+ self.assertEqual("string", rs.fields[0].type)
+ self.assertEqual({
+ "type": "record",
+ "fields": [{
+ "name": "http://example.com/#hello",
+ "doc": "Hello test case",
+ "type": "string"
+ }]
+ }, rs.save())
+
+ def test_err(self):
+ doc = {
+ "doc": "Hello test case",
+ "type": "string"
+ }
+ with self.assertRaises(cg_metaschema.ValidationException):
+ rf = cg_metaschema.RecordField(doc, "", cg_metaschema.LoadingOptions())
+
+ def test_include(self):
+ doc = {
+ "name": "hello",
+ "doc": [{"$include": "hello.txt"}],
+ "type": "documentation"
+ }
+ rf = cg_metaschema.Documentation(doc, "http://example.com/",
+ cg_metaschema.LoadingOptions(fileuri=file_uri(get_data("tests/_"))))
+ self.assertEqual("http://example.com/#hello", rf.name)
+ self.assertEqual(["hello world!\n"], rf.doc)
+ self.assertEqual("documentation", rf.type)
+ self.assertEqual({
+ "name": "http://example.com/#hello",
+ "doc": ["hello world!\n"],
+ "type": "documentation"
+ }, rf.save())
+
+ def test_import(self):
+ doc = {
+ "type": "record",
+ "fields": [{
+ "$import": "hellofield.yml"
+ }]
+ }
+ lead = file_uri(os.path.normpath(get_data("tests")))
+ rs = cg_metaschema.RecordSchema(doc, "http://example.com/", cg_metaschema.LoadingOptions(fileuri=lead+"/_"))
+ self.assertEqual("record", rs.type)
+ self.assertEqual(lead+"/hellofield.yml#hello", rs.fields[0].name)
+ self.assertEqual("hello world!\n", rs.fields[0].doc)
+ self.assertEqual("string", rs.fields[0].type)
+ self.assertEqual({
+ "type": "record",
+ "fields": [{
+ "name": lead+"/hellofield.yml#hello",
+ "doc": "hello world!\n",
+ "type": "string"
+ }]
+ }, rs.save())
+
+
+ maxDiff = None
+
+ def test_import2(self):
+ rs = cg_metaschema.load_document(file_uri(get_data("tests/docimp/d1.yml")), "", cg_metaschema.LoadingOptions())
+ self.assertEqual([{'doc': [u'*Hello*', 'hello 2', u'*dee dee dee five*',
+ 'hello 3', 'hello 4', u'*dee dee dee five*',
+ 'hello 5'],
+ 'type': 'documentation',
+ 'name': file_uri(get_data("tests/docimp/d1.yml"))+"#Semantic_Annotations_for_Linked_Avro_Data"}],
+ [r.save() for r in rs])
+
+ def test_err2(self):
+ doc = {
+ "type": "rucord",
+ "fields": [{
+ "name": "hello",
+ "doc": "Hello test case",
+ "type": "string"
+ }]
+ }
+ with self.assertRaises(cg_metaschema.ValidationException):
+ rs = cg_metaschema.RecordSchema(doc, "", cg_metaschema.LoadingOptions())
+
+ def test_idmap(self):
+ doc = {
+ "type": "record",
+ "fields": {
+ "hello": {
+ "doc": "Hello test case",
+ "type": "string"
+ }
+ }
+ }
+ rs = cg_metaschema.RecordSchema(doc, "http://example.com/", cg_metaschema.LoadingOptions())
+ self.assertEqual("record", rs.type)
+ self.assertEqual("http://example.com/#hello", rs.fields[0].name)
+ self.assertEqual("Hello test case", rs.fields[0].doc)
+ self.assertEqual("string", rs.fields[0].type)
+ self.assertEqual({
+ "type": "record",
+ "fields": [{
+ "name": "http://example.com/#hello",
+ "doc": "Hello test case",
+ "type": "string"
+ }]
+ }, rs.save())
+
+ def test_idmap2(self):
+ doc = {
+ "type": "record",
+ "fields": {
+ "hello": "string"
+ }
+ }
+ rs = cg_metaschema.RecordSchema(doc, "http://example.com/", cg_metaschema.LoadingOptions())
+ self.assertEqual("record", rs.type)
+ self.assertEqual("http://example.com/#hello", rs.fields[0].name)
+ self.assertEqual(None, rs.fields[0].doc)
+ self.assertEqual("string", rs.fields[0].type)
+ self.assertEqual({
+ "type": "record",
+ "fields": [{
+ "name": "http://example.com/#hello",
+ "type": "string"
+ }]
+ }, rs.save())
+
+ def test_load_pt(self):
+ doc = cg_metaschema.load_document(file_uri(get_data("tests/pt.yml")), "", cg_metaschema.LoadingOptions())
+ self.assertEqual(['https://w3id.org/cwl/salad#null',
+ 'http://www.w3.org/2001/XMLSchema#boolean',
+ 'http://www.w3.org/2001/XMLSchema#int',
+ 'http://www.w3.org/2001/XMLSchema#long',
+ 'http://www.w3.org/2001/XMLSchema#float',
+ 'http://www.w3.org/2001/XMLSchema#double',
+ 'http://www.w3.org/2001/XMLSchema#string'], doc.symbols)
+
+ def test_load_metaschema(self):
+ doc = cg_metaschema.load_document(file_uri(get_data("metaschema/metaschema.yml")), "", cg_metaschema.LoadingOptions())
+ with open(get_data("tests/metaschema-pre.yml")) as f:
+ pre = json.load(f)
+ saved = [d.save() for d in doc]
+ self.assertEqual(saved, JsonDiffMatcher(pre))
+
+ def test_load_cwlschema(self):
+ doc = cg_metaschema.load_document(file_uri(get_data("tests/test_schema/CommonWorkflowLanguage.yml")), "", cg_metaschema.LoadingOptions())
+ with open(get_data("tests/cwl-pre.yml")) as f:
+ pre = json.load(f)
+ saved = [d.save() for d in doc]
+ self.assertEqual(saved, JsonDiffMatcher(pre))
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_cli_args.py b/v1.1.0-dev1/salad/schema_salad/tests/test_cli_args.py
new file mode 100644
index 000000000..16b7e5cfa
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_cli_args.py
@@ -0,0 +1,41 @@
+from __future__ import absolute_import
+import unittest
+import sys
+
+import schema_salad.main as cli_parser
+
+# for capturing print() output
+from contextlib import contextmanager
+from six import StringIO
+
+@contextmanager
+def captured_output():
+ new_out, new_err = StringIO(), StringIO()
+ old_out, old_err = sys.stdout, sys.stderr
+ try:
+ sys.stdout, sys.stderr = new_out, new_err
+ yield sys.stdout, sys.stderr
+ finally:
+ sys.stdout, sys.stderr = old_out, old_err
+
+
+""" test different sets of command line arguments"""
+class ParseCliArgs(unittest.TestCase):
+
+ def test_version(self):
+ args = [["--version"], ["-v"]]
+ for arg in args:
+ with captured_output() as (out, err):
+ cli_parser.main(arg)
+
+ response = out.getvalue().strip() # capture output and strip newline
+ self.assertTrue("Current version" in response)
+
+ def test_empty_input(self):
+ # running schema_salad tool wihtout any args
+ args = []
+ with captured_output() as (out, err):
+ cli_parser.main(args)
+
+ response = out.getvalue().strip()
+ self.assertTrue("error: too few arguments" in response)
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_errors.py b/v1.1.0-dev1/salad/schema_salad/tests/test_errors.py
new file mode 100644
index 000000000..4578ba149
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_errors.py
@@ -0,0 +1,36 @@
+from __future__ import absolute_import
+from __future__ import print_function
+from .util import get_data
+import unittest
+from schema_salad.schema import load_schema, load_and_validate
+from schema_salad.validate import ValidationException
+from avro.schema import Names
+import six
+
+class TestErrors(unittest.TestCase):
+ def test_errors(self):
+ document_loader, avsc_names, schema_metadata, metaschema_loader = load_schema(
+ get_data(u"tests/test_schema/CommonWorkflowLanguage.yml"))
+
+ for t in ("test_schema/test1.cwl",
+ "test_schema/test2.cwl",
+ "test_schema/test3.cwl",
+ "test_schema/test4.cwl",
+ "test_schema/test5.cwl",
+ "test_schema/test6.cwl",
+ "test_schema/test7.cwl",
+ "test_schema/test8.cwl",
+ "test_schema/test9.cwl",
+ "test_schema/test10.cwl",
+ "test_schema/test11.cwl",
+ "test_schema/test12.cwl",
+ "test_schema/test13.cwl",
+ "test_schema/test14.cwl",
+ "test_schema/test15.cwl"):
+ with self.assertRaises(ValidationException):
+ try:
+ load_and_validate(document_loader, avsc_names,
+ six.text_type(get_data("tests/"+t)), True)
+ except ValidationException as e:
+ print("\n", e)
+ raise
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_examples.py b/v1.1.0-dev1/salad/schema_salad/tests/test_examples.py
index c63b81efc..facd6c625 100644
--- a/v1.1.0-dev1/salad/schema_salad/tests/test_examples.py
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_examples.py
@@ -1,44 +1,37 @@
+from __future__ import absolute_import
+from __future__ import print_function
+from .util import get_data
import unittest
import schema_salad.ref_resolver
import schema_salad.main
import schema_salad.schema
from schema_salad.jsonld_context import makerdf
-from pkg_resources import Requirement, resource_filename, ResolutionError # type: ignore
import rdflib
-import ruamel.yaml as yaml
+import ruamel.yaml
import json
import os
+from schema_salad.sourceline import cmap, SourceLine
try:
from ruamel.yaml import CSafeLoader as SafeLoader
except ImportError:
from ruamel.yaml import SafeLoader # type: ignore
-
-def get_data(filename):
- filepath = None
- try:
- filepath = resource_filename(
- Requirement.parse("schema-salad"), filename)
- except ResolutionError:
- pass
- if not filepath or not os.path.isfile(filepath):
- filepath = os.path.join(os.path.dirname(__file__), os.pardir, filename)
- return filepath
+from ruamel.yaml.comments import CommentedSeq, CommentedMap
class TestSchemas(unittest.TestCase):
def test_schemas(self):
- l = schema_salad.ref_resolver.Loader({})
+ loader = schema_salad.ref_resolver.Loader({})
- ra, _ = l.resolve_all({
- u"$schemas": ["file://" + get_data("tests/EDAM.owl")],
+ ra, _ = loader.resolve_all(cmap({
+ u"$schemas": [schema_salad.ref_resolver.file_uri(get_data("tests/EDAM.owl"))],
u"$namespaces": {u"edam": u"http://edamontology.org/"},
u"edam:has_format": u"edam:format_1915"
- }, "")
+ }), "")
self.assertEqual({
- u"$schemas": ["file://" + get_data("tests/EDAM.owl")],
+ u"$schemas": [schema_salad.ref_resolver.file_uri(get_data("tests/EDAM.owl"))],
u"$namespaces": {u"edam": u"http://edamontology.org/"},
u'http://edamontology.org/has_format': u'http://edamontology.org/format_1915'
}, ra)
@@ -56,7 +49,7 @@ def test_schemas(self):
# "edam:has_format": "edam:format_1915"
# }, "")
- # self.assertEquals(ra, {
+ # self.assertEqual(ra, {
# "$schemas": ["tests/EDAM.owl"],
# "$namespaces": {"edam": "http://edamontology.org/"},
# 'http://edamontology.org/has_format': 'http://edamontology.org/format_1915'
@@ -74,7 +67,7 @@ def test_avro_regression(self):
argsl=[get_data("tests/Process.yml")]))
def test_jsonld_ctx(self):
- ldr, _, _, _ = schema_salad.schema.load_schema({
+ ldr, _, _, _ = schema_salad.schema.load_schema(cmap({
"$base": "Y",
"name": "X",
"$namespaces": {
@@ -84,9 +77,9 @@ def test_jsonld_ctx(self):
"name": "ExampleType",
"type": "enum",
"symbols": ["asym", "bsym"]}]
- })
+ }))
- ra, _ = ldr.resolve_all({"foo:bar": "asym"}, "X")
+ ra, _ = ldr.resolve_all(cmap({"foo:bar": "asym"}), "X")
self.assertEqual(ra, {
'http://example.com/foo#bar': 'asym'
@@ -106,7 +99,7 @@ def test_idmap(self):
},
"id": "@id"})
- ra, _ = ldr.resolve_all({
+ ra, _ = ldr.resolve_all(cmap({
"id": "stuff",
"inputs": {
"zip": 1,
@@ -116,17 +109,17 @@ def test_idmap(self):
"other": {
'n': 9
}
- }, "http://example2.com/")
+ }), "http://example2.com/")
self.assertEqual("http://example2.com/#stuff", ra["id"])
for item in ra["inputs"]:
if item["a"] == 2:
- self.assertEquals(
+ self.assertEqual(
'http://example2.com/#stuff/zing', item["id"])
else:
- self.assertEquals('http://example2.com/#stuff/zip', item["id"])
- self.assertEquals(['http://example2.com/#stuff/out'], ra['outputs'])
- self.assertEquals({'n': 9}, ra['other'])
+ self.assertEqual('http://example2.com/#stuff/zip', item["id"])
+ self.assertEqual(['http://example2.com/#stuff/out'], ra['outputs'])
+ self.assertEqual({'n': 9}, ra['other'])
def test_scoped_ref(self):
ldr = schema_salad.ref_resolver.Loader({})
@@ -159,7 +152,7 @@ def test_scoped_ref(self):
},
"id": "@id"})
- ra, _ = ldr.resolve_all({
+ ra, _ = ldr.resolve_all(cmap({
"inputs": {
"inp": "string",
"inp2": "string"
@@ -188,9 +181,9 @@ def test_scoped_ref(self):
"out": ["out"]
}
}
- }, "http://example2.com/")
+ }), "http://example2.com/")
- self.assertEquals(
+ self.assertEqual(
{'inputs': [{
'id': 'http://example2.com/#inp',
'type': 'string'
@@ -234,13 +227,15 @@ def test_examples(self):
get_data("metaschema/%s_schema.yml" % a))
with open(get_data("metaschema/%s_src.yml" % a)) as src_fp:
src = ldr.resolve_all(
- yaml.load(src_fp, Loader=SafeLoader), "", checklinks=False)[0]
+ ruamel.yaml.round_trip_load(src_fp), "",
+ checklinks=False)[0]
with open(get_data("metaschema/%s_proc.yml" % a)) as src_proc:
- proc = yaml.load(src_proc, Loader=SafeLoader)
+ proc = ruamel.yaml.safe_load(src_proc)
self.assertEqual(proc, src)
def test_yaml_float_test(self):
- self.assertEqual(yaml.load("float-test: 2e-10")["float-test"], 2e-10)
+ self.assertEqual(ruamel.yaml.safe_load("float-test: 2e-10")["float-test"],
+ 2e-10)
def test_typedsl_ref(self):
ldr = schema_salad.ref_resolver.Loader({})
@@ -254,16 +249,16 @@ def test_typedsl_ref(self):
}
})
- ra, _ = ldr.resolve_all({"type": "File"}, "")
+ ra, _ = ldr.resolve_all(cmap({"type": "File"}), "")
self.assertEqual({'type': 'File'}, ra)
- ra, _ = ldr.resolve_all({"type": "File?"}, "")
+ ra, _ = ldr.resolve_all(cmap({"type": "File?"}), "")
self.assertEqual({'type': ['null', 'File']}, ra)
- ra, _ = ldr.resolve_all({"type": "File[]"}, "")
+ ra, _ = ldr.resolve_all(cmap({"type": "File[]"}), "")
self.assertEqual({'type': {'items': 'File', 'type': 'array'}}, ra)
- ra, _ = ldr.resolve_all({"type": "File[]?"}, "")
+ ra, _ = ldr.resolve_all(cmap({"type": "File[]?"}), "")
self.assertEqual(
{'type': ['null', {'items': 'File', 'type': 'array'}]}, ra)
@@ -280,12 +275,12 @@ def test_scoped_id(self):
}
ldr.add_context(ctx)
- ra, _ = ldr.resolve_all({
+ ra, _ = ldr.resolve_all(cmap({
"id": "foo",
"bar": {
"id": "baz"
}
- }, "http://example.com")
+ }), "http://example.com")
self.assertEqual({'id': 'http://example.com/#foo',
'bar': {
'id': 'http://example.com/#foo/baz'},
@@ -294,12 +289,12 @@ def test_scoped_id(self):
g = makerdf(None, ra, ctx)
print(g.serialize(format="n3"))
- ra, _ = ldr.resolve_all({
+ ra, _ = ldr.resolve_all(cmap({
"location": "foo",
"bar": {
"location": "baz"
}
- }, "http://example.com", checklinks=False)
+ }), "http://example.com", checklinks=False)
self.assertEqual({'location': 'http://example.com/foo',
'bar': {
'location': 'http://example.com/baz'},
@@ -308,12 +303,12 @@ def test_scoped_id(self):
g = makerdf(None, ra, ctx)
print(g.serialize(format="n3"))
- ra, _ = ldr.resolve_all({
+ ra, _ = ldr.resolve_all(cmap({
"id": "foo",
"bar": {
"location": "baz"
}
- }, "http://example.com", checklinks=False)
+ }), "http://example.com", checklinks=False)
self.assertEqual({'id': 'http://example.com/#foo',
'bar': {
'location': 'http://example.com/baz'},
@@ -322,12 +317,12 @@ def test_scoped_id(self):
g = makerdf(None, ra, ctx)
print(g.serialize(format="n3"))
- ra, _ = ldr.resolve_all({
+ ra, _ = ldr.resolve_all(cmap({
"location": "foo",
"bar": {
"id": "baz"
}
- }, "http://example.com", checklinks=False)
+ }), "http://example.com", checklinks=False)
self.assertEqual({'location': 'http://example.com/foo',
'bar': {
'id': 'http://example.com/#baz'},
@@ -336,21 +331,43 @@ def test_scoped_id(self):
g = makerdf(None, ra, ctx)
print(g.serialize(format="n3"))
+ def test_subscoped_id(self):
+ ldr = schema_salad.ref_resolver.Loader({})
+ ctx = {
+ "id": "@id",
+ "bar": {
+ "subscope": "bar",
+ }
+ }
+ ldr.add_context(ctx)
+
+ ra, _ = ldr.resolve_all(cmap({
+ "id": "foo",
+ "bar": {
+ "id": "baz"
+ }
+ }), "http://example.com")
+ self.assertEqual({'id': 'http://example.com/#foo',
+ 'bar': {
+ 'id': 'http://example.com/#foo/bar/baz'},
+ }, ra)
+
+
def test_mixin(self):
+ base_url = schema_salad.ref_resolver.file_uri(os.path.join(os.getcwd(), "tests"))
ldr = schema_salad.ref_resolver.Loader({})
- ra = ldr.resolve_ref({"$mixin": get_data("tests/mixin.yml"), "one": "five"},
- base_url="file://" + os.getcwd() + "/tests/")
+ ra = ldr.resolve_ref(cmap({"$mixin": get_data("tests/mixin.yml"), "one": "five"}),
+ base_url=base_url)
self.assertEqual({'id': 'four', 'one': 'five'}, ra[0])
-
ldr = schema_salad.ref_resolver.Loader({"id": "@id"})
- base_url = "file://" + os.getcwd() + "/tests/"
- ra = ldr.resolve_all([{
+
+ ra = ldr.resolve_all(cmap([{
"id": "a",
"m": {"$mixin": get_data("tests/mixin.yml")}
}, {
"id": "b",
"m": {"$mixin": get_data("tests/mixin.yml")}
- }], base_url=base_url)
+ }]), base_url=base_url)
self.assertEqual([{
'id': base_url + '#a',
'm': {
@@ -364,5 +381,47 @@ def test_mixin(self):
'one': 'two'}
}], ra[0])
+ def test_fragment(self):
+ ldr = schema_salad.ref_resolver.Loader({"id": "@id"})
+ b, _ = ldr.resolve_ref(get_data("tests/frag.yml#foo2"))
+ self.assertEqual({"id": b["id"], "bar":"b2"}, b)
+
+ def test_file_uri(self):
+ # Note: this test probably won't pass on Windows. Someone with a
+ # windows box should add an alternate test.
+ self.assertEqual("file:///foo/bar%20baz/quux", schema_salad.ref_resolver.file_uri("/foo/bar baz/quux"))
+ self.assertEqual(os.path.normpath("/foo/bar baz/quux"),
+ schema_salad.ref_resolver.uri_file_path("file:///foo/bar%20baz/quux"))
+ self.assertEqual("file:///foo/bar%20baz/quux%23zing%20zong", schema_salad.ref_resolver.file_uri("/foo/bar baz/quux#zing zong"))
+ self.assertEqual("file:///foo/bar%20baz/quux#zing%20zong", schema_salad.ref_resolver.file_uri("/foo/bar baz/quux#zing zong", split_frag=True))
+ self.assertEqual(os.path.normpath("/foo/bar baz/quux#zing zong"),
+ schema_salad.ref_resolver.uri_file_path("file:///foo/bar%20baz/quux#zing%20zong"))
+
+
+class SourceLineTest(unittest.TestCase):
+ def test_sourceline(self):
+ ldr = schema_salad.ref_resolver.Loader({"id": "@id"})
+ b, _ = ldr.resolve_ref(get_data("tests/frag.yml"))
+
+ class TestExp(Exception):
+ pass
+
+ try:
+ with SourceLine(b, 1, TestExp, False):
+ raise Exception("Whoops")
+ except TestExp as e:
+ self.assertTrue(str(e).endswith("frag.yml:3:3: Whoops"))
+ except Exception:
+ self.assertFail()
+
+ try:
+ with SourceLine(b, 1, TestExp, True):
+ raise Exception("Whoops")
+ except TestExp as e:
+ self.assertTrue(str(e).splitlines()[0].endswith("frag.yml:3:3: Traceback (most recent call last):"))
+ except Exception:
+ self.assertFail()
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_fetch.py b/v1.1.0-dev1/salad/schema_salad/tests/test_fetch.py
new file mode 100644
index 000000000..7396e37c7
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_fetch.py
@@ -0,0 +1,61 @@
+from __future__ import absolute_import
+from __future__ import print_function
+import unittest
+import schema_salad.ref_resolver
+import schema_salad.main
+import schema_salad.schema
+from schema_salad.jsonld_context import makerdf
+import rdflib
+import ruamel.yaml as yaml
+import json
+import os
+from typing import Text
+from six.moves import urllib
+
+class TestFetcher(unittest.TestCase):
+ def test_fetcher(self):
+ class TestFetcher(schema_salad.ref_resolver.Fetcher):
+ def __init__(self, a, b):
+ pass
+
+ def fetch_text(self, url): # type: (Text) -> Text
+ if url == "keep:abc+123/foo.txt":
+ return "hello: keepfoo"
+ if url.endswith("foo.txt"):
+ return "hello: foo"
+ else:
+ raise RuntimeError("Not foo.txt")
+
+ def check_exists(self, url): # type: (Text) -> bool
+ if url.endswith("foo.txt"):
+ return True
+ else:
+ return False
+
+ def urljoin(self, base, url):
+ urlsp = urllib.parse.urlsplit(url)
+ if urlsp.scheme:
+ return url
+ basesp = urllib.parse.urlsplit(base)
+
+ if basesp.scheme == "keep":
+ return base + "/" + url
+ return urllib.parse.urljoin(base, url)
+
+ loader = schema_salad.ref_resolver.Loader({}, fetcher_constructor=TestFetcher)
+ self.assertEqual({"hello": "foo"}, loader.resolve_ref("foo.txt")[0])
+ self.assertEqual({"hello": "keepfoo"}, loader.resolve_ref("foo.txt", base_url="keep:abc+123")[0])
+ self.assertTrue(loader.check_exists("foo.txt"))
+
+ with self.assertRaises(RuntimeError):
+ loader.resolve_ref("bar.txt")
+ self.assertFalse(loader.check_exists("bar.txt"))
+
+ def test_cache(self):
+ loader = schema_salad.ref_resolver.Loader({})
+ foo = os.path.join(os.getcwd(), "foo.txt")
+ foo = schema_salad.ref_resolver.file_uri(foo)
+ loader.cache.update({foo: "hello: foo"})
+ print(loader.cache)
+ self.assertEqual({"hello": "foo"}, loader.resolve_ref("foo.txt")[0])
+ self.assertTrue(loader.check_exists(foo))
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_print_oneline.py b/v1.1.0-dev1/salad/schema_salad/tests/test_print_oneline.py
new file mode 100644
index 000000000..9e442ea0d
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_print_oneline.py
@@ -0,0 +1,120 @@
+from .util import get_data
+import unittest
+from schema_salad.main import to_one_line_messages, reformat_yaml_exception_message
+from schema_salad.schema import load_schema, load_and_validate
+from schema_salad.sourceline import strip_dup_lineno
+from schema_salad.validate import ValidationException
+from os.path import normpath
+import re
+import six
+
+class TestPrintOneline(unittest.TestCase):
+ def test_print_oneline(self):
+ # Issue #135
+ document_loader, avsc_names, schema_metadata, metaschema_loader = load_schema(
+ get_data(u"tests/test_schema/CommonWorkflowLanguage.yml"))
+
+ src = "test15.cwl"
+ with self.assertRaises(ValidationException):
+ try:
+ load_and_validate(document_loader, avsc_names,
+ six.text_type(get_data("tests/test_schema/"+src)), True)
+ except ValidationException as e:
+ msgs = to_one_line_messages(str(e)).splitlines()
+ self.assertEqual(len(msgs), 2)
+ m = re.match(r'^(.+:\d+:\d+:)(.+)$', msgs[0])
+ self.assertTrue(msgs[0].endswith(src+":11:7: invalid field `invalid_field`, expected one of: 'loadContents', 'position', 'prefix', 'separate', 'itemSeparator', 'valueFrom', 'shellQuote'"))
+ self.assertTrue(msgs[1].endswith(src+":12:7: invalid field `another_invalid_field`, expected one of: 'loadContents', 'position', 'prefix', 'separate', 'itemSeparator', 'valueFrom', 'shellQuote'"))
+ print("\n", e)
+ raise
+
+ def test_print_oneline_for_invalid_yaml(self):
+ # Issue #137
+ document_loader, avsc_names, schema_metadata, metaschema_loader = load_schema(
+ get_data(u"tests/test_schema/CommonWorkflowLanguage.yml"))
+
+ src = "test16.cwl"
+ with self.assertRaises(RuntimeError):
+ try:
+ load_and_validate(document_loader, avsc_names,
+ six.text_type(get_data("tests/test_schema/"+src)), True)
+ except RuntimeError as e:
+ msg = reformat_yaml_exception_message(strip_dup_lineno(six.text_type(e)))
+ msg = to_one_line_messages(msg)
+ self.assertTrue(msg.endswith(src+":10:1: could not find expected \':\'"))
+ print("\n", e)
+ raise
+
+ def test_print_oneline_for_errors_in_the_same_line(self):
+ # Issue #136
+ document_loader, avsc_names, schema_metadata, metaschema_loader = load_schema(
+ get_data(u"tests/test_schema/CommonWorkflowLanguage.yml"))
+
+ src = "test17.cwl"
+ with self.assertRaises(ValidationException):
+ try:
+ load_and_validate(document_loader, avsc_names,
+ six.text_type(get_data("tests/test_schema/"+src)), True)
+ except ValidationException as e:
+ msgs = to_one_line_messages(str(e)).splitlines()
+ self.assertEqual(len(msgs), 2)
+ self.assertTrue(msgs[0].endswith(src+":13:5: missing required field `id`"))
+ self.assertTrue(msgs[1].endswith(src+":13:5: invalid field `aa`, expected one of: 'label', 'secondaryFiles', 'format', 'streamable', 'doc', 'id', 'outputBinding', 'type'"))
+ print("\n", e)
+ raise
+
+ def test_print_oneline_for_errors_in_resolve_ref(self):
+ # Issue #141
+ document_loader, avsc_names, schema_metadata, metaschema_loader = load_schema(
+ get_data(u"tests/test_schema/CommonWorkflowLanguage.yml"))
+
+ src = "test18.cwl"
+ fullpath = normpath(get_data("tests/test_schema/"+src))
+ with self.assertRaises(ValidationException):
+ try:
+ load_and_validate(document_loader, avsc_names,
+ six.text_type(fullpath), True)
+ except ValidationException as e:
+ msgs = to_one_line_messages(str(strip_dup_lineno(six.text_type(e)))).splitlines()
+ # convert Windows path to Posix path
+ if '\\' in fullpath:
+ fullpath = '/'+fullpath.replace('\\', '/')
+ self.assertEqual(len(msgs), 1)
+ self.assertTrue(msgs[0].endswith(src+':13:5: Field `type` references unknown identifier `Filea`, tried file://%s#Filea' % (fullpath)))
+ print("\n", e)
+ raise
+
+ def test_for_invalid_yaml1(self):
+ # Issue 143
+ document_loader, avsc_names, schema_metadata, metaschema_loader = load_schema(
+ get_data(u"tests/test_schema/CommonWorkflowLanguage.yml"))
+
+ src = "test16.cwl"
+ with self.assertRaises(RuntimeError):
+ try:
+ load_and_validate(document_loader, avsc_names,
+ six.text_type(get_data("tests/test_schema/"+src)), True)
+ except RuntimeError as e:
+ msg = reformat_yaml_exception_message(strip_dup_lineno(six.text_type(e)))
+ msgs = msg.splitlines()
+ self.assertEqual(len(msgs), 2)
+ self.assertTrue(msgs[0].endswith(src+":9:7: while scanning a simple key"))
+ self.assertTrue(msgs[1].endswith(src+":10:1: could not find expected ':'"))
+ print("\n", e)
+ raise
+
+ def test_for_invalid_yaml2(self):
+ # Issue 143
+ document_loader, avsc_names, schema_metadata, metaschema_loader = load_schema(
+ get_data(u"tests/test_schema/CommonWorkflowLanguage.yml"))
+
+ src = "test19.cwl"
+ with self.assertRaises(RuntimeError):
+ try:
+ load_and_validate(document_loader, avsc_names,
+ six.text_type(get_data("tests/test_schema/"+src)), True)
+ except RuntimeError as e:
+ msg = reformat_yaml_exception_message(strip_dup_lineno(six.text_type(e)))
+ self.assertTrue(msg.endswith(src+":1:1: expected , but found ':'"))
+ print("\n", e)
+ raise
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_ref_resolver.py b/v1.1.0-dev1/salad/schema_salad/tests/test_ref_resolver.py
new file mode 100644
index 000000000..84a35bd7e
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_ref_resolver.py
@@ -0,0 +1,159 @@
+"""Test the ref_resolver module."""
+
+from __future__ import absolute_import
+import shutil
+import tempfile
+
+import pytest # type: ignore
+
+@pytest.fixture
+def tmp_dir_fixture(request):
+ d = tempfile.mkdtemp()
+
+ @request.addfinalizer
+ def teardown():
+ shutil.rmtree(d)
+ return d
+
+def test_Loader_initialisation_for_HOME_env_var(tmp_dir_fixture):
+ import os
+ from schema_salad.ref_resolver import Loader
+ from requests import Session
+
+ # Ensure HOME is set.
+ os.environ["HOME"] = tmp_dir_fixture
+
+ loader = Loader(ctx={})
+ assert isinstance(loader.session, Session)
+
+def test_Loader_initialisation_for_TMP_env_var(tmp_dir_fixture):
+ import os
+ from schema_salad.ref_resolver import Loader
+ from requests import Session
+
+ # Ensure HOME is missing.
+ if "HOME" in os.environ:
+ del os.environ["HOME"]
+ # Ensure TMP is present.
+ os.environ["TMP"] = tmp_dir_fixture
+
+ loader = Loader(ctx={})
+ assert isinstance(loader.session, Session)
+
+def test_Loader_initialisation_with_neither_TMP_HOME_set(tmp_dir_fixture):
+ import os
+ from schema_salad.ref_resolver import Loader
+ from requests import Session
+
+ # Ensure HOME is missing.
+ if "HOME" in os.environ:
+ del os.environ["HOME"]
+ if "TMP" in os.environ:
+ del os.environ["TMP"]
+
+ loader = Loader(ctx={})
+ assert isinstance(loader.session, Session)
+
+def test_DefaultFetcher_urljoin_win32(tmp_dir_fixture):
+ import os
+ import sys
+ from schema_salad.ref_resolver import DefaultFetcher
+ from requests import Session
+
+ # Ensure HOME is set.
+ os.environ["HOME"] = tmp_dir_fixture
+
+ actual_platform = sys.platform
+ try:
+ # For this test always pretend we're on Windows
+ sys.platform = "win32"
+ fetcher = DefaultFetcher({}, None)
+ # Relative path, same folder
+ url = fetcher.urljoin("file:///C:/Users/fred/foo.cwl", "soup.cwl")
+ assert url == "file:///C:/Users/fred/soup.cwl"
+ # Relative path, sub folder
+ url = fetcher.urljoin("file:///C:/Users/fred/foo.cwl", "foo/soup.cwl")
+ assert url == "file:///C:/Users/fred/foo/soup.cwl"
+ # relative climb-up path
+ url = fetcher.urljoin("file:///C:/Users/fred/foo.cwl", "../alice/soup.cwl")
+ assert url == "file:///C:/Users/alice/soup.cwl"
+
+ # Path with drive: should not be treated as relative to directory
+ # Note: \ would already have been converted to / by resolve_ref()
+ url = fetcher.urljoin("file:///C:/Users/fred/foo.cwl", "c:/bar/soup.cwl")
+ assert url == "file:///c:/bar/soup.cwl"
+ # /C:/ (regular URI absolute path)
+ url = fetcher.urljoin("file:///C:/Users/fred/foo.cwl", "/c:/bar/soup.cwl")
+ assert url == "file:///c:/bar/soup.cwl"
+ # Relative, change drive
+ url = fetcher.urljoin("file:///C:/Users/fred/foo.cwl", "D:/baz/soup.cwl")
+ assert url == "file:///d:/baz/soup.cwl"
+ # Relative from root of base's D: drive
+ url = fetcher.urljoin("file:///d:/baz/soup.cwl", "/foo/soup.cwl")
+ assert url == "file:///d:/foo/soup.cwl"
+
+ # resolving absolute non-drive URIs still works
+ url = fetcher.urljoin("file:///C:/Users/fred/foo.cwl", "http://example.com/bar/soup.cwl")
+ assert url == "http://example.com/bar/soup.cwl"
+ # and of course relative paths from http://
+ url = fetcher.urljoin("http://example.com/fred/foo.cwl", "soup.cwl")
+ assert url == "http://example.com/fred/soup.cwl"
+
+ # Stay on http:// and same host
+ url = fetcher.urljoin("http://example.com/fred/foo.cwl", "/bar/soup.cwl")
+ assert url == "http://example.com/bar/soup.cwl"
+
+
+ # Security concern - can't resolve file: from http:
+ with pytest.raises(ValueError):
+ url = fetcher.urljoin("http://example.com/fred/foo.cwl", "file:///c:/bar/soup.cwl")
+ # Drive-relative -- should NOT return "absolute" URI c:/bar/soup.cwl"
+ # as that is a potential remote exploit
+ with pytest.raises(ValueError):
+ url = fetcher.urljoin("http://example.com/fred/foo.cwl", "c:/bar/soup.cwl")
+
+ finally:
+ sys.platform = actual_platform
+
+def test_DefaultFetcher_urljoin_linux(tmp_dir_fixture):
+ import os
+ import sys
+ from schema_salad.ref_resolver import DefaultFetcher
+ from requests import Session
+
+ # Ensure HOME is set.
+ os.environ["HOME"] = tmp_dir_fixture
+
+ actual_platform = sys.platform
+ try:
+ # Pretend it's Linux (e.g. not win32)
+ sys.platform = "linux2"
+ fetcher = DefaultFetcher({}, None)
+ url = fetcher.urljoin("file:///home/fred/foo.cwl", "soup.cwl")
+ assert url == "file:///home/fred/soup.cwl"
+
+ url = fetcher.urljoin("file:///home/fred/foo.cwl", "../alice/soup.cwl")
+ assert url == "file:///home/alice/soup.cwl"
+ # relative from root
+ url = fetcher.urljoin("file:///home/fred/foo.cwl", "/baz/soup.cwl")
+ assert url == "file:///baz/soup.cwl"
+
+ url = fetcher.urljoin("file:///home/fred/foo.cwl", "http://example.com/bar/soup.cwl")
+ assert url == "http://example.com/bar/soup.cwl"
+
+ url = fetcher.urljoin("http://example.com/fred/foo.cwl", "soup.cwl")
+ assert url == "http://example.com/fred/soup.cwl"
+
+ # Root-relative -- here relative to http host, not file:///
+ url = fetcher.urljoin("http://example.com/fred/foo.cwl", "/bar/soup.cwl")
+ assert url == "http://example.com/bar/soup.cwl"
+
+ # Security concern - can't resolve file: from http:
+ with pytest.raises(ValueError):
+ url = fetcher.urljoin("http://example.com/fred/foo.cwl", "file:///bar/soup.cwl")
+
+ # But this one is not "dangerous" on Linux
+ fetcher.urljoin("http://example.com/fred/foo.cwl", "c:/bar/soup.cwl")
+
+ finally:
+ sys.platform = actual_platform
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/CommandLineTool.yml b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/CommandLineTool.yml
new file mode 100644
index 000000000..6a5de9805
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/CommandLineTool.yml
@@ -0,0 +1,895 @@
+$base: "https://w3id.org/cwl/cwl#"
+
+$namespaces:
+ cwl: "https://w3id.org/cwl/cwl#"
+ sld: "https://w3id.org/cwl/salad#"
+
+$graph:
+
+- name: CommandLineToolDoc
+ type: documentation
+ doc:
+ - |
+ # Common Workflow Language (CWL) Command Line Tool Description, v1.0
+
+ This version:
+ * https://w3id.org/cwl/v1.0/
+
+ Current version:
+ * https://w3id.org/cwl/
+ - "\n\n"
+ - {$include: contrib.md}
+ - "\n\n"
+ - |
+ # Abstract
+
+ A Command Line Tool is a non-interactive executable program that reads
+ some input, performs a computation, and terminates after producing some
+ output. Command line programs are a flexible unit of code sharing and
+ reuse, unfortunately the syntax and input/output semantics among command
+ line programs is extremely heterogeneous. A common layer for describing
+ the syntax and semantics of programs can reduce this incidental
+ complexity by providing a consistent way to connect programs together.
+ This specification defines the Common Workflow Language (CWL) Command
+ Line Tool Description, a vendor-neutral standard for describing the
+ syntax and input/output semantics of command line programs.
+
+ - {$include: intro.md}
+
+ - |
+ ## Introduction to v1.0
+
+ This specification represents the first full release from the CWL group.
+ Since draft-3, version 1.0 introduces the following changes and additions:
+
+ * The [Directory](#Directory) type.
+ * Syntax simplifcations: denoted by the `map<>` syntax. Example: inputs
+ contains a list of items, each with an id. Now one can specify
+ a mapping of that identifier to the corresponding
+ `CommandInputParamater`.
+ ```
+ inputs:
+ - id: one
+ type: string
+ doc: First input parameter
+ - id: two
+ type: int
+ doc: Second input parameter
+ ```
+ can be
+ ```
+ inputs:
+ one:
+ type: string
+ doc: First input parameter
+ two:
+ type: int
+ doc: Second input parameter
+ ```
+ * [InitialWorkDirRequirement](#InitialWorkDirRequirement): list of
+ files and subdirectories to be present in the output directory prior
+ to execution.
+ * Shortcuts for specifying the standard [output](#stdout) and/or
+ [error](#stderr) streams as a (streamable) File output.
+ * [SoftwareRequirement](#SoftwareRequirement) for describing software
+ dependencies of a tool.
+ * The common `description` field has been renamed to `doc`.
+
+ ## Errata
+
+ Post v1.0 release changes to the spec.
+
+ * 13 July 2016: Mark `baseCommand` as optional and update descriptive text.
+
+ ## Purpose
+
+ Standalone programs are a flexible and interoperable form of code reuse.
+ Unlike monolithic applications, applications and analysis workflows which
+ are composed of multiple separate programs can be written in multiple
+ languages and execute concurrently on multiple hosts. However, POSIX
+ does not dictate computer-readable grammar or semantics for program input
+ and output, resulting in extremely heterogeneous command line grammar and
+ input/output semantics among program. This is a particular problem in
+ distributed computing (multi-node compute clusters) and virtualized
+ environments (such as Docker containers) where it is often necessary to
+ provision resources such as input files before executing the program.
+
+ Often this gap is filled by hard coding program invocation and
+ implicitly assuming requirements will be met, or abstracting program
+ invocation with wrapper scripts or descriptor documents. Unfortunately,
+ where these approaches are application or platform specific it creates a
+ significant barrier to reproducibility and portability, as methods
+ developed for one platform must be manually ported to be used on new
+ platforms. Similarly it creates redundant work, as wrappers for popular
+ tools must be rewritten for each application or platform in use.
+
+ The Common Workflow Language Command Line Tool Description is designed to
+ provide a common standard description of grammar and semantics for
+ invoking programs used in data-intensive fields such as Bioinformatics,
+ Chemistry, Physics, Astronomy, and Statistics. This specification
+ defines a precise data and execution model for Command Line Tools that
+ can be implemented on a variety of computing platforms, ranging from a
+ single workstation to cluster, grid, cloud, and high performance
+ computing platforms.
+
+ - {$include: concepts.md}
+ - {$include: invocation.md}
+
+
+- type: record
+ name: EnvironmentDef
+ doc: |
+ Define an environment variable that will be set in the runtime environment
+ by the workflow platform when executing the command line tool. May be the
+ result of executing an expression, such as getting a parameter from input.
+ fields:
+ - name: envName
+ type: string
+ doc: The environment variable name
+ - name: envValue
+ type: [string, Expression]
+ doc: The environment variable value
+
+- type: record
+ name: CommandLineBinding
+ extends: InputBinding
+ doc: |
+
+ When listed under `inputBinding` in the input schema, the term
+ "value" refers to the the corresponding value in the input object. For
+ binding objects listed in `CommandLineTool.arguments`, the term "value"
+ refers to the effective value after evaluating `valueFrom`.
+
+ The binding behavior when building the command line depends on the data
+ type of the value. If there is a mismatch between the type described by
+ the input schema and the effective value, such as resulting from an
+ expression evaluation, an implementation must use the data type of the
+ effective value.
+
+ - **string**: Add `prefix` and the string to the command line.
+
+ - **number**: Add `prefix` and decimal representation to command line.
+
+ - **boolean**: If true, add `prefix` to the command line. If false, add
+ nothing.
+
+ - **File**: Add `prefix` and the value of
+ [`File.path`](#File) to the command line.
+
+ - **array**: If `itemSeparator` is specified, add `prefix` and the join
+ the array into a single string with `itemSeparator` separating the
+ items. Otherwise first add `prefix`, then recursively process
+ individual elements.
+
+ - **object**: Add `prefix` only, and recursively add object fields for
+ which `inputBinding` is specified.
+
+ - **null**: Add nothing.
+
+ fields:
+ - name: position
+ type: int?
+ doc: "The sorting key. Default position is 0."
+ - name: prefix
+ type: string?
+ doc: "Command line prefix to add before the value."
+ - name: separate
+ type: boolean?
+ doc: |
+ If true (default), then the prefix and value must be added as separate
+ command line arguments; if false, prefix and value must be concatenated
+ into a single command line argument.
+ - name: itemSeparator
+ type: string?
+ doc: |
+ Join the array elements into a single string with the elements
+ separated by by `itemSeparator`.
+ - name: valueFrom
+ type:
+ - "null"
+ - string
+ - Expression
+ jsonldPredicate: "cwl:valueFrom"
+ doc: |
+ If `valueFrom` is a constant string value, use this as the value and
+ apply the binding rules above.
+
+ If `valueFrom` is an expression, evaluate the expression to yield the
+ actual value to use to build the command line and apply the binding
+ rules above. If the inputBinding is associated with an input
+ parameter, the value of `self` in the expression will be the value of the
+ input parameter.
+
+ When a binding is part of the `CommandLineTool.arguments` field,
+ the `valueFrom` field is required.
+ - name: shellQuote
+ type: boolean?
+ doc: |
+ If `ShellCommandRequirement` is in the requirements for the current command,
+ this controls whether the value is quoted on the command line (default is true).
+ Use `shellQuote: false` to inject metacharacters for operations such as pipes.
+
+- type: record
+ name: CommandOutputBinding
+ extends: OutputBinding
+ doc: |
+ Describes how to generate an output parameter based on the files produced
+ by a CommandLineTool.
+
+ The output parameter is generated by applying these operations in
+ the following order:
+
+ - glob
+ - loadContents
+ - outputEval
+ fields:
+ - name: glob
+ type:
+ - "null"
+ - string
+ - Expression
+ - type: array
+ items: string
+ doc: |
+ Find files relative to the output directory, using POSIX glob(3)
+ pathname matching. If an array is provided, find files that match any
+ pattern in the array. If an expression is provided, the expression must
+ return a string or an array of strings, which will then be evaluated as
+ one or more glob patterns. Must only match and return files which
+ actually exist.
+ - name: loadContents
+ type:
+ - "null"
+ - boolean
+ jsonldPredicate: "cwl:loadContents"
+ doc: |
+ For each file matched in `glob`, read up to
+ the first 64 KiB of text from the file and place it in the `contents`
+ field of the file object for manipulation by `outputEval`.
+ - name: outputEval
+ type:
+ - "null"
+ - string
+ - Expression
+ doc: |
+ Evaluate an expression to generate the output value. If `glob` was
+ specified, the value of `self` must be an array containing file objects
+ that were matched. If no files were matched, `self` must be a zero
+ length array; if a single file was matched, the value of `self` is an
+ array of a single element. Additionally, if `loadContents` is `true`,
+ the File objects must include up to the first 64 KiB of file contents
+ in the `contents` field.
+
+
+- name: CommandInputRecordField
+ type: record
+ extends: InputRecordField
+ specialize:
+ - specializeFrom: InputRecordSchema
+ specializeTo: CommandInputRecordSchema
+ - specializeFrom: InputEnumSchema
+ specializeTo: CommandInputEnumSchema
+ - specializeFrom: InputArraySchema
+ specializeTo: CommandInputArraySchema
+ - specializeFrom: InputBinding
+ specializeTo: CommandLineBinding
+
+
+- name: CommandInputRecordSchema
+ type: record
+ extends: InputRecordSchema
+ specialize:
+ - specializeFrom: InputRecordField
+ specializeTo: CommandInputRecordField
+
+
+- name: CommandInputEnumSchema
+ type: record
+ extends: InputEnumSchema
+ specialize:
+ - specializeFrom: InputBinding
+ specializeTo: CommandLineBinding
+
+
+- name: CommandInputArraySchema
+ type: record
+ extends: InputArraySchema
+ specialize:
+ - specializeFrom: InputRecordSchema
+ specializeTo: CommandInputRecordSchema
+ - specializeFrom: InputEnumSchema
+ specializeTo: CommandInputEnumSchema
+ - specializeFrom: InputArraySchema
+ specializeTo: CommandInputArraySchema
+ - specializeFrom: InputBinding
+ specializeTo: CommandLineBinding
+
+
+- name: CommandOutputRecordField
+ type: record
+ extends: OutputRecordField
+ specialize:
+ - specializeFrom: OutputRecordSchema
+ specializeTo: CommandOutputRecordSchema
+ - specializeFrom: OutputEnumSchema
+ specializeTo: CommandOutputEnumSchema
+ - specializeFrom: OutputArraySchema
+ specializeTo: CommandOutputArraySchema
+ - specializeFrom: OutputBinding
+ specializeTo: CommandOutputBinding
+
+
+- name: CommandOutputRecordSchema
+ type: record
+ extends: OutputRecordSchema
+ specialize:
+ - specializeFrom: OutputRecordField
+ specializeTo: CommandOutputRecordField
+
+
+- name: CommandOutputEnumSchema
+ type: record
+ extends: OutputEnumSchema
+ specialize:
+ - specializeFrom: OutputRecordSchema
+ specializeTo: CommandOutputRecordSchema
+ - specializeFrom: OutputEnumSchema
+ specializeTo: CommandOutputEnumSchema
+ - specializeFrom: OutputArraySchema
+ specializeTo: CommandOutputArraySchema
+ - specializeFrom: OutputBinding
+ specializeTo: CommandOutputBinding
+
+
+- name: CommandOutputArraySchema
+ type: record
+ extends: OutputArraySchema
+ specialize:
+ - specializeFrom: OutputRecordSchema
+ specializeTo: CommandOutputRecordSchema
+ - specializeFrom: OutputEnumSchema
+ specializeTo: CommandOutputEnumSchema
+ - specializeFrom: OutputArraySchema
+ specializeTo: CommandOutputArraySchema
+ - specializeFrom: OutputBinding
+ specializeTo: CommandOutputBinding
+
+
+- type: record
+ name: CommandInputParameter
+ extends: InputParameter
+ doc: An input parameter for a CommandLineTool.
+ specialize:
+ - specializeFrom: InputRecordSchema
+ specializeTo: CommandInputRecordSchema
+ - specializeFrom: InputEnumSchema
+ specializeTo: CommandInputEnumSchema
+ - specializeFrom: InputArraySchema
+ specializeTo: CommandInputArraySchema
+ - specializeFrom: InputBinding
+ specializeTo: CommandLineBinding
+
+- type: record
+ name: CommandOutputParameter
+ extends: OutputParameter
+ doc: An output parameter for a CommandLineTool.
+ specialize:
+ - specializeFrom: OutputBinding
+ specializeTo: CommandOutputBinding
+ fields:
+ - name: type
+ type:
+ - "null"
+ - CWLType
+ - stdout
+ - stderr
+ - CommandOutputRecordSchema
+ - CommandOutputEnumSchema
+ - CommandOutputArraySchema
+ - string
+ - type: array
+ items:
+ - CWLType
+ - CommandOutputRecordSchema
+ - CommandOutputEnumSchema
+ - CommandOutputArraySchema
+ - string
+ jsonldPredicate:
+ "_id": "sld:type"
+ "_type": "@vocab"
+ refScope: 2
+ typeDSL: True
+ doc: |
+ Specify valid types of data that may be assigned to this parameter.
+
+- name: stdout
+ type: enum
+ symbols: [ "cwl:stdout" ]
+ docParent: "#CommandOutputParameter"
+ doc: |
+ Only valid as a `type` for a `CommandLineTool` output with no
+ `outputBinding` set.
+
+ The following
+ ```
+ outputs:
+ an_output_name:
+ type: stdout
+
+ stdout: a_stdout_file
+ ```
+ is equivalent to
+ ```
+ outputs:
+ an_output_name:
+ type: File
+ streamable: true
+ outputBinding:
+ glob: a_stdout_file
+
+ stdout: a_stdout_file
+ ```
+
+ If there is no `stdout` name provided, a random filename will be created.
+ For example, the following
+ ```
+ outputs:
+ an_output_name:
+ type: stdout
+ ```
+ is equivalent to
+ ```
+ outputs:
+ an_output_name:
+ type: File
+ streamable: true
+ outputBinding:
+ glob: random_stdout_filenameABCDEFG
+
+ stdout: random_stdout_filenameABCDEFG
+ ```
+
+
+- name: stderr
+ type: enum
+ symbols: [ "cwl:stderr" ]
+ docParent: "#CommandOutputParameter"
+ doc: |
+ Only valid as a `type` for a `CommandLineTool` output with no
+ `outputBinding` set.
+
+ The following
+ ```
+ outputs:
+ an_output_name:
+ type: stderr
+
+ stderr: a_stderr_file
+ ```
+ is equivalent to
+ ```
+ outputs:
+ an_output_name:
+ type: File
+ streamable: true
+ outputBinding:
+ glob: a_stderr_file
+
+ stderr: a_stderr_file
+ ```
+
+ If there is no `stderr` name provided, a random filename will be created.
+ For example, the following
+ ```
+ outputs:
+ an_output_name:
+ type: stderr
+ ```
+ is equivalent to
+ ```
+ outputs:
+ an_output_name:
+ type: File
+ streamable: true
+ outputBinding:
+ glob: random_stderr_filenameABCDEFG
+
+ stderr: random_stderr_filenameABCDEFG
+ ```
+
+
+- type: record
+ name: CommandLineTool
+ extends: Process
+ documentRoot: true
+ specialize:
+ - specializeFrom: InputParameter
+ specializeTo: CommandInputParameter
+ - specializeFrom: OutputParameter
+ specializeTo: CommandOutputParameter
+ doc: |
+ This defines the schema of the CWL Command Line Tool Description document.
+
+ fields:
+ - name: class
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ type: string
+ - name: baseCommand
+ doc: |
+ Specifies the program to execute. If an array, the first element of
+ the array is the command to execute, and subsequent elements are
+ mandatory command line arguments. The elements in `baseCommand` must
+ appear before any command line bindings from `inputBinding` or
+ `arguments`.
+
+ If `baseCommand` is not provided or is an empty array, the first
+ element of the command line produced after processing `inputBinding` or
+ `arguments` must be used as the program to execute.
+
+ If the program includes a path separator character it must
+ be an absolute path, otherwise it is an error. If the program does not
+ include a path separator, search the `$PATH` variable in the runtime
+ environment of the workflow runner find the absolute path of the
+ executable.
+ type:
+ - string?
+ - string[]?
+ jsonldPredicate:
+ "_id": "cwl:baseCommand"
+ "_container": "@list"
+ - name: arguments
+ doc: |
+ Command line bindings which are not directly associated with input parameters.
+ type:
+ - "null"
+ - type: array
+ items: [string, Expression, CommandLineBinding]
+ jsonldPredicate:
+ "_id": "cwl:arguments"
+ "_container": "@list"
+ - name: stdin
+ type: ["null", string, Expression]
+ doc: |
+ A path to a file whose contents must be piped into the command's
+ standard input stream.
+ - name: stderr
+ type: ["null", string, Expression]
+ jsonldPredicate: "https://w3id.org/cwl/cwl#stderr"
+ doc: |
+ Capture the command's standard error stream to a file written to
+ the designated output directory.
+
+ If `stderr` is a string, it specifies the file name to use.
+
+ If `stderr` is an expression, the expression is evaluated and must
+ return a string with the file name to use to capture stderr. If the
+ return value is not a string, or the resulting path contains illegal
+ characters (such as the path separator `/`) it is an error.
+ - name: stdout
+ type: ["null", string, Expression]
+ jsonldPredicate: "https://w3id.org/cwl/cwl#stdout"
+ doc: |
+ Capture the command's standard output stream to a file written to
+ the designated output directory.
+
+ If `stdout` is a string, it specifies the file name to use.
+
+ If `stdout` is an expression, the expression is evaluated and must
+ return a string with the file name to use to capture stdout. If the
+ return value is not a string, or the resulting path contains illegal
+ characters (such as the path separator `/`) it is an error.
+ - name: successCodes
+ type: int[]?
+ doc: |
+ Exit codes that indicate the process completed successfully.
+
+ - name: temporaryFailCodes
+ type: int[]?
+ doc: |
+ Exit codes that indicate the process failed due to a possibly
+ temporary condition, where executing the process with the same
+ runtime environment and inputs may produce different results.
+
+ - name: permanentFailCodes
+ type: int[]?
+ doc:
+ Exit codes that indicate the process failed due to a permanent logic
+ error, where executing the process with the same runtime environment and
+ same inputs is expected to always fail.
+
+
+- type: record
+ name: DockerRequirement
+ extends: ProcessRequirement
+ doc: |
+ Indicates that a workflow component should be run in a
+ [Docker](http://docker.com) container, and specifies how to fetch or build
+ the image.
+
+ If a CommandLineTool lists `DockerRequirement` under
+ `hints` (or `requirements`), it may (or must) be run in the specified Docker
+ container.
+
+ The platform must first acquire or install the correct Docker image as
+ specified by `dockerPull`, `dockerImport`, `dockerLoad` or `dockerFile`.
+
+ The platform must execute the tool in the container using `docker run` with
+ the appropriate Docker image and tool command line.
+
+ The workflow platform may provide input files and the designated output
+ directory through the use of volume bind mounts. The platform may rewrite
+ file paths in the input object to correspond to the Docker bind mounted
+ locations.
+
+ When running a tool contained in Docker, the workflow platform must not
+ assume anything about the contents of the Docker container, such as the
+ presence or absence of specific software, except to assume that the
+ generated command line represents a valid command within the runtime
+ environment of the container.
+
+ ## Interaction with other requirements
+
+ If [EnvVarRequirement](#EnvVarRequirement) is specified alongside a
+ DockerRequirement, the environment variables must be provided to Docker
+ using `--env` or `--env-file` and interact with the container's preexisting
+ environment as defined by Docker.
+
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'DockerRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: dockerPull
+ type: string?
+ doc: "Specify a Docker image to retrieve using `docker pull`."
+ - name: dockerLoad
+ type: string?
+ doc: "Specify a HTTP URL from which to download a Docker image using `docker load`."
+ - name: dockerFile
+ type: string?
+ doc: "Supply the contents of a Dockerfile which will be built using `docker build`."
+ - name: dockerImport
+ type: string?
+ doc: "Provide HTTP URL to download and gunzip a Docker images using `docker import."
+ - name: dockerImageId
+ type: string?
+ doc: |
+ The image id that will be used for `docker run`. May be a
+ human-readable image name or the image identifier hash. May be skipped
+ if `dockerPull` is specified, in which case the `dockerPull` image id
+ must be used.
+ - name: dockerOutputDirectory
+ type: string?
+ doc: |
+ Set the designated output directory to a specific location inside the
+ Docker container.
+
+
+- type: record
+ name: SoftwareRequirement
+ extends: ProcessRequirement
+ doc: |
+ A list of software packages that should be configured in the environment of
+ the defined process.
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'SoftwareRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: packages
+ type: SoftwarePackage[]
+ doc: "The list of software to be configured."
+ jsonldPredicate:
+ mapSubject: package
+ mapPredicate: specs
+
+- name: SoftwarePackage
+ type: record
+ fields:
+ - name: package
+ type: string
+ doc: "The common name of the software to be configured."
+ - name: version
+ type: string[]?
+ doc: "The (optional) version of the software to configured."
+ - name: specs
+ type: string[]?
+ doc: |
+ Must be one or more IRIs identifying resources for installing or
+ enabling the software. Implementations may provide resolvers which map
+ well-known software spec IRIs to some configuration action.
+
+ For example, an IRI `https://packages.debian.org/jessie/bowtie` could
+ be resolved with `apt-get install bowtie`. An IRI
+ `https://anaconda.org/bioconda/bowtie` could be resolved with `conda
+ install -c bioconda bowtie`.
+
+ Tools may also provide IRIs to index entries such as
+ [RRID](http://www.identifiers.org/rrid/), such as
+ `http://identifiers.org/rrid/RRID:SCR_005476`
+
+
+- name: Dirent
+ type: record
+ doc: |
+ Define a file or subdirectory that must be placed in the designated output
+ directory prior to executing the command line tool. May be the result of
+ executing an expression, such as building a configuration file from a
+ template.
+ fields:
+ - name: entryname
+ type: ["null", string, Expression]
+ jsonldPredicate:
+ _id: cwl:entryname
+ doc: |
+ The name of the file or subdirectory to create in the output directory.
+ If `entry` is a File or Directory, this overrides `basename`. Optional.
+ - name: entry
+ type: [string, Expression]
+ jsonldPredicate:
+ _id: cwl:entry
+ doc: |
+ If the value is a string literal or an expression which evaluates to a
+ string, a new file must be created with the string as the file contents.
+
+ If the value is an expression that evaluates to a `File` object, this
+ indicates the referenced file should be added to the designated output
+ directory prior to executing the tool.
+
+ If the value is an expression that evaluates to a `Dirent` object, this
+ indicates that the File or Directory in `entry` should be added to the
+ designated output directory with the name in `entryname`.
+
+ If `writable` is false, the file may be made available using a bind
+ mount or file system link to avoid unnecessary copying of the input
+ file.
+ - name: writable
+ type: boolean?
+ doc: |
+ If true, the file or directory must be writable by the tool. Changes
+ to the file or directory must be isolated and not visible by any other
+ CommandLineTool process. This may be implemented by making a copy of
+ the original file or directory. Default false (files and directories
+ read-only by default).
+
+
+- name: InitialWorkDirRequirement
+ type: record
+ extends: ProcessRequirement
+ doc:
+ Define a list of files and subdirectories that must be created by the
+ workflow platform in the designated output directory prior to executing the
+ command line tool.
+ fields:
+ - name: class
+ type: string
+ doc: InitialWorkDirRequirement
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: listing
+ type:
+ - type: array
+ items: [File, Directory, Dirent, string, Expression]
+ - string
+ - Expression
+ jsonldPredicate:
+ _id: "cwl:listing"
+ doc: |
+ The list of files or subdirectories that must be placed in the
+ designated output directory prior to executing the command line tool.
+
+ May be an expression. If so, the expression return value must validate
+ as `{type: array, items: [File, Directory]}`.
+
+
+- name: EnvVarRequirement
+ type: record
+ extends: ProcessRequirement
+ doc: |
+ Define a list of environment variables which will be set in the
+ execution environment of the tool. See `EnvironmentDef` for details.
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'EnvVarRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: envDef
+ type: EnvironmentDef[]
+ doc: The list of environment variables.
+ jsonldPredicate:
+ mapSubject: envName
+ mapPredicate: envValue
+
+
+- type: record
+ name: ShellCommandRequirement
+ extends: ProcessRequirement
+ doc: |
+ Modify the behavior of CommandLineTool to generate a single string
+ containing a shell command line. Each item in the argument list must be
+ joined into a string separated by single spaces and quoted to prevent
+ intepretation by the shell, unless `CommandLineBinding` for that argument
+ contains `shellQuote: false`. If `shellQuote: false` is specified, the
+ argument is joined into the command string without quoting, which allows
+ the use of shell metacharacters such as `|` for pipes.
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'ShellCommandRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+
+
+- type: record
+ name: ResourceRequirement
+ extends: ProcessRequirement
+ doc: |
+ Specify basic hardware resource requirements.
+
+ "min" is the minimum amount of a resource that must be reserved to schedule
+ a job. If "min" cannot be satisfied, the job should not be run.
+
+ "max" is the maximum amount of a resource that the job shall be permitted
+ to use. If a node has sufficient resources, multiple jobs may be scheduled
+ on a single node provided each job's "max" resource requirements are
+ met. If a job attempts to exceed its "max" resource allocation, an
+ implementation may deny additional resources, which may result in job
+ failure.
+
+ If "min" is specified but "max" is not, then "max" == "min"
+ If "max" is specified by "min" is not, then "min" == "max".
+
+ It is an error if max < min.
+
+ It is an error if the value of any of these fields is negative.
+
+ If neither "min" nor "max" is specified for a resource, an implementation may provide a default.
+
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'ResourceRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: coresMin
+ type: ["null", long, string, Expression]
+ doc: Minimum reserved number of CPU cores
+
+ - name: coresMax
+ type: ["null", int, string, Expression]
+ doc: Maximum reserved number of CPU cores
+
+ - name: ramMin
+ type: ["null", long, string, Expression]
+ doc: Minimum reserved RAM in mebibytes (2**20)
+
+ - name: ramMax
+ type: ["null", long, string, Expression]
+ doc: Maximum reserved RAM in mebibytes (2**20)
+
+ - name: tmpdirMin
+ type: ["null", long, string, Expression]
+ doc: Minimum reserved filesystem based storage for the designated temporary directory, in mebibytes (2**20)
+
+ - name: tmpdirMax
+ type: ["null", long, string, Expression]
+ doc: Maximum reserved filesystem based storage for the designated temporary directory, in mebibytes (2**20)
+
+ - name: outdirMin
+ type: ["null", long, string, Expression]
+ doc: Minimum reserved filesystem based storage for the designated output directory, in mebibytes (2**20)
+
+ - name: outdirMax
+ type: ["null", long, string, Expression]
+ doc: Maximum reserved filesystem based storage for the designated output directory, in mebibytes (2**20)
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/CommonWorkflowLanguage.yml b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/CommonWorkflowLanguage.yml
new file mode 100644
index 000000000..73921e899
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/CommonWorkflowLanguage.yml
@@ -0,0 +1,11 @@
+$base: "https://w3id.org/cwl/cwl#"
+
+$namespaces:
+ cwl: "https://w3id.org/cwl/cwl#"
+ sld: "https://w3id.org/cwl/salad#"
+
+$graph:
+
+- $import: Process.yml
+- $import: CommandLineTool.yml
+- $import: Workflow.yml
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/Process.yml b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/Process.yml
new file mode 100644
index 000000000..8b9bce5f0
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/Process.yml
@@ -0,0 +1,743 @@
+$base: "https://w3id.org/cwl/cwl#"
+
+$namespaces:
+ cwl: "https://w3id.org/cwl/cwl#"
+ sld: "https://w3id.org/cwl/salad#"
+
+$graph:
+
+- name: "Common Workflow Language, v1.0"
+ type: documentation
+ doc: {$include: concepts.md}
+
+- $import: "metaschema_base.yml"
+
+- name: BaseTypesDoc
+ type: documentation
+ doc: |
+ ## Base types
+ docChild:
+ - "#CWLType"
+ - "#Process"
+
+- type: enum
+ name: CWLVersion
+ doc: "Version symbols for published CWL document versions."
+ symbols:
+ - cwl:draft-2
+ - cwl:draft-3.dev1
+ - cwl:draft-3.dev2
+ - cwl:draft-3.dev3
+ - cwl:draft-3.dev4
+ - cwl:draft-3.dev5
+ - cwl:draft-3
+ - cwl:draft-4.dev1
+ - cwl:draft-4.dev2
+ - cwl:draft-4.dev3
+ - cwl:v1.0.dev4
+ - cwl:v1.0
+
+- name: CWLType
+ type: enum
+ extends: "sld:PrimitiveType"
+ symbols:
+ - cwl:File
+ - cwl:Directory
+ doc:
+ - "Extends primitive types with the concept of a file and directory as a builtin type."
+ - "File: A File object"
+ - "Directory: A Directory object"
+
+- name: File
+ type: record
+ docParent: "#CWLType"
+ doc: |
+ Represents a file (or group of files if `secondaryFiles` is specified) that
+ must be accessible by tools using standard POSIX file system call API such as
+ open(2) and read(2).
+ fields:
+ - name: class
+ type:
+ type: enum
+ name: File_class
+ symbols:
+ - cwl:File
+ jsonldPredicate:
+ _id: "@type"
+ _type: "@vocab"
+ doc: Must be `File` to indicate this object describes a file.
+ - name: location
+ type: string?
+ doc: |
+ An IRI that identifies the file resource. This may be a relative
+ reference, in which case it must be resolved using the base IRI of the
+ document. The location may refer to a local or remote resource; the
+ implementation must use the IRI to retrieve file content. If an
+ implementation is unable to retrieve the file content stored at a
+ remote resource (due to unsupported protocol, access denied, or other
+ issue) it must signal an error.
+
+ If the `location` field is not provided, the `contents` field must be
+ provided. The implementation must assign a unique identifier for
+ the `location` field.
+
+ If the `path` field is provided but the `location` field is not, an
+ implementation may assign the value of the `path` field to `location`,
+ then follow the rules above.
+ jsonldPredicate:
+ _id: "@id"
+ _type: "@id"
+ - name: path
+ type: string?
+ doc: |
+ The local host path where the File is available when a CommandLineTool is
+ executed. This field must be set by the implementation. The final
+ path component must match the value of `basename`. This field
+ must not be used in any other context. The command line tool being
+ executed must be able to to access the file at `path` using the POSIX
+ `open(2)` syscall.
+
+ As a special case, if the `path` field is provided but the `location`
+ field is not, an implementation may assign the value of the `path`
+ field to `location`, and remove the `path` field.
+
+ If the `path` contains [POSIX shell metacharacters](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02)
+ (`|`,`&`, `;`, `<`, `>`, `(`,`)`, `$`,`` ` ``, `\`, `"`, `'`,
+ ``, ``, and ``) or characters
+ [not allowed](http://www.iana.org/assignments/idna-tables-6.3.0/idna-tables-6.3.0.xhtml)
+ for [Internationalized Domain Names for Applications](https://tools.ietf.org/html/rfc6452)
+ then implementations may terminate the process with a
+ `permanentFailure`.
+ jsonldPredicate:
+ "_id": "cwl:path"
+ "_type": "@id"
+ - name: basename
+ type: string?
+ doc: |
+ The base name of the file, that is, the name of the file without any
+ leading directory path. The base name must not contain a slash `/`.
+
+ If not provided, the implementation must set this field based on the
+ `location` field by taking the final path component after parsing
+ `location` as an IRI. If `basename` is provided, it is not required to
+ match the value from `location`.
+
+ When this file is made available to a CommandLineTool, it must be named
+ with `basename`, i.e. the final component of the `path` field must match
+ `basename`.
+ jsonldPredicate: "cwl:basename"
+ - name: dirname
+ type: string?
+ doc: |
+ The name of the directory containing file, that is, the path leading up
+ to the final slash in the path such that `dirname + '/' + basename ==
+ path`.
+
+ The implementation must set this field based on the value of `path`
+ prior to evaluating parameter references or expressions in a
+ CommandLineTool document. This field must not be used in any other
+ context.
+ - name: nameroot
+ type: string?
+ doc: |
+ The basename root such that `nameroot + nameext == basename`, and
+ `nameext` is empty or begins with a period and contains at most one
+ period. For the purposess of path splitting leading periods on the
+ basename are ignored; a basename of `.cshrc` will have a nameroot of
+ `.cshrc`.
+
+ The implementation must set this field automatically based on the value
+ of `basename` prior to evaluating parameter references or expressions.
+ - name: nameext
+ type: string?
+ doc: |
+ The basename extension such that `nameroot + nameext == basename`, and
+ `nameext` is empty or begins with a period and contains at most one
+ period. Leading periods on the basename are ignored; a basename of
+ `.cshrc` will have an empty `nameext`.
+
+ The implementation must set this field automatically based on the value
+ of `basename` prior to evaluating parameter references or expressions.
+ - name: checksum
+ type: string?
+ doc: |
+ Optional hash code for validating file integrity. Currently must be in the form
+ "sha1$ + hexadecimal string" using the SHA-1 algorithm.
+ - name: size
+ type: long?
+ doc: Optional file size
+ - name: "secondaryFiles"
+ type:
+ - "null"
+ - type: array
+ items: [File, Directory]
+ jsonldPredicate: "cwl:secondaryFiles"
+ doc: |
+ A list of additional files that are associated with the primary file
+ and must be transferred alongside the primary file. Examples include
+ indexes of the primary file, or external references which must be
+ included when loading primary document. A file object listed in
+ `secondaryFiles` may itself include `secondaryFiles` for which the same
+ rules apply.
+ - name: format
+ type: string?
+ jsonldPredicate:
+ _id: cwl:format
+ _type: "@id"
+ identity: true
+ doc: |
+ The format of the file: this must be an IRI of a concept node that
+ represents the file format, preferrably defined within an ontology.
+ If no ontology is available, file formats may be tested by exact match.
+
+ Reasoning about format compatability must be done by checking that an
+ input file format is the same, `owl:equivalentClass` or
+ `rdfs:subClassOf` the format required by the input parameter.
+ `owl:equivalentClass` is transitive with `rdfs:subClassOf`, e.g. if
+ ` owl:equivalentClass ` and ` owl:subclassOf ` then infer
+ ` owl:subclassOf `.
+
+ File format ontologies may be provided in the "$schema" metadata at the
+ root of the document. If no ontologies are specified in `$schema`, the
+ runtime may perform exact file format matches.
+ - name: contents
+ type: string?
+ doc: |
+ File contents literal. Maximum of 64 KiB.
+
+ If neither `location` nor `path` is provided, `contents` must be
+ non-null. The implementation must assign a unique identifier for the
+ `location` field. When the file is staged as input to CommandLineTool,
+ the value of `contents` must be written to a file.
+
+ If `loadContents` of `inputBinding` or `outputBinding` is true and
+ `location` is valid, the implementation must read up to the first 64
+ KiB of text from the file and place it in the "contents" field.
+
+
+- name: Directory
+ type: record
+ docAfter: "#File"
+ doc: |
+ Represents a directory to present to a command line tool.
+ fields:
+ - name: class
+ type:
+ type: enum
+ name: Directory_class
+ symbols:
+ - cwl:Directory
+ jsonldPredicate:
+ _id: "@type"
+ _type: "@vocab"
+ doc: Must be `Directory` to indicate this object describes a Directory.
+ - name: location
+ type: string?
+ doc: |
+ An IRI that identifies the directory resource. This may be a relative
+ reference, in which case it must be resolved using the base IRI of the
+ document. The location may refer to a local or remote resource. If
+ the `listing` field is not set, the implementation must use the
+ location IRI to retrieve directory listing. If an implementation is
+ unable to retrieve the directory listing stored at a remote resource (due to
+ unsupported protocol, access denied, or other issue) it must signal an
+ error.
+
+ If the `location` field is not provided, the `listing` field must be
+ provided. The implementation must assign a unique identifier for
+ the `location` field.
+
+ If the `path` field is provided but the `location` field is not, an
+ implementation may assign the value of the `path` field to `location`,
+ then follow the rules above.
+ jsonldPredicate:
+ _id: "@id"
+ _type: "@id"
+ - name: path
+ type: string?
+ doc: |
+ The local path where the Directory is made available prior to executing a
+ CommandLineTool. This must be set by the implementation. This field
+ must not be used in any other context. The command line tool being
+ executed must be able to to access the directory at `path` using the POSIX
+ `opendir(2)` syscall.
+
+ If the `path` contains [POSIX shell metacharacters](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02)
+ (`|`,`&`, `;`, `<`, `>`, `(`,`)`, `$`,`` ` ``, `\`, `"`, `'`,
+ ``, ``, and ``) or characters
+ [not allowed](http://www.iana.org/assignments/idna-tables-6.3.0/idna-tables-6.3.0.xhtml)
+ for [Internationalized Domain Names for Applications](https://tools.ietf.org/html/rfc6452)
+ then implementations may terminate the process with a
+ `permanentFailure`.
+ jsonldPredicate:
+ _id: "cwl:path"
+ _type: "@id"
+ - name: basename
+ type: string?
+ doc: |
+ The base name of the directory, that is, the name of the file without any
+ leading directory path. The base name must not contain a slash `/`.
+
+ If not provided, the implementation must set this field based on the
+ `location` field by taking the final path component after parsing
+ `location` as an IRI. If `basename` is provided, it is not required to
+ match the value from `location`.
+
+ When this file is made available to a CommandLineTool, it must be named
+ with `basename`, i.e. the final component of the `path` field must match
+ `basename`.
+ jsonldPredicate: "cwl:basename"
+ - name: listing
+ type:
+ - "null"
+ - type: array
+ items: [File, Directory]
+ doc: |
+ List of files or subdirectories contained in this directory. The name
+ of each file or subdirectory is determined by the `basename` field of
+ each `File` or `Directory` object. It is an error if a `File` shares a
+ `basename` with any other entry in `listing`. If two or more
+ `Directory` object share the same `basename`, this must be treated as
+ equivalent to a single subdirectory with the listings recursively
+ merged.
+ jsonldPredicate:
+ _id: "cwl:listing"
+
+- name: SchemaBase
+ type: record
+ abstract: true
+ fields:
+ - name: label
+ type:
+ - "null"
+ - string
+ jsonldPredicate: "rdfs:label"
+ doc: "A short, human-readable label of this object."
+
+
+- name: Parameter
+ type: record
+ extends: SchemaBase
+ abstract: true
+ doc: |
+ Define an input or output parameter to a process.
+
+ fields:
+ - name: secondaryFiles
+ type:
+ - "null"
+ - string
+ - Expression
+ - type: array
+ items: [string, Expression]
+ jsonldPredicate: "cwl:secondaryFiles"
+ doc: |
+ Only valid when `type: File` or is an array of `items: File`.
+
+ Describes files that must be included alongside the primary file(s).
+
+ If the value is an expression, the value of `self` in the expression
+ must be the primary input or output File to which this binding applies.
+
+ If the value is a string, it specifies that the following pattern
+ should be applied to the primary file:
+
+ 1. If string begins with one or more caret `^` characters, for each
+ caret, remove the last file extension from the path (the last
+ period `.` and all following characters). If there are no file
+ extensions, the path is unchanged.
+ 2. Append the remainder of the string to the end of the file path.
+
+ - name: format
+ type:
+ - "null"
+ - string
+ - type: array
+ items: string
+ - Expression
+ jsonldPredicate:
+ _id: cwl:format
+ _type: "@id"
+ identity: true
+ doc: |
+ Only valid when `type: File` or is an array of `items: File`.
+
+ For input parameters, this must be one or more IRIs of concept nodes
+ that represents file formats which are allowed as input to this
+ parameter, preferrably defined within an ontology. If no ontology is
+ available, file formats may be tested by exact match.
+
+ For output parameters, this is the file format that will be assigned to
+ the output parameter.
+
+ - name: streamable
+ type: boolean?
+ doc: |
+ Only valid when `type: File` or is an array of `items: File`.
+
+ A value of `true` indicates that the file is read or written
+ sequentially without seeking. An implementation may use this flag to
+ indicate whether it is valid to stream file contents using a named
+ pipe. Default: `false`.
+
+ - name: doc
+ type:
+ - string?
+ - string[]?
+ doc: "A documentation string for this type, or an array of strings which should be concatenated."
+ jsonldPredicate: "rdfs:comment"
+
+
+- type: enum
+ name: Expression
+ doc: |
+ 'Expression' is not a real type. It indicates that a field must allow
+ runtime parameter references. If [InlineJavascriptRequirement](#InlineJavascriptRequirement)
+ is declared and supported by the platform, the field must also allow
+ Javascript expressions.
+ symbols:
+ - cwl:ExpressionPlaceholder
+
+
+- name: InputBinding
+ type: record
+ abstract: true
+ fields:
+ - name: loadContents
+ type:
+ - "null"
+ - boolean
+ jsonldPredicate: "cwl:loadContents"
+ doc: |
+ Only valid when `type: File` or is an array of `items: File`.
+
+ Read up to the first 64 KiB of text from the file and place it in the
+ "contents" field of the file object for use by expressions.
+
+
+- name: OutputBinding
+ type: record
+ abstract: true
+
+
+- name: InputSchema
+ extends: SchemaBase
+ type: record
+ abstract: true
+
+
+- name: OutputSchema
+ extends: SchemaBase
+ type: record
+ abstract: true
+
+
+- name: InputRecordField
+ type: record
+ extends: "sld:RecordField"
+ specialize:
+ - specializeFrom: "sld:RecordSchema"
+ specializeTo: InputRecordSchema
+ - specializeFrom: "sld:EnumSchema"
+ specializeTo: InputEnumSchema
+ - specializeFrom: "sld:ArraySchema"
+ specializeTo: InputArraySchema
+ - specializeFrom: "sld:PrimitiveType"
+ specializeTo: CWLType
+ fields:
+ - name: inputBinding
+ type: InputBinding?
+ jsonldPredicate: "cwl:inputBinding"
+ - name: label
+ type: string?
+ jsonldPredicate: "rdfs:label"
+ doc: "A short, human-readable label of this process object."
+
+
+- name: InputRecordSchema
+ type: record
+ extends: ["sld:RecordSchema", InputSchema]
+ specialize:
+ - specializeFrom: "sld:RecordField"
+ specializeTo: InputRecordField
+
+
+- name: InputEnumSchema
+ type: record
+ extends: ["sld:EnumSchema", InputSchema]
+ fields:
+ - name: inputBinding
+ type: InputBinding?
+ jsonldPredicate: "cwl:inputBinding"
+
+
+- name: InputArraySchema
+ type: record
+ extends: ["sld:ArraySchema", InputSchema]
+ specialize:
+ - specializeFrom: "sld:RecordSchema"
+ specializeTo: InputRecordSchema
+ - specializeFrom: "sld:EnumSchema"
+ specializeTo: InputEnumSchema
+ - specializeFrom: "sld:ArraySchema"
+ specializeTo: InputArraySchema
+ - specializeFrom: "sld:PrimitiveType"
+ specializeTo: CWLType
+ fields:
+ - name: inputBinding
+ type: InputBinding?
+ jsonldPredicate: "cwl:inputBinding"
+
+
+- name: OutputRecordField
+ type: record
+ extends: "sld:RecordField"
+ specialize:
+ - specializeFrom: "sld:RecordSchema"
+ specializeTo: OutputRecordSchema
+ - specializeFrom: "sld:EnumSchema"
+ specializeTo: OutputEnumSchema
+ - specializeFrom: "sld:ArraySchema"
+ specializeTo: OutputArraySchema
+ - specializeFrom: "sld:PrimitiveType"
+ specializeTo: CWLType
+ fields:
+ - name: outputBinding
+ type: OutputBinding?
+ jsonldPredicate: "cwl:outputBinding"
+
+
+- name: OutputRecordSchema
+ type: record
+ extends: ["sld:RecordSchema", "#OutputSchema"]
+ docParent: "#OutputParameter"
+ specialize:
+ - specializeFrom: "sld:RecordField"
+ specializeTo: OutputRecordField
+
+
+- name: OutputEnumSchema
+ type: record
+ extends: ["sld:EnumSchema", OutputSchema]
+ docParent: "#OutputParameter"
+ fields:
+ - name: outputBinding
+ type: OutputBinding?
+ jsonldPredicate: "cwl:outputBinding"
+
+- name: OutputArraySchema
+ type: record
+ extends: ["sld:ArraySchema", OutputSchema]
+ docParent: "#OutputParameter"
+ specialize:
+ - specializeFrom: "sld:RecordSchema"
+ specializeTo: OutputRecordSchema
+ - specializeFrom: "sld:EnumSchema"
+ specializeTo: OutputEnumSchema
+ - specializeFrom: "sld:ArraySchema"
+ specializeTo: OutputArraySchema
+ - specializeFrom: "sld:PrimitiveType"
+ specializeTo: CWLType
+ fields:
+ - name: outputBinding
+ type: OutputBinding?
+ jsonldPredicate: "cwl:outputBinding"
+
+
+- name: InputParameter
+ type: record
+ extends: Parameter
+ fields:
+ - name: id
+ type: string
+ jsonldPredicate: "@id"
+ doc: "The unique identifier for this parameter object."
+
+ - name: inputBinding
+ type: InputBinding?
+ jsonldPredicate: "cwl:inputBinding"
+ doc: |
+ Describes how to handle the inputs of a process and convert them
+ into a concrete form for execution, such as command line parameters.
+
+ - name: default
+ type: Any?
+ jsonldPredicate: "cwl:default"
+ doc: |
+ The default value for this parameter if not provided in the input
+ object.
+
+ - name: type
+ type:
+ - "null"
+ - CWLType
+ - InputRecordSchema
+ - InputEnumSchema
+ - InputArraySchema
+ - string
+ - type: array
+ items:
+ - CWLType
+ - InputRecordSchema
+ - InputEnumSchema
+ - InputArraySchema
+ - string
+ jsonldPredicate:
+ "_id": "sld:type"
+ "_type": "@vocab"
+ refScope: 2
+ typeDSL: True
+ doc: |
+ Specify valid types of data that may be assigned to this parameter.
+
+- name: OutputParameter
+ type: record
+ extends: Parameter
+ fields:
+ - name: id
+ type: string
+ jsonldPredicate: "@id"
+ doc: "The unique identifier for this parameter object."
+ - name: outputBinding
+ type: OutputBinding?
+ jsonldPredicate: "cwl:outputBinding"
+ doc: |
+ Describes how to handle the outputs of a process.
+
+
+- type: record
+ name: ProcessRequirement
+ abstract: true
+ doc: |
+ A process requirement declares a prerequisite that may or must be fulfilled
+ before executing a process. See [`Process.hints`](#process) and
+ [`Process.requirements`](#process).
+
+ Process requirements are the primary mechanism for specifying extensions to
+ the CWL core specification.
+
+
+- type: record
+ name: Process
+ abstract: true
+ doc: |
+
+ The base executable type in CWL is the `Process` object defined by the
+ document. Note that the `Process` object is abstract and cannot be
+ directly executed.
+
+ fields:
+ - name: id
+ type: string?
+ jsonldPredicate: "@id"
+ doc: "The unique identifier for this process object."
+ - name: inputs
+ type:
+ type: array
+ items: InputParameter
+ jsonldPredicate:
+ _id: "cwl:inputs"
+ mapSubject: id
+ mapPredicate: type
+ doc: |
+ Defines the input parameters of the process. The process is ready to
+ run when all required input parameters are associated with concrete
+ values. Input parameters include a schema for each parameter which is
+ used to validate the input object. It may also be used to build a user
+ interface for constructing the input object.
+ - name: outputs
+ type:
+ type: array
+ items: OutputParameter
+ jsonldPredicate:
+ _id: "cwl:outputs"
+ mapSubject: id
+ mapPredicate: type
+ doc: |
+ Defines the parameters representing the output of the process. May be
+ used to generate and/or validate the output object.
+ - name: requirements
+ type: ProcessRequirement[]?
+ jsonldPredicate:
+ _id: "cwl:requirements"
+ mapSubject: class
+ doc: |
+ Declares requirements that apply to either the runtime environment or the
+ workflow engine that must be met in order to execute this process. If
+ an implementation cannot satisfy all requirements, or a requirement is
+ listed which is not recognized by the implementation, it is a fatal
+ error and the implementation must not attempt to run the process,
+ unless overridden at user option.
+ - name: hints
+ type: Any[]?
+ doc: |
+ Declares hints applying to either the runtime environment or the
+ workflow engine that may be helpful in executing this process. It is
+ not an error if an implementation cannot satisfy all hints, however
+ the implementation may report a warning.
+ jsonldPredicate:
+ _id: cwl:hints
+ noLinkCheck: true
+ mapSubject: class
+ - name: label
+ type: string?
+ jsonldPredicate: "rdfs:label"
+ doc: "A short, human-readable label of this process object."
+ - name: doc
+ type: string?
+ jsonldPredicate: "rdfs:comment"
+ doc: "A long, human-readable description of this process object."
+ - name: cwlVersion
+ type: CWLVersion?
+ doc: |
+ CWL document version. Always required at the document root. Not
+ required for a Process embedded inside another Process.
+ jsonldPredicate:
+ "_id": "cwl:cwlVersion"
+ "_type": "@vocab"
+
+- name: InlineJavascriptRequirement
+ type: record
+ extends: ProcessRequirement
+ doc: |
+ Indicates that the workflow platform must support inline Javascript expressions.
+ If this requirement is not present, the workflow platform must not perform expression
+ interpolatation.
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'InlineJavascriptRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: expressionLib
+ type: string[]?
+ doc: |
+ Additional code fragments that will also be inserted
+ before executing the expression code. Allows for function definitions that may
+ be called from CWL expressions.
+
+
+- name: SchemaDefRequirement
+ type: record
+ extends: ProcessRequirement
+ doc: |
+ This field consists of an array of type definitions which must be used when
+ interpreting the `inputs` and `outputs` fields. When a `type` field
+ contain a IRI, the implementation must check if the type is defined in
+ `schemaDefs` and use that definition. If the type is not found in
+ `schemaDefs`, it is an error. The entries in `schemaDefs` must be
+ processed in the order listed such that later schema definitions may refer
+ to earlier schema definitions.
+ fields:
+ - name: class
+ type: string
+ doc: "Always 'SchemaDefRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ - name: types
+ type:
+ type: array
+ items: InputSchema
+ doc: The list of type definitions.
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/Workflow.yml b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/Workflow.yml
new file mode 100644
index 000000000..6c2b06281
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/Workflow.yml
@@ -0,0 +1,583 @@
+$base: "https://w3id.org/cwl/cwl#"
+
+$namespaces:
+ sld: "https://w3id.org/cwl/salad#"
+ cwl: "https://w3id.org/cwl/cwl#"
+
+$graph:
+
+- name: "WorkflowDoc"
+ type: documentation
+ doc:
+ - |
+ # Common Workflow Language (CWL) Workflow Description, v1.0
+
+ This version:
+ * https://w3id.org/cwl/v1.0/
+
+ Current version:
+ * https://w3id.org/cwl/
+ - "\n\n"
+ - {$include: contrib.md}
+ - "\n\n"
+ - |
+ # Abstract
+
+ One way to define a workflow is: an analysis task represented by a
+ directed graph describing a sequence of operations that transform an
+ input data set to output. This specification defines the Common Workflow
+ Language (CWL) Workflow description, a vendor-neutral standard for
+ representing workflows intended to be portable across a variety of
+ computing platforms.
+
+ - {$include: intro.md}
+
+ - |
+
+ ## Introduction to v1.0
+
+ This specification represents the first full release from the CWL group.
+ Since draft-3, this draft introduces the following changes and additions:
+
+ * The `inputs` and `outputs` fields have been renamed `in` and `out`.
+ * Syntax simplifcations: denoted by the `map<>` syntax. Example: `in`
+ contains a list of items, each with an id. Now one can specify
+ a mapping of that identifier to the corresponding
+ `InputParameter`.
+ ```
+ in:
+ - id: one
+ type: string
+ doc: First input parameter
+ - id: two
+ type: int
+ doc: Second input parameter
+ ```
+ can be
+ ```
+ in:
+ one:
+ type: string
+ doc: First input parameter
+ two:
+ type: int
+ doc: Second input parameter
+ ```
+ * The common field `description` has been renamed to `doc`.
+
+ ## Purpose
+
+ The Common Workflow Language Command Line Tool Description express
+ workflows for data-intensive science, such as Bioinformatics, Chemistry,
+ Physics, and Astronomy. This specification is intended to define a data
+ and execution model for Workflows that can be implemented on top of a
+ variety of computing platforms, ranging from an individual workstation to
+ cluster, grid, cloud, and high performance computing systems.
+
+ - {$include: concepts.md}
+
+- name: ExpressionToolOutputParameter
+ type: record
+ extends: OutputParameter
+ fields:
+ - name: type
+ type:
+ - "null"
+ - "#CWLType"
+ - "#OutputRecordSchema"
+ - "#OutputEnumSchema"
+ - "#OutputArraySchema"
+ - string
+ - type: array
+ items:
+ - "#CWLType"
+ - "#OutputRecordSchema"
+ - "#OutputEnumSchema"
+ - "#OutputArraySchema"
+ - string
+ jsonldPredicate:
+ "_id": "sld:type"
+ "_type": "@vocab"
+ refScope: 2
+ typeDSL: True
+ doc: |
+ Specify valid types of data that may be assigned to this parameter.
+
+- type: record
+ name: ExpressionTool
+ extends: Process
+ specialize:
+ - specializeFrom: "#OutputParameter"
+ specializeTo: "#ExpressionToolOutputParameter"
+ documentRoot: true
+ doc: |
+ Execute an expression as a Workflow step.
+ fields:
+ - name: "class"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ type: string
+ - name: expression
+ type: [string, Expression]
+ doc: |
+ The expression to execute. The expression must return a JSON object which
+ matches the output parameters of the ExpressionTool.
+
+- name: LinkMergeMethod
+ type: enum
+ docParent: "#WorkflowStepInput"
+ doc: The input link merge method, described in [WorkflowStepInput](#WorkflowStepInput).
+ symbols:
+ - merge_nested
+ - merge_flattened
+
+
+- name: WorkflowOutputParameter
+ type: record
+ extends: OutputParameter
+ docParent: "#Workflow"
+ doc: |
+ Describe an output parameter of a workflow. The parameter must be
+ connected to one or more parameters defined in the workflow that will
+ provide the value of the output parameter.
+ fields:
+ - name: outputSource
+ doc: |
+ Specifies one or more workflow parameters that supply the value of to
+ the output parameter.
+ jsonldPredicate:
+ "_id": "cwl:outputSource"
+ "_type": "@id"
+ refScope: 0
+ type:
+ - string?
+ - string[]?
+ - name: linkMerge
+ type: ["null", "#LinkMergeMethod"]
+ jsonldPredicate: "cwl:linkMerge"
+ doc: |
+ The method to use to merge multiple sources into a single array.
+ If not specified, the default method is "merge_nested".
+ - name: type
+ type:
+ - "null"
+ - "#CWLType"
+ - "#OutputRecordSchema"
+ - "#OutputEnumSchema"
+ - "#OutputArraySchema"
+ - string
+ - type: array
+ items:
+ - "#CWLType"
+ - "#OutputRecordSchema"
+ - "#OutputEnumSchema"
+ - "#OutputArraySchema"
+ - string
+ jsonldPredicate:
+ "_id": "sld:type"
+ "_type": "@vocab"
+ refScope: 2
+ typeDSL: True
+ doc: |
+ Specify valid types of data that may be assigned to this parameter.
+
+
+- name: Sink
+ type: record
+ abstract: true
+ fields:
+ - name: source
+ doc: |
+ Specifies one or more workflow parameters that will provide input to
+ the underlying step parameter.
+ jsonldPredicate:
+ "_id": "cwl:source"
+ "_type": "@id"
+ refScope: 2
+ type:
+ - string?
+ - string[]?
+ - name: linkMerge
+ type: LinkMergeMethod?
+ jsonldPredicate: "cwl:linkMerge"
+ doc: |
+ The method to use to merge multiple inbound links into a single array.
+ If not specified, the default method is "merge_nested".
+
+
+- type: record
+ name: WorkflowStepInput
+ extends: Sink
+ docParent: "#WorkflowStep"
+ doc: |
+ The input of a workflow step connects an upstream parameter (from the
+ workflow inputs, or the outputs of other workflows steps) with the input
+ parameters of the underlying step.
+
+ ## Input object
+
+ A WorkflowStepInput object must contain an `id` field in the form
+ `#fieldname` or `#stepname.fieldname`. When the `id` field contains a
+ period `.` the field name consists of the characters following the final
+ period. This defines a field of the workflow step input object with the
+ value of the `source` parameter(s).
+
+ ## Merging
+
+ To merge multiple inbound data links,
+ [MultipleInputFeatureRequirement](#MultipleInputFeatureRequirement) must be specified
+ in the workflow or workflow step requirements.
+
+ If the sink parameter is an array, or named in a [workflow
+ scatter](#WorkflowStep) operation, there may be multiple inbound data links
+ listed in the `source` field. The values from the input links are merged
+ depending on the method specified in the `linkMerge` field. If not
+ specified, the default method is "merge_nested".
+
+ * **merge_nested**
+
+ The input must be an array consisting of exactly one entry for each
+ input link. If "merge_nested" is specified with a single link, the value
+ from the link must be wrapped in a single-item list.
+
+ * **merge_flattened**
+
+ 1. The source and sink parameters must be compatible types, or the source
+ type must be compatible with single element from the "items" type of
+ the destination array parameter.
+ 2. Source parameters which are arrays are concatenated.
+ Source parameters which are single element types are appended as
+ single elements.
+
+ fields:
+ - name: id
+ type: string
+ jsonldPredicate: "@id"
+ doc: "A unique identifier for this workflow input parameter."
+ - name: default
+ type: ["null", Any]
+ doc: |
+ The default value for this parameter if there is no `source`
+ field.
+ jsonldPredicate: "cwl:default"
+ - name: valueFrom
+ type:
+ - "null"
+ - "string"
+ - "#Expression"
+ jsonldPredicate: "cwl:valueFrom"
+ doc: |
+ To use valueFrom, [StepInputExpressionRequirement](#StepInputExpressionRequirement) must
+ be specified in the workflow or workflow step requirements.
+
+ If `valueFrom` is a constant string value, use this as the value for
+ this input parameter.
+
+ If `valueFrom` is a parameter reference or expression, it must be
+ evaluated to yield the actual value to be assiged to the input field.
+
+ The `self` value of in the parameter reference or expression must be
+ the value of the parameter(s) specified in the `source` field, or
+ null if there is no `source` field.
+
+ The value of `inputs` in the parameter reference or expression must be
+ the input object to the workflow step after assigning the `source`
+ values and then scattering. The order of evaluating `valueFrom` among
+ step input parameters is undefined and the result of evaluating
+ `valueFrom` on a parameter must not be visible to evaluation of
+ `valueFrom` on other parameters.
+
+
+- type: record
+ name: WorkflowStepOutput
+ docParent: "#WorkflowStep"
+ doc: |
+ Associate an output parameter of the underlying process with a workflow
+ parameter. The workflow parameter (given in the `id` field) be may be used
+ as a `source` to connect with input parameters of other workflow steps, or
+ with an output parameter of the process.
+ fields:
+ - name: id
+ type: string
+ jsonldPredicate: "@id"
+ doc: |
+ A unique identifier for this workflow output parameter. This is the
+ identifier to use in the `source` field of `WorkflowStepInput` to
+ connect the output value to downstream parameters.
+
+
+- name: ScatterMethod
+ type: enum
+ docParent: "#WorkflowStep"
+ doc: The scatter method, as described in [workflow step scatter](#WorkflowStep).
+ symbols:
+ - dotproduct
+ - nested_crossproduct
+ - flat_crossproduct
+
+
+- name: WorkflowStep
+ type: record
+ docParent: "#Workflow"
+ doc: |
+ A workflow step is an executable element of a workflow. It specifies the
+ underlying process implementation (such as `CommandLineTool` or another
+ `Workflow`) in the `run` field and connects the input and output parameters
+ of the underlying process to workflow parameters.
+
+ # Scatter/gather
+
+ To use scatter/gather,
+ [ScatterFeatureRequirement](#ScatterFeatureRequirement) must be specified
+ in the workflow or workflow step requirements.
+
+ A "scatter" operation specifies that the associated workflow step or
+ subworkflow should execute separately over a list of input elements. Each
+ job making up a scatter operation is independent and may be executed
+ concurrently.
+
+ The `scatter` field specifies one or more input parameters which will be
+ scattered. An input parameter may be listed more than once. The declared
+ type of each input parameter is implicitly wrapped in an array for each
+ time it appears in the `scatter` field. As a result, upstream parameters
+ which are connected to scattered parameters may be arrays.
+
+ All output parameter types are also implicitly wrapped in arrays. Each job
+ in the scatter results in an entry in the output array.
+
+ If `scatter` declares more than one input parameter, `scatterMethod`
+ describes how to decompose the input into a discrete set of jobs.
+
+ * **dotproduct** specifies that each of the input arrays are aligned and one
+ element taken from each array to construct each job. It is an error
+ if all input arrays are not the same length.
+
+ * **nested_crossproduct** specifies the Cartesian product of the inputs,
+ producing a job for every combination of the scattered inputs. The
+ output must be nested arrays for each level of scattering, in the
+ order that the input arrays are listed in the `scatter` field.
+
+ * **flat_crossproduct** specifies the Cartesian product of the inputs,
+ producing a job for every combination of the scattered inputs. The
+ output arrays must be flattened to a single level, but otherwise listed in the
+ order that the input arrays are listed in the `scatter` field.
+
+ # Subworkflows
+
+ To specify a nested workflow as part of a workflow step,
+ [SubworkflowFeatureRequirement](#SubworkflowFeatureRequirement) must be
+ specified in the workflow or workflow step requirements.
+
+ fields:
+ - name: id
+ type: string
+ jsonldPredicate: "@id"
+ doc: "The unique identifier for this workflow step."
+ - name: in
+ type: WorkflowStepInput[]
+ jsonldPredicate:
+ _id: "cwl:in"
+ mapSubject: id
+ mapPredicate: source
+ doc: |
+ Defines the input parameters of the workflow step. The process is ready to
+ run when all required input parameters are associated with concrete
+ values. Input parameters include a schema for each parameter which is
+ used to validate the input object. It may also be used build a user
+ interface for constructing the input object.
+ - name: out
+ type:
+ - type: array
+ items: [string, WorkflowStepOutput]
+ jsonldPredicate:
+ _id: "cwl:out"
+ _type: "@id"
+ identity: true
+ doc: |
+ Defines the parameters representing the output of the process. May be
+ used to generate and/or validate the output object.
+ - name: requirements
+ type: ProcessRequirement[]?
+ jsonldPredicate:
+ _id: "cwl:requirements"
+ mapSubject: class
+ doc: |
+ Declares requirements that apply to either the runtime environment or the
+ workflow engine that must be met in order to execute this workflow step. If
+ an implementation cannot satisfy all requirements, or a requirement is
+ listed which is not recognized by the implementation, it is a fatal
+ error and the implementation must not attempt to run the process,
+ unless overridden at user option.
+ - name: hints
+ type: Any[]?
+ jsonldPredicate:
+ _id: "cwl:hints"
+ noLinkCheck: true
+ mapSubject: class
+ doc: |
+ Declares hints applying to either the runtime environment or the
+ workflow engine that may be helpful in executing this workflow step. It is
+ not an error if an implementation cannot satisfy all hints, however
+ the implementation may report a warning.
+ - name: label
+ type: string?
+ jsonldPredicate: "rdfs:label"
+ doc: "A short, human-readable label of this process object."
+ - name: doc
+ type: string?
+ jsonldPredicate: "rdfs:comment"
+ doc: "A long, human-readable description of this process object."
+ - name: run
+ type: [string, Process]
+ jsonldPredicate:
+ "_id": "cwl:run"
+ "_type": "@id"
+ doc: |
+ Specifies the process to run.
+ - name: scatter
+ type:
+ - string?
+ - string[]?
+ jsonldPredicate:
+ "_id": "cwl:scatter"
+ "_type": "@id"
+ "_container": "@list"
+ refScope: 0
+ - name: scatterMethod
+ doc: |
+ Required if `scatter` is an array of more than one element.
+ type: ScatterMethod?
+ jsonldPredicate:
+ "_id": "cwl:scatterMethod"
+ "_type": "@vocab"
+
+
+- name: Workflow
+ type: record
+ extends: "#Process"
+ documentRoot: true
+ specialize:
+ - specializeFrom: "#OutputParameter"
+ specializeTo: "#WorkflowOutputParameter"
+ doc: |
+ A workflow describes a set of **steps** and the **dependencies** between
+ those steps. When a step produces output that will be consumed by a
+ second step, the first step is a dependency of the second step.
+
+ When there is a dependency, the workflow engine must execute the preceeding
+ step and wait for it to successfully produce output before executing the
+ dependent step. If two steps are defined in the workflow graph that
+ are not directly or indirectly dependent, these steps are **independent**,
+ and may execute in any order or execute concurrently. A workflow is
+ complete when all steps have been executed.
+
+ Dependencies between parameters are expressed using the `source` field on
+ [workflow step input parameters](#WorkflowStepInput) and [workflow output
+ parameters](#WorkflowOutputParameter).
+
+ The `source` field expresses the dependency of one parameter on another
+ such that when a value is associated with the parameter specified by
+ `source`, that value is propagated to the destination parameter. When all
+ data links inbound to a given step are fufilled, the step is ready to
+ execute.
+
+ ## Workflow success and failure
+
+ A completed step must result in one of `success`, `temporaryFailure` or
+ `permanentFailure` states. An implementation may choose to retry a step
+ execution which resulted in `temporaryFailure`. An implementation may
+ choose to either continue running other steps of a workflow, or terminate
+ immediately upon `permanentFailure`.
+
+ * If any step of a workflow execution results in `permanentFailure`, then
+ the workflow status is `permanentFailure`.
+
+ * If one or more steps result in `temporaryFailure` and all other steps
+ complete `success` or are not executed, then the workflow status is
+ `temporaryFailure`.
+
+ * If all workflow steps are executed and complete with `success`, then the
+ workflow status is `success`.
+
+ # Extensions
+
+ [ScatterFeatureRequirement](#ScatterFeatureRequirement) and
+ [SubworkflowFeatureRequirement](#SubworkflowFeatureRequirement) are
+ available as standard [extensions](#Extensions_and_Metadata) to core
+ workflow semantics.
+
+ fields:
+ - name: "class"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+ type: string
+ - name: steps
+ doc: |
+ The individual steps that make up the workflow. Each step is executed when all of its
+ input data links are fufilled. An implementation may choose to execute
+ the steps in a different order than listed and/or execute steps
+ concurrently, provided that dependencies between steps are met.
+ type:
+ - type: array
+ items: "#WorkflowStep"
+ jsonldPredicate:
+ mapSubject: id
+
+
+- type: record
+ name: SubworkflowFeatureRequirement
+ extends: ProcessRequirement
+ doc: |
+ Indicates that the workflow platform must support nested workflows in
+ the `run` field of [WorkflowStep](#WorkflowStep).
+ fields:
+ - name: "class"
+ type: "string"
+ doc: "Always 'SubworkflowFeatureRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+
+- name: ScatterFeatureRequirement
+ type: record
+ extends: ProcessRequirement
+ doc: |
+ Indicates that the workflow platform must support the `scatter` and
+ `scatterMethod` fields of [WorkflowStep](#WorkflowStep).
+ fields:
+ - name: "class"
+ type: "string"
+ doc: "Always 'ScatterFeatureRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+
+- name: MultipleInputFeatureRequirement
+ type: record
+ extends: ProcessRequirement
+ doc: |
+ Indicates that the workflow platform must support multiple inbound data links
+ listed in the `source` field of [WorkflowStepInput](#WorkflowStepInput).
+ fields:
+ - name: "class"
+ type: "string"
+ doc: "Always 'MultipleInputFeatureRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
+
+- type: record
+ name: StepInputExpressionRequirement
+ extends: ProcessRequirement
+ doc: |
+ Indicate that the workflow platform must support the `valueFrom` field
+ of [WorkflowStepInput](#WorkflowStepInput).
+ fields:
+ - name: "class"
+ type: "string"
+ doc: "Always 'StepInputExpressionRequirement'"
+ jsonldPredicate:
+ "_id": "@type"
+ "_type": "@vocab"
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/concepts.md b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/concepts.md
new file mode 100644
index 000000000..8b1378917
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/concepts.md
@@ -0,0 +1 @@
+
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/contrib.md b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/contrib.md
new file mode 100644
index 000000000..8b1378917
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/contrib.md
@@ -0,0 +1 @@
+
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/intro.md b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/intro.md
new file mode 100644
index 000000000..8b1378917
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/intro.md
@@ -0,0 +1 @@
+
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/invocation.md b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/invocation.md
new file mode 100644
index 000000000..8b1378917
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/invocation.md
@@ -0,0 +1 @@
+
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/metaschema_base.yml b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/metaschema_base.yml
new file mode 100644
index 000000000..73511d141
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/metaschema_base.yml
@@ -0,0 +1,164 @@
+$base: "https://w3id.org/cwl/salad#"
+
+$namespaces:
+ sld: "https://w3id.org/cwl/salad#"
+ dct: "http://purl.org/dc/terms/"
+ rdf: "http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+ rdfs: "http://www.w3.org/2000/01/rdf-schema#"
+ xsd: "http://www.w3.org/2001/XMLSchema#"
+
+$graph:
+- name: PrimitiveType
+ type: enum
+ symbols:
+ - "sld:null"
+ - "xsd:boolean"
+ - "xsd:int"
+ - "xsd:long"
+ - "xsd:float"
+ - "xsd:double"
+ - "xsd:string"
+ doc:
+ - |
+ Salad data types are based on Avro schema declarations. Refer to the
+ [Avro schema declaration documentation](https://avro.apache.org/docs/current/spec.html#schemas) for
+ detailed information.
+ - "null: no value"
+ - "boolean: a binary value"
+ - "int: 32-bit signed integer"
+ - "long: 64-bit signed integer"
+ - "float: single precision (32-bit) IEEE 754 floating-point number"
+ - "double: double precision (64-bit) IEEE 754 floating-point number"
+ - "string: Unicode character sequence"
+
+
+- name: Any
+ type: enum
+ symbols: ["#Any"]
+ doc: |
+ The **Any** type validates for any non-null value.
+
+
+- name: RecordField
+ type: record
+ doc: A field of a record.
+ fields:
+ - name: name
+ type: string
+ jsonldPredicate: "@id"
+ doc: |
+ The name of the field
+
+ - name: doc
+ type: string?
+ doc: |
+ A documentation string for this field
+ jsonldPredicate: "rdfs:comment"
+
+ - name: type
+ type:
+ - PrimitiveType
+ - RecordSchema
+ - EnumSchema
+ - ArraySchema
+ - string
+ - type: array
+ items:
+ - PrimitiveType
+ - RecordSchema
+ - EnumSchema
+ - ArraySchema
+ - string
+ jsonldPredicate:
+ _id: sld:type
+ _type: "@vocab"
+ typeDSL: true
+ refScope: 2
+ doc: |
+ The field type
+
+
+- name: RecordSchema
+ type: record
+ fields:
+ type:
+ doc: "Must be `record`"
+ type:
+ name: Record_symbol
+ type: enum
+ symbols:
+ - "sld:record"
+ jsonldPredicate:
+ _id: "sld:type"
+ _type: "@vocab"
+ typeDSL: true
+ refScope: 2
+ fields:
+ type: RecordField[]?
+ jsonldPredicate:
+ _id: sld:fields
+ mapSubject: name
+ mapPredicate: type
+ doc: "Defines the fields of the record."
+
+
+- name: EnumSchema
+ type: record
+ doc: |
+ Define an enumerated type.
+ fields:
+ type:
+ doc: "Must be `enum`"
+ type:
+ name: Enum_symbol
+ type: enum
+ symbols:
+ - "sld:enum"
+ jsonldPredicate:
+ _id: "sld:type"
+ _type: "@vocab"
+ typeDSL: true
+ refScope: 2
+ symbols:
+ type: string[]
+ jsonldPredicate:
+ _id: "sld:symbols"
+ _type: "@id"
+ identity: true
+ doc: "Defines the set of valid symbols."
+
+
+- name: ArraySchema
+ type: record
+ fields:
+ type:
+ doc: "Must be `array`"
+ type:
+ name: Array_symbol
+ type: enum
+ symbols:
+ - "sld:array"
+ jsonldPredicate:
+ _id: "sld:type"
+ _type: "@vocab"
+ typeDSL: true
+ refScope: 2
+ items:
+ type:
+ - PrimitiveType
+ - RecordSchema
+ - EnumSchema
+ - ArraySchema
+ - string
+ - type: array
+ items:
+ - PrimitiveType
+ - RecordSchema
+ - EnumSchema
+ - ArraySchema
+ - string
+ jsonldPredicate:
+ _id: "sld:items"
+ _type: "@vocab"
+ refScope: 2
+ doc: "Defines the type of the array elements."
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test1.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test1.cwl
new file mode 100644
index 000000000..2406c8648
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test1.cwl
@@ -0,0 +1 @@
+class: Workflow
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test10.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test10.cwl
new file mode 100644
index 000000000..28608072e
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test10.cwl
@@ -0,0 +1,10 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps:
+ step1:
+ scatterMethod: [record]
+ in: []
+ out: [out]
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test11.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test11.cwl
new file mode 100644
index 000000000..43281fbc5
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test11.cwl
@@ -0,0 +1,10 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps:
+ step1:
+ run: blub.cwl
+ in: []
+ out: [out]
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test12.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test12.cwl
new file mode 100644
index 000000000..d994e7cbc
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test12.cwl
@@ -0,0 +1,16 @@
+cwlVersion: v1.0
+class: CommandLineTool
+baseCommand: echo
+inputs:
+ - id: example_flag
+ type: boolean
+ inputBinding:
+ position: 1
+ prefix: -f
+ - id: example_flag
+ type: int
+ inputBinding:
+ position: 3
+ prefix: --example-string
+
+outputs: []
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test13.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test13.cwl
new file mode 100644
index 000000000..caa274d2a
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test13.cwl
@@ -0,0 +1,20 @@
+cwlVersion: v1.0
+class: Workflow
+inputs:
+ example_flag:
+ type: boolean
+ inputBinding:
+ position: 1
+ prefix: -f
+
+outputs: []
+
+steps:
+ example_flag:
+ in: []
+ out: []
+ run:
+ id: blah
+ class: CommandLineTool
+ inputs: []
+ outputs: []
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test14.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test14.cwl
new file mode 100644
index 000000000..729ee83df
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test14.cwl
@@ -0,0 +1,11 @@
+cwlVersion: v1.0
+class: CommandLineTool
+baseCommand: echo
+inputs:
+ example_flag:
+ type: boolean
+ inputBinding:
+ position: 1
+ prefix: -f
+outputs:
+ example_flag: int
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test15.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test15.cwl
new file mode 100755
index 000000000..0fc7f7efa
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test15.cwl
@@ -0,0 +1,13 @@
+#!/usr/bin/env cwl-runner
+
+cwlVersion: v1.0
+class: CommandLineTool
+baseCommand: echo
+inputs:
+ message:
+ type: string
+ inputBinding:
+ position: 1
+ invalid_field: it_is_invalid_field
+ another_invalid_field: invalid
+outputs: []
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test16.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test16.cwl
new file mode 100644
index 000000000..6bcfcc11c
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test16.cwl
@@ -0,0 +1,15 @@
+cwlVersion: v1.0
+class: CommandLineTool
+baseCommand: echo
+inputs:
+ message:
+ type: string
+ inputBinding:
+ position: 1
+ posi
+outputs:
+ hello_output:
+ type: File
+ outputBinding:
+ glob: hello-out.txt
+stdout: hello-out.txt
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test17.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test17.cwl
new file mode 100644
index 000000000..51cbbf1f3
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test17.cwl
@@ -0,0 +1,13 @@
+class: CommandLineTool
+cwlVersion: v1.0
+baseCommand: cowsay
+inputs:
+ - id: input
+ type: string?
+ inputBinding:
+ position: 0
+outputs:
+ - id: output
+ type: string?
+ outputBinding: {}
+ - aa: moa
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test18.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test18.cwl
new file mode 100644
index 000000000..e849a3d3f
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test18.cwl
@@ -0,0 +1,13 @@
+class: CommandLineTool
+cwlVersion: v1.0
+baseCommand: echo
+inputs:
+ - id: input
+ type: string?
+ inputBinding: {}
+outputs:
+ - id: output
+ type: string?
+ outputBinding: {}
+ - id: output1
+ type: Filea
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test19.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test19.cwl
new file mode 100644
index 000000000..a78d2876e
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test19.cwl
@@ -0,0 +1,15 @@
+: aaa
+cwlVersion: v1.0
+class: CommandLineTool
+baseCommand: echo
+inputs:
+ message:
+ type: string
+ inputBinding:
+ position: 1
+outputs:
+ hello_output:
+ type: File
+ outputBinding:
+ glob: hello-out.txt
+stdout: hello-out.txt
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test2.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test2.cwl
new file mode 100644
index 000000000..96ae14014
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test2.cwl
@@ -0,0 +1 @@
+class: xWorkflow
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test3.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test3.cwl
new file mode 100644
index 000000000..517e920d2
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test3.cwl
@@ -0,0 +1,6 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: xstring
+steps: []
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test4.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test4.cwl
new file mode 100644
index 000000000..e57292d36
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test4.cwl
@@ -0,0 +1,6 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: 12
+steps: []
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test5.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test5.cwl
new file mode 100644
index 000000000..8a7ba2220
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test5.cwl
@@ -0,0 +1,6 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps: [12]
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test6.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test6.cwl
new file mode 100644
index 000000000..eff4ac5c7
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test6.cwl
@@ -0,0 +1,5 @@
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps: [12]
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test7.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test7.cwl
new file mode 100644
index 000000000..0e12c1295
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test7.cwl
@@ -0,0 +1,10 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps:
+ step1:
+ scatter_method: blub
+ in: []
+ out: [out]
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test8.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test8.cwl
new file mode 100644
index 000000000..128cb4a75
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test8.cwl
@@ -0,0 +1,10 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps:
+ step1:
+ scatterMethod: abc
+ in: []
+ out: [out]
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test9.cwl b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test9.cwl
new file mode 100644
index 000000000..2d7ff4cd9
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/test_schema/test9.cwl
@@ -0,0 +1,10 @@
+class: Workflow
+inputs:
+ foo: string
+outputs:
+ bar: string
+steps:
+ step1:
+ scatterMethod: 12
+ in: []
+ out: [out]
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/schema_salad/tests/util.py b/v1.1.0-dev1/salad/schema_salad/tests/util.py
new file mode 100644
index 000000000..110daa5cb
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/tests/util.py
@@ -0,0 +1,16 @@
+from __future__ import absolute_import
+from pkg_resources import Requirement, resource_filename, ResolutionError # type: ignore
+from typing import Optional, Text
+import os
+
+def get_data(filename): # type: (Text) -> Optional[Text]
+ filename = os.path.normpath(filename) # normalizing path depending on OS or else it will cause problem when joining path
+ filepath = None
+ try:
+ filepath = resource_filename(
+ Requirement.parse("schema-salad"), filename)
+ except ResolutionError:
+ pass
+ if not filepath or not os.path.isfile(filepath):
+ filepath = os.path.join(os.path.dirname(__file__), os.pardir, filename)
+ return filepath
diff --git a/v1.1.0-dev1/salad/schema_salad/utils.py b/v1.1.0-dev1/salad/schema_salad/utils.py
new file mode 100644
index 000000000..438bab588
--- /dev/null
+++ b/v1.1.0-dev1/salad/schema_salad/utils.py
@@ -0,0 +1,46 @@
+from __future__ import absolute_import
+import os
+from typing import Any, Dict, List
+
+
+def add_dictlist(di, key, val): # type: (Dict, Any, Any) -> None
+ if key not in di:
+ di[key] = []
+ di[key].append(val)
+
+
+def aslist(l): # type: (Any) -> List
+ """Convenience function to wrap single items and lists, and return lists unchanged."""
+
+ if isinstance(l, list):
+ return l
+ else:
+ return [l]
+
+# http://rightfootin.blogspot.com/2006/09/more-on-python-flatten.html
+
+def flatten(l, ltypes=(list, tuple)):
+ # type: (Any, Any) -> Any
+ if l is None:
+ return []
+ if not isinstance(l, ltypes):
+ return [l]
+
+ ltype = type(l)
+ lst = list(l)
+ i = 0
+ while i < len(lst):
+ while isinstance(lst[i], ltypes):
+ if not lst[i]:
+ lst.pop(i)
+ i -= 1
+ break
+ else:
+ lst[i:i + 1] = lst[i]
+ i += 1
+ return ltype(lst)
+
+# Check if we are on windows OS
+def onWindows():
+ # type: () -> (bool)
+ return os.name == 'nt'
diff --git a/v1.1.0-dev1/salad/schema_salad/validate.py b/v1.1.0-dev1/salad/schema_salad/validate.py
index 90b02e97f..ed8b8a306 100644
--- a/v1.1.0-dev1/salad/schema_salad/validate.py
+++ b/v1.1.0-dev1/salad/schema_salad/validate.py
@@ -1,29 +1,45 @@
+from __future__ import absolute_import
import pprint
import avro.schema
+from avro.schema import Schema
import sys
-import urlparse
-from typing import Any
+import re
+import logging
+
+import six
+from six.moves import urllib
+from six.moves import range
+
+from typing import Any, List, Set, Union, Text
+from .sourceline import SourceLine, lineno_re, bullets, indent
+
+_logger = logging.getLogger("salad")
class ValidationException(Exception):
pass
-def validate(expected_schema, datum, identifiers=set(), strict=False, foreign_properties=set()):
- # type: (avro.schema.Schema, Any, Set[unicode], bool, Set[unicode]) -> bool
- try:
- return validate_ex(expected_schema, datum, identifiers, strict=strict, foreign_properties=foreign_properties)
- except ValidationException:
- return False
+
+class ClassValidationException(ValidationException):
+ pass
+
+
+def validate(expected_schema, # type: Schema
+ datum, # type: Any
+ identifiers=[], # type: List[Text]
+ strict=False, # type: bool
+ foreign_properties=set() # type: Set[Text]
+ ):
+ # type: (...) -> bool
+ return validate_ex(
+ expected_schema, datum, identifiers, strict=strict,
+ foreign_properties=foreign_properties, raise_ex=False)
+
INT_MIN_VALUE = -(1 << 31)
INT_MAX_VALUE = (1 << 31) - 1
LONG_MIN_VALUE = -(1 << 63)
LONG_MAX_VALUE = (1 << 63) - 1
-def indent(v, nolead=False): # type: (str, bool) -> str
- if nolead:
- return v.splitlines()[0] + "\n".join([" " + l for l in v.splitlines()[1:]])
- else:
- return "\n".join([" " + l for l in v.splitlines()])
def friendly(v): # type: (Any) -> Any
if isinstance(v, avro.schema.NamedSchema):
@@ -37,11 +53,6 @@ def friendly(v): # type: (Any) -> Any
else:
return v
-def multi(v, q=""): # type: (str, str) -> str
- if '\n' in v:
- return "%s%s%s\n" % (q, v, q)
- else:
- return "%s%s%s" % (q, v, q)
def vpformat(datum): # type: (Any) -> str
a = pprint.pformat(datum)
@@ -49,13 +60,22 @@ def vpformat(datum): # type: (Any) -> str
a = a[0:160] + "[...]"
return a
-def validate_ex(expected_schema, datum, identifiers=None, strict=False,
- foreign_properties=None):
- # type: (avro.schema.Schema, Any, Set[unicode], bool, Set[unicode]) -> bool
+
+def validate_ex(expected_schema, # type: Schema
+ datum, # type: Any
+ identifiers=None, # type: List[Text]
+ strict=False, # type: bool
+ foreign_properties=None, # type: Set[Text]
+ raise_ex=True, # type: bool
+ strict_foreign_properties=False, # type: bool
+ logger=_logger, # type: logging.Logger
+ skip_foreign_properties=False # type: bool
+ ):
+ # type: (...) -> bool
"""Determine if a python datum is an instance of a schema."""
if not identifiers:
- identifiers = set()
+ identifiers = []
if not foreign_properties:
foreign_properties = set()
@@ -66,93 +86,190 @@ def validate_ex(expected_schema, datum, identifiers=None, strict=False,
if datum is None:
return True
else:
- raise ValidationException("the value `%s` is not null" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(u"the value is not null")
+ else:
+ return False
elif schema_type == 'boolean':
if isinstance(datum, bool):
return True
else:
- raise ValidationException("the value `%s` is not boolean" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(u"the value is not boolean")
+ else:
+ return False
elif schema_type == 'string':
- if isinstance(datum, basestring):
+ if isinstance(datum, six.string_types):
return True
elif isinstance(datum, bytes):
- datum = datum.decode("utf-8")
+ datum = datum.decode(u"utf-8")
return True
else:
- raise ValidationException("the value `%s` is not string" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(u"the value is not string")
+ else:
+ return False
elif schema_type == 'bytes':
if isinstance(datum, str):
return True
else:
- raise ValidationException("the value `%s` is not bytes" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(
+ u"the value `%s` is not bytes" % vpformat(datum))
+ else:
+ return False
elif schema_type == 'int':
- if ((isinstance(datum, int) or isinstance(datum, long))
- and INT_MIN_VALUE <= datum <= INT_MAX_VALUE):
+ if (isinstance(datum, six.integer_types)
+ and INT_MIN_VALUE <= datum <= INT_MAX_VALUE):
return True
else:
- raise ValidationException("`%s` is not int" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(u"`%s` is not int" % vpformat(datum))
+ else:
+ return False
elif schema_type == 'long':
- if ((isinstance(datum, int) or isinstance(datum, long))
- and LONG_MIN_VALUE <= datum <= LONG_MAX_VALUE):
+ if ((isinstance(datum, six.integer_types))
+ and LONG_MIN_VALUE <= datum <= LONG_MAX_VALUE):
return True
else:
- raise ValidationException("the value `%s` is not long" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(
+ u"the value `%s` is not long" % vpformat(datum))
+ else:
+ return False
elif schema_type in ['float', 'double']:
- if (isinstance(datum, int) or isinstance(datum, long)
- or isinstance(datum, float)):
- return True
- else:
- raise ValidationException("the value `%s` is not float or double" % vpformat(datum))
- elif isinstance(expected_schema, avro.schema.FixedSchema):
- if isinstance(datum, str) and len(datum) == expected_schema.size:
+ if (isinstance(datum, six.integer_types)
+ or isinstance(datum, float)):
return True
else:
- raise ValidationException("the value `%s` is not fixed" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(
+ u"the value `%s` is not float or double" % vpformat(datum))
+ else:
+ return False
elif isinstance(expected_schema, avro.schema.EnumSchema):
if expected_schema.name == "Any":
if datum is not None:
return True
else:
- raise ValidationException("Any type must be non-null")
+ if raise_ex:
+ raise ValidationException(u"'Any' type must be non-null")
+ else:
+ return False
+ if not isinstance(datum, six.string_types):
+ if raise_ex:
+ raise ValidationException(
+ u"value is a %s but expected a string" % (type(datum).__name__))
+ else:
+ return False
if datum in expected_schema.symbols:
return True
else:
- raise ValidationException("the value `%s`\n is not a valid symbol in enum %s, expected one of %s" % (vpformat(datum), expected_schema.name, "'" + "', '".join(expected_schema.symbols) + "'"))
+ if raise_ex:
+ raise ValidationException(u"the value %s is not a valid %s, expected %s%s" % (vpformat(datum), expected_schema.name,
+ "one of " if len(
+ expected_schema.symbols) > 1 else "",
+ "'" + "', '".join(expected_schema.symbols) + "'"))
+ else:
+ return False
elif isinstance(expected_schema, avro.schema.ArraySchema):
if isinstance(datum, list):
for i, d in enumerate(datum):
try:
- validate_ex(expected_schema.items, d, identifiers, strict=strict, foreign_properties=foreign_properties)
+ sl = SourceLine(datum, i, ValidationException)
+ if not validate_ex(expected_schema.items, d, identifiers,
+ strict=strict,
+ foreign_properties=foreign_properties,
+ raise_ex=raise_ex,
+ strict_foreign_properties=strict_foreign_properties,
+ logger=logger,
+ skip_foreign_properties=skip_foreign_properties):
+ return False
except ValidationException as v:
- raise ValidationException("At position %i\n%s" % (i, indent(str(v))))
- return True
- else:
- raise ValidationException("the value `%s` is not a list, expected list of %s" % (vpformat(datum), friendly(expected_schema.items)))
- elif isinstance(expected_schema, avro.schema.MapSchema):
- if (isinstance(datum, dict) and
- False not in [isinstance(k, basestring) for k in datum.keys()] and
- False not in [validate(expected_schema.values, v, strict=strict) for v in datum.values()]):
+ if raise_ex:
+ raise sl.makeError(
+ six.text_type("item is invalid because\n%s" % (indent(str(v)))))
+ else:
+ return False
return True
else:
- raise ValidationException("`%s` is not a valid map value, expected\n %s" % (vpformat(datum), vpformat(expected_schema.values)))
+ if raise_ex:
+ raise ValidationException(u"the value %s is not a list, expected list of %s" % (
+ vpformat(datum), friendly(expected_schema.items)))
+ else:
+ return False
elif isinstance(expected_schema, avro.schema.UnionSchema):
- if True in [validate(s, datum, identifiers, strict=strict) for s in expected_schema.schemas]:
- return True
+ for s in expected_schema.schemas:
+ if validate_ex(s, datum, identifiers, strict=strict, raise_ex=False,
+ strict_foreign_properties=strict_foreign_properties,
+ logger=logger, skip_foreign_properties=skip_foreign_properties):
+ return True
+
+ if not raise_ex:
+ return False
+
+ errors = [] # type: List[Text]
+ checked = []
+ for s in expected_schema.schemas:
+ if isinstance(datum, list) and not isinstance(s, avro.schema.ArraySchema):
+ continue
+ elif isinstance(datum, dict) and not isinstance(s, avro.schema.RecordSchema):
+ continue
+ elif (isinstance(datum, (bool, six.integer_types, float, six.string_types)) and # type: ignore
+ isinstance(s, (avro.schema.ArraySchema, avro.schema.RecordSchema))):
+ continue
+ elif datum is not None and s.type == "null":
+ continue
+
+ checked.append(s)
+ try:
+ validate_ex(s, datum, identifiers, strict=strict,
+ foreign_properties=foreign_properties,
+ raise_ex=True,
+ strict_foreign_properties=strict_foreign_properties,
+ logger=logger, skip_foreign_properties=skip_foreign_properties)
+ except ClassValidationException as e:
+ raise
+ except ValidationException as e:
+ errors.append(six.text_type(e))
+ if bool(errors):
+ raise ValidationException(bullets(["tried %s but\n%s" % (friendly(
+ checked[i]), indent(errors[i])) for i in range(0, len(errors))], "- "))
else:
- errors = []
- for s in expected_schema.schemas:
- try:
- validate_ex(s, datum, identifiers, strict=strict, foreign_properties=foreign_properties)
- except ValidationException as e:
- errors.append(str(e))
- raise ValidationException("the value %s is not a valid type in the union, expected one of:\n%s" % (multi(vpformat(datum), '`'), "\n".join(["- %s, but\n %s" % (friendly(expected_schema.schemas[i]), indent(multi(errors[i]))) for i in range(0, len(expected_schema.schemas))])))
+ raise ValidationException("value is a %s, expected %s" % (
+ type(datum).__name__, friendly(expected_schema)))
elif isinstance(expected_schema, avro.schema.RecordSchema):
if not isinstance(datum, dict):
- raise ValidationException("`%s`\n is not a dict" % vpformat(datum))
+ if raise_ex:
+ raise ValidationException(u"is not a dict")
+ else:
+ return False
+
+ classmatch = None
+ for f in expected_schema.fields:
+ if f.name in ("class",):
+ d = datum.get(f.name)
+ if not d:
+ if raise_ex:
+ raise ValidationException(
+ u"Missing '%s' field" % (f.name))
+ else:
+ return False
+ if expected_schema.name != d:
+ if raise_ex:
+ raise ValidationException(
+ u"Expected class '%s' but this is '%s'" % (expected_schema.name, d))
+ else:
+ return False
+ classmatch = d
+ break
errors = []
for f in expected_schema.fields:
+ if f.name in ("class",):
+ continue
+
if f.name in datum:
fieldval = datum[f.name]
else:
@@ -162,28 +279,62 @@ def validate_ex(expected_schema, datum, identifiers=None, strict=False,
fieldval = None
try:
- validate_ex(f.type, fieldval, identifiers, strict=strict, foreign_properties=foreign_properties)
+ sl = SourceLine(datum, f.name, six.text_type)
+ if not validate_ex(f.type, fieldval, identifiers, strict=strict,
+ foreign_properties=foreign_properties,
+ raise_ex=raise_ex,
+ strict_foreign_properties=strict_foreign_properties,
+ logger=logger, skip_foreign_properties=skip_foreign_properties):
+ return False
except ValidationException as v:
if f.name not in datum:
- errors.append("missing required field `%s`" % f.name)
+ errors.append(u"missing required field `%s`" % f.name)
else:
- errors.append("could not validate field `%s` because\n%s" % (f.name, multi(indent(str(v)))))
- if strict:
- for d in datum:
- found = False
- for f in expected_schema.fields:
- if d == f.name:
- found = True
- if not found:
- if d not in identifiers and d not in foreign_properties and d[0] not in ("@", "$"):
- split = urlparse.urlsplit(d)
- if split.scheme:
- errors.append("could not validate extension field `%s` because it is not recognized and strict is True. Did you include a $schemas section?" % (d))
+ errors.append(sl.makeError(u"the `%s` field is not valid because\n%s" % (
+ f.name, indent(str(v)))))
+
+ for d in datum:
+ found = False
+ for f in expected_schema.fields:
+ if d == f.name:
+ found = True
+ if not found:
+ sl = SourceLine(datum, d, six.text_type)
+ if d not in identifiers and d not in foreign_properties and d[0] not in ("@", "$"):
+ if (d not in identifiers and strict) and (
+ d not in foreign_properties and strict_foreign_properties) and not raise_ex:
+ return False
+ split = urllib.parse.urlsplit(d)
+ if split.scheme:
+ if not skip_foreign_properties:
+ err = sl.makeError(u"unrecognized extension field `%s`%s."
+ " Did you include "
+ "a $schemas section?" % (
+ d, " and strict_foreign_properties is True" if strict_foreign_properties else ""))
+ if strict_foreign_properties:
+ errors.append(err)
+ else:
+ logger.warn(err)
+ logger.warn("foreign properties %s", foreign_properties)
+ else:
+ err = sl.makeError(u"invalid field `%s`, expected one of: %s" % (
+ d, ", ".join("'%s'" % fn.name for fn in expected_schema.fields)))
+ if strict:
+ errors.append(err)
else:
- errors.append("could not validate field `%s` because it is not recognized and strict is True, valid fields are: %s" % (d, ", ".join(fn.name for fn in expected_schema.fields)))
+ logger.warn(err)
- if errors:
- raise ValidationException("\n".join(errors))
+ if bool(errors):
+ if raise_ex:
+ if classmatch:
+ raise ClassValidationException(bullets(errors, "* "))
+ else:
+ raise ValidationException(bullets(errors, "* "))
+ else:
+ return False
else:
return True
- raise ValidationException("Unrecognized schema_type %s" % schema_type)
+ if raise_ex:
+ raise ValidationException(u"Unrecognized schema_type %s" % schema_type)
+ else:
+ return False
diff --git a/v1.1.0-dev1/salad/setup.cfg b/v1.1.0-dev1/salad/setup.cfg
index a559cd8c3..903b12df4 100644
--- a/v1.1.0-dev1/salad/setup.cfg
+++ b/v1.1.0-dev1/salad/setup.cfg
@@ -7,5 +7,5 @@ universal = 1
[aliases]
test=pytest
-[pytest]
+[tool:pytest]
addopts=--pyarg schema_salad
diff --git a/v1.1.0-dev1/salad/setup.py b/v1.1.0-dev1/salad/setup.py
index a67730968..4179b41c5 100755
--- a/v1.1.0-dev1/salad/setup.py
+++ b/v1.1.0-dev1/salad/setup.py
@@ -1,11 +1,10 @@
#!/usr/bin/env python
+# -*- coding: utf-8 -*-
import os
import sys
-import shutil
import setuptools.command.egg_info as egg_info_cmd
-
from setuptools import setup, find_packages
SETUP_DIR = os.path.dirname(__file__)
@@ -28,25 +27,24 @@
requirements = []
install_requires = [
- 'requests',
- 'ruamel.yaml == 0.12.4',
- 'rdflib >= 4.1.0',
- 'rdflib-jsonld >= 0.3.0',
- 'mistune',
- 'typing >= 3.5.2',
- 'CacheControl',
- 'lockfile']
-
-install_requires.append("avro") # TODO: remove me once cwltool is
-# available in Debian Stable, Ubuntu 12.04 LTS
+ 'setuptools',
+ 'requests >= 1.0',
+ 'ruamel.yaml >= 0.12.4, < 0.15',
+ 'rdflib >= 4.2.2, < 4.3.0',
+ 'rdflib-jsonld >= 0.3.0, < 0.5.0',
+ 'mistune >= 0.7.3, < 0.8',
+ 'typing >= 3.5.3',
+ 'CacheControl >= 0.11.7, < 0.12',
+ 'lockfile >= 0.9',
+ 'six >= 1.8.0']
-# extras_require={ # TODO: uncomment me, same conditions as above
-# ':python_version<"3"': ['avro'],
-# ':python_version>="3"': ['avro-python3']}
-extras_require = {} # TODO: to be removed when the above is added
+extras_require={
+ ':python_version<"3"': ['avro == 1.8.1'],
+ ':python_version>="3"': ['future', 'avro-cwl == 1.8.4'] # fork of avro for working with python3
+}
setup(name='schema-salad',
- version='1.17',
+ version='2.7', # update the VERSION prefix in the Makefile as well 🙂
description='Schema Annotations for Linked Avro Data (SALAD)',
long_description=open(README).read(),
author='Common workflow language working group',
@@ -63,19 +61,21 @@
test_suite='tests',
tests_require=['pytest'],
entry_points={
- 'console_scripts': ["schema-salad-tool=schema_salad.main:main"]
+ 'console_scripts': ["schema-salad-tool=schema_salad.main:main", "schema-salad-doc=schema_salad.makedoc:main"]
},
zip_safe=True,
cmdclass={'egg_info': tagger},
classifiers=[
"Environment :: Console",
"Intended Audience :: Science/Research",
- "Operating System :: POSIX :: Linux",
+ "License :: OSI Approved :: Apache Software License",
+ "Operating System :: POSIX",
"Operating System :: MacOS :: MacOS X",
- "Development Status :: 4 - Beta",
+ "Operating System :: Microsoft :: Windows",
+ "Development Status :: 5 - Production/Stable",
"Programming Language :: Python :: 2.7",
- #"Programming Language :: Python :: 3.3", # TODO: uncomment these
- #"Programming Language :: Python :: 3.4", # lines
- #"Programming Language :: Python :: 3.5"
+ "Programming Language :: Python :: 3.4",
+ "Programming Language :: Python :: 3.5",
+ "Programming Language :: Python :: 3.6"
]
)
diff --git a/v1.1.0-dev1/salad/tox.ini b/v1.1.0-dev1/salad/tox.ini
index acd2ee831..44fa3c2f8 100644
--- a/v1.1.0-dev1/salad/tox.ini
+++ b/v1.1.0-dev1/salad/tox.ini
@@ -1,51 +1,48 @@
[tox]
-#envlist = py35-lint,py34-lint,py33-lint,py27-lint,py35-unit,py34-unit,py33-unit,py27-unit
-envlist = py27-lint, py27-unit, py35-mypy
+envlist =
+ py{27,34,35,36}-lint,
+ py{27,34,35,36}-unit,
+ py35-mypy{2,3},
+ py27-pipconflictchecker
+
skipsdist = True
+skip_missing_interpreters = True
-[tox:travis]
-2.7 = py27
-3.5 = py35-mypy
+[travis]
+python=
+ 2.7: py27
+ 3.5: py35
+ 3.6: py36
[testenv]
-deps = -rrequirements.txt
-
-[testenv:py35-mypy]
-commands = make mypy
-whitelist_externals = make
deps =
- mypy-lang>=0.4
- typed-ast
-rrequirements.txt
+ py{27,34,35,36}-lint: flake8
-[testenv:py35-lint]
-commands = flake8 schema_salad setup.py
-whitelist_externals = flake8
-deps = flake8
-
-[testenv:py34-lint]
-commands = flake8 schema_salad setup.py
-whitelist_externals = flake8
-deps = flake8
-
-[testenv:py33-lint]
-commands = flake8 schema_salad setup.py
-whitelist_externals = flake8
-deps = flake8
-
-[testenv:py27-lint]
-commands = flake8 schema_salad setup.py
-whitelist_externals = flake8
-deps = flake8
+commands =
+ py{27,34,35,36}-unit: python setup.py test
+ py{27,34,35,36}-lint: flake8 schema_salad setup.py
-[testenv:py35-unit]
-commands = python setup.py test
+whitelist_externals =
+ py{27,34,35,36}-lint: flake8
-[testenv:py34-unit]
-commands = python setup.py test
+[testenv:py35-mypy2]
+commands =
+ make mypy2
+whitelist_externals = make
+deps =
+ -rmypy_requirements.txt
+ -rrequirements.txt
-[testenv:py33-unit]
-commands = python setup.py test
+[testenv:py35-mypy3]
+commands =
+ make mypy3
+whitelist_externals = make
+deps =
+ -rmypy_requirements.txt
+ -rrequirements.txt
-[testenv:py27-unit]
-commands = python setup.py test
+[testenv:py27-pipconflictchecker]
+commands = pipconflictchecker
+whitelist_externals = pipconflictchecker
+deps = pip-conflict-checker
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/argparse.pyi b/v1.1.0-dev1/salad/typeshed/2.7/argparse.pyi
deleted file mode 100644
index 13a7b15e5..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/argparse.pyi
+++ /dev/null
@@ -1,176 +0,0 @@
-# Stubs for argparse (Python 2)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any, Callable, Dict, List, IO, Iterable, Sequence, Union
-
-SUPPRESS = ... # type: Any
-OPTIONAL = ... # type: Any
-ZERO_OR_MORE = ... # type: Any
-ONE_OR_MORE = ... # type: Any
-PARSER = ... # type: Any
-REMAINDER = ... # type: Any
-
-class _AttributeHolder: ...
-
-class HelpFormatter:
- def __init__(self, prog, indent_increment=..., max_help_position=..., width=...) -> None: ...
- class _Section:
- formatter = ... # type: Any
- parent = ... # type: Any
- heading = ... # type: Any
- items = ... # type: Any
- def __init__(self, formatter, parent, heading=...) -> None: ...
- def format_help(self): ...
- def start_section(self, heading): ...
- def end_section(self): ...
- def add_text(self, text): ...
- def add_usage(self, usage, actions, groups, prefix=...): ...
- def add_argument(self, action): ...
- def add_arguments(self, actions): ...
- def format_help(self): ...
-
-class RawDescriptionHelpFormatter(HelpFormatter): ...
-class RawTextHelpFormatter(RawDescriptionHelpFormatter): ...
-class ArgumentDefaultsHelpFormatter(HelpFormatter): ...
-
-class ArgumentError(Exception):
- argument_name = ... # type: Any
- message = ... # type: Any
- def __init__(self, argument, message) -> None: ...
-
-class ArgumentTypeError(Exception): ...
-
-class Action(_AttributeHolder):
- option_strings = ... # type: Any
- dest = ... # type: Any
- nargs = ... # type: Any
- const = ... # type: Any
- default = ... # type: Any
- type = ... # type: Any
- choices = ... # type: Any
- required = ... # type: Any
- help = ... # type: Any
- metavar = ... # type: Any
- def __init__(self, option_strings: List[str], dest=str, nargs: Union[int, str]=..., const: Any =..., default: Any =..., type: Callable[[str], Any] =..., choices: Iterable[Any] =..., required: bool=..., help: str=..., metavar: str =...) -> None: ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _StoreAction(Action):
- def __init__(self, option_strings, dest, nargs=..., const=..., default=..., type=...,
- choices=..., required=..., help=..., metavar=...): ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _StoreConstAction(Action):
- def __init__(self, option_strings, dest, const, default=..., required=..., help=...,
- metavar=...): ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _StoreTrueAction(_StoreConstAction):
- def __init__(self, option_strings, dest, default=..., required=..., help=...) -> None: ...
-
-class _StoreFalseAction(_StoreConstAction):
- def __init__(self, option_strings, dest, default=..., required=..., help=...) -> None: ...
-
-class _AppendAction(Action):
- def __init__(self, option_strings, dest, nargs=..., const=..., default=..., type=...,
- choices=..., required=..., help=..., metavar=...): ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _AppendConstAction(Action):
- def __init__(self, option_strings, dest, const, default=..., required=..., help=...,
- metavar=...): ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _CountAction(Action):
- def __init__(self, option_strings, dest, default=..., required=..., help=...) -> None: ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _HelpAction(Action):
- def __init__(self, option_strings, dest=..., default=..., help=...) -> None: ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _VersionAction(Action):
- version = ... # type: Any
- def __init__(self, option_strings, version=..., dest=..., default=..., help=...) -> None: ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class _SubParsersAction(Action):
- class _ChoicesPseudoAction(Action):
- def __init__(self, name, help) -> None: ...
- def __init__(self, option_strings, prog, parser_class, dest=..., help=..., metavar=...) -> None: ...
- def add_parser(self, name, **kwargs): ...
- def __call__(self, parser, namespace, values, option_string=...): ...
-
-class FileType:
- def __init__(self, mode=..., bufsize=...) -> None: ...
- def __call__(self, string): ...
-
-class Namespace(_AttributeHolder):
- def __init__(self, **kwargs) -> None: ...
- __hash__ = ... # type: Any
- def __eq__(self, other): ...
- def __ne__(self, other): ...
- def __contains__(self, key): ...
- def __getattr__(self, name: str) -> Any: ...
-
-class _ActionsContainer:
- description = ... # type: Any
- argument_default = ... # type: Any
- prefix_chars = ... # type: Any
- conflict_handler = ... # type: Any
- def __init__(self, description, prefix_chars, argument_default, conflict_handler) -> None: ...
- def register(self, registry_name, value, object): ...
- def set_defaults(self, **kwargs): ...
- def get_default(self, dest): ...
- def add_argument(self,
- *args: Union[str, unicode],
- action: Union[str, Action] = ...,
- nargs: str = ...,
- const: Any = ...,
- default: Any = ...,
- type: Any = ...,
- choices: Any = ..., # TODO: Container?
- required: bool = ...,
- help: str = ...,
- metavar: str = ...,
- dest: str = ...,
- version: str = ...
- ) -> None: ...
- def add_argument_group(self, *args, **kwargs): ...
- def add_mutually_exclusive_group(self, **kwargs) -> _MutuallyExclusiveGroup: ...
-
-class _ArgumentGroup(_ActionsContainer):
- title = ... # type: Any
- def __init__(self, container, title=..., description=..., **kwargs) -> None: ...
-
-class _MutuallyExclusiveGroup(_ArgumentGroup):
- required = ... # type: Any
- def __init__(self, container, required=...) -> None: ...
-
-class ArgumentParser(_AttributeHolder, _ActionsContainer):
- prog = ... # type: Any
- usage = ... # type: Any
- epilog = ... # type: Any
- version = ... # type: Any
- formatter_class = ... # type: Any
- fromfile_prefix_chars = ... # type: Any
- add_help = ... # type: Any
- def __init__(self, prog: str=..., usage: str=..., description: str=...,
- epilog: str=..., version: None=...,
- parents: Iterable[ArgumentParser]=...,
- formatter_class: HelpFormatter=..., prefix_chars: str=...,
- fromfile_prefix_chars: str=...,
- argument_default: str=..., conflict_handler: str=...,
- add_help: bool=...) -> None: ...
- def add_subparsers(self, **kwargs): ...
- def parse_args(self, args: Sequence[str] = ..., namespace=...): ...
- def parse_known_args(self, args=..., namespace=...): ...
- def convert_arg_line_to_args(self, arg_line): ...
- def format_usage(self): ...
- def format_help(self): ...
- def format_version(self): ...
- def print_usage(self, file=...): ...
- def print_help(self, file: IO[Any] = None) -> None: ...
- def print_version(self, file=...): ...
- def exit(self, status=..., message=...): ...
- def error(self, message): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/mistune.pyi b/v1.1.0-dev1/salad/typeshed/2.7/mistune.pyi
deleted file mode 100644
index 0b9b0d95d..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/mistune.pyi
+++ /dev/null
@@ -1,164 +0,0 @@
-# Stubs for mistune (Python 2)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-
-def escape(text, quote=False, smart_amp=True): ...
-
-class BlockGrammar:
- def_links = ... # type: Any
- def_footnotes = ... # type: Any
- newline = ... # type: Any
- block_code = ... # type: Any
- fences = ... # type: Any
- hrule = ... # type: Any
- heading = ... # type: Any
- lheading = ... # type: Any
- block_quote = ... # type: Any
- list_block = ... # type: Any
- list_item = ... # type: Any
- list_bullet = ... # type: Any
- paragraph = ... # type: Any
- block_html = ... # type: Any
- table = ... # type: Any
- nptable = ... # type: Any
- text = ... # type: Any
-
-class BlockLexer:
- grammar_class = ... # type: Any
- default_rules = ... # type: Any
- list_rules = ... # type: Any
- footnote_rules = ... # type: Any
- tokens = ... # type: Any
- def_links = ... # type: Any
- def_footnotes = ... # type: Any
- rules = ... # type: Any
- def __init__(self, rules=None, **kwargs): ...
- def __call__(self, text, rules=None): ...
- def parse(self, text, rules=None): ...
- def parse_newline(self, m): ...
- def parse_block_code(self, m): ...
- def parse_fences(self, m): ...
- def parse_heading(self, m): ...
- def parse_lheading(self, m): ...
- def parse_hrule(self, m): ...
- def parse_list_block(self, m): ...
- def parse_block_quote(self, m): ...
- def parse_def_links(self, m): ...
- def parse_def_footnotes(self, m): ...
- def parse_table(self, m): ...
- def parse_nptable(self, m): ...
- def parse_block_html(self, m): ...
- def parse_paragraph(self, m): ...
- def parse_text(self, m): ...
-
-class InlineGrammar:
- escape = ... # type: Any
- inline_html = ... # type: Any
- autolink = ... # type: Any
- link = ... # type: Any
- reflink = ... # type: Any
- nolink = ... # type: Any
- url = ... # type: Any
- double_emphasis = ... # type: Any
- emphasis = ... # type: Any
- code = ... # type: Any
- linebreak = ... # type: Any
- strikethrough = ... # type: Any
- footnote = ... # type: Any
- text = ... # type: Any
- def hard_wrap(self): ...
-
-class InlineLexer:
- grammar_class = ... # type: Any
- default_rules = ... # type: Any
- inline_html_rules = ... # type: Any
- renderer = ... # type: Any
- links = ... # type: Any
- footnotes = ... # type: Any
- footnote_index = ... # type: Any
- rules = ... # type: Any
- def __init__(self, renderer, rules=None, **kwargs): ...
- def __call__(self, text, rules=None): ...
- def setup(self, links, footnotes): ...
- line_match = ... # type: Any
- line_started = ... # type: Any
- def output(self, text, rules=None): ...
- def output_escape(self, m): ...
- def output_autolink(self, m): ...
- def output_url(self, m): ...
- def output_inline_html(self, m): ...
- def output_footnote(self, m): ...
- def output_link(self, m): ...
- def output_reflink(self, m): ...
- def output_nolink(self, m): ...
- def output_double_emphasis(self, m): ...
- def output_emphasis(self, m): ...
- def output_code(self, m): ...
- def output_linebreak(self, m): ...
- def output_strikethrough(self, m): ...
- def output_text(self, m): ...
-
-class Renderer:
- options = ... # type: Any
- def __init__(self, **kwargs) -> None: ...
- def placeholder(self): ...
- def block_code(self, code, lang=None): ...
- def block_quote(self, text): ...
- def block_html(self, html): ...
- def header(self, text, level, raw=None): ...
- def hrule(self): ...
- def list(self, body, ordered=True): ...
- def list_item(self, text): ...
- def paragraph(self, text): ...
- def table(self, header, body): ...
- def table_row(self, content): ...
- def table_cell(self, content, **flags): ...
- def double_emphasis(self, text): ...
- def emphasis(self, text): ...
- def codespan(self, text): ...
- def linebreak(self): ...
- def strikethrough(self, text): ...
- def text(self, text): ...
- def autolink(self, link, is_email=False): ...
- def link(self, link, title, text): ...
- def image(self, src, title, text): ...
- def inline_html(self, html): ...
- def newline(self): ...
- def footnote_ref(self, key, index): ...
- def footnote_item(self, key, text): ...
- def footnotes(self, text): ...
-
-class Markdown:
- renderer = ... # type: Any
- inline = ... # type: Any
- block = ... # type: Any
- footnotes = ... # type: Any
- tokens = ... # type: Any
- def __init__(self, renderer=None, inline=None, block=None, **kwargs): ...
- def __call__(self, text): ...
- def render(self, text): ...
- def parse(self, text): ...
- token = ... # type: Any
- def pop(self): ...
- def peek(self): ...
- def output(self, text, rules=None): ...
- def tok(self): ...
- def tok_text(self): ...
- def output_newline(self): ...
- def output_hrule(self): ...
- def output_heading(self): ...
- def output_code(self): ...
- def output_table(self): ...
- def output_block_quote(self): ...
- def output_list(self): ...
- def output_list_item(self): ...
- def output_loose_item(self): ...
- def output_footnote(self): ...
- def output_close_html(self): ...
- def output_open_html(self): ...
- def output_paragraph(self): ...
- def output_text(self): ...
-
-def markdown(text: str, escape: bool=True, **kwargs) -> str: ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/pathlib2.pyi b/v1.1.0-dev1/salad/typeshed/2.7/pathlib2.pyi
new file mode 100644
index 000000000..30bbe2cbd
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2.7/pathlib2.pyi
@@ -0,0 +1,188 @@
+# Stubs for pathlib2 (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Any, AnyStr, Type, TypeVar, Optional, Union
+from collections import Sequence
+
+_P = TypeVar('_P', bound='PurePath')
+
+intern = ... # type: Any
+basestring = ... # type: Any
+supports_symlinks = ... # type: bool
+nt = ... # type: Any
+
+class _Flavour:
+ join = ... # type: Any
+ def __init__(self) -> None: ...
+ def parse_parts(self, parts): ...
+ def join_parsed_parts(self, drv, root, parts, drv2, root2, parts2): ...
+
+class _WindowsFlavour(_Flavour):
+ sep = ... # type: str
+ altsep = ... # type: str
+ has_drv = ... # type: bool
+ pathmod = ... # type: Any
+ is_supported = ... # type: Any
+ drive_letters = ... # type: Any
+ ext_namespace_prefix = ... # type: str
+ reserved_names = ... # type: Any
+ def splitroot(self, part, sep: Any = ...): ...
+ def casefold(self, s): ...
+ def casefold_parts(self, parts): ...
+ def resolve(self, path): ...
+ def is_reserved(self, parts): ...
+ def make_uri(self, path): ...
+ def gethomedir(self, username): ...
+
+class _PosixFlavour(_Flavour):
+ sep = ... # type: str
+ altsep = ... # type: str
+ has_drv = ... # type: bool
+ pathmod = ... # type: Any
+ is_supported = ... # type: Any
+ def splitroot(self, part, sep: Any = ...): ...
+ def casefold(self, s): ...
+ def casefold_parts(self, parts): ...
+ def resolve(self, path): ...
+ def is_reserved(self, parts): ...
+ def make_uri(self, path): ...
+ def gethomedir(self, username): ...
+
+class _Accessor: ...
+
+class _NormalAccessor(_Accessor):
+ stat = ... # type: Any
+ lstat = ... # type: Any
+ open = ... # type: Any
+ listdir = ... # type: Any
+ chmod = ... # type: Any
+ lchmod = ... # type: Any
+ #def lchmod(self, pathobj, mode): ...
+ mkdir = ... # type: Any
+ unlink = ... # type: Any
+ rmdir = ... # type: Any
+ rename = ... # type: Any
+ replace = ... # type: Any
+ symlink = ... # type: Any
+ #def symlink(a, b, target_is_directory): ...
+ #@staticmethod
+ #def symlink(a, b, target_is_directory): ...
+ utime = ... # type: Any
+ def readlink(self, path): ...
+
+class _Selector:
+ child_parts = ... # type: Any
+ successor = ... # type: Any
+ def __init__(self, child_parts) -> None: ...
+ def select_from(self, parent_path): ...
+
+class _TerminatingSelector: ...
+
+class _PreciseSelector(_Selector):
+ name = ... # type: Any
+ def __init__(self, name, child_parts) -> None: ...
+
+class _WildcardSelector(_Selector):
+ pat = ... # type: Any
+ def __init__(self, pat, child_parts) -> None: ...
+
+class _RecursiveWildcardSelector(_Selector):
+ def __init__(self, pat, child_parts) -> None: ...
+
+class _PathParents(Sequence):
+ def __init__(self, path) -> None: ...
+ def __len__(self): ...
+ def __getitem__(self, idx): ...
+
+class PurePath:
+ def __new__(cls, *args): ...
+ def __reduce__(self): ...
+ def as_posix(self): ...
+ def __bytes__(self): ...
+ def as_uri(self) -> str: ...
+ def __eq__(self, other): ...
+ def __ne__(self, other): ...
+ def __hash__(self): ...
+ def __lt__(self, other): ...
+ def __le__(self, other): ...
+ def __gt__(self, other): ...
+ def __ge__(self, other): ...
+ drive = ... # type: Any
+ root = ... # type: Any
+ @property
+ def anchor(self): ...
+ @property
+ def name(self): ...
+ @property
+ def suffix(self): ...
+ @property
+ def suffixes(self): ...
+ @property
+ def stem(self): ...
+ def with_name(self, name): ...
+ def with_suffix(self, suffix): ...
+ def relative_to(self, *other): ...
+ @property
+ def parts(self): ...
+ def joinpath(self, *args): ...
+ def __truediv__(self, key): ...
+ def __rtruediv__(self, key): ...
+ __div__ = ... # type: Any
+ __rdiv__ = ... # type: Any
+ @property
+ def parent(self): ...
+ @property
+ def parents(self): ...
+ def is_absolute(self): ...
+ def is_reserved(self): ...
+ def match(self, path_pattern): ...
+
+class PurePosixPath(PurePath): ...
+class PureWindowsPath(PurePath): ...
+
+class Path(PurePath):
+ def __new__(cls: Type[_P], *args: Union[AnyStr, PurePath],
+ **kwargs: Any) -> _P: ...
+ def __enter__(self): ...
+ def __exit__(self, t, v, tb): ...
+ @classmethod
+ def cwd(cls): ...
+ @classmethod
+ def home(cls): ...
+ def samefile(self, other_path): ...
+ def iterdir(self): ...
+ def glob(self, pattern): ...
+ def rglob(self, pattern): ...
+ def absolute(self): ...
+ def resolve(self): ...
+ def stat(self): ...
+ def owner(self): ...
+ def group(self): ...
+ def open(self, mode: str = ..., buffering: int = ..., encoding: Optional[Any] = ..., errors: Optional[Any] = ..., newline: Optional[Any] = ...): ...
+ def read_bytes(self): ...
+ def read_text(self, encoding: Optional[Any] = ..., errors: Optional[Any] = ...): ...
+ def write_bytes(self, data): ...
+ def write_text(self, data, encoding: Optional[Any] = ..., errors: Optional[Any] = ...): ...
+ def touch(self, mode: int = ..., exist_ok: bool = ...): ...
+ def mkdir(self, mode: int = ..., parents: bool = ..., exist_ok: bool = ...): ...
+ def chmod(self, mode): ...
+ def lchmod(self, mode): ...
+ def unlink(self): ...
+ def rmdir(self): ...
+ def lstat(self): ...
+ def rename(self, target): ...
+ def replace(self, target): ...
+ def symlink_to(self, target, target_is_directory: bool = ...): ...
+ def exists(self): ...
+ def is_dir(self): ...
+ def is_file(self): ...
+ def is_symlink(self): ...
+ def is_block_device(self): ...
+ def is_char_device(self): ...
+ def is_fifo(self): ...
+ def is_socket(self): ...
+ def expanduser(self): ...
+
+class PosixPath(Path, PurePosixPath): ...
+class WindowsPath(Path, PureWindowsPath): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/pprint.pyi b/v1.1.0-dev1/salad/typeshed/2.7/pprint.pyi
deleted file mode 100644
index 1452a69ab..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/pprint.pyi
+++ /dev/null
@@ -1,21 +0,0 @@
-# Stubs for pprint (Python 2)
-#
-# NOTE: Based on a dynamically typed automatically generated by stubgen.
-
-from typing import IO, Any
-
-def pprint(object: Any, stream: IO[Any] = ..., indent: int = ..., width: int = ...,
- depth: int = ...) -> None: ...
-def pformat(object: Any, indent: int =..., width: int =..., depth: int =...) -> str: ...
-def saferepr(object): ...
-def isreadable(object): ...
-def isrecursive(object): ...
-
-class PrettyPrinter:
- def __init__(self, indent: int = ..., width: int = ..., depth: int = ...,
- stream: IO[Any] = ...) -> None: ...
- def pprint(self, object: Any) -> str: ...
- def pformat(self, object): ...
- def isrecursive(self, object): ...
- def isreadable(self, object): ...
- def format(self, object, context, maxlevels, level): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/re.pyi b/v1.1.0-dev1/salad/typeshed/2.7/re.pyi
deleted file mode 100644
index 8b88a7822..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/re.pyi
+++ /dev/null
@@ -1,99 +0,0 @@
-# Stubs for re
-# Ron Murawski
-# 'bytes' support added by Jukka Lehtosalo
-
-# based on: http://docs.python.org/2.7/library/re.html
-
-from typing import (
- List, Iterator, overload, Callable, Tuple, Sequence, Dict,
- Generic, AnyStr, Match, Pattern, Any
-)
-
-# ----- re variables and constants -----
-DEBUG = 0
-I = 0
-IGNORECASE = 0
-L = 0
-LOCALE = 0
-M = 0
-MULTILINE = 0
-S = 0
-DOTALL = 0
-X = 0
-VERBOSE = 0
-U = 0
-UNICODE = 0
-T = 0
-TEMPLATE = 0
-
-class error(Exception): ...
-
-@overload
-def compile(pattern: AnyStr, flags: int = ...) -> Pattern[AnyStr]: ...
-@overload
-def compile(pattern: Pattern[AnyStr], flags: int = ...) -> Pattern[AnyStr]: ...
-
-@overload
-def search(pattern: AnyStr, string: AnyStr, flags: int = ...) -> Match[AnyStr]: ...
-@overload
-def search(pattern: Pattern[AnyStr], string: AnyStr, flags: int = ...) -> Match[AnyStr]: ...
-
-@overload
-def match(pattern: AnyStr, string: AnyStr, flags: int = ...) -> Match[AnyStr]: ...
-@overload
-def match(pattern: Pattern[AnyStr], string: AnyStr, flags: int = ...) -> Match[AnyStr]: ...
-
-@overload
-def split(pattern: AnyStr, string: AnyStr,
- maxsplit: int = ..., flags: int = ...) -> List[AnyStr]: ...
-@overload
-def split(pattern: Pattern[AnyStr], string: AnyStr,
- maxsplit: int = ..., flags: int = ...) -> List[AnyStr]: ...
-
-@overload
-def findall(pattern: AnyStr, string: AnyStr, flags: int = ...) -> List[Any]: ...
-@overload
-def findall(pattern: Pattern[AnyStr], string: AnyStr, flags: int = ...) -> List[Any]: ...
-
-# Return an iterator yielding match objects over all non-overlapping matches
-# for the RE pattern in string. The string is scanned left-to-right, and
-# matches are returned in the order found. Empty matches are included in the
-# result unless they touch the beginning of another match.
-@overload
-def finditer(pattern: AnyStr, string: AnyStr,
- flags: int = ...) -> Iterator[Match[AnyStr]]: ...
-@overload
-def finditer(pattern: Pattern[AnyStr], string: AnyStr,
- flags: int = ...) -> Iterator[Match[AnyStr]]: ...
-
-@overload
-def sub(pattern: AnyStr, repl: AnyStr, string: AnyStr, count: int = ...,
- flags: int = ...) -> AnyStr: ...
-@overload
-def sub(pattern: AnyStr, repl: Callable[[Match[AnyStr]], AnyStr],
- string: AnyStr, count: int = ..., flags: int = ...) -> AnyStr: ...
-@overload
-def sub(pattern: Pattern[AnyStr], repl: AnyStr, string: AnyStr, count: int = ...,
- flags: int = ...) -> AnyStr: ...
-@overload
-def sub(pattern: Pattern[AnyStr], repl: Callable[[Match[AnyStr]], AnyStr],
- string: AnyStr, count: int = ..., flags: int = ...) -> AnyStr: ...
-
-@overload
-def subn(pattern: AnyStr, repl: AnyStr, string: AnyStr, count: int = ...,
- flags: int = ...) -> Tuple[AnyStr, int]: ...
-@overload
-def subn(pattern: AnyStr, repl: Callable[[Match[AnyStr]], AnyStr],
- string: AnyStr, count: int = ...,
- flags: int = ...) -> Tuple[AnyStr, int]: ...
-@overload
-def subn(pattern: Pattern[AnyStr], repl: AnyStr, string: AnyStr, count: int = ...,
- flags: int = ...) -> Tuple[AnyStr, int]: ...
-@overload
-def subn(pattern: Pattern[AnyStr], repl: Callable[[Match[AnyStr]], AnyStr],
- string: AnyStr, count: int = ...,
- flags: int = ...) -> Tuple[AnyStr, int]: ...
-
-def escape(string: AnyStr) -> AnyStr: ...
-
-def purge() -> None: ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/__init__.pyi
deleted file mode 100644
index 6ea56efcc..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/__init__.pyi
+++ /dev/null
@@ -1,38 +0,0 @@
-# Stubs for requests (based on version 2.6.0, Python 3)
-
-from typing import Any
-from requests import models
-from requests import api
-from requests import sessions
-from requests import status_codes
-from requests import exceptions
-import logging
-
-__title__ = ... # type: Any
-__build__ = ... # type: Any
-__license__ = ... # type: Any
-__copyright__ = ... # type: Any
-
-Request = models.Request
-Response = models.Response
-PreparedRequest = models.PreparedRequest
-request = api.request
-get = api.get
-head = api.head
-post = api.post
-patch = api.patch
-put = api.put
-delete = api.delete
-options = api.options
-session = sessions.session
-Session = sessions.Session
-codes = status_codes.codes
-RequestException = exceptions.RequestException
-Timeout = exceptions.Timeout
-URLRequired = exceptions.URLRequired
-TooManyRedirects = exceptions.TooManyRedirects
-HTTPError = exceptions.HTTPError
-ConnectionError = exceptions.ConnectionError
-
-class NullHandler(logging.Handler):
- def emit(self, record): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/adapters.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/adapters.pyi
deleted file mode 100644
index 109dc9a3e..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/adapters.pyi
+++ /dev/null
@@ -1,69 +0,0 @@
-# Stubs for requests.adapters (Python 3)
-
-from typing import Any
-from . import models
-from .packages.urllib3 import poolmanager
-from .packages.urllib3 import response
-from .packages.urllib3.util import retry
-from . import compat
-from . import utils
-from . import structures
-from .packages.urllib3 import exceptions as urllib3_exceptions
-from . import cookies
-from . import exceptions
-from . import auth
-
-Response = models.Response
-PoolManager = poolmanager.PoolManager
-proxy_from_url = poolmanager.proxy_from_url
-HTTPResponse = response.HTTPResponse
-Retry = retry.Retry
-DEFAULT_CA_BUNDLE_PATH = utils.DEFAULT_CA_BUNDLE_PATH
-get_encoding_from_headers = utils.get_encoding_from_headers
-prepend_scheme_if_needed = utils.prepend_scheme_if_needed
-get_auth_from_url = utils.get_auth_from_url
-urldefragauth = utils.urldefragauth
-CaseInsensitiveDict = structures.CaseInsensitiveDict
-ConnectTimeoutError = urllib3_exceptions.ConnectTimeoutError
-MaxRetryError = urllib3_exceptions.MaxRetryError
-ProtocolError = urllib3_exceptions.ProtocolError
-ReadTimeoutError = urllib3_exceptions.ReadTimeoutError
-ResponseError = urllib3_exceptions.ResponseError
-extract_cookies_to_jar = cookies.extract_cookies_to_jar
-ConnectionError = exceptions.ConnectionError
-ConnectTimeout = exceptions.ConnectTimeout
-ReadTimeout = exceptions.ReadTimeout
-SSLError = exceptions.SSLError
-ProxyError = exceptions.ProxyError
-RetryError = exceptions.RetryError
-
-DEFAULT_POOLBLOCK = ... # type: Any
-DEFAULT_POOLSIZE = ... # type: Any
-DEFAULT_RETRIES = ... # type: Any
-
-class BaseAdapter:
- def __init__(self) -> None: ...
- # TODO: "request" parameter not actually supported, added to please mypy.
- def send(self, request=...): ...
- def close(self): ...
-
-class HTTPAdapter(BaseAdapter):
- __attrs__ = ... # type: Any
- max_retries = ... # type: Any
- config = ... # type: Any
- proxy_manager = ... # type: Any
- def __init__(self, pool_connections=..., pool_maxsize=..., max_retries=...,
- pool_block=...): ...
- poolmanager = ... # type: Any
- def init_poolmanager(self, connections, maxsize, block=..., **pool_kwargs): ...
- def proxy_manager_for(self, proxy, **proxy_kwargs): ...
- def cert_verify(self, conn, url, verify, cert): ...
- def build_response(self, req, resp): ...
- def get_connection(self, url, proxies=...): ...
- def close(self): ...
- def request_url(self, request, proxies): ...
- def add_headers(self, request, **kwargs): ...
- def proxy_headers(self, proxy): ...
- # TODO: "request" is not actually optional, modified to please mypy.
- def send(self, request=..., stream=..., timeout=..., verify=..., cert=...,
- proxies=...): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/api.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/api.pyi
deleted file mode 100644
index 44853f72b..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/api.pyi
+++ /dev/null
@@ -1,14 +0,0 @@
-# Stubs for requests.api (Python 3)
-
-from typing import Union
-
-from .models import Response
-
-def request(method: str, url: str, **kwargs) -> Response: ...
-def get(url: Union[str, unicode], **kwargs) -> Response: ...
-def options(url: Union[str, unicode], **kwargs) -> Response: ...
-def head(url: Union[str, unicode], **kwargs) -> Response: ...
-def post(url: Union[str, unicode], data=..., json=..., **kwargs) -> Response: ...
-def put(url: Union[str, unicode], data=..., **kwargs) -> Response: ...
-def patch(url: Union[str, unicode], data=..., **kwargs) -> Response: ...
-def delete(url: Union[str, unicode], **kwargs) -> Response: ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/auth.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/auth.pyi
deleted file mode 100644
index 8eea2b0e0..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/auth.pyi
+++ /dev/null
@@ -1,41 +0,0 @@
-# Stubs for requests.auth (Python 3)
-
-from typing import Any
-from . import compat
-from . import cookies
-from . import utils
-from . import status_codes
-
-extract_cookies_to_jar = cookies.extract_cookies_to_jar
-parse_dict_header = utils.parse_dict_header
-to_native_string = utils.to_native_string
-codes = status_codes.codes
-
-CONTENT_TYPE_FORM_URLENCODED = ... # type: Any
-CONTENT_TYPE_MULTI_PART = ... # type: Any
-
-class AuthBase:
- def __call__(self, r): ...
-
-class HTTPBasicAuth(AuthBase):
- username = ... # type: Any
- password = ... # type: Any
- def __init__(self, username, password) -> None: ...
- def __call__(self, r): ...
-
-class HTTPProxyAuth(HTTPBasicAuth):
- def __call__(self, r): ...
-
-class HTTPDigestAuth(AuthBase):
- username = ... # type: Any
- password = ... # type: Any
- last_nonce = ... # type: Any
- nonce_count = ... # type: Any
- chal = ... # type: Any
- pos = ... # type: Any
- num_401_calls = ... # type: Any
- def __init__(self, username, password) -> None: ...
- def build_digest_header(self, method, url): ...
- def handle_redirect(self, r, **kwargs): ...
- def handle_401(self, r, **kwargs): ...
- def __call__(self, r): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/compat.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/compat.pyi
deleted file mode 100644
index 63b92f6fe..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/compat.pyi
+++ /dev/null
@@ -1,6 +0,0 @@
-# Stubs for requests.compat (Python 3.4)
-
-from typing import Any
-import collections
-
-OrderedDict = collections.OrderedDict
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/cookies.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/cookies.pyi
deleted file mode 100644
index 6f56c82fb..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/cookies.pyi
+++ /dev/null
@@ -1,61 +0,0 @@
-# Stubs for requests.cookies (Python 3)
-
-from typing import Any, MutableMapping
-import collections
-from . import compat
-
-class MockRequest:
- type = ... # type: Any
- def __init__(self, request) -> None: ...
- def get_type(self): ...
- def get_host(self): ...
- def get_origin_req_host(self): ...
- def get_full_url(self): ...
- def is_unverifiable(self): ...
- def has_header(self, name): ...
- def get_header(self, name, default=...): ...
- def add_header(self, key, val): ...
- def add_unredirected_header(self, name, value): ...
- def get_new_headers(self): ...
- @property
- def unverifiable(self): ...
- @property
- def origin_req_host(self): ...
- @property
- def host(self): ...
-
-class MockResponse:
- def __init__(self, headers) -> None: ...
- def info(self): ...
- def getheaders(self, name): ...
-
-def extract_cookies_to_jar(jar, request, response): ...
-def get_cookie_header(jar, request): ...
-def remove_cookie_by_name(cookiejar, name, domain=..., path=...): ...
-
-class CookieConflictError(RuntimeError): ...
-
-class RequestsCookieJar(MutableMapping):
- def get(self, name, default=..., domain=..., path=...): ...
- def set(self, name, value, **kwargs): ...
- def iterkeys(self): ...
- def keys(self): ...
- def itervalues(self): ...
- def values(self): ...
- def iteritems(self): ...
- def items(self): ...
- def list_domains(self): ...
- def list_paths(self): ...
- def multiple_domains(self): ...
- def get_dict(self, domain=..., path=...): ...
- def __getitem__(self, name): ...
- def __setitem__(self, name, value): ...
- def __delitem__(self, name): ...
- def set_cookie(self, cookie, *args, **kwargs): ...
- def update(self, other): ...
- def copy(self): ...
-
-def create_cookie(name, value, **kwargs): ...
-def morsel_to_cookie(morsel): ...
-def cookiejar_from_dict(cookie_dict, cookiejar=..., overwrite=...): ...
-def merge_cookies(cookiejar, cookies): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/exceptions.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/exceptions.pyi
deleted file mode 100644
index ff0c32883..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/exceptions.pyi
+++ /dev/null
@@ -1,26 +0,0 @@
-# Stubs for requests.exceptions (Python 3)
-
-from typing import Any
-from .packages.urllib3.exceptions import HTTPError as BaseHTTPError
-
-class RequestException(IOError):
- response = ... # type: Any
- request = ... # type: Any
- def __init__(self, *args, **kwargs) -> None: ...
-
-class HTTPError(RequestException): ...
-class ConnectionError(RequestException): ...
-class ProxyError(ConnectionError): ...
-class SSLError(ConnectionError): ...
-class Timeout(RequestException): ...
-class ConnectTimeout(ConnectionError, Timeout): ...
-class ReadTimeout(Timeout): ...
-class URLRequired(RequestException): ...
-class TooManyRedirects(RequestException): ...
-class MissingSchema(RequestException, ValueError): ...
-class InvalidSchema(RequestException, ValueError): ...
-class InvalidURL(RequestException, ValueError): ...
-class ChunkedEncodingError(RequestException): ...
-class ContentDecodingError(RequestException, BaseHTTPError): ...
-class StreamConsumedError(RequestException, TypeError): ...
-class RetryError(RequestException): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/hooks.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/hooks.pyi
deleted file mode 100644
index 3367d9a48..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/hooks.pyi
+++ /dev/null
@@ -1,8 +0,0 @@
-# Stubs for requests.hooks (Python 3)
-
-from typing import Any
-
-HOOKS = ... # type: Any
-
-def default_hooks(): ...
-def dispatch_hook(key, hooks, hook_data, **kwargs): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/models.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/models.pyi
deleted file mode 100644
index d400d4a06..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/models.pyi
+++ /dev/null
@@ -1,133 +0,0 @@
-# Stubs for requests.models (Python 3)
-
-from typing import Any, List, MutableMapping, Iterator, Dict
-import datetime
-
-from . import hooks
-from . import structures
-from . import auth
-from . import cookies
-from .cookies import RequestsCookieJar
-from .packages.urllib3 import fields
-from .packages.urllib3 import filepost
-from .packages.urllib3 import util
-from .packages.urllib3 import exceptions as urllib3_exceptions
-from . import exceptions
-from . import utils
-from . import compat
-from . import status_codes
-
-default_hooks = hooks.default_hooks
-CaseInsensitiveDict = structures.CaseInsensitiveDict
-HTTPBasicAuth = auth.HTTPBasicAuth
-cookiejar_from_dict = cookies.cookiejar_from_dict
-get_cookie_header = cookies.get_cookie_header
-RequestField = fields.RequestField
-encode_multipart_formdata = filepost.encode_multipart_formdata
-DecodeError = urllib3_exceptions.DecodeError
-ReadTimeoutError = urllib3_exceptions.ReadTimeoutError
-ProtocolError = urllib3_exceptions.ProtocolError
-LocationParseError = urllib3_exceptions.LocationParseError
-HTTPError = exceptions.HTTPError
-MissingSchema = exceptions.MissingSchema
-InvalidURL = exceptions.InvalidURL
-ChunkedEncodingError = exceptions.ChunkedEncodingError
-ContentDecodingError = exceptions.ContentDecodingError
-ConnectionError = exceptions.ConnectionError
-StreamConsumedError = exceptions.StreamConsumedError
-guess_filename = utils.guess_filename
-get_auth_from_url = utils.get_auth_from_url
-requote_uri = utils.requote_uri
-stream_decode_response_unicode = utils.stream_decode_response_unicode
-to_key_val_list = utils.to_key_val_list
-parse_header_links = utils.parse_header_links
-iter_slices = utils.iter_slices
-guess_json_utf = utils.guess_json_utf
-super_len = utils.super_len
-to_native_string = utils.to_native_string
-codes = status_codes.codes
-
-REDIRECT_STATI = ... # type: Any
-DEFAULT_REDIRECT_LIMIT = ... # type: Any
-CONTENT_CHUNK_SIZE = ... # type: Any
-ITER_CHUNK_SIZE = ... # type: Any
-json_dumps = ... # type: Any
-
-class RequestEncodingMixin:
- @property
- def path_url(self): ...
-
-class RequestHooksMixin:
- def register_hook(self, event, hook): ...
- def deregister_hook(self, event, hook): ...
-
-class Request(RequestHooksMixin):
- hooks = ... # type: Any
- method = ... # type: Any
- url = ... # type: Any
- headers = ... # type: Any
- files = ... # type: Any
- data = ... # type: Any
- json = ... # type: Any
- params = ... # type: Any
- auth = ... # type: Any
- cookies = ... # type: Any
- def __init__(self, method=..., url=..., headers=..., files=..., data=..., params=...,
- auth=..., cookies=..., hooks=..., json=...): ...
- def prepare(self): ...
-
-class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
- method = ... # type: Any
- url = ... # type: Any
- headers = ... # type: Any
- body = ... # type: Any
- hooks = ... # type: Any
- def __init__(self) -> None: ...
- def prepare(self, method=..., url=..., headers=..., files=..., data=..., params=...,
- auth=..., cookies=..., hooks=..., json=...): ...
- def copy(self): ...
- def prepare_method(self, method): ...
- def prepare_url(self, url, params): ...
- def prepare_headers(self, headers): ...
- def prepare_body(self, data, files, json=...): ...
- def prepare_content_length(self, body): ...
- def prepare_auth(self, auth, url=...): ...
- def prepare_cookies(self, cookies): ...
- def prepare_hooks(self, hooks): ...
-
-class Response:
- __attrs__ = ... # type: Any
- status_code = ... # type: int
- headers = ... # type: MutableMapping[str, str]
- raw = ... # type: Any
- url = ... # type: str
- encoding = ... # type: str
- history = ... # type: List[Response]
- reason = ... # type: str
- cookies = ... # type: RequestsCookieJar
- elapsed = ... # type: datetime.timedelta
- request = ... # type: PreparedRequest
- def __init__(self) -> None: ...
- def __bool__(self) -> bool: ...
- def __nonzero__(self) -> bool: ...
- def __iter__(self) -> Iterator[str]: ...
- @property
- def ok(self) -> bool: ...
- @property
- def is_redirect(self) -> bool: ...
- @property
- def is_permanent_redirect(self) -> bool: ...
- @property
- def apparent_encoding(self) -> str: ...
- def iter_content(self, chunk_size: int = ...,
- decode_unicode: bool = ...) -> Iterator[Any]: ...
- def iter_lines(self, chunk_size=..., decode_unicode=..., delimiter=...): ...
- @property
- def content(self) -> str: ...
- @property
- def text(self) -> str: ...
- def json(self, **kwargs) -> Any: ...
- @property
- def links(self) -> Dict[Any, Any]: ...
- def raise_for_status(self) -> None: ...
- def close(self) -> None: ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/__init__.pyi
deleted file mode 100644
index 2b1bff828..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/__init__.pyi
+++ /dev/null
@@ -1,8 +0,0 @@
-# Stubs for requests.packages (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-class VendorAlias:
- def __init__(self, package_names) -> None: ...
- def find_module(self, fullname, path=...): ...
- def load_module(self, name): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/__init__.pyi
deleted file mode 100644
index 38cf6729c..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/__init__.pyi
+++ /dev/null
@@ -1,12 +0,0 @@
-# Stubs for requests.packages.urllib3 (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-import logging
-
-class NullHandler(logging.Handler):
- def emit(self, record): ...
-
-def add_stderr_logger(level=...): ...
-def disable_warnings(category=...): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/_collections.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/_collections.pyi
deleted file mode 100644
index 58aa94422..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/_collections.pyi
+++ /dev/null
@@ -1,51 +0,0 @@
-# Stubs for requests.packages.urllib3._collections (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from collections import MutableMapping
-
-class RLock:
- def __enter__(self): ...
- def __exit__(self, exc_type, exc_value, traceback): ...
-
-class RecentlyUsedContainer(MutableMapping):
- ContainerCls = ... # type: Any
- dispose_func = ... # type: Any
- lock = ... # type: Any
- def __init__(self, maxsize=..., dispose_func=...) -> None: ...
- def __getitem__(self, key): ...
- def __setitem__(self, key, value): ...
- def __delitem__(self, key): ...
- def __len__(self): ...
- def __iter__(self): ...
- def clear(self): ...
- def keys(self): ...
-
-class HTTPHeaderDict(dict):
- def __init__(self, headers=..., **kwargs) -> None: ...
- def __setitem__(self, key, val): ...
- def __getitem__(self, key): ...
- def __delitem__(self, key): ...
- def __contains__(self, key): ...
- def __eq__(self, other): ...
- def __ne__(self, other): ...
- values = ... # type: Any
- get = ... # type: Any
- update = ... # type: Any
- iterkeys = ... # type: Any
- itervalues = ... # type: Any
- def pop(self, key, default=...): ...
- def discard(self, key): ...
- def add(self, key, val): ...
- def extend(*args, **kwargs): ...
- def getlist(self, key): ...
- getheaders = ... # type: Any
- getallmatchingheaders = ... # type: Any
- iget = ... # type: Any
- def copy(self): ...
- def iteritems(self): ...
- def itermerged(self): ...
- def items(self): ...
- @classmethod
- def from_httplib(cls, message, duplicates=...): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/connection.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/connection.pyi
deleted file mode 100644
index 289fd1836..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/connection.pyi
+++ /dev/null
@@ -1,51 +0,0 @@
-# Stubs for requests.packages.urllib3.connection (Python 3.4)
-
-from typing import Any
-from httplib import HTTPException
-from . import packages
-from . import exceptions
-from . import util
-
-class DummyConnection: ...
-
-ConnectTimeoutError = exceptions.ConnectTimeoutError
-SystemTimeWarning = exceptions.SystemTimeWarning
-SecurityWarning = exceptions.SecurityWarning
-
-port_by_scheme = ... # type: Any
-RECENT_DATE = ... # type: Any
-
-class HTTPConnection(object):
- default_port = ... # type: Any
- default_socket_options = ... # type: Any
- is_verified = ... # type: Any
- source_address = ... # type: Any
- socket_options = ... # type: Any
- def __init__(self, *args, **kw) -> None: ...
- def connect(self): ...
-
-class HTTPSConnection(HTTPConnection):
- default_port = ... # type: Any
- key_file = ... # type: Any
- cert_file = ... # type: Any
- def __init__(self, host, port=..., key_file=..., cert_file=..., strict=..., timeout=..., **kw) -> None: ...
- sock = ... # type: Any
- def connect(self): ...
-
-class VerifiedHTTPSConnection(HTTPSConnection):
- cert_reqs = ... # type: Any
- ca_certs = ... # type: Any
- ssl_version = ... # type: Any
- assert_fingerprint = ... # type: Any
- key_file = ... # type: Any
- cert_file = ... # type: Any
- assert_hostname = ... # type: Any
- def set_cert(self, key_file=..., cert_file=..., cert_reqs=..., ca_certs=..., assert_hostname=..., assert_fingerprint=...): ...
- sock = ... # type: Any
- auto_open = ... # type: Any
- is_verified = ... # type: Any
- def connect(self): ...
-
-UnverifiedHTTPSConnection = ... # type: Any
-
-class ConnectionError(Exception): pass
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/connectionpool.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/connectionpool.pyi
deleted file mode 100644
index 03c3140f2..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/connectionpool.pyi
+++ /dev/null
@@ -1,87 +0,0 @@
-# Stubs for requests.packages.urllib3.connectionpool (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from ssl import SSLError as BaseSSLError
-from . import exceptions
-from .packages import ssl_match_hostname
-from . import packages
-from . import connection
-from . import request
-from . import response
-from .util import connection as _connection
-from .util import retry
-from .util import timeout
-from .util import url
-
-ClosedPoolError = exceptions.ClosedPoolError
-ProtocolError = exceptions.ProtocolError
-EmptyPoolError = exceptions.EmptyPoolError
-HostChangedError = exceptions.HostChangedError
-LocationValueError = exceptions.LocationValueError
-MaxRetryError = exceptions.MaxRetryError
-ProxyError = exceptions.ProxyError
-ReadTimeoutError = exceptions.ReadTimeoutError
-SSLError = exceptions.SSLError
-TimeoutError = exceptions.TimeoutError
-InsecureRequestWarning = exceptions.InsecureRequestWarning
-CertificateError = ssl_match_hostname.CertificateError
-port_by_scheme = connection.port_by_scheme
-DummyConnection = connection.DummyConnection
-HTTPConnection = connection.HTTPConnection
-HTTPSConnection = connection.HTTPSConnection
-VerifiedHTTPSConnection = connection.VerifiedHTTPSConnection
-HTTPException = connection.HTTPException
-ConnectionError = connection.ConnectionError
-RequestMethods = request.RequestMethods
-HTTPResponse = response.HTTPResponse
-is_connection_dropped = _connection.is_connection_dropped
-Retry = retry.Retry
-Timeout = timeout.Timeout
-get_host = url.get_host
-
-xrange = ... # type: Any
-log = ... # type: Any
-
-class ConnectionPool:
- scheme = ... # type: Any
- QueueCls = ... # type: Any
- host = ... # type: Any
- port = ... # type: Any
- def __init__(self, host, port=...) -> None: ...
- def __enter__(self): ...
- def __exit__(self, exc_type, exc_val, exc_tb): ...
- def close(self): ...
-
-class HTTPConnectionPool(ConnectionPool, RequestMethods):
- scheme = ... # type: Any
- ConnectionCls = ... # type: Any
- strict = ... # type: Any
- timeout = ... # type: Any
- retries = ... # type: Any
- pool = ... # type: Any
- block = ... # type: Any
- proxy = ... # type: Any
- proxy_headers = ... # type: Any
- num_connections = ... # type: Any
- num_requests = ... # type: Any
- conn_kw = ... # type: Any
- def __init__(self, host, port=..., strict=..., timeout=..., maxsize=..., block=..., headers=..., retries=..., _proxy=..., _proxy_headers=..., **conn_kw) -> None: ...
- def close(self): ...
- def is_same_host(self, url): ...
- def urlopen(self, method, url, body=..., headers=..., retries=..., redirect=..., assert_same_host=..., timeout=..., pool_timeout=..., release_conn=..., **response_kw): ...
-
-class HTTPSConnectionPool(HTTPConnectionPool):
- scheme = ... # type: Any
- ConnectionCls = ... # type: Any
- key_file = ... # type: Any
- cert_file = ... # type: Any
- cert_reqs = ... # type: Any
- ca_certs = ... # type: Any
- ssl_version = ... # type: Any
- assert_hostname = ... # type: Any
- assert_fingerprint = ... # type: Any
- def __init__(self, host, port=..., strict=..., timeout=..., maxsize=..., block=..., headers=..., retries=..., _proxy=..., _proxy_headers=..., key_file=..., cert_file=..., cert_reqs=..., ca_certs=..., ssl_version=..., assert_hostname=..., assert_fingerprint=..., **conn_kw) -> None: ...
-
-def connection_from_url(url, **kw): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/contrib/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/contrib/__init__.pyi
deleted file mode 100644
index 17d26bb13..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/contrib/__init__.pyi
+++ /dev/null
@@ -1,4 +0,0 @@
-# Stubs for requests.packages.urllib3.contrib (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/exceptions.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/exceptions.pyi
deleted file mode 100644
index 3e7d0f6c9..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/exceptions.pyi
+++ /dev/null
@@ -1,54 +0,0 @@
-# Stubs for requests.packages.urllib3.exceptions (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-
-class HTTPError(Exception): ...
-class HTTPWarning(Warning): ...
-
-class PoolError(HTTPError):
- pool = ... # type: Any
- def __init__(self, pool, message) -> None: ...
- def __reduce__(self): ...
-
-class RequestError(PoolError):
- url = ... # type: Any
- def __init__(self, pool, url, message) -> None: ...
- def __reduce__(self): ...
-
-class SSLError(HTTPError): ...
-class ProxyError(HTTPError): ...
-class DecodeError(HTTPError): ...
-class ProtocolError(HTTPError): ...
-
-ConnectionError = ... # type: Any
-
-class MaxRetryError(RequestError):
- reason = ... # type: Any
- def __init__(self, pool, url, reason=...) -> None: ...
-
-class HostChangedError(RequestError):
- retries = ... # type: Any
- def __init__(self, pool, url, retries=...) -> None: ...
-
-class TimeoutStateError(HTTPError): ...
-class TimeoutError(HTTPError): ...
-class ReadTimeoutError(TimeoutError, RequestError): ...
-class ConnectTimeoutError(TimeoutError): ...
-class EmptyPoolError(PoolError): ...
-class ClosedPoolError(PoolError): ...
-class LocationValueError(ValueError, HTTPError): ...
-
-class LocationParseError(LocationValueError):
- location = ... # type: Any
- def __init__(self, location) -> None: ...
-
-class ResponseError(HTTPError):
- GENERIC_ERROR = ... # type: Any
- SPECIFIC_ERROR = ... # type: Any
-
-class SecurityWarning(HTTPWarning): ...
-class InsecureRequestWarning(SecurityWarning): ...
-class SystemTimeWarning(SecurityWarning): ...
-class InsecurePlatformWarning(SecurityWarning): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/fields.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/fields.pyi
deleted file mode 100644
index cdc7734e2..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/fields.pyi
+++ /dev/null
@@ -1,16 +0,0 @@
-# Stubs for requests.packages.urllib3.fields (Python 3.4)
-
-from typing import Any
-from . import packages
-
-def guess_content_type(filename, default=...): ...
-def format_header_param(name, value): ...
-
-class RequestField:
- data = ... # type: Any
- headers = ... # type: Any
- def __init__(self, name, data, filename=..., headers=...) -> None: ...
- @classmethod
- def from_tuples(cls, fieldname, value): ...
- def render_headers(self): ...
- def make_multipart(self, content_disposition=..., content_type=..., content_location=...): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/filepost.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/filepost.pyi
deleted file mode 100644
index c6fefa618..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/filepost.pyi
+++ /dev/null
@@ -1,19 +0,0 @@
-# Stubs for requests.packages.urllib3.filepost (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from . import packages
-#from .packages import six
-from . import fields
-
-#six = packages.six
-#b = six.b
-RequestField = fields.RequestField
-
-writer = ... # type: Any
-
-def choose_boundary(): ...
-def iter_field_objects(fields): ...
-def iter_fields(fields): ...
-def encode_multipart_formdata(fields, boundary=...): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/packages/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/packages/__init__.pyi
deleted file mode 100644
index 231463649..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/packages/__init__.pyi
+++ /dev/null
@@ -1,4 +0,0 @@
-# Stubs for requests.packages.urllib3.packages (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/__init__.pyi
deleted file mode 100644
index 05c03dc08..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/__init__.pyi
+++ /dev/null
@@ -1 +0,0 @@
-class CertificateError(ValueError): pass
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.pyi
deleted file mode 100644
index 5abbc9dd5..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.pyi
+++ /dev/null
@@ -1,7 +0,0 @@
-# Stubs for requests.packages.urllib3.packages.ssl_match_hostname._implementation (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-class CertificateError(ValueError): ...
-
-def match_hostname(cert, hostname): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/poolmanager.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/poolmanager.pyi
deleted file mode 100644
index a65f66497..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/poolmanager.pyi
+++ /dev/null
@@ -1,31 +0,0 @@
-# Stubs for requests.packages.urllib3.poolmanager (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from .request import RequestMethods
-
-class PoolManager(RequestMethods):
- proxy = ... # type: Any
- connection_pool_kw = ... # type: Any
- pools = ... # type: Any
- def __init__(self, num_pools=..., headers=..., **connection_pool_kw) -> None: ...
- def __enter__(self): ...
- def __exit__(self, exc_type, exc_val, exc_tb): ...
- def clear(self): ...
- def connection_from_host(self, host, port=..., scheme=...): ...
- def connection_from_url(self, url): ...
- # TODO: This was the original signature -- copied another one from base class to fix complaint.
- # def urlopen(self, method, url, redirect=True, **kw): ...
- def urlopen(self, method, url, body=..., headers=..., encode_multipart=..., multipart_boundary=..., **kw): ...
-
-class ProxyManager(PoolManager):
- proxy = ... # type: Any
- proxy_headers = ... # type: Any
- def __init__(self, proxy_url, num_pools=..., headers=..., proxy_headers=..., **connection_pool_kw) -> None: ...
- def connection_from_host(self, host, port=..., scheme=...): ...
- # TODO: This was the original signature -- copied another one from base class to fix complaint.
- # def urlopen(self, method, url, redirect=True, **kw): ...
- def urlopen(self, method, url, body=..., headers=..., encode_multipart=..., multipart_boundary=..., **kw): ...
-
-def proxy_from_url(url, **kw): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/request.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/request.pyi
deleted file mode 100644
index 788c759c5..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/request.pyi
+++ /dev/null
@@ -1,13 +0,0 @@
-# Stubs for requests.packages.urllib3.request (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-
-class RequestMethods:
- headers = ... # type: Any
- def __init__(self, headers=...) -> None: ...
- def urlopen(self, method, url, body=..., headers=..., encode_multipart=..., multipart_boundary=..., **kw): ...
- def request(self, method, url, fields=..., headers=..., **urlopen_kw): ...
- def request_encode_url(self, method, url, fields=..., **urlopen_kw): ...
- def request_encode_body(self, method, url, fields=..., headers=..., encode_multipart=..., multipart_boundary=..., **urlopen_kw): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/response.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/response.pyi
deleted file mode 100644
index c84f7e91f..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/response.pyi
+++ /dev/null
@@ -1,58 +0,0 @@
-# Stubs for requests.packages.urllib3.response (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any, IO
-import io
-from . import _collections
-from . import exceptions
-#from .packages import six
-from . import connection
-from .util import response
-
-HTTPHeaderDict = _collections.HTTPHeaderDict
-ProtocolError = exceptions.ProtocolError
-DecodeError = exceptions.DecodeError
-ReadTimeoutError = exceptions.ReadTimeoutError
-binary_type = str # six.binary_type
-PY3 = True # six.PY3
-is_fp_closed = response.is_fp_closed
-
-class DeflateDecoder:
- def __init__(self) -> None: ...
- def __getattr__(self, name): ...
- def decompress(self, data): ...
-
-class GzipDecoder:
- def __init__(self) -> None: ...
- def __getattr__(self, name): ...
- def decompress(self, data): ...
-
-class HTTPResponse(IO[Any]):
- CONTENT_DECODERS = ... # type: Any
- REDIRECT_STATUSES = ... # type: Any
- headers = ... # type: Any
- status = ... # type: Any
- version = ... # type: Any
- reason = ... # type: Any
- strict = ... # type: Any
- decode_content = ... # type: Any
- def __init__(self, body=..., headers=..., status=..., version=..., reason=..., strict=..., preload_content=..., decode_content=..., original_response=..., pool=..., connection=...) -> None: ...
- def get_redirect_location(self): ...
- def release_conn(self): ...
- @property
- def data(self): ...
- def tell(self): ...
- def read(self, amt=..., decode_content=..., cache_content=...): ...
- def stream(self, amt=..., decode_content=...): ...
- @classmethod
- def from_httplib(ResponseCls, r, **response_kw): ...
- def getheaders(self): ...
- def getheader(self, name, default=...): ...
- def close(self): ...
- @property
- def closed(self): ...
- def fileno(self): ...
- def flush(self): ...
- def readable(self): ...
- def readinto(self, b): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/__init__.pyi
deleted file mode 100644
index eca2ea93d..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/__init__.pyi
+++ /dev/null
@@ -1,7 +0,0 @@
-# Stubs for requests.packages.urllib3.util (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from . import connection
-from . import request
-
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/connection.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/connection.pyi
deleted file mode 100644
index cd673098c..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/connection.pyi
+++ /dev/null
@@ -1,11 +0,0 @@
-# Stubs for requests.packages.urllib3.util.connection (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-
-poll = ... # type: Any
-select = ... # type: Any
-
-def is_connection_dropped(conn): ...
-def create_connection(address, timeout=..., source_address=..., socket_options=...): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/request.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/request.pyi
deleted file mode 100644
index 20a6ea277..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/request.pyi
+++ /dev/null
@@ -1,12 +0,0 @@
-# Stubs for requests.packages.urllib3.util.request (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-#from ..packages import six
-
-#b = six.b
-
-ACCEPT_ENCODING = ... # type: Any
-
-def make_headers(keep_alive=..., accept_encoding=..., user_agent=..., basic_auth=..., proxy_basic_auth=..., disable_cache=...): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/response.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/response.pyi
deleted file mode 100644
index 761a00679..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/response.pyi
+++ /dev/null
@@ -1,5 +0,0 @@
-# Stubs for requests.packages.urllib3.util.response (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-def is_fp_closed(obj): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/retry.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/retry.pyi
deleted file mode 100644
index e958d9061..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/retry.pyi
+++ /dev/null
@@ -1,36 +0,0 @@
-# Stubs for requests.packages.urllib3.util.retry (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from .. import exceptions
-from .. import packages
-
-ConnectTimeoutError = exceptions.ConnectTimeoutError
-MaxRetryError = exceptions.MaxRetryError
-ProtocolError = exceptions.ProtocolError
-ReadTimeoutError = exceptions.ReadTimeoutError
-ResponseError = exceptions.ResponseError
-
-log = ... # type: Any
-
-class Retry:
- DEFAULT_METHOD_WHITELIST = ... # type: Any
- BACKOFF_MAX = ... # type: Any
- total = ... # type: Any
- connect = ... # type: Any
- read = ... # type: Any
- redirect = ... # type: Any
- status_forcelist = ... # type: Any
- method_whitelist = ... # type: Any
- backoff_factor = ... # type: Any
- raise_on_redirect = ... # type: Any
- def __init__(self, total=..., connect=..., read=..., redirect=..., method_whitelist=..., status_forcelist=..., backoff_factor=..., raise_on_redirect=..., _observed_errors=...) -> None: ...
- def new(self, **kw): ...
- @classmethod
- def from_int(cls, retries, redirect=..., default=...): ...
- def get_backoff_time(self): ...
- def sleep(self): ...
- def is_forced_retry(self, method, status_code): ...
- def is_exhausted(self): ...
- def increment(self, method=..., url=..., response=..., error=..., _pool=..., _stacktrace=...): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/timeout.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/timeout.pyi
deleted file mode 100644
index 0a7653c5f..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/timeout.pyi
+++ /dev/null
@@ -1,24 +0,0 @@
-# Stubs for requests.packages.urllib3.util.timeout (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from .. import exceptions
-
-TimeoutStateError = exceptions.TimeoutStateError
-
-def current_time(): ...
-
-class Timeout:
- DEFAULT_TIMEOUT = ... # type: Any
- total = ... # type: Any
- def __init__(self, total=..., connect=..., read=...) -> None: ...
- @classmethod
- def from_float(cls, timeout): ...
- def clone(self): ...
- def start_connect(self): ...
- def get_connect_duration(self): ...
- @property
- def connect_timeout(self): ...
- @property
- def read_timeout(self): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/url.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/url.pyi
deleted file mode 100644
index 9877b4a17..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/packages/urllib3/util/url.pyi
+++ /dev/null
@@ -1,26 +0,0 @@
-# Stubs for requests.packages.urllib3.util.url (Python 3.4)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from .. import exceptions
-
-LocationParseError = exceptions.LocationParseError
-
-url_attrs = ... # type: Any
-
-class Url:
- slots = ... # type: Any
- def __new__(cls, scheme=..., auth=..., host=..., port=..., path=..., query=..., fragment=...): ...
- @property
- def hostname(self): ...
- @property
- def request_uri(self): ...
- @property
- def netloc(self): ...
- @property
- def url(self): ...
-
-def split_first(s, delims): ...
-def parse_url(url): ...
-def get_host(url): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/sessions.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/sessions.pyi
deleted file mode 100644
index 7c51e9736..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/sessions.pyi
+++ /dev/null
@@ -1,92 +0,0 @@
-# Stubs for requests.sessions (Python 3)
-
-from typing import Any, AnyStr, Union, MutableMapping
-from . import auth
-from . import compat
-from . import cookies
-from . import models
-from .models import Response
-from . import hooks
-from . import utils
-from . import exceptions
-from .packages.urllib3 import _collections
-from . import structures
-from . import adapters
-from . import status_codes
-
-OrderedDict = compat.OrderedDict
-cookiejar_from_dict = cookies.cookiejar_from_dict
-extract_cookies_to_jar = cookies.extract_cookies_to_jar
-RequestsCookieJar = cookies.RequestsCookieJar
-merge_cookies = cookies.merge_cookies
-Request = models.Request
-PreparedRequest = models.PreparedRequest
-DEFAULT_REDIRECT_LIMIT = models.DEFAULT_REDIRECT_LIMIT
-default_hooks = hooks.default_hooks
-dispatch_hook = hooks.dispatch_hook
-to_key_val_list = utils.to_key_val_list
-default_headers = utils.default_headers
-to_native_string = utils.to_native_string
-TooManyRedirects = exceptions.TooManyRedirects
-InvalidSchema = exceptions.InvalidSchema
-ChunkedEncodingError = exceptions.ChunkedEncodingError
-ContentDecodingError = exceptions.ContentDecodingError
-RecentlyUsedContainer = _collections.RecentlyUsedContainer
-CaseInsensitiveDict = structures.CaseInsensitiveDict
-HTTPAdapter = adapters.HTTPAdapter
-requote_uri = utils.requote_uri
-get_environ_proxies = utils.get_environ_proxies
-get_netrc_auth = utils.get_netrc_auth
-should_bypass_proxies = utils.should_bypass_proxies
-get_auth_from_url = utils.get_auth_from_url
-codes = status_codes.codes
-REDIRECT_STATI = models.REDIRECT_STATI
-
-REDIRECT_CACHE_SIZE = ... # type: Any
-
-def merge_setting(request_setting, session_setting, dict_class=...): ...
-def merge_hooks(request_hooks, session_hooks, dict_class=...): ...
-
-class SessionRedirectMixin:
- def resolve_redirects(self, resp, req, stream=..., timeout=..., verify=..., cert=...,
- proxies=...): ...
- def rebuild_auth(self, prepared_request, response): ...
- def rebuild_proxies(self, prepared_request, proxies): ...
-
-class Session(SessionRedirectMixin):
- __attrs__ = ... # type: Any
- headers = ... # type: MutableMapping[str, str]
- auth = ... # type: Any
- proxies = ... # type: Any
- hooks = ... # type: Any
- params = ... # type: Any
- stream = ... # type: Any
- verify = ... # type: Any
- cert = ... # type: Any
- max_redirects = ... # type: Any
- trust_env = ... # type: Any
- cookies = ... # type: Any
- adapters = ... # type: Any
- redirect_cache = ... # type: Any
- def __init__(self) -> None: ...
- def __enter__(self) -> 'Session': ...
- def __exit__(self, *args) -> None: ...
- def prepare_request(self, request): ...
- def request(self, method: str, url: str, params=..., data=..., headers=...,
- cookies=..., files=..., auth=..., timeout=..., allow_redirects=...,
- proxies=..., hooks=..., stream=..., verify=..., cert=...,
- json=...) -> Response: ...
- def get(self, url: AnyStr, **kwargs) -> Response: ...
- def options(self, url: str, **kwargs) -> Response: ...
- def head(self, url: str, **kwargs) -> Response: ...
- def post(self, url: str, data=..., json=..., **kwargs) -> Response: ...
- def put(self, url: str, data=..., **kwargs) -> Response: ...
- def patch(self, url: str, data=..., **kwargs) -> Response: ...
- def delete(self, url: str, **kwargs) -> Response: ...
- def send(self, request, **kwargs): ...
- def merge_environment_settings(self, url, proxies, stream, verify, cert): ...
- def get_adapter(self, url): ...
- def close(self) -> None: ...
- def mount(self, prefix, adapter): ...
-
-def session() -> Session: ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/status_codes.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/status_codes.pyi
deleted file mode 100644
index e3035eb91..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/status_codes.pyi
+++ /dev/null
@@ -1,8 +0,0 @@
-# Stubs for requests.status_codes (Python 3)
-#
-# NOTE: This dynamically typed stub was automatically generated by stubgen.
-
-from typing import Any
-from .structures import LookupDict
-
-codes = ... # type: Any
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/structures.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/structures.pyi
deleted file mode 100644
index 837cf2501..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/structures.pyi
+++ /dev/null
@@ -1,21 +0,0 @@
-# Stubs for requests.structures (Python 3)
-
-from typing import Any
-import collections
-
-class CaseInsensitiveDict(collections.MutableMapping):
- def __init__(self, data=..., **kwargs) -> None: ...
- def __setitem__(self, key, value): ...
- def __getitem__(self, key): ...
- def __delitem__(self, key): ...
- def __iter__(self): ...
- def __len__(self): ...
- def lower_items(self): ...
- def __eq__(self, other): ...
- def copy(self): ...
-
-class LookupDict(dict):
- name = ... # type: Any
- def __init__(self, name=...) -> None: ...
- def __getitem__(self, key): ...
- def get(self, key, default=...): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/requests/utils.pyi b/v1.1.0-dev1/salad/typeshed/2.7/requests/utils.pyi
deleted file mode 100644
index 945277afc..000000000
--- a/v1.1.0-dev1/salad/typeshed/2.7/requests/utils.pyi
+++ /dev/null
@@ -1,52 +0,0 @@
-# Stubs for requests.utils (Python 3)
-
-from typing import Any
-from . import compat
-from . import cookies
-from . import structures
-from . import exceptions
-
-OrderedDict = compat.OrderedDict
-RequestsCookieJar = cookies.RequestsCookieJar
-cookiejar_from_dict = cookies.cookiejar_from_dict
-CaseInsensitiveDict = structures.CaseInsensitiveDict
-InvalidURL = exceptions.InvalidURL
-
-NETRC_FILES = ... # type: Any
-DEFAULT_CA_BUNDLE_PATH = ... # type: Any
-
-def dict_to_sequence(d): ...
-def super_len(o): ...
-def get_netrc_auth(url): ...
-def guess_filename(obj): ...
-def from_key_val_list(value): ...
-def to_key_val_list(value): ...
-def parse_list_header(value): ...
-def parse_dict_header(value): ...
-def unquote_header_value(value, is_filename=...): ...
-def dict_from_cookiejar(cj): ...
-def add_dict_to_cookiejar(cj, cookie_dict): ...
-def get_encodings_from_content(content): ...
-def get_encoding_from_headers(headers): ...
-def stream_decode_response_unicode(iterator, r): ...
-def iter_slices(string, slice_length): ...
-def get_unicode_from_response(r): ...
-
-UNRESERVED_SET = ... # type: Any
-
-def unquote_unreserved(uri): ...
-def requote_uri(uri): ...
-def address_in_network(ip, net): ...
-def dotted_netmask(mask): ...
-def is_ipv4_address(string_ip): ...
-def is_valid_cidr(string_network): ...
-def should_bypass_proxies(url): ...
-def get_environ_proxies(url): ...
-def default_user_agent(name=...): ...
-def default_headers(): ...
-def parse_header_links(value): ...
-def guess_json_utf(data): ...
-def prepend_scheme_if_needed(url, new_scheme): ...
-def get_auth_from_url(url): ...
-def to_native_string(string, encoding=...): ...
-def urldefragauth(url): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/avro/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2and3/avro/__init__.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/avro/__init__.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/avro/__init__.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/avro/schema.pyi b/v1.1.0-dev1/salad/typeshed/2and3/avro/schema.pyi
similarity index 91%
rename from v1.1.0-dev1/salad/typeshed/2.7/avro/schema.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/avro/schema.pyi
index fcc43db39..05f48e516 100644
--- a/v1.1.0-dev1/salad/typeshed/2.7/avro/schema.pyi
+++ b/v1.1.0-dev1/salad/typeshed/2and3/avro/schema.pyi
@@ -2,7 +2,7 @@
#
# NOTE: This dynamically typed stub was automatically generated by stubgen.
-from typing import Any, Union, Dict, List
+from typing import Any, Union, Dict, List, Text
PRIMITIVE_TYPES = ... # type: Any
NAMED_TYPES = ... # type: Any
@@ -32,9 +32,9 @@ class Name:
class Names:
names = ... # type: Any
default_namespace = ... # type: Any
- def __init__(self, default_namespace: Union[str, unicode] = None) -> None: ...
- def has_name(self, name_attr: Union[str, unicode], space_attr: Union[str, unicode]) -> bool: ...
- def get_name(self, name_attr: Union[str, unicode], space_attr: Union[str, unicode]) -> Schema: ...
+ def __init__(self, default_namespace: Union[str, Text] = None) -> None: ...
+ def has_name(self, name_attr: Union[str, Text], space_attr: Union[str, Text]) -> bool: ...
+ def get_name(self, name_attr: Union[str, Text], space_attr: Union[str, Text]) -> Schema: ...
def prune_namespace(self, properties): ...
def add_name(self, name_attr, space_attr, new_schema): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/__init__.pyi
new file mode 100644
index 000000000..dba67bc37
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/__init__.pyi
@@ -0,0 +1,10 @@
+# Stubs for cachecontrol (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Any
+from .wrapper import CacheControl as CacheControl
+#from .adapter import CacheControlAdapter as CacheControlAdapter
+#from .controller import CacheController as CacheController
+
+__email__ = ... # type: Any
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/adapter.pyi b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/adapter.pyi
new file mode 100644
index 000000000..734fe1448
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/adapter.pyi
@@ -0,0 +1,19 @@
+# Stubs for cachecontrol.adapter (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Any
+from requests.adapters import HTTPAdapter
+from .controller import CacheController as CacheController
+from .cache import DictCache as DictCache
+from .filewrapper import CallbackFileWrapper as CallbackFileWrapper
+
+class CacheControlAdapter(HTTPAdapter):
+ invalidating_methods = ... # type: Any
+ cache = ... # type: Any
+ heuristic = ... # type: Any
+ controller = ... # type: Any
+ def __init__(self, cache=None, cache_etags=True, controller_class=None, serializer=None, heuristic=None, *args, **kw): ...
+ def send(self, request, **kw): ...
+ def build_response(self, request, response, from_cache=False): ...
+ def close(self): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/cache.pyi b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/cache.pyi
new file mode 100644
index 000000000..c4f507b41
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/cache.pyi
@@ -0,0 +1,19 @@
+# Stubs for cachecontrol.cache (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Any
+
+class BaseCache:
+ def get(self, key): ...
+ def set(self, key, value): ...
+ def delete(self, key): ...
+ def close(self): ...
+
+class DictCache(BaseCache):
+ lock = ... # type: Any
+ data = ... # type: Any
+ def __init__(self, init_dict=None): ...
+ def get(self, key): ...
+ def set(self, key, value): ...
+ def delete(self, key): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/caches/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/caches/__init__.pyi
new file mode 100644
index 000000000..b7784c2bc
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/caches/__init__.pyi
@@ -0,0 +1,9 @@
+# Stubs for cachecontrol.caches (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Any
+from .file_cache import FileCache as FileCache
+#from .redis_cache import RedisCache as RedisCache
+
+notice = ... # type: Any
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/caches/file_cache.pyi b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/caches/file_cache.pyi
new file mode 100644
index 000000000..2d6ca0c54
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/caches/file_cache.pyi
@@ -0,0 +1,22 @@
+# Stubs for cachecontrol.caches.file_cache (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Any, Callable
+from ..cache import BaseCache as BaseCache
+from ..controller import CacheController as CacheController
+
+class FileCache(BaseCache):
+ directory = ... # type: str
+ forever = ... # type: bool
+ filemode = ... # type: str
+ dirmode = ... # type: str
+ lock_class = ... # type: Callable
+ def __init__(self, directory: str, forever=False, filemode=384, dirmode=448, use_dir_lock=None, lock_class=None) -> None: ...
+ @staticmethod
+ def encode(x): ...
+ def get(self, key): ...
+ def set(self, key, value): ...
+ def delete(self, key): ...
+
+def url_to_file_path(url, filecache): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/compat.pyi b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/compat.pyi
new file mode 100644
index 000000000..0926ecc5e
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/compat.pyi
@@ -0,0 +1,7 @@
+# Stubs for cachecontrol.compat (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Any
+
+text_type = ... # type: Any
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/controller.pyi b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/controller.pyi
new file mode 100644
index 000000000..1161e797d
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/controller.pyi
@@ -0,0 +1,25 @@
+# Stubs for cachecontrol.controller (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Any
+#from .cache import DictCache as DictCache
+#from .serialize import Serializer as Serializer
+
+logger = ... # type: Any
+URI = ... # type: Any
+
+def parse_uri(uri): ...
+
+class CacheController:
+ cache = ... # type: Any
+ cache_etags = ... # type: Any
+ serializer = ... # type: Any
+ def __init__(self, cache=None, cache_etags=True, serializer=None): ...
+ @classmethod
+ def cache_url(cls, uri): ...
+ def parse_cache_control(self, headers): ...
+ def cached_request(self, request): ...
+ def conditional_headers(self, request): ...
+ def cache_response(self, request, response, body=None): ...
+ def update_cached_response(self, request, response): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/filewrapper.pyi b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/filewrapper.pyi
new file mode 100644
index 000000000..d21df1508
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/filewrapper.pyi
@@ -0,0 +1,8 @@
+# Stubs for cachecontrol.filewrapper (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+class CallbackFileWrapper:
+ def __init__(self, fp, callback): ...
+ def __getattr__(self, name): ...
+ def read(self, amt=None): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/serialize.pyi b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/serialize.pyi
new file mode 100644
index 000000000..a0f9eec87
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/serialize.pyi
@@ -0,0 +1,10 @@
+# Stubs for cachecontrol.serialize (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from .compat import HTTPResponse as HTTPResponse, pickle as pickle, text_type as text_type
+
+class Serializer:
+ def dumps(self, request, response, body=None): ...
+ def loads(self, request, data): ...
+ def prepare_response(self, request, cached): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/wrapper.pyi b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/wrapper.pyi
new file mode 100644
index 000000000..3544f51d3
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/cachecontrol/wrapper.pyi
@@ -0,0 +1,14 @@
+# Stubs for cachecontrol.wrapper (Python 2)
+#
+# NOTE: This dynamically typed stub was automatically generated by stubgen.
+
+from typing import Union
+from .adapter import CacheControlAdapter as CacheControlAdapter
+from .cache import DictCache, BaseCache
+import requests
+
+def CacheControl(sess: requests.sessions.Session,
+ cache: Union[DictCache, BaseCache] = None,
+ cache_etags: bool = True,
+ serializer=None,
+ heuristic=None) -> requests.sessions.Session: ...
diff --git a/v1.1.0-dev1/salad/typeshed/2and3/mistune.pyi b/v1.1.0-dev1/salad/typeshed/2and3/mistune.pyi
new file mode 100644
index 000000000..d267d9216
--- /dev/null
+++ b/v1.1.0-dev1/salad/typeshed/2and3/mistune.pyi
@@ -0,0 +1,271 @@
+__author__ = "Aleksandr Slepchenkov"
+__email__ = "Sl.aleksandr28@gmail.com"
+
+from typing import Any, Optional, Pattern, List, Text, Tuple, Dict, Match, Type, Sequence, Iterable
+
+Tokens = List[Dict[Text, Any]]
+# There are too much levels of optional unions of lists of text in cell and align 385 and 396 lines in mistune
+
+
+def escape(text: Text, quote: bool = ..., smart_amp: bool = ...) -> Text: ...
+
+
+class BlockGrammar:
+ def_links = ... # type: Pattern
+ def_footnotes = ... # type: Pattern
+ newline = ... # type: Pattern
+ block_code = ... # type: Pattern
+ fences = ... # type: Pattern
+ hrule = ... # type: Pattern
+ heading = ... # type: Pattern
+ lheading = ... # type: Pattern
+ block_quote = ... # type: Pattern
+ list_block = ... # type: Pattern
+ list_item = ... # type: Pattern
+ list_bullet = ... # type: Pattern
+ paragraph = ... # type: Pattern
+ block_html = ... # type: Pattern
+ table = ... # type: Pattern
+ nptable = ... # type: Pattern
+ text = ... # type: Pattern
+
+
+class BlockLexer:
+ grammar_class = ... # type: Type[BlockGrammar]
+ default_rules = ... # type: List[Text]
+ list_rules = ... # type: Tuple[Text]
+ footnote_rules = ... # type: Tuple[Text]
+ tokens = ... # type: Tokens
+ def_links = ... # type: Dict[Text, Dict[Text, Text]]
+ def_footnotes = ... # type: Dict[Text, int]
+ rules = ... # type: BlockGrammar
+
+ def __init__(self, rules: Optional[BlockGrammar] = ..., **kwargs) -> None: ...
+
+ def __call__(self, text: Text, rules: Optional[Sequence[Text]] = ...) -> Tokens: ...
+
+ def parse(self, text: Text, rules: Optional[Sequence[Text]] = ...) -> Tokens: ...
+
+ def parse_newline(self, m: Match) -> None: ...
+
+ def parse_block_code(self, m: Match) -> None: ...
+
+ def parse_fences(self, m: Match) -> None: ...
+
+ def parse_heading(self, m: Match) -> None: ...
+
+ def parse_lheading(self, m: Match) -> None: ...
+
+ def parse_hrule(self, m: Match) -> None: ...
+
+ def parse_list_block(self, m: Match) -> None: ...
+
+ def parse_block_quote(self, m: Match) -> None: ...
+
+ def parse_def_links(self, m: Match) -> None: ...
+
+ def parse_def_footnotes(self, m: Match) -> None: ...
+
+ def parse_table(self, m: Match) -> None: ...
+
+ def parse_nptable(self, m: Match) -> None: ...
+
+ def parse_block_html(self, m: Match) -> None: ...
+
+ def parse_paragraph(self, m: Match) -> None: ...
+
+ def parse_text(self, m: Match) -> None: ...
+
+
+class InlineGrammar:
+ escape = ... # type: Pattern
+ inline_html = ... # type: Pattern
+ autolink = ... # type: Pattern
+ link = ... # type: Pattern
+ reflink = ... # type: Pattern
+ nolink = ... # type: Pattern
+ url = ... # type: Pattern
+ double_emphasis = ... # type: Pattern
+ emphasis = ... # type: Pattern
+ code = ... # type: Pattern
+ linebreak = ... # type: Pattern
+ strikethrough = ... # type: Pattern
+ footnote = ... # type: Pattern
+ text = ... # type: Pattern
+
+ def hard_wrap(self) -> None: ...
+
+
+class InlineLexer:
+ grammar_class = ... # type: Type[InlineGrammar]
+ default_rules = ... # type: List[Text]
+ inline_html_rules = ... # type: List[Text]
+ renderer = ... # type: Renderer
+ links = ... # type: Dict[Any, Dict]
+ footnotes = ... # type: Dict[Text, int]
+ footnote_index = ... # type: int
+ _in_link = ... # type: bool
+ _in_footnote = ... # type: bool
+ _parse_inline_html = ... # type: bool
+ rules = ... # type: InlineGrammar
+
+ def __init__(self, renderer: Renderer, rules: Optional[InlineGrammar] = ..., **kwargs) -> None: ...
+
+ def __call__(self, text: Text, rules: Optional[Sequence[Text]] = ...) -> Text: ...
+
+ def setup(self, links: Optional[Dict[Any, Dict]], footnotes: Optional[Dict[Text, int]]) -> None: ...
+
+ line_match = ... # type: Match
+ line_started = ... # type: bool
+
+ def output(self, text: Text, rules: Optional[Sequence[Text]] = ...) -> Text: ...
+
+ def output_escape(self, m: Match) -> Text: ...
+
+ def output_autolink(self, m: Match) -> Text: ...
+
+ def output_url(self, m: Match) -> Text: ...
+
+ def output_inline_html(self, m: Match) -> Text: ...
+
+ def output_footnote(self, m: Match) -> Optional[Text]: ...
+
+ def output_link(self, m: Match) -> Text: ...
+
+ def output_reflink(self, m: Match) -> Optional[Text]: ...
+
+ def output_nolink(self, m: Match) -> Optional[Text]: ...
+
+ def output_double_emphasis(self, m: Match) -> Text: ...
+
+ def output_emphasis(self, m: Match) -> Text: ...
+
+ def output_code(self, m: Match) -> Text: ...
+
+ def output_linebreak(self, m: Match) -> Text: ...
+
+ def output_strikethrough(self, m: Match) -> Text: ...
+
+ def output_text(self, m: Match) -> Text: ...
+
+
+class Renderer:
+ options = ... # type: Dict
+
+ def __init__(self, **kwargs) -> None: ...
+
+ def placeholder(self) -> Text: ...
+
+ def block_code(self, code: Text,
+ lang: Any = ...) -> Text: ... # It seems that lang should be string, however other types are valid as well
+
+ def block_quote(self, text: Text) -> Text: ...
+
+ def block_html(self, html: Text) -> Text: ...
+
+ def header(self, text: Text, level: int, raw: Optional[Text] = ...) -> Text: ...
+
+ def hrule(self) -> Text: ...
+
+ def list(self, body: Any,
+ ordered: bool = ...) -> Text: ... # body - same reason as for lang above, and for other Any in this class
+
+ def list_item(self, text: Any) -> Text: ...
+
+ def paragraph(self, text: Text) -> Text: ...
+
+ def table(self, header: Any, body: Any) -> Text: ...
+
+ def table_row(self, content: Any) -> Text: ...
+
+ def table_cell(self, content: Any, **flags) -> Text: ...
+
+ def double_emphasis(self, text: Any) -> Text: ...
+
+ def emphasis(self, text: Any) -> Text: ...
+
+ def codespan(self, text: Text) -> Text: ...
+
+ def linebreak(self) -> Text: ...
+
+ def strikethrough(self, text: Any) -> Text: ...
+
+ def text(self, text: Any) -> Text: ...
+
+ def escape(self, text: Any) -> Text: ...
+
+ def autolink(self, link: Any, is_email: bool = ...) -> Text: ...
+
+ def link(self, link: Any, title: Any, text: Any) -> Text: ...
+
+ def image(self, src: Any, title: Any, text: Any) -> Text: ...
+
+ def inline_html(self, html: Any) -> Text: ...
+
+ def newline(self) -> Text: ...
+
+ def footnote_ref(self, key: Any, index: int) -> Text: ...
+
+ def footnote_item(self, key: Any, text: Text) -> Text: ...
+
+ def footnotes(self, text: Any) -> Text: ...
+
+
+class Markdown:
+ renderer = ... # type: Renderer
+ inline = ... # type: InlineLexer
+ block = ... # type: BlockLexer
+ footnotes = ... # type: List[Dict[Text, Any]]
+ tokens = ... # type: Tokens
+
+ def __init__(self, renderer: Optional[Renderer] = ..., inline: Optional[InlineLexer] = ...,
+ block: Optional[BlockLexer] = ..., **kwargs) -> None: ...
+
+ def __call__(self, text: Text) -> Text: ...
+
+ def render(self, text: Text) -> Text: ...
+
+ def parse(self, text: Text) -> Text: ...
+
+ token = ... # type: Dict[Text, Any]
+
+ def pop(self) -> Optional[Dict[Text, Any]]: ...
+
+ def peek(self) -> Optional[Dict[Text, Any]]: ...
+
+ def output(self, text: Text, rules: Optional[Sequence[Text]] = ...): ...
+
+ def tok(self) -> Text: ...
+
+ def tok_text(self) -> Text: ...
+
+ def output_newline(self) -> Text: ...
+
+ def output_hrule(self) -> Text: ...
+
+ def output_heading(self) -> Text: ...
+
+ def output_code(self) -> Text: ...
+
+ def output_table(self) -> Text: ...
+
+ def output_block_quote(self) -> Text: ...
+
+ def output_list(self) -> Text: ...
+
+ def output_list_item(self) -> Text: ...
+
+ def output_loose_item(self) -> Text: ...
+
+ def output_footnote(self) -> Text: ...
+
+ def output_close_html(self) -> Text: ...
+
+ def output_open_html(self) -> Text: ...
+
+ def output_paragraph(self) -> Text: ...
+
+ def output_text(self) -> Text: ...
+
+
+def markdown(text: Text, escape: bool = ..., **kwargs) -> Text: ...
\ No newline at end of file
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/__init__.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/__init__.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/__init__.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/events.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/events.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/events.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/events.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/exceptions.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/exceptions.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/exceptions.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/exceptions.pyi
diff --git a/v1.0/salad/typeshed/2.7/rdflib/graph.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/graph.pyi
similarity index 93%
rename from v1.0/salad/typeshed/2.7/rdflib/graph.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/graph.pyi
index 94a9f3f8f..6972611ea 100644
--- a/v1.0/salad/typeshed/2.7/rdflib/graph.pyi
+++ b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/graph.pyi
@@ -2,8 +2,8 @@
#
# NOTE: This dynamically typed stub was automatically generated by stubgen.
-from typing import Any, AnyStr, Union, IO, Tuple, Iterator
-from StringIO import StringIO as BytesIO
+from typing import Any, AnyStr, Dict, Union, IO, Tuple, Iterator, Text
+from io import StringIO as BytesIO
from rdflib.term import Node, URIRef
from rdflib.store import Store
from rdflib.namespace import NamespaceManager
@@ -25,8 +25,8 @@ class Graph(Node):
def close(self, commit_pending_transaction=False): ...
def add(self, __tuple_arg_2: Tuple[Node, Node, Node]) -> None: ...
def addN(self, quads): ...
- def remove(self, __tuple_arg_2): ...
- def triples(self, __tuple_arg_2: Tuple[Union[AnyStr, URIRef], AnyStr, Union[AnyStr, URIRef]]) -> Iterator[Tuple[AnyStr, AnyStr, AnyStr]]: ...
+ def remove(self, __tuple_arg_2: Tuple[Union[AnyStr, URIRef], AnyStr, Union[AnyStr, URIRef]]) -> None: ...
+ def triples(self, __tuple_arg_2: Tuple[Union[AnyStr, URIRef, None], Union[AnyStr, None], Union[AnyStr, URIRef, None]]) -> Iterator[Tuple[AnyStr, AnyStr, AnyStr]]: ...
def __getitem__(self, item): ...
def __len__(self): ...
def __iter__(self): ...
@@ -71,8 +71,8 @@ class Graph(Node):
def absolutize(self, uri, defrag=1): ...
def serialize(self, destination: Union[str, IO[Any]]=None, format: str='', base: str=None, encoding: str=None, **args) -> Union[bytes, None]: ...
def parse(self, source: str = None, publicID: str = None,
- format: Union[str, unicode] = None,
- location: Union[str, unicode] = None, file: IO[Any] = None,
+ format: Text = None,
+ location: Text = None, file: IO[Any] = None,
data: str = None, **args): ...
def load(self, source, publicID=None, format=''): ...
def query(self, query_object, processor: str = '', result: str = '', initNs: Dict = None, initBindings: Dict = None, use_store_provided: bool = True, **kwargs) -> Result: ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/namespace.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/namespace.pyi
similarity index 90%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/namespace.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/namespace.pyi
index 01197c745..2a0a6816d 100644
--- a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/namespace.pyi
+++ b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/namespace.pyi
@@ -2,9 +2,9 @@
#
# NOTE: This dynamically typed stub was automatically generated by stubgen.
-from typing import Any, Tuple, Union
+from typing import Any, Tuple, Union, Text
-class Namespace(unicode):
+class Namespace(Text):
__doc__ = ... # type: Any
def __new__(cls, value): ...
@property
@@ -13,7 +13,7 @@ class Namespace(unicode):
def __getitem__(self, key, default=None): ...
def __getattr__(self, name): ...
-class URIPattern(unicode):
+class URIPattern(Text):
__doc__ = ... # type: Any
def __new__(cls, value): ...
def __mod__(self, *args, **kwargs): ...
@@ -57,4 +57,4 @@ def is_ncname(name): ...
XMLNS = ... # type: Any
-def split_uri(uri: Union[str, unicode]) -> Tuple[str, str]: ...
+def split_uri(uri: Text) -> Tuple[str, str]: ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/parser.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/parser.pyi
similarity index 84%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/parser.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/parser.pyi
index 3ff074d8c..fa1ec9a74 100644
--- a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/parser.pyi
+++ b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/parser.pyi
@@ -3,7 +3,7 @@
# NOTE: This dynamically typed stub was automatically generated by stubgen.
from typing import Any
-from StringIO import StringIO as BytesIO
+from io import StringIO as BytesIO
class Parser:
def __init__(self): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/plugin.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/plugin.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/plugin.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/plugin.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/plugins/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/plugins/__init__.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/plugins/__init__.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/plugins/__init__.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/plugins/parsers/__init__.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/plugins/parsers/__init__.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/plugins/parsers/__init__.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/plugins/parsers/__init__.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/plugins/parsers/notation3.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/plugins/parsers/notation3.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/plugins/parsers/notation3.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/plugins/parsers/notation3.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/query.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/query.pyi
similarity index 97%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/query.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/query.pyi
index db002b8b0..fad3c3770 100644
--- a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/query.pyi
+++ b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/query.pyi
@@ -3,7 +3,7 @@
# NOTE: This dynamically typed stub was automatically generated by stubgen.
from typing import Any
-from StringIO import StringIO as BytesIO
+from io import StringIO as BytesIO
import collections
class Processor:
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/serializer.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/serializer.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/serializer.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/serializer.pyi
diff --git a/v1.0/salad/typeshed/2.7/rdflib/store.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/store.pyi
similarity index 97%
rename from v1.0/salad/typeshed/2.7/rdflib/store.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/store.pyi
index 4a98a44b0..6886321b0 100644
--- a/v1.0/salad/typeshed/2.7/rdflib/store.pyi
+++ b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/store.pyi
@@ -4,7 +4,7 @@
from typing import Any
from rdflib.events import Event
-from cStringIO import StringIO as BytesIO
+from io import StringIO as BytesIO
class StoreCreatedEvent(Event): ...
class TripleAddedEvent(Event): ...
diff --git a/v1.0/salad/typeshed/2.7/rdflib/term.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/term.pyi
similarity index 97%
rename from v1.0/salad/typeshed/2.7/rdflib/term.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/term.pyi
index 698218123..0021d14be 100644
--- a/v1.0/salad/typeshed/2.7/rdflib/term.pyi
+++ b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/term.pyi
@@ -2,11 +2,11 @@
#
# NOTE: This dynamically typed stub was automatically generated by stubgen.
-from typing import Any
+from typing import Any, Text
class Node: ...
-class Identifier(Node, unicode):
+class Identifier(Node, Text):
def __new__(cls, value): ...
def eq(self, other): ...
def neq(self, other): ...
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/rdflib/util.pyi b/v1.1.0-dev1/salad/typeshed/2and3/rdflib/util.pyi
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/rdflib/util.pyi
rename to v1.1.0-dev1/salad/typeshed/2and3/rdflib/util.pyi
diff --git a/v1.1.0-dev1/salad/typeshed/2.7/ruamel/__init__.py b/v1.1.0-dev1/salad/typeshed/2and3/ruamel/__init__.py
similarity index 100%
rename from v1.1.0-dev1/salad/typeshed/2.7/ruamel/__init__.py
rename to v1.1.0-dev1/salad/typeshed/2and3/ruamel/__init__.py
diff --git a/v1.1.0-dev1/v1.1.0-dev1/binding-test.cwl b/v1.1.0-dev1/v1.1.0-dev1/binding-test.cwl
index 66a74b8b9..304a13dfc 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/binding-test.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/binding-test.cwl
@@ -2,7 +2,9 @@
class: CommandLineTool
cwlVersion: v1.1.0-dev1
-
+hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
inputs:
- id: reference
type: File
diff --git a/v1.1.0-dev1/v1.1.0-dev1/bwa-mem-tool.cwl b/v1.1.0-dev1/v1.1.0-dev1/bwa-mem-tool.cwl
index b431df0ee..1bf551453 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/bwa-mem-tool.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/bwa-mem-tool.cwl
@@ -6,7 +6,7 @@ class: CommandLineTool
hints:
- class: ResourceRequirement
- coresMin: 4
+ coresMin: 2
- class: DockerRequirement
dockerPull: python:2-slim
diff --git a/v1.1.0-dev1/v1.1.0-dev1/cat1-testcli.cwl b/v1.1.0-dev1/v1.1.0-dev1/cat1-testcli.cwl
index aa40435c9..888e4a984 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/cat1-testcli.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/cat1-testcli.cwl
@@ -3,6 +3,12 @@
"class": "CommandLineTool",
"cwlVersion": "v1.1.0-dev1",
"doc": "Print the contents of a file to stdout using 'cat' running in a docker container.",
+ "hints": [
+ {
+ "class": "DockerRequirement",
+ "dockerPull": "python:2-slim"
+ }
+ ],
"inputs": [
{
"id": "file1",
diff --git a/v1.1.0-dev1/v1.1.0-dev1/cat1-tool.cwl b/v1.1.0-dev1/v1.1.0-dev1/cat1-tool.cwl
deleted file mode 100755
index dcfd3fe48..000000000
--- a/v1.1.0-dev1/v1.1.0-dev1/cat1-tool.cwl
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/usr/bin/env cwl-runner
-cwlVersion: v1.1.0-dev1
-class: CommandLineTool
-doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
-hints:
- DockerRequirement:
- dockerPull: debian:wheezy
- SoftwareRequirement:
- packages:
- - name: cat
-inputs:
- file1:
- type: File
- inputBinding: {position: 1}
- numbering:
- type: boolean?
- inputBinding:
- position: 0
- prefix: -n
-outputs: []
-baseCommand: cat
diff --git a/v1.1.0-dev1/v1.1.0-dev1/cat3-tool-mediumcut.cwl b/v1.1.0-dev1/v1.1.0-dev1/cat3-tool-mediumcut.cwl
index 2b4835010..15f6ec695 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/cat3-tool-mediumcut.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/cat3-tool-mediumcut.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.1.0-dev1
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1:
type: File
diff --git a/v1.1.0-dev1/v1.1.0-dev1/cat3-tool-shortcut.cwl b/v1.1.0-dev1/v1.1.0-dev1/cat3-tool-shortcut.cwl
index 4909ef79d..36a823fab 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/cat3-tool-shortcut.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/cat3-tool-shortcut.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.1.0-dev1
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1:
type: File
diff --git a/v1.1.0-dev1/v1.1.0-dev1/cat3-tool.cwl b/v1.1.0-dev1/v1.1.0-dev1/cat3-tool.cwl
index e19b986e9..c83f5296c 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/cat3-tool.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/cat3-tool.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.1.0-dev1
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1:
type: File
diff --git a/v1.1.0-dev1/v1.1.0-dev1/cat4-tool.cwl b/v1.1.0-dev1/v1.1.0-dev1/cat4-tool.cwl
index b07b54b93..8f3b19ad6 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/cat4-tool.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/cat4-tool.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.1.0-dev1
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1: File
outputs:
diff --git a/v1.1.0-dev1/v1.1.0-dev1/cat5-tool.cwl b/v1.1.0-dev1/v1.1.0-dev1/cat5-tool.cwl
index 6125644f9..dab01d0b8 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/cat5-tool.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/cat5-tool.cwl
@@ -6,7 +6,7 @@ class: CommandLineTool
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: "debian:wheezy"
+ dockerPull: "debian:stretch-slim"
ex:BlibberBlubberFakeRequirement:
fakeField: fraggleFroogle
inputs:
diff --git a/v1.1.0-dev1/v1.1.0-dev1/count-lines11-wf.cwl b/v1.1.0-dev1/v1.1.0-dev1/count-lines11-wf.cwl
new file mode 100755
index 000000000..0f3ba779f
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/count-lines11-wf.cwl
@@ -0,0 +1,28 @@
+#!/usr/bin/env cwl-runner
+class: Workflow
+cwlVersion: v1.1.0-dev1
+
+inputs:
+ file1: File?
+
+outputs:
+ count_output:
+ type: int
+ outputSource: step2/output
+
+steps:
+ step1:
+ run: wc-tool.cwl
+ in:
+ file1:
+ source: file1
+ default:
+ class: File
+ location: whale.txt
+ out: [output]
+
+ step2:
+ run: parseInt-tool.cwl
+ in:
+ file1: step1/output
+ out: [output]
diff --git a/v1.1.0-dev1/v1.1.0-dev1/default_path.cwl b/v1.1.0-dev1/v1.1.0-dev1/default_path.cwl
index 58d6dced3..f27f00479 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/default_path.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/default_path.cwl
@@ -1,4 +1,4 @@
-cwlVersion: v1.0
+cwlVersion: v1.1.0-dev1
class: CommandLineTool
inputs:
- id: "file1"
diff --git a/v1.1.0-dev1/v1.1.0-dev1/dir2.cwl b/v1.1.0-dev1/v1.1.0-dev1/dir2.cwl
index 34600bdfa..5d78d788d 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/dir2.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/dir2.cwl
@@ -2,7 +2,7 @@ class: CommandLineTool
cwlVersion: v1.1.0-dev1
hints:
DockerRequirement:
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
ShellCommandRequirement: {}
inputs:
indir: Directory
diff --git a/v1.1.0-dev1/v1.1.0-dev1/dir5.cwl b/v1.1.0-dev1/v1.1.0-dev1/dir5.cwl
index 177d98172..ebfe9a049 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/dir5.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/dir5.cwl
@@ -11,7 +11,7 @@ outputs:
type: File
outputBinding:
glob: output.txt
-arguments: ["find", "-L", ".",
+arguments: ["find", "-L", ".", "!", "-path", "*.txt",
{shellQuote: false, valueFrom: "|"},
"sort"]
stdout: output.txt
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/docker-array-secondaryfiles.cwl b/v1.1.0-dev1/v1.1.0-dev1/docker-array-secondaryfiles.cwl
index 8fedb9eda..67321278e 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/docker-array-secondaryfiles.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/docker-array-secondaryfiles.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.1.0-dev1
requirements:
- class: DockerRequirement
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
- class: InlineJavascriptRequirement
- class: ShellCommandRequirement
diff --git a/v1.1.0-dev1/v1.1.0-dev1/docker-output-dir.cwl b/v1.1.0-dev1/v1.1.0-dev1/docker-output-dir.cwl
index e50e1d5f2..f4deb24ea 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/docker-output-dir.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/docker-output-dir.cwl
@@ -2,7 +2,7 @@ class: CommandLineTool
cwlVersion: v1.1.0-dev1
requirements:
DockerRequirement:
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
dockerOutputDirectory: /other
inputs: []
outputs:
diff --git a/v1.1.0-dev1/v1.1.0-dev1/empty-array-input.cwl b/v1.1.0-dev1/v1.1.0-dev1/empty-array-input.cwl
new file mode 100644
index 000000000..237d1516e
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/empty-array-input.cwl
@@ -0,0 +1,32 @@
+#!/usr/bin/env cwl-runner
+
+cwlVersion: v1.1.0-dev1
+class: CommandLineTool
+
+hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
+
+inputs:
+ - id: array
+ type: { type: array, items: int }
+ inputBinding:
+ position: 1
+ prefix: -I
+ itemSeparator: ","
+
+ - id: args.py
+ type: File
+ default:
+ class: File
+ location: args.py
+ inputBinding:
+ position: -1
+
+outputs:
+ - id: args
+ type:
+ type: array
+ items: string
+
+baseCommand: python
diff --git a/v1.1.0-dev1/v1.1.0-dev1/empty-array-job.json b/v1.1.0-dev1/v1.1.0-dev1/empty-array-job.json
new file mode 100644
index 000000000..038229f2b
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/empty-array-job.json
@@ -0,0 +1,3 @@
+{
+ "array": []
+}
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/envvar2.cwl b/v1.1.0-dev1/v1.1.0-dev1/envvar2.cwl
index a3bfa7045..b7fe75b53 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/envvar2.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/envvar2.cwl
@@ -6,7 +6,7 @@ requirements:
ShellCommandRequirement: {}
hints:
DockerRequirement:
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
arguments: [
echo, {valueFrom: '"HOME=$HOME"', shellQuote: false}, {valueFrom: '"TMPDIR=$TMPDIR"', shellQuote: false},
{valueFrom: '&&', shellQuote: false},
diff --git a/v1.1.0-dev1/v1.1.0-dev1/file1-null.json b/v1.1.0-dev1/v1.1.0-dev1/file1-null.json
new file mode 100644
index 000000000..cc6b36193
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/file1-null.json
@@ -0,0 +1,3 @@
+{
+ "file1": null
+}
diff --git a/v1.1.0-dev1/v1.1.0-dev1/formattest2.cwl b/v1.1.0-dev1/v1.1.0-dev1/formattest2.cwl
index 4ee74fc6f..2118caa23 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/formattest2.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/formattest2.cwl
@@ -5,6 +5,9 @@ $schemas:
class: CommandLineTool
cwlVersion: v1.1.0-dev1
doc: "Reverse each line using the `rev` command"
+hints:
+ DockerRequirement:
+ dockerPull: "debian:stretch-slim"
inputs:
input:
diff --git a/v1.1.0-dev1/v1.1.0-dev1/formattest3.cwl b/v1.1.0-dev1/v1.1.0-dev1/formattest3.cwl
index 7d6046412..46cad7567 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/formattest3.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/formattest3.cwl
@@ -7,6 +7,9 @@ $schemas:
class: CommandLineTool
cwlVersion: v1.1.0-dev1
doc: "Reverse each line using the `rev` command"
+hints:
+ DockerRequirement:
+ dockerPull: "debian:stretch-slim"
inputs:
input:
diff --git a/v1.1.0-dev1/v1.1.0-dev1/index.py b/v1.1.0-dev1/v1.1.0-dev1/index.py
index 65acd921b..77918b5e5 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/index.py
+++ b/v1.1.0-dev1/v1.1.0-dev1/index.py
@@ -31,3 +31,7 @@
open(sys.argv[1] + ".idx3", "w")
open(sys.argv[1] + ".idx4", "w")
open(sys.argv[1] + ".idx5", "w")
+open(os.path.splitext(sys.argv[1])[0] + ".idx6" + os.path.splitext(sys.argv[1])[1], "w")
+open(sys.argv[1] + ".idx7", "w")
+os.mkdir(sys.argv[1] + "_idx8")
+open(sys.argv[1] + "_idx8/index", "w")
diff --git a/v1.1.0-dev1/v1.1.0-dev1/initialworkdirrequirement-docker-out.cwl b/v1.1.0-dev1/v1.1.0-dev1/initialworkdirrequirement-docker-out.cwl
index a80a0ae3f..5a0bb4c9b 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/initialworkdirrequirement-docker-out.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/initialworkdirrequirement-docker-out.cwl
@@ -4,7 +4,7 @@ cwlVersion: v1.1.0-dev1
requirements:
- class: DockerRequirement
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
- class: InitialWorkDirRequirement
listing:
- $(inputs.INPUT)
diff --git a/v1.1.0-dev1/v1.1.0-dev1/metadata.cwl b/v1.1.0-dev1/v1.1.0-dev1/metadata.cwl
index 7f222498d..2ee5ccc26 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/metadata.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/metadata.cwl
@@ -18,7 +18,7 @@ dct:creator:
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1:
type: File
diff --git a/v1.1.0-dev1/v1.1.0-dev1/optional-output.cwl b/v1.1.0-dev1/v1.1.0-dev1/optional-output.cwl
index df51801df..16cf36a1a 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/optional-output.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/optional-output.cwl
@@ -1,10 +1,10 @@
#!/usr/bin/env cwl-runner
class: CommandLineTool
-cwlVersion: "v1.1.0-dev1"
+cwlVersion: "v1.0"
doc: "Print the contents of a file to stdout using 'cat' running in a docker container."
hints:
DockerRequirement:
- dockerPull: debian:wheezy
+ dockerPull: debian:stretch-slim
inputs:
file1:
type: File
diff --git a/v1.1.0-dev1/v1.1.0-dev1/params_inc.yml b/v1.1.0-dev1/v1.1.0-dev1/params_inc.yml
index 9ecffdb74..930d2e89c 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/params_inc.yml
+++ b/v1.1.0-dev1/v1.1.0-dev1/params_inc.yml
@@ -108,3 +108,13 @@
type: Any
outputBinding:
outputEval: $(inputs.bar.buz[1]) $(inputs.bar.buz[1])
+
+ - id: t27
+ type: "null"
+ outputBinding:
+ outputEval: $(null)
+
+ - id: t28
+ type: int
+ outputBinding:
+ outputEval: $(inputs.bar.buz.length)
diff --git a/v1.1.0-dev1/v1.1.0-dev1/rec/A b/v1.1.0-dev1/v1.1.0-dev1/rec/A
new file mode 100644
index 000000000..e69de29bb
diff --git a/v1.1.0-dev1/v1.1.0-dev1/rec/A.s2 b/v1.1.0-dev1/v1.1.0-dev1/rec/A.s2
new file mode 100644
index 000000000..e69de29bb
diff --git a/v1.1.0-dev1/v1.1.0-dev1/rec/B b/v1.1.0-dev1/v1.1.0-dev1/rec/B
new file mode 100644
index 000000000..e69de29bb
diff --git a/v1.1.0-dev1/v1.1.0-dev1/rec/B.s3 b/v1.1.0-dev1/v1.1.0-dev1/rec/B.s3
new file mode 100644
index 000000000..e69de29bb
diff --git a/v1.1.0-dev1/v1.1.0-dev1/rec/C b/v1.1.0-dev1/v1.1.0-dev1/rec/C
new file mode 100644
index 000000000..e69de29bb
diff --git a/v1.1.0-dev1/v1.1.0-dev1/rec/C.s3 b/v1.1.0-dev1/v1.1.0-dev1/rec/C.s3
new file mode 100644
index 000000000..e69de29bb
diff --git a/v1.1.0-dev1/v1.1.0-dev1/rec/D b/v1.1.0-dev1/v1.1.0-dev1/rec/D
new file mode 100644
index 000000000..e69de29bb
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-format-job.yml b/v1.1.0-dev1/v1.1.0-dev1/record-format-job.yml
new file mode 100644
index 000000000..57b67f55b
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-format-job.yml
@@ -0,0 +1,16 @@
+regular_input:
+ class: File
+ location: rec/D
+ format: http://example.com/format1
+record_input:
+ f1:
+ class: File
+ location: rec/A
+ format: http://example.com/format1
+ f2:
+ - class: File
+ location: rec/B
+ format: http://example.com/format2
+ - class: File
+ location: rec/C
+ format: http://example.com/format2
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-format-job2.yml b/v1.1.0-dev1/v1.1.0-dev1/record-format-job2.yml
new file mode 100644
index 000000000..dbae50f8e
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-format-job2.yml
@@ -0,0 +1,16 @@
+regular_input:
+ class: File
+ location: rec/D
+ format: http://example.com/formatZ
+record_input:
+ f1:
+ class: File
+ location: rec/A
+ format: http://example.com/format1
+ f2:
+ - class: File
+ location: rec/B
+ format: http://example.com/format2
+ - class: File
+ location: rec/C
+ format: http://example.com/format2
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-format-job3.yml b/v1.1.0-dev1/v1.1.0-dev1/record-format-job3.yml
new file mode 100644
index 000000000..bd94aed8d
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-format-job3.yml
@@ -0,0 +1,16 @@
+regular_input:
+ class: File
+ location: rec/D
+ format: http://example.com/format1
+record_input:
+ f1:
+ class: File
+ location: rec/A
+ format: http://example.com/formatZ
+ f2:
+ - class: File
+ location: rec/B
+ format: http://example.com/format2
+ - class: File
+ location: rec/C
+ format: http://example.com/format2
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-format-job4.yml b/v1.1.0-dev1/v1.1.0-dev1/record-format-job4.yml
new file mode 100644
index 000000000..b19ba1963
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-format-job4.yml
@@ -0,0 +1,16 @@
+regular_input:
+ class: File
+ location: rec/D
+ format: http://example.com/format1
+record_input:
+ f1:
+ class: File
+ location: rec/A
+ format: http://example.com/format1
+ f2:
+ - class: File
+ location: rec/B
+ format: http://example.com/format2
+ - class: File
+ location: rec/C
+ format: http://example.com/formatZ
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-in-format.cwl b/v1.1.0-dev1/v1.1.0-dev1/record-in-format.cwl
new file mode 100644
index 000000000..576cc1507
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-in-format.cwl
@@ -0,0 +1,20 @@
+class: CommandLineTool
+cwlVersion: v1.1.0-dev1
+inputs:
+ regular_input:
+ type: File
+ format: http://example.com/format1
+ record_input:
+ type:
+ type: record
+ fields:
+ f1:
+ type: File
+ format: http://example.com/format1
+ f2:
+ type:
+ type: array
+ items: File
+ format: http://example.com/format2
+outputs: []
+arguments: ['true']
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-in-secondaryFiles-missing-wf.cwl b/v1.1.0-dev1/v1.1.0-dev1/record-in-secondaryFiles-missing-wf.cwl
new file mode 100644
index 000000000..5dd34f519
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-in-secondaryFiles-missing-wf.cwl
@@ -0,0 +1,20 @@
+class: Workflow
+cwlVersion: v1.1.0-dev1
+inputs:
+ record_input:
+ type:
+ type: record
+ fields:
+ f1:
+ type: File
+ f2:
+ type:
+ type: array
+ items: File
+outputs: []
+steps:
+ step1:
+ in:
+ record_input: record_input
+ out: []
+ run: record-in-secondaryFiles.cwl
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-in-secondaryFiles-wf.cwl b/v1.1.0-dev1/v1.1.0-dev1/record-in-secondaryFiles-wf.cwl
new file mode 100644
index 000000000..e76f43c3e
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-in-secondaryFiles-wf.cwl
@@ -0,0 +1,22 @@
+class: Workflow
+cwlVersion: v1.1.0-dev1
+inputs:
+ record_input:
+ type:
+ type: record
+ fields:
+ f1:
+ type: File
+ secondaryFiles: .s2
+ f2:
+ type:
+ type: array
+ items: File
+ secondaryFiles: .s3
+outputs: []
+steps:
+ step1:
+ in:
+ record_input: record_input
+ out: []
+ run: record-in-secondaryFiles.cwl
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-in-secondaryFiles.cwl b/v1.1.0-dev1/v1.1.0-dev1/record-in-secondaryFiles.cwl
new file mode 100644
index 000000000..6a987b18a
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-in-secondaryFiles.cwl
@@ -0,0 +1,20 @@
+class: CommandLineTool
+cwlVersion: v1.1.0-dev1
+inputs:
+ record_input:
+ type:
+ type: record
+ fields:
+ f1:
+ type: File
+ secondaryFiles: .s2
+ f2:
+ type:
+ type: array
+ items: File
+ secondaryFiles: .s3
+outputs: []
+baseCommand: test
+arguments: [-f, $(inputs.record_input.f1.path).s2,
+ '-a', '-f', '$(inputs.record_input.f2[0].path).s3',
+ '-a', '-f', '$(inputs.record_input.f2[1].path).s3']
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-out-format.cwl b/v1.1.0-dev1/v1.1.0-dev1/record-out-format.cwl
new file mode 100644
index 000000000..a972a1fa1
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-out-format.cwl
@@ -0,0 +1,25 @@
+class: CommandLineTool
+cwlVersion: v1.1.0-dev1
+inputs:
+ record_input:
+ type:
+ type: record
+ fields:
+ f1: File
+ f2: File[]
+outputs:
+ f1out:
+ type: File
+ format: http://example.com/format1
+ outputBinding:
+ outputEval: $(inputs.record_input.f1)
+ record_output:
+ type:
+ type: record
+ fields:
+ f2out:
+ type: File
+ format: http://example.com/format2
+ outputBinding:
+ outputEval: $(inputs.record_input.f2[0])
+arguments: ['true']
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-out-secondaryFiles.cwl b/v1.1.0-dev1/v1.1.0-dev1/record-out-secondaryFiles.cwl
new file mode 100644
index 000000000..44af429e4
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-out-secondaryFiles.cwl
@@ -0,0 +1,23 @@
+class: CommandLineTool
+cwlVersion: v1.1.0-dev1
+inputs: []
+outputs:
+ record_output:
+ type:
+ type: record
+ fields:
+ f1:
+ type: File
+ secondaryFiles: .s2
+ outputBinding:
+ glob: A
+ f2:
+ type:
+ type: array
+ items: File
+ secondaryFiles: .s3
+ outputBinding:
+ glob: [B, C]
+
+baseCommand: touch
+arguments: [A, A.s2, B, B.s3, C, C.s3]
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-sd-secondaryFiles.cwl b/v1.1.0-dev1/v1.1.0-dev1/record-sd-secondaryFiles.cwl
new file mode 100644
index 000000000..9b73a0cec
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-sd-secondaryFiles.cwl
@@ -0,0 +1,26 @@
+class: CommandLineTool
+cwlVersion: v1.1.0-dev1
+requirements:
+ SchemaDefRequirement:
+ types:
+ - name: RecordTestType
+ type: record
+ doc: Type test record
+ fields:
+ f1:
+ type: File
+ secondaryFiles: .s2
+ f2:
+ type:
+ type: array
+ items: File
+ secondaryFiles: .s3
+inputs:
+ record_input:
+ type: RecordTestType
+
+outputs: []
+baseCommand: test
+arguments: [-f, $(inputs.record_input.f1.path).s2,
+ '-a', '-f', '$(inputs.record_input.f2[0].path).s3',
+ '-a', '-f', '$(inputs.record_input.f2[1].path).s3']
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/record-secondaryFiles-job.yml b/v1.1.0-dev1/v1.1.0-dev1/record-secondaryFiles-job.yml
new file mode 100644
index 000000000..c33bffbdf
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/record-secondaryFiles-job.yml
@@ -0,0 +1,9 @@
+record_input:
+ f1:
+ class: File
+ location: rec/A
+ f2:
+ - class: File
+ location: rec/B
+ - class: File
+ location: rec/C
diff --git a/v1.1.0-dev1/v1.1.0-dev1/recursive-input-directory.cwl b/v1.1.0-dev1/v1.1.0-dev1/recursive-input-directory.cwl
new file mode 100644
index 000000000..4cb5f3115
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/recursive-input-directory.cwl
@@ -0,0 +1,34 @@
+cwlVersion: v1.1.0-dev1
+class: CommandLineTool
+requirements:
+ - class: InlineJavascriptRequirement
+ - class: InitialWorkDirRequirement
+ listing:
+ - entry: $(inputs.input_dir)
+ entryname: work_dir
+ writable: true
+ - class: ShellCommandRequirement
+stdout: output.txt
+arguments:
+ - shellQuote: false
+ valueFrom: |
+ touch work_dir/e;
+ if [ ! -w work_dir ]; then echo work_dir not writable; fi;
+ if [ -L work_dir ]; then echo work_dir is a symlink; fi;
+ if [ ! -w work_dir/a ]; then echo work_dir/a not writable; fi;
+ if [ -L work_dir/a ]; then echo work_dir/a is a symlink; fi;
+ if [ ! -w work_dir/c ]; then echo work_dir/c not writable; fi;
+ if [ -L work_dir/c ]; then echo work_dir/c is a symlink; fi;
+ if [ ! -w work_dir/c/d ]; then echo work_dir/c/d not writable; fi;
+ if [ -L work_dir/c/d ]; then echo work_dir/c/d is a symlink; fi;
+ if [ ! -w work_dir/e ]; then echo work_dir/e not writable; fi;
+ if [ -L work_dir/e ]; then echo work_dir/e is a symlink ; fi;
+inputs:
+ input_dir: Directory
+outputs:
+ output_dir:
+ type: Directory
+ outputBinding:
+ glob: work_dir
+ test_result:
+ type: stdout
diff --git a/v1.1.0-dev1/v1.1.0-dev1/recursive-input-directory.yml b/v1.1.0-dev1/v1.1.0-dev1/recursive-input-directory.yml
new file mode 100644
index 000000000..7c83f3f26
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/recursive-input-directory.yml
@@ -0,0 +1,3 @@
+input_dir:
+ class: Directory
+ location: testdir
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/revsort.cwl b/v1.1.0-dev1/v1.1.0-dev1/revsort.cwl
index c5d692b7f..f40c34434 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/revsort.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/revsort.cwl
@@ -10,7 +10,7 @@ cwlVersion: v1.1.0-dev1
# in which the command line tools will execute.
hints:
- class: DockerRequirement
- dockerPull: debian:8
+ dockerPull: debian:stretch-slim
# The inputs array defines the structure of the input object that describes
diff --git a/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job1.json b/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job1.json
new file mode 100644
index 000000000..dfd8a4904
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job1.json
@@ -0,0 +1,3 @@
+{
+ "inp": []
+}
diff --git a/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job2.json b/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job2.json
new file mode 100644
index 000000000..a73a3785f
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job2.json
@@ -0,0 +1,4 @@
+{
+ "inp1": ["one", "two"],
+ "inp2": []
+}
diff --git a/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job3.json b/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job3.json
new file mode 100644
index 000000000..6aa7b6d5e
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job3.json
@@ -0,0 +1,4 @@
+{
+ "inp1": [],
+ "inp2": ["one", "two"]
+}
diff --git a/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job4.json b/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job4.json
new file mode 100644
index 000000000..e90c2665a
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/scatter-empty-job4.json
@@ -0,0 +1,4 @@
+{
+ "inp1": [],
+ "inp2": []
+}
diff --git a/v1.1.0-dev1/v1.1.0-dev1/search-job.json b/v1.1.0-dev1/v1.1.0-dev1/search-job.json
index b5a1b61e5..a5f9d9805 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/search-job.json
+++ b/v1.1.0-dev1/v1.1.0-dev1/search-job.json
@@ -3,5 +3,9 @@
"class": "File",
"location": "whale.txt"
},
+ "secondfile": {
+ "class": "File",
+ "location": "hello.txt"
+ },
"term": "find"
}
diff --git a/v1.1.0-dev1/v1.1.0-dev1/search.cwl b/v1.1.0-dev1/v1.1.0-dev1/search.cwl
index d0ff147d6..f5a85f2f6 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/search.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/search.cwl
@@ -2,7 +2,7 @@ cwlVersion: v1.1.0-dev1
$graph:
- id: index
class: CommandLineTool
- baseCommand: python2
+ baseCommand: python
arguments:
- valueFrom: input.txt
position: 1
@@ -12,9 +12,13 @@ $graph:
- entryname: input.txt
entry: $(inputs.file)
- class: InlineJavascriptRequirement
+ hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
inputs:
file: File
+ secondfile: File
index.py:
type: File
default:
@@ -30,15 +34,21 @@ $graph:
secondaryFiles:
- ".idx1"
- "^.idx2"
- - '$(self.location+".idx3")'
- - '$({"location": self.location+".idx4", "class": "File"})'
- - '${ return self.location+".idx5"; }'
+ - '$(self.basename).idx3'
+ - '${ return self.basename+".idx4"; }'
+ - '$({"path": self.path+".idx5", "class": "File"})'
+ - '$(self.nameroot).idx6$(self.nameext)'
+ - '${ return [self.basename+".idx7", inputs.secondfile]; }'
+ - "_idx8"
- id: search
class: CommandLineTool
- baseCommand: python2
+ baseCommand: python
requirements:
- class: InlineJavascriptRequirement
+ hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
inputs:
file:
type: File
@@ -47,9 +57,11 @@ $graph:
secondaryFiles:
- ".idx1"
- "^.idx2"
- - '$(self.location+".idx3")'
- - '$({"location": self.location+".idx4", "class": "File"})'
- - '${ return self.location+".idx5"; }'
+ - '$(self.basename).idx3'
+ - '${ return self.basename+".idx4"; }'
+ - '$(self.nameroot).idx6$(self.nameext)'
+ - '${ return [self.basename+".idx7"]; }'
+ - "_idx8"
search.py:
type: File
default:
@@ -72,6 +84,7 @@ $graph:
class: Workflow
inputs:
infile: File
+ secondfile: File
term: string
outputs:
outfile:
@@ -85,12 +98,13 @@ $graph:
index:
run: "#index"
in:
- file: "#main/infile"
+ file: infile
+ secondfile: secondfile
out: [result]
search:
run: "#search"
in:
- file: "#main/index/result"
- term: "#main/term"
+ file: index/result
+ term: term
out: [result]
diff --git a/v1.1.0-dev1/v1.1.0-dev1/shellchar.cwl b/v1.1.0-dev1/v1.1.0-dev1/shellchar.cwl
new file mode 100644
index 000000000..df288d8ce
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/shellchar.cwl
@@ -0,0 +1,13 @@
+#!/usr/bin/env cwl-runner
+class: CommandLineTool
+cwlVersion: v1.1.0-dev1
+doc: |
+ Ensure that arguments containing shell directives are not interpreted and
+ that `shellQuote: false` has no effect when ShellCommandRequirement is not in
+ effect.
+inputs: []
+outputs:
+ stdout_file: stdout
+ stderr_file: stderr
+baseCommand: echo
+arguments: [{valueFrom: "foo 1>&2", shellQuote: false}]
diff --git a/v1.1.0-dev1/v1.1.0-dev1/shellchar2.cwl b/v1.1.0-dev1/v1.1.0-dev1/shellchar2.cwl
new file mode 100644
index 000000000..b715deb65
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/shellchar2.cwl
@@ -0,0 +1,14 @@
+#!/usr/bin/env cwl-runner
+class: CommandLineTool
+cwlVersion: v1.1.0-dev1
+doc: |
+ Ensure that `shellQuote: true` is the default behavior when
+ ShellCommandRequirement is in effect.
+requirements:
+ ShellCommandRequirement: {}
+inputs: []
+outputs:
+ stdout_file: stdout
+ stderr_file: stderr
+baseCommand: echo
+arguments: ["foo 1>&2"]
diff --git a/v1.1.0-dev1/v1.1.0-dev1/stagefile.cwl b/v1.1.0-dev1/v1.1.0-dev1/stagefile.cwl
index 3e383e95b..542725739 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/stagefile.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/stagefile.cwl
@@ -1,5 +1,8 @@
class: CommandLineTool
cwlVersion: v1.1.0-dev1
+hints:
+ - class: DockerRequirement
+ dockerPull: python:2-slim
requirements:
InitialWorkDirRequirement:
listing:
diff --git a/v1.1.0-dev1/v1.1.0-dev1/steplevel-resreq.cwl b/v1.1.0-dev1/v1.1.0-dev1/steplevel-resreq.cwl
new file mode 100644
index 000000000..63a8c2d02
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/steplevel-resreq.cwl
@@ -0,0 +1,33 @@
+#!/usr/bin/env cwl-runner
+class: Workflow
+cwlVersion: v1.1.0-dev1
+
+requirements:
+ ResourceRequirement:
+ coresMin: 4
+ coresMax: 4
+
+inputs: []
+
+steps:
+ step1:
+ requirements:
+ ResourceRequirement:
+ coresMin: 1
+ coresMax: 1
+ run:
+ class: CommandLineTool
+ inputs: []
+ outputs:
+ output:
+ type: stdout
+ baseCommand: echo
+ stdout: cores.txt
+ arguments: [ $(runtime.cores) ]
+ in: []
+ out: [output]
+
+outputs:
+ out:
+ type: File
+ outputSource: step1/output
diff --git a/v1.1.0-dev1/v1.1.0-dev1/template-tool.cwl b/v1.1.0-dev1/v1.1.0-dev1/template-tool.cwl
index 88b90166d..7d4322d12 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/template-tool.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/template-tool.cwl
@@ -5,14 +5,14 @@ requirements:
- class: InlineJavascriptRequirement
expressionLib:
- { $include: underscore.js }
- - "var t = function(s) { return _.template(s)({'inputs': inputs}); };"
+ - "var t = function(s) { return _.template(s, {variable: 'data'})({'inputs': inputs}); };"
- class: InitialWorkDirRequirement
listing:
- entryname: foo.txt
- entry: $(t("The file is <%= inputs.file1.path.split('/').slice(-1)[0] %>\n"))
+ entry: $(t("The file is <%= data.inputs.file1.path.split('/').slice(-1)[0] %>\n"))
hints:
DockerRequirement:
- dockerPull: "debian:8"
+ dockerPull: "debian:stretch-slim"
inputs:
- id: file1
type: File
diff --git a/v1.1.0-dev1/v1.1.0-dev1/test-cwl-out.cwl b/v1.1.0-dev1/v1.1.0-dev1/test-cwl-out.cwl
index 0a9464df8..40cd12a2c 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/test-cwl-out.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/test-cwl-out.cwl
@@ -4,7 +4,7 @@ requirements:
- class: ShellCommandRequirement
hints:
DockerRequirement:
- dockerPull: "debian:wheezy"
+ dockerPull: "debian:stretch-slim"
inputs: []
diff --git a/v1.1.0-dev1/v1.1.0-dev1/test-cwl-out2.cwl b/v1.1.0-dev1/v1.1.0-dev1/test-cwl-out2.cwl
index d8c5f9ac5..7b7994ef5 100644
--- a/v1.1.0-dev1/v1.1.0-dev1/test-cwl-out2.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/test-cwl-out2.cwl
@@ -4,7 +4,7 @@ requirements:
- class: ShellCommandRequirement
hints:
DockerRequirement:
- dockerPull: "debian:wheezy"
+ dockerPull: "debian:stretch-slim"
inputs: []
diff --git a/v1.1.0-dev1/v1.1.0-dev1/tmap-tool.cwl b/v1.1.0-dev1/v1.1.0-dev1/tmap-tool.cwl
index d9ea0fb75..a6d0b0626 100755
--- a/v1.1.0-dev1/v1.1.0-dev1/tmap-tool.cwl
+++ b/v1.1.0-dev1/v1.1.0-dev1/tmap-tool.cwl
@@ -3,7 +3,12 @@
"cwlVersion": "v1.1.0-dev1",
"class": "CommandLineTool",
-
+ "hints": [
+ {
+ "class": "DockerRequirement",
+ "dockerPull": "python:2-slim"
+ }
+ ],
"inputs": [
{
"id": "reads",
diff --git a/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents-job.yml b/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents-job.yml
new file mode 100644
index 000000000..1b8442b6f
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents-job.yml
@@ -0,0 +1,3 @@
+my_file:
+ class: File
+ location: answer
diff --git a/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents.cwl b/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents.cwl
new file mode 100644
index 000000000..978f1f698
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents.cwl
@@ -0,0 +1,30 @@
+cwlVersion: v1.1.0-dev1
+class: Workflow
+requirements:
+ StepInputExpressionRequirement: {}
+ InlineJavascriptRequirement: {}
+inputs:
+ my_file:
+ type: File
+ inputBinding:
+ loadContents: true
+
+steps:
+ one:
+ run:
+ class: ExpressionTool
+ requirements: { InlineJavascriptRequirement: {} }
+ inputs: { my_number: int }
+ outputs: { my_int: int }
+ expression: |
+ ${ return { "my_int": inputs.my_number }; }
+ in:
+ my_number:
+ source: my_file
+ valueFrom: $(parseInt(self.contents))
+ out: [ my_int ]
+
+outputs:
+ my_int:
+ type: int
+ outputSource: one/my_int
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents2.cwl b/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents2.cwl
new file mode 100644
index 000000000..99951f5b5
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents2.cwl
@@ -0,0 +1,29 @@
+cwlVersion: v1.1.0-dev1
+class: Workflow
+requirements:
+ StepInputExpressionRequirement: {}
+ InlineJavascriptRequirement: {}
+inputs:
+ my_file:
+ type: File
+ loadContents: true
+
+steps:
+ one:
+ run:
+ class: ExpressionTool
+ requirements: { InlineJavascriptRequirement: {} }
+ inputs: { my_number: int }
+ outputs: { my_int: int }
+ expression: |
+ ${ return { "my_int": inputs.my_number }; }
+ in:
+ my_number:
+ source: my_file
+ valueFrom: $(parseInt(self.contents))
+ out: [ my_int ]
+
+outputs:
+ my_int:
+ type: int
+ outputSource: one/my_int
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents3.cwl b/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents3.cwl
new file mode 100644
index 000000000..e014230f4
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents3.cwl
@@ -0,0 +1,28 @@
+cwlVersion: v1.1.0-dev1
+class: Workflow
+requirements:
+ StepInputExpressionRequirement: {}
+ InlineJavascriptRequirement: {}
+inputs:
+ my_file: File
+
+steps:
+ one:
+ run:
+ class: ExpressionTool
+ requirements: { InlineJavascriptRequirement: {} }
+ inputs:
+ my_number:
+ type: File
+ loadContents: true
+ outputs: { my_int: int }
+ expression: |
+ ${ return { "my_int": parseInt(inputs.my_number.contents) }; }
+ in:
+ my_number: my_file
+ out: [ my_int ]
+
+outputs:
+ my_int:
+ type: int
+ outputSource: one/my_int
\ No newline at end of file
diff --git a/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents4.cwl b/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents4.cwl
new file mode 100644
index 000000000..2dd5d7e16
--- /dev/null
+++ b/v1.1.0-dev1/v1.1.0-dev1/wf-loadContents4.cwl
@@ -0,0 +1,28 @@
+cwlVersion: v1.1.0-dev1
+class: Workflow
+requirements:
+ StepInputExpressionRequirement: {}
+ InlineJavascriptRequirement: {}
+inputs:
+ my_file: File
+
+steps:
+ one:
+ run:
+ class: ExpressionTool
+ requirements: { InlineJavascriptRequirement: {} }
+ inputs: { my_number: int }
+ outputs: { my_int: int }
+ expression: |
+ ${ return { "my_int": inputs.my_number }; }
+ in:
+ my_number:
+ source: my_file
+ loadContents: true
+ valueFrom: $(parseInt(self.contents))
+ out: [ my_int ]
+
+outputs:
+ my_int:
+ type: int
+ outputSource: one/my_int
\ No newline at end of file
![[Video] Common Workflow Language explained in 64 seconds](https://github.com/common-workflow-language/logo/raw/master/intro_video_screenshot_413x193.png){kind=logo}
",
- "primtype": null
+ "primtype": null,
+ "extra": { "class": "File", "path": "empty" }
},
{
"source": {
@@ -24,7 +25,8 @@
"redirect": [],
"brandlink": "index.html",
"brandimg": "