v0.27.0

Summary of changes

• Move to new config structure Closes #310 #314 e8b377d [BREAKING CHANGE]
• Include preliminary ATAC-seq workflows #362
• Add version check #355
• Fix bug sra pipeline not working for scATAC-seq data Closes #348
• Add multi-sample QC workflow 7f7c630
• [sratoolkit] Add maxSize option (-X option of prefetch) b5c7bae. This option should be set accordingly when running sra pipeline.
• Update Scanpy version to 1.8.1 #324 8370443
• [scanpy] Allow to set perplexity parameter for t-SNE e3b256a
• [scanpy] Implement additional flavors for computing HVG. Now possible to use: seurat, seurat_v3 and cell_ranger
• [scrublet] Update Scrublet Dockerfile to 0.2.3 8370443
• [harmony] Update Harmony to 1.0-3 74371bf
• [pcacv] Save PRESS errors in case it fails finding the optimal number of PCs 742f3bd

v0.26.1

Summary of changes

• Fixes #345
• Closes #329
• Add NXF workflow to download from SRA

v0.26.0

Summary of changes

• File converter cleanup (py, R)
• Use multiple batch variable for harmony (#297)
• Add harmony/scenic workflow (#312)
• Additional arguments possible for harmony (#313)
• Fix reproducibility of cell normalization (#295)
• Fix bugs in scrublet (#306, #308)
• Cell annotation filter fixes (#309)
  • Allow metadata to match to fewer than the full set of barcodes in the
    adata object (80% or more cells allowed)
  • Add CI test for cell annotation method
• Add quiet mode (#305)
• Cellranger updates (#304) and profile updates (#320)
• Remove exception on Scenic multiruns that prevents the workflow running in a local executor
• Remove hard links from the publishing process (#318)
• getDataChannel updates:
  • Implement file tagging (#291)
  • Fixes to account for the tag parameter in the remaining file types
  • Bugfix for parsing multiple paths separated by a comma
• Popscle (freemuxlet/demuxlet) workflow updates (#330)
  • Allow input files (bam, barcodes) other than those from Cellranger formatted sources
  • Allow analysis on barcodes other than 'CB'
• Add VSC/HPC specific config
• Docker image updates
  • Include samtools and a base image with zlibng for faster compression/decompression (#325)
  • Use this base image for other tools (popscle, bwamaptools, singlecelltoolkit, ...)
• Documentation updates
• Fix incorrectly applied compute resources settings (#335)

Updates to scATAC-seq preprocessing v1 (#334)

• Update bwa mapping approach
  • skip samtools sort step, turn off compression for piped commands, reduce samtools threads
  • Re-work add_barcode_as_tag mawk script to include both corrected and uncorrected barcode tags in final bam
  • Split bwamaptools config file into discrete parts that can be loaded separately
• Update fragments sorting command with locale
• BAP updates
  • Implement multiplet detection pipeline
  • Add ipynb report to summarize bap results
  • Fix bap install in Docker image
• Enable skipping of barcode correction (when whitelist parameters are empty)
• scATAC-seq docs updates and fixes

v0.25.0

Major updates:

• Update project to run on Nextflow >= 20.10.0 (#253)
• Cell barcode -1 suffixes are no longer removed by default (#288)

Publishing system updates:

• Streamline publish workflow (remove proxy process)
• Avoid copying internally when publishing a file
• Allow files to be published across filesystems when params.publish.mode is 'copy' (#265)

Other:

• Documentation updates
• Disable loom validation in file converter (#286)
• Fix process labels and run times in cellranger, pcacv
• Add CI test for parameter exploration
• Fixes for CI tests failing when using hard links
• Set CI tests to run on pushes and PRs to develop
• Use sparse loading from loom files
• Update utilities Docker image

v0.24.0

Major changes:

• Convert from submodule-based structure to a monorepo. The old submodule repositories have been archived and all code and commit history has been moved into vib-singlecell-nf/vsn-pipelines. The overall directory structure remains unchanged (modules are stored in src/<tool-name>).

Minor changes:

• Documentation changes to reflect the switch to a single unified repository
• Cleanup files
• Update .gitignore

v0.23.0

Major changes:

Implement the scATAC-seq preprocessing workflow (#245):

• Barcode correction [singlecelltoolkit, #249]
• "Debarcode" fastq files:
  • 10x [single_cell_toolkit, #249]
  • BioRad [BAP, #246]
• Adapter Trimming [Trim Galore / Cutadapt, #242]
• Mapping [bwa mem, #243]
  • mark duplicates [samtools markdup, #243]
• Generate fragments file [Sinto, #244]

Other scATAC-seq notes:

• Currently compatible with 10x-format fastq files (R1, R2[barcode reads], R3), and BioRad (R1, R2), which are processed with BAP.
• Added documentation for preprocessing steps
• The scATAC-seq workflows are in main_atac.nf and need to be referenced with the full path: vib-singlecell-nf/vsn-pipelines/main_atac.nf when running the pipeline.
• Added two additional input data types to getDataChannel: bam and fragments, both of which are loaded with their associated index file.

Other scRNA-seq tool changes:

• celeda
  • Add ability to remove empty cells after applying DecontX
  • Fix plotting bug when no outliers are found
  • Add cell_annotate config entry
• scanpy
  • Add clustering preflight checks (vib-singlecell-nf/scanpy/issues/48)
  • Fixes to limit cpu usage in various processes.
  • Adaptive threshold cell filtering threshold (vib-singlecell-nf/scanpy/issues/47)
  • Add qc plots for n_counts filter in report notebook
  • Add single_sample_qc workflow
  • Bug fixes for mnncorrect, clustering, pca inputs
• pcacv
  • Adapt *-n-pc and k parameters if violating requirement of data dimensionality
  • Handle cases where no optimal number of PCs are found (vib-singlecell-nf/pcacv/issues/15)
  • Fix arguments passed to run_pca_cv.R
  • Fix vib-singlecell-nf/pcacv/issues/14
• sratoolkit
  • Fix missing assignment operator in config

v0.22.0

Summary of changes:

General changes

• Pre-filtering/pre-annotating (FILTER_AND_ANNOTATE_AND_CLEAN) is now performed outside of QC_FILTER 839735e a463b8a
• Add sanity check - Check if column containing sample ID exists in metadata file c32cfce
• *scrublet workflows - Publish data with doublets removed 47e611e
• Support to tag final output files (loom, h5ad) with the batch variabe for the harmony pipelines 822fc95
• Reformat popscle inputs Merge pull request #256 Closes #255
• Bug fix - Pipeline should output scopeloom instead of FINALIZE.out.scopeloom so that directs output will be included in final loom 22d587c
• Bug fix - Fix bug leading to out of sync issues of the input params of getChannel when calling it multiple times f50074a d0eb654
• Bug fix - Closes #260 2b1bd4e

Tools/Modules

• celda

  • Bug fix - Fix bug when plotting density plot when no outliers found 099b2f0
  • Add support to remove empty cells after DecontX matrix decontamination with the filterEmptyCells parameter 648b4df

• scanpy

  • Add plots for the n_counts filter in filter QC report python notebook and Closes vib-singlecell-nf/scanpy#47 b1923bb
  • Add single_sample_qc workflow 4915786
  • Set the n_jobs argument in regress out step using compute_resources__cpu label 23520e0
  • Disable BLAS multithreading in regress out step to reduce cpu usage (especially if n_jobs is > 1) 262c219
  • Add clustering preflight checks Closes vib-singlecell-nf/scanpy#48 debb6a1
  • Bug fix - Fix bug belong to scanpy not sc in config 26e57f2

• scenic

  • Bug fix - Use np.array_equel for array comparison instead of equality operator which won't work if array have different lengths d723bdd
  • Disable multi-threading for cisTarget and AUCell step to reduce cpu usage 2d9fa8f
  • Add TF-gene correlations to the SCENIC GRN output Merge pull request #257 Closes #254

• pcacv

  • Closes vib-singlecell-nf/pcacv#14 b543a8b
  • Closes vib-singlecell-nf/pcacv#8 ef7ab83
  • Closes vib-singlecell-nf/pcacv#15 b04d121
  • Adapt *-n-pc and k parameters if violating requirement of data dimensionality 228276f

• sratoolkit

  • Bug fix - Fix config: missing assignment operator 4d12b74

Docs

• Specify current Nextflow version to use in getting started docs Closes #250 923baad

v0.21.0

• Merge #230 Closes #228 Add DecontX
• Merge #237 Closes #229 Add SoupX
• Closes #233 3a66a0c Improve interoperability with Scanpy
• Closes #234 Merge final h5ad files when in param exploration mode
• Closes #235 Store log files in log folder
• c1cf114 Support for h5ad built with anndata>=0.7.x for h5ad to loom conversion
• Bug fixes #227 #236
• Bug fix 0bbffbc Regressing out variables should be performed after HVG and before feature scaling
• New label for report NXF processes: compute_resources__report
• [scenic] 455a165 Allow to use --all_modules argument when running the cisTarget step
• [scanpy] 06f4a41 Fix bug cells not matching when running mnncorrect batch effect correction pipeline

v0.20.0

Summary of changes

Major changes:

• Add pcacv workflow to run PCACV 2741eb5
• Specify compute resources using process labels. Merge #219 Closes #154
• Remove the need of defining explicitly the helper functions (e.g.: parseConfig) inside the config dfa6af2 1ab9efb 5d74cb5
• Implement feature to run pipelines on h5ad files with different suffixes (aka multi-labeled group strategy) 44d9aef de1f896
• Allow user to run a workflow on a subset of cells from the data 83d6122 Closes #221
• Don't include scope profile by default ad8115f
• Publish pipeline final output only if params.utils.publish = true is in the config 33f643a

General changes:

• Implement feature to beautify a h5ad file 1c41b42 6260b79

• Add harmony workflow and related config 9c30059

• Publish a loom with doublets info b46832f Closes #220

• Add workflow cellranger_count_demuxlet to run Cell Ranger and Demuxlet at once 5240226

• Add utility workflow to annotate data using a cell-based .tsv metadata file 39069dd

• Add utility workflow to annotate and subsequently filter data using a cell-based .tsv metadata file

• Add example config for SRA_TO_METADATA process 5adcbe9

• Rename cellranger_metadata to cellranger_count_metadata 5abc06a

• Abstract away code block used for cell filter, cell annotation and sample annotation into workflow ff1ab81

• Add publish flag in cell_annotate and cell_filter 5bb233f

• Allow to skip for a given sample the sample-based metadata annotation if not present in metadata 23e188c

• Fixes bug where some columns from h5ad files were wrongly saved to either metric or annotation in loom 5711b35

• Fixes #214

• Other bug fixes:

  • 83d2d6b f15f71e bc16a94 1c8aca5

• Miscellaneous

  • c2c3041 6c227cb f5efb00 6b6edcb 67b4c92 a0436ad 43d068c edcda97 239b229 16c5dbf

• Containers

  • Bump utils container version from 0.2.2 to 0.3.0 f262e88
  • Bump scrublet container version from 0.1.3 to 0.1.4 2ed2447
  • Bump scanpy container version from 0.5.0 to 0.5.2 2ed2447
  • Bump scenic container version from 0.10.0 to 0.10.3 8648758

Tools/Modules:

• directs [NEW]
  • Merge #218 implements #216: module to automatically select default clustering
  • Allow the user to specify the cell embeddings (via cellEmbeddingsIndex param) to use for the density based clustering dfa1b27
• scenic [UPDATE]
  • Allow to skip generating the reports e723e11
• cellranger [UPDATE]
  • Allow n/a value in fastqs_dir_name column of metadata table 4866e82
  • With cellranger_count_metadata workflow, sample ID is defined as short_uuid + "__" + sample_name 4866e82
  • Allow the user to specify the chemistry parameter when running cellranger_count_metadata workflow 334137d
• pcacv [UPDATE]
  • Set nCores to 1 in default config and "disable" multi-threading 9e52a96
  • Publish output files in data/pcacv folder as hard links 63428f1
  • Use dopar for parallelization 9b8d447
  • Increase max size of object to share with future package from 500Mb to 2Gb 8263f8c
• scrublet [UPDATE]
  • Add sensible default values for Scrublet 57dd7d3

Docs:

• Add docs for cellranger_count_metadata workflow 55626f0
• Add docs regarding relevant output files generated by single_sample_scrublet pipeline cf73c3e a4e596d
• Update docs related to cell-based metadata annotation feature 61ecbc7 a2c8e45

v0.19.0

Summary of changes:

• Upgrade to Nextflow v20.04.1 Closes #182
• Fixes #210 #165
  • To have reproducible results with pipelines involving Harmony, this version of the pipelines is required.
• Improve annotation detection when convert h5ad to loom d599b9a
• Publish files to be staged through PUBLISH module eb4d6c4
  • Important intermediate files are published (hardlinked) in out/data:
    • e1821a5
    • vib-singlecell-nf/harmony@bfac8ff
    • vib-singlecell-nf/scanpy@86d1456
    • vib-singlecell-nf/scanpy@15eed53
    • vib-singlecell-nf/scanpy@5321fc5
  • They undergo additional compression ad15e59
    • Compression level can be set with params.utils.publish.compressionLevel
  • In exploration mode, some published (*final_output*) files have been reduced in size Closes #211
• Fix bug in cell tagging 9becc28 fdeeab1
• Closes vib-singlecell-nf/pcacv#11
• Fixes vib-singlecell-nf/pcacv#9
• Update vib-singlecell-nf/pcacv Dockerfile
  • Bump version from 0.1.0 to 0.2.0
• In bbknn, Run UMAP before cluster identification vib-singlecell-nf/scanpy@b5fbb64
• Fix bug where additional plots in report were empty vib-singlecell-nf/scanpy@98bff00
• Properly transfer the seed when running clustering vib-singlecell-nf/scanpy@44fc0b9 vib-singlecell-nf/scanpy@7220ed9
• Update Create module docs with latest code changes 02f75a3