Add --add-n5-container option to Paintera main class #273

hanslovsky · 2019-07-31T22:11:46Z

With this PR it will be possible to add N5 (file system and HDF5) to Paintera on start-up, e.g. it will be possible to start Paintera like this:

paintera \
    --default-to-temp-directory \
    --add-n5-container $HOME/Downloads/sample_A_20160501.hdf $HOME/Downloads/sample_B_20160501.hdf \
    -d volumes/raw volumes/labels/neuron_ids \
    -R \
    --id-service-fallback=none \
    --label-block-lookup-fallback=none

or

paintera \
    --default-to-temp-directory \
    --add-n5-container $HOME/Downloads/sample_A_20160501.hdf $HOME/Downloads/sample_B_20160501.hdf \
    -R \
    --id-service-fallback=none \
    --label-block-lookup-fallback=none \
    --entire-container \
    --exclude=.* \
    --include .*raw .*neuron_ids \

This would add raw and neuron_ids for both samples A and B to Paintera at startup.

Fixes #179 because PainteraShowContainer will become obsolete.
Fixes #176 because while doing this, some of the private static helper methods have been ported in a more general way into N5Helpers and N5Data.

This is possible with argument groups that have become available with the 4.0.0 release of picocli.

TODO:

test more thoroughly
improve make max Id fallback dialog
add option to specify label block lookup fallback
add option to load all datasets from container with include/exclude pattern

For example, this works: ``` --add-dataset --container=/data/hanslovskyp/cremi/sample_A_padded_20160501.n5 -d volumes/raw volumes/labels/neuron_ids ```

This is neccessary because hdf5 can be opened only in one mode at a time.

The container(s) are now specified in the --add-n5-container flag (instead of an additional --container flag)

hanslovsky · 2019-08-02T20:42:47Z

Updated help message:

$ paintera --help
Usage: Paintera [--add-n5-container=<container>...
                [--add-n5-container=<container>...]... (-d=DATASET...
                [-d=DATASET...]... [-r=RESOLUTION] [-r=RESOLUTION]...
                [-o=OFFSET] [-o=OFFSET]... [-R] [--min=MIN] [--max=MAX]
                [--channel-dimension=CHANNEL_DIMENSION]
                [--channels=CHANNELS...] [--channels=CHANNELS...]...
                [--name=NAME] [--name=NAME]...
                [--id-service-fallback=ID_SERVICE_FALLBACK]
                [--label-block-lookup-fallback=LABEL_BLOCK_LOOKUP_FALLBACK]
                [--entire-container] [--exclude=EXCLUDE...]
                [--exclude=EXCLUDE...]... [--include=INCLUDE...]
                [--include=INCLUDE...]... [--only-explicitly-included])]...
                [-h] [--default-to-temp-directory] [--print-error-codes]
                [--version] [--height=HEIGHT]
                [--highest-screen-scale=HIGHEST_SCREEN_SCALE]
                [--num-screen-scales=NUM_SCREEN_SCALES]
                [--screen-scale-factor=SCREEN_SCALE_FACTOR] [--width=WIDTH]
                [--screen-scales=SCREEN_SCALES[,SCREEN_SCALES...]...]...
                [PROJECT]
      [PROJECT]              Optional project N5 root (N5 or FileSystem).
      --add-n5-container=<container>...
                             Container of dataset(s) to be added. If none is
                               provided, default to Paintera project (if any).
                               Currently N5 file system and HDF5 containers are
                               supported.
      --channel-dimension=CHANNEL_DIMENSION
                             Defines the dimension of a 4D dataset to be
                               interpreted as channel axis. 0 <=
                               CHANNEL_DIMENSION <= 3
                               Default: 3
      --channels=CHANNELS... Use only this subset of channels for channel (4D)
                               data. Multiple subsets can be specified. If no
                               channels are specified, use all channels.
  -d, --dataset=DATASET...   Dataset(s) within CONTAINER to be added. TODO: If
                               no datasets are specified, all datasets will be
                               added (or use a separate option for this).
      --default-to-temp-directory
                             Default to temporary directory instead of showing
                               dialog when PROJECT is not specified.
      --entire-container     If set to true, discover all datasets (Paintera
                               format, multi-scale group, and N5 dataset)
                               inside CONTAINER and add to Paintera. The -d,
                               --dataset and --name options will be ignored if
                               ENTIRE_CONTAINER is set. Datasets can be
                               excluded through the --exclude option. The
                               --include option overrides any exclusions.
      --exclude=EXCLUDE...   Exclude any data set that matches any of EXCLUDE
                               regex patterns.
  -h, --help                 Display this help message.
      --height=HEIGHT        Initial height of viewer. Defaults to 600.
                               Overrides height stored in project.
                               Default: -1
      --highest-screen-scale=HIGHEST_SCREEN_SCALE
                             Highest screen scale, restricted to the interval
                               (0,1], defaults to 1. If no scale option is
                               specified, scales default to [1.0, 0.5, 0.25,
                               0.125, 0.0625].
      --id-service-fallback=ID_SERVICE_FALLBACK
                             Set a fallback id service for scenarios in which
                               an id service is not provided by the data
                               backend, e.g. when no `maxId' attribute is
                               specified in an N5 dataset. Valid options are
                               (case insensitive): from-data — infer the max id
                               and id service from the dataset (may take a long
                               time for large datasets), none — do not use an
                               id service (requesting new ids will not be
                               possible), and ask — show a dialog to choose
                               between those two options
                               Default: ask
      --include=INCLUDE...   Include any data set that matches any of INCLUDE
                               regex patterns. Takes precedence over EXCLUDE.
      --label-block-lookup-fallback=LABEL_BLOCK_LOOKUP_FALLBACK
                             Set a fallback label block lookup for scenarios in
                               which a label block lookup is not provided by
                               the data backend. The label block lookup is used
                               to process only relevant data during on-the-fly
                               mesh generation. Valid options are: `complete' —
                               always process the entire dataset (slow for
                               large data), `none' — do not process at all (no
                               3D representations/meshes available), and `ask'
                               — show a dialog to choose between those two
                               options
                               Default: ask
      --max=MAX              Maximum value of contrast range for raw and
                               channel data.
      --min=MIN              Minimum value of contrast range for raw and
                               channel data.
      --name=NAME            Specify name for dataset(s). The names are
                               assigned to datasets in the same order as
                               specified. If more datasets than names are
                               specified, the remaining dataset names will
                               default to the last segment of the dataset path.
      --num-screen-scales=NUM_SCREEN_SCALES
                             Number of screen scales, defaults to 3. If no
                               scale option is specified, scales default to
                               [1.0, 0.5, 0.25, 0.125, 0.0625].
  -o, --offset=OFFSET        Spatial offset for all dataset(s) specified by
                               DATASET. Takes meta-data over resolution
                               specified in meta data of DATASET
      --only-explicitly-included
                             When this option is set, use only data sets that
                               were explicitly included via INCLUDE. Equivalent
                               to --exclude '.*'
      --print-error-codes    List all error codes and exit.
  -r, --resolution=RESOLUTION
                             Spatial resolution for all dataset(s) specified by
                               DATASET. Takes meta-data over resolution
                               specified in meta data of DATASET
  -R, --revert-array-attributes
                             Revert array attributes found in meta data of
                               attributes of DATASET. Does not affect any array
                               attributes set explicitly through the RESOLUTION
                               or OFFSET options.
      --screen-scale-factor=SCREEN_SCALE_FACTOR
                             Scalar value from the open interval (0,1) that
                               defines how screen scales diminish in each
                               dimension. Defaults to 0.5. If no scale option
                               is specified, scales default to [1.0, 0.5, 0.25,
                               0.125, 0.0625].
      --screen-scales=SCREEN_SCALES[,SCREEN_SCALES...]...
                             Explicitly set screen scales. Must be strictly
                               monotonically decreasing values in from the
                               interval (0,1]. Overrides all other screen scale
                               options. If no scale option is specified, scales
                               default to [1.0, 0.5, 0.25, 0.125, 0.0625].
      --version              Print version string and exit
      --width=WIDTH          Initial width of viewer. Defaults to 800.
                               Overrides width stored in project.
                               Default: -1

hanslovsky · 2019-08-02T20:46:40Z

I will test a little more next week and then merge this PR.

hanslovsky · 2019-08-02T21:02:01Z

Or (requiring bash expansion of *):

jgo org.janelia.saalfeldlab:paintera:0.17.1-SNAPSHOT+org.slf4j:slf4j-simple:1.7.25 \
    --default-to-temp-directory \
    --add-n5-container $HOME/Downloads/cropped_samples/*.hdf \
    -d volumes/raw volumes/labels/neuron_ids \
    -R \
    --id-service-fallback=none \
    --label-block-lookup-fallback=none

instanceof takes care of that already.

hanslovsky · 2019-08-05T15:42:08Z

@igorpisarev I have tested this quite a bit on my machine. Can you try on your machine, as well. A few example commands (replace paintera with jgo org.janelia.saalfeldlab:paintera:0.17.1-SNAPSHOT+org.slf4j:slf4j-simple:1.7.25 after mvn clean install):

paintera --help
paintera --add-n5-container
paintera --add-n5-container=$HOME/local/tmp/sample_a_padded_20160501.n5 
paintera --add-n5-container=$HOME/local/tmp/sample_a_padded_20160501.n5  --entire-container
paintera --add-n5-container=$HOME/local/tmp/sample_a_padded_20160501.n5  -d volumes/raw volumes/labels/neuron_ids --default-to-temp-directory
paintera --add-n5-container=$HOME/local/tmp/sample_a_padded_20160501.n5  --entire-container --default-to-temp-directory
paintera --add-n5-container=$HOME/local/tmp/sample_a_padded_20160501.n5  --entire-container --exclude='.*' --default-to-temp-directory
paintera --add-n5-container=$HOME/local/tmp/sample_a_padded_20160501.n5  --entire-container --exclude='.*' --include '.*raw' --default-to-temp-directory
paintera --add-n5-container=$HOME/local/tmp/sample_a_padded_20160501.n5  --entire-container --exclude='.*' --include '.*raw' '*neuron_ids' --default-to-temp-directory
paintera --add-n5-container=$HOME/local/tmp/sample_a_padded_20160501.n5  --entire-container --exclude='.*' --include '.*raw' '.*neuron_ids' --default-to-temp-directory

Ideally with existing project as well (not with --default-to-temp-directory).

Thank you!

Use new options for Paintera main entry point instead

igorpisarev · 2019-08-05T20:29:57Z

@hanslovsky I tested it and it seems to work really well! Tried with both temp directory and a real path for storing the project, and different modes for specifying datasets such as -d and --entire-container with various --include and --exclude options.

hanslovsky · 2019-08-05T20:30:18Z

Thanks @igorpisarev I'll take care of the conflicts and then go ahead and merge

hanslovsky · 2019-08-05T21:13:25Z

I will go ahead and merge.

Obsolete as of saalfeldlab/paintera#273

hanslovsky added 10 commits July 31, 2019 15:54

Update picocli and throwing-function dependencies

f35784a

Add first working version of add-dataset option

a1e52b2

For example, this works: ``` --add-dataset --container=/data/hanslovskyp/cremi/sample_A_padded_20160501.n5 -d volumes/raw volumes/labels/neuron_ids ```

Add fall-back for N5 id service

de3feb2

Add converter for long[] arrays

edaab4c

Update label text for id serivce prompt

ae7b1ac

Move painteraArgs.addToViewer to just before stage is showing

795e106

Allow to not pass any --add-dataset arguments

c875953

Return a writer in N5HDF5Meta.reader()

53fae45

This is neccessary because hdf5 can be opened only in one mode at a time.

Some formatting fixes

4f258d8

Deprecate PainteraShowContainer

6db2be7

hanslovsky mentioned this pull request Aug 1, 2019

Update CLI with more flexible argument groups in picocli 4.0.0 saalfeldlab/paintera-conversion-helper#26

Closed

hanslovsky added 7 commits August 2, 2019 12:19

Improve maxId dialog

cc621b0

Parallelize over celss if cell img

ecfe422

Add id service fallback options to CLI

de30876

Add CLI option for label block lookup fallback

3683c05

Move cli specific methods from N5Helpers to PainteraCommandLineArgs

ac61bdd

Add option to load entire container

e1a2a37

Add inclulde/exclude patterns

5ef407a

hanslovsky changed the title ~~Add --add-dataset option to Paintera main class~~ Add --add-n5-container option to Paintera main class Aug 2, 2019

hanslovsky added 2 commits August 2, 2019 16:18

Remove obsolete logging

ea5e2fd

Improve options syntax

680d4ab

The container(s) are now specified in the --add-n5-container flag (instead of an additional --container flag)

hanslovsky added 5 commits August 2, 2019 17:02

Skip datasets that don't exist in a container

30c9955

Update help message

bc02799

Merge branch 'master' into update-picocli-dependency

6c17629

Set default screen scales to 1.0, 0.5, 0.25, 0.125, 0.0625

c542489

Remove unnecessary null check

b4938cd

instanceof takes care of that already.

hanslovsky added 2 commits August 5, 2019 14:38

Add generate-bash-completion script

0c3723d

Remove obsolete class PainteraShowContainer

604f252

Use new options for Paintera main entry point instead

Merge branch 'master' into update-picocli-dependency

914cbf9

hanslovsky merged commit 7525eef into master Aug 5, 2019

hanslovsky deleted the update-picocli-dependency branch August 5, 2019 21:15

hanslovsky added a commit to saalfeldlab/paintera-cmd that referenced this pull request Aug 5, 2019

Remove obsolete paintera-show-container command

d7ec3d6

Obsolete as of saalfeldlab/paintera#273

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add --add-n5-container option to Paintera main class #273

Add --add-n5-container option to Paintera main class #273

hanslovsky commented Jul 31, 2019 •

edited

Loading

hanslovsky commented Aug 2, 2019

hanslovsky commented Aug 2, 2019

hanslovsky commented Aug 2, 2019

hanslovsky commented Aug 5, 2019

igorpisarev commented Aug 5, 2019

hanslovsky commented Aug 5, 2019

hanslovsky commented Aug 5, 2019

Add --add-n5-container option to Paintera main class #273

Add --add-n5-container option to Paintera main class #273

Conversation

hanslovsky commented Jul 31, 2019 • edited Loading

hanslovsky commented Aug 2, 2019

hanslovsky commented Aug 2, 2019

hanslovsky commented Aug 2, 2019

hanslovsky commented Aug 5, 2019

igorpisarev commented Aug 5, 2019

hanslovsky commented Aug 5, 2019

hanslovsky commented Aug 5, 2019

hanslovsky commented Jul 31, 2019 •

edited

Loading