Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update version script to remove bump2version dependency #41

Merged
merged 4 commits into from
Jun 4, 2021

Conversation

gigony
Copy link
Contributor

@gigony gigony commented May 20, 2021

Refactor update-version.sh to not depend on bump2version utility.

@gigony gigony added breaking Introduces a breaking change improvement Improves an existing functionality labels May 20, 2021
@gigony gigony requested a review from ajschmidt8 May 20, 2021 19:19
@gigony gigony requested review from a team as code owners May 20, 2021 19:19
@@ -11,7 +11,7 @@
NEXT_FULL_TAG=$1

# Get current version
CURRENT_TAG=$(git tag | grep -xE 'v[0-9\.]+' | sort --version-sort | tail -n 1 | tr -d 'v')
CURRENT_TAG=$(cat VERSION)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change actually brought to my attention some issues with this script. I have some changes locally that I will put into a new PR that you can merge in here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this PR ( #42 )?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, @jakirkham. you beat me to it 🙂. Ray and I had to fix some things in this script. We'll be pushing those changes to all the other repos as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ajschmidt8 cuCIM project depends on VERSION file that is used in building/packaging (cmake/python).
According to update-version.sh's use (executed for each branching), I think getting the current tag (without 'v' or 'a' in it) from VERSION file does not impact on existing workflow. Please let me know if there is an issue with this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using the tag should be okay. I will test the update-version.sh script locally first before approving this PR.

Comment on lines 36 to 42
sed_runner "s/${CURRENT_TAG}/${NEXT_FULL_TAG}/g" python/cucim/VERSION
sed_runner "s/${CURRENT_TAG}/${NEXT_FULL_TAG}/g" cpp/plugins/cucim.kit.cuslide/VERSION
sed_runner "s#\[Version ${CURRENT_TAG}\](release_notes/v${CURRENT_TAG}.md)#\[Version ${NEXT_FULL_TAG}\](release_notes/v${NEXT_FULL_TAG}.md)#g" python/cucim/docs/index.md
sed_runner "s/v${CURRENT_TAG}/v${NEXT_FULL_TAG}/g" python/cucim/docs/getting_started/index.md
sed_runner "s#cucim.kit.cuslide@${CURRENT_TAG}.so#cucim.kit.cuslide@${NEXT_FULL_TAG}.so#g" python/cucim/docs/getting_started/index.md
sed_runner "s#cucim.kit.cuslide@${CURRENT_TAG}.so#cucim.kit.cuslide@${NEXT_FULL_TAG}.so#g" cucim.code-workspace
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
sed_runner "s/${CURRENT_TAG}/${NEXT_FULL_TAG}/g" VERSION
sed_runner "s/${CURRENT_TAG}/${NEXT_FULL_TAG}/g" python/cucim/VERSION
sed_runner "s/${CURRENT_TAG}/${NEXT_FULL_TAG}/g" cpp/plugins/cucim.kit.cuslide/VERSION
sed_runner "s#\[Version ${CURRENT_TAG}\](release_notes/v${CURRENT_TAG}.md)#\[Version ${NEXT_FULL_TAG}\](release_notes/v${NEXT_FULL_TAG}.md)#g" python/cucim/docs/index.md
sed_runner "s/v${CURRENT_TAG}/v${NEXT_FULL_TAG}/g" python/cucim/docs/getting_started/index.md
sed_runner "s#cucim.kit.cuslide@${CURRENT_TAG}.so#cucim.kit.cuslide@${NEXT_FULL_TAG}.so#g" python/cucim/docs/getting_started/index.md
sed_runner "s#cucim.kit.cuslide@${CURRENT_TAG}.so#cucim.kit.cuslide@${NEXT_FULL_TAG}.so#g" cucim.code-workspace
sed_runner "s/${CURRENT_LONG_TAG}/${NEXT_FULL_TAG}/g" VERSION
sed_runner "s/${CURRENT_LONG_TAG}/${NEXT_FULL_TAG}/g" python/cucim/VERSION
sed_runner "s/${CURRENT_LONG_TAG}/${NEXT_FULL_TAG}/g" cpp/plugins/cucim.kit.cuslide/VERSION
sed_runner "s#\[Version ${CURRENT_LONG_TAG}\](release_notes/v${CURRENT_LONG_TAG}.md)#\[Version ${NEXT_FULL_TAG}\](release_notes/v${NEXT_FULL_TAG}.md)#g" python/cucim/docs/index.md
sed_runner "s/v${CURRENT_LONG_TAG}/v${NEXT_FULL_TAG}/g" python/cucim/docs/getting_started/index.md
sed_runner "s#cucim.kit.cuslide@${CURRENT_LONG_TAG}.so#cucim.kit.cuslide@${NEXT_FULL_TAG}.so#g" python/cucim/docs/getting_started/index.md
sed_runner "s#cucim.kit.cuslide@${CURRENT_LONG_TAG}.so#cucim.kit.cuslide@${NEXT_FULL_TAG}.so#g" cucim.code-workspace

@gigony, can you merge the changes from #42 into your PR and then switch all of these to CURRENT_TAG variables to CURRENT_LOG_TAG?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Will do.

@@ -11,7 +11,7 @@
NEXT_FULL_TAG=$1

# Get current version
CURRENT_TAG=$(git tag | grep -xE 'v[0-9\.]+' | sort --version-sort | tail -n 1 | tr -d 'v')
CURRENT_TAG=$(cat VERSION)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you revert the cat VERSION change here and make sure this line matches the change that was done in #42 after you merge those changes into your PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, to make the statement more robust to tag changes from other branches (e.g., adding a patch tag[21.06.01] to branch-21.06 after creating tag[21.08.00]),

CURRENT_TAG=$(git tag | grep -xE 'v[0-9\.]+' | sort --version-sort | tail -n 1 | tr -d 'v')

to

CURRENT_TAG=$(git tag --merged HEAD | grep -xE 'v[0-9\.]+' | sort --version-sort | tail -n 1 | tr -d 'v')

what do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, that seems like a good idea. feel free to add it to this PR.

@ajschmidt8
Copy link
Member

@gigony, looks like you've got some merge conflicts. can you pull the latest from branch-21.06?

Remove scripts/docs related to bump2version because
we don't need it anymore since versioning scheme is
changed to CalVer.
@jakirkham
Copy link
Member

jakirkham commented May 21, 2021

We've been seeing some installation conflicts (also in CI currently). Unfortunately CI stops short of showing the conflict.

Initially suspected these were related to version discrepancies between cuCIM and the rest of RAPIDS. Maybe that is still true and we are just missing something here? Though maybe we are missing something else that is going on?

@ajschmidt8
Copy link
Member

rerun tests

@ajschmidt8
Copy link
Member

We've been seeing some installation conflicts (also in CI currently). Unfortunately CI stops short of showing the conflict.

Initially suspected these were related to version discrepancies between cuCIM and the rest of RAPIDS. Maybe that is still true and we are just missing something here? Though maybe we are missing something else that is going on?

Can't say I have any ideas about this. I just reran CI, so let's see if we run into the issue again.

@ajschmidt8
Copy link
Member

@gigony, any reason you used the breaking label for this PR? does this PR have any functional changes that would break things for cucim consumers? if not, you can switch the label to non-breaking.

@jakirkham
Copy link
Member

Yeah it's still failing. Unfortunately the builds are being terminated before we get the conflict report. Would it be possible to extend the build time so we could get this information?

@ajschmidt8
Copy link
Member

ajschmidt8 commented May 24, 2021

Yeah it's still failing. Unfortunately the builds are being terminated before we get the conflict report. Would it be possible to extend the build time so we could get this information?

sure. I just bumped it from 40 minutes to 2 hours.

rerun tests

@gigony gigony added non-breaking Introduces a non-breaking change and removed breaking Introduces a breaking change labels May 24, 2021
@gigony
Copy link
Contributor Author

gigony commented May 24, 2021

@gigony, any reason you used the breaking label for this PR? does this PR have any functional changes that would break things for cucim consumers? if not, you can switch the label to non-breaking.

Ah, I thought the 'breaking' label needs to be added if we touch some code but looks like we add it if it breaks things for 'user'?

In either case, it doesn't touch internal code and I removed the label.

@gigony
Copy link
Contributor Author

gigony commented May 26, 2021

@jakirkham @ajschmidt8 Do you have any idea on the following build error messages?

https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci-v0.20/job/cucim/job/prb/job/cucim-cpu-python-build/CUDA=11.2,PYTHON=3.7/62/console

14:19:07 # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
14:19:07 

14:19:07     Traceback (most recent call last):
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 251, in _load
14:19:07         raise UnavailableInvalidChannel(self.url_w_repodata_fn, 404)
14:19:07     conda.exceptions.UnavailableInvalidChannel: The channel is not accessible or is invalid.
14:19:07       channel name: rapidsai-nightly
14:19:07       channel url: https://conda.anaconda.org/rapidsai-nightly
14:19:07       error code: 404
14:19:07     
14:19:07     You will need to adjust your conda configuration to proceed.
14:19:07     Use `conda config --show channels` to view your configuration's current state,
14:19:07     and use `conda config --show-sources` to view config file locations.
14:19:07     
14:19:07     
14:19:07     During handling of the above exception, another exception occurred:
14:19:07     
14:19:07     Traceback (most recent call last):
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
14:19:07         return func(*args, **kwargs)
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/cli/main.py", line 84, in _main
14:19:07         exit_code = do_call(args, p)
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 82, in do_call
14:19:07         return getattr(module, func_name)(args, parser)
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/cli/main_install.py", line 20, in execute
14:19:07         install(args, parser, 'install')
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/cli/install.py", line 261, in install
14:19:07         unlink_link_transaction = solver.solve_for_transaction(
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/solve.py", line 114, in solve_for_transaction
14:19:07         unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/solve.py", line 157, in solve_for_diff
14:19:07         final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/solve.py", line 262, in solve_final_state
14:19:07         ssc = self._collect_all_metadata(ssc)
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
14:19:07         return f(*args, **kwds)
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/solve.py", line 415, in _collect_all_metadata
14:19:07         index, r = self._prepare(prepared_specs)
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/solve.py", line 1010, in _prepare
14:19:07         reduced_index = get_reduced_index(self.prefix, self.channels,
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/index.py", line 227, in get_reduced_index
14:19:07         new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 105, in query_all
14:19:07         result = tuple(concat(executor.map(subdir_query, channel_urls)))
14:19:07       File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
14:19:07         yield fs.pop().result()
14:19:07       File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 432, in result
14:19:07         return self.__get_result()
14:19:07       File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
14:19:07         raise self._exception
14:19:07       File "/opt/conda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
14:19:07         result = self.fn(*self.args, **self.kwargs)
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 97, in <lambda>
14:19:07         subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query(
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 110, in query
14:19:07         self.load()
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 174, in load
14:19:07         _internal_state = self._load()
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 255, in _load
14:19:07         return self._load()
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 276, in _load
14:19:07         _internal_state = self._process_raw_repodata_str(raw_repodata_str)
14:19:07       File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 349, in _process_raw_repodata_str
14:19:07         json_obj = json.loads(raw_repodata_str or '{}')
14:19:07       File "/opt/conda/lib/python3.8/json/__init__.py", line 357, in loads
14:19:07         return _default_decoder.decode(s)
14:19:07       File "/opt/conda/lib/python3.8/json/decoder.py", line 337, in decode
14:19:07         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
14:19:07       File "/opt/conda/lib/python3.8/json/decoder.py", line 353, in raw_decode
14:19:07         obj, end = self.scan_once(s, idx)
14:19:07     json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 917721 column 78 (char 26270959)
14:19:07 

14:19:07 `$ /opt/conda/bin/conda install -c conda-forge conda-build`
14:19:07 

14:19:07   environment variables:
14:19:07                  CIO_TEST=<not set>
14:19:07                 CLASSPATH=
14:19:07             CONDA_BLD_DIR=/workspace/.conda-bld
14:19:07                 CONDA_DIR=/opt/conda
14:19:07                 CONDA_EXE=/opt/conda/bin/conda
14:19:07          CONDA_PYTHON_EXE=/opt/conda/bin/python
14:19:07                CONDA_ROOT=/opt/conda
14:19:07               CONDA_SHLVL=0
14:19:07            CONDA_USERNAME=
14:19:07           LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/u
14:19:07                           sr/local/lib
14:19:07              LIBRARY_PATH=/usr/local/cuda/lib64/stubs
14:19:07                      PATH=/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/cuda/bin:
14:19:07                           /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/game
14:19:07                           s:/usr/local/games:/snap/bin
14:19:07                    PYTHON=3.7
14:19:07                PYTHON_VER=3.7
14:19:07        REQUESTS_CA_BUNDLE=<not set>
14:19:07             SSL_CERT_FILE=<not set>
14:19:07 

14:19:07      active environment : None
14:19:07             shell level : 0
14:19:07        user config file : /workspace/.condarc
14:19:07  populated config files : /opt/conda/.condarc
14:19:07           conda version : 4.8.3
14:19:07     conda-build version : not installed
14:19:07          python version : 3.8.3.final.0
14:19:07        virtual packages : __glibc=2.23
14:19:07        base environment : /opt/conda  (writable)
14:19:07            channel URLs : https://conda.anaconda.org/conda-forge/linux-64
14:19:07                           https://conda.anaconda.org/conda-forge/noarch
14:19:07                           https://conda.anaconda.org/gpuci/linux-64
14:19:07                           https://conda.anaconda.org/gpuci/noarch
14:19:07                           https://conda.anaconda.org/rapidsai-nightly/linux-64
14:19:07                           https://conda.anaconda.org/rapidsai-nightly/noarch
14:19:07                           https://conda.anaconda.org/rapidsai/linux-64
14:19:07                           https://conda.anaconda.org/rapidsai/noarch
14:19:07                           https://conda.anaconda.org/nvidia/linux-64
14:19:07                           https://conda.anaconda.org/nvidia/noarch
14:19:07                           https://conda.anaconda.org/pytorch/linux-64
14:19:07                           https://conda.anaconda.org/pytorch/noarch
14:19:07           package cache : /opt/conda/pkgs
14:19:07                           /workspace/.conda/pkgs
14:19:07        envs directories : /opt/conda/envs
14:19:07                           /workspace/.conda/envs
14:19:07                platform : linux-64
14:19:07              user-agent : conda/4.8.3 requests/2.23.0 CPython/3.8.3 Linux/5.4.0-1038-aws ubuntu/16.04.7 glibc/2.23
14:19:07                 UID:GID : 1000:1000
14:19:07              netrc file : None
14:19:07            offline mode : False
14:19:07 

14:19:07 

14:19:07 An unexpected error has occurred. Conda has prepared the above report.
14:19:07 

14:19:08 Build step 'Execute shell' marked build as failure
14:19:14 [Set GitHub commit status (universal)] ERROR on repos [GHRepository@362b03d2[nodeId=MDEwOlJlcG9zaXRvcnkzNTYzNTYzMzY=,description=<null>,homepage=,name=cucim,fork=false,archived=false,size=18229,milestones={},language=Jupyter Notebook,commits={},source=<null>,parent=<null>,isTemplate=<null>,url=https://api.github.com/repos/rapidsai/cucim,id=356356336,nodeId=<null>,createdAt=2021-04-09T17:54:18Z,updatedAt=2021-05-24T13:13:32Z]] (sha:46ca77b) with context:gpuCI/cucim/build/python/3.7/cuda/11.2

And

https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci-v0.20/job/cucim/job/prb/job/cucim-gpu-test/CUDA=11.0,GPU_LABEL=gpu,OS=ubuntu16.04,PYTHON=3.7/62/console

shows bunch of conflicts.

@ajschmidt8
Copy link
Member

ajschmidt8 commented May 26, 2021

@gigony, I think the first error log you posted was a random conda server error. the real issue here is likely with the conflicts in the second log you posted.

@ajschmidt8
Copy link
Member

from quickly looking at the logs, I noticed this discrepancy in Python versions. this might warrant further investigation.

image

@ajschmidt8 ajschmidt8 merged commit 17eff5f into rapidsai:branch-21.06 Jun 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improves an existing functionality non-breaking Introduces a non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants