Package info from meta.yaml not updated on anaconda.org

[lindstro]

I have a question about getting package metadata updated on anaconda.org. Our zfp/zfpy packages were updated recently and contain the following in meta.yaml:

about:
  home: https://zfp.llnl.gov
  license: BSD-3-Clause
  license_family: BSD
  license_file: LICENSE
  summary: Library for compressed numerical arrays that support high throughput read and write random access
  description: |
    zfp is an open source C/C++ library for compressed numerical arrays
    that support high throughput read and write random access. zfp also
    supports streaming compression of integer and floating-point data,
    e.g., for applications that read and write large data sets to and
    from disk. zfp is primarily written in C and C++ but also includes
    Python and Fortran bindings.
  doc_url: https://zfp.readthedocs.io/
  dev_url: https://github.com/LLNL/zfp

The anaconda package website does not reflect the updated home, license, or doc_url fields, however:

The Files tab shows the package uploaded a couple of weeks ago. I'd appreciate any suggestions on how to get the metadata updated.

[yshmatov-anaconda]

You should be able to update meta data of existing release with anaconda update command (available since anaconda-client 1.9.0).

To do this, create JSON file with your metadata, e.g.:

{
  "home": "https://zfp.llnl.gov",
  "license": "BSD-3-Clause",
  "license_family": "BSD",
  "license_file": "LICENSE",
  "summary": "Library for compressed numerical arrays that support high throughput read and write random access",
  "description": "zfp is an open source C/C++ library for compressed numerical arrays that support high throughput read and write random access. zfp also supports streaming compression of integer and floating-point data, e.g., for applications that read and write large data sets to and from disk. zfp is primarily written in C and C++ but also includes Python and Fortran bindings.",
  "doc_url": "https://zfp.readthedocs.io/",
  "dev_url": "https://github.com/LLNL/zfp"
}

And run anaconda update command:

anaconda update --release conda-forge/zfp/1.0.0 new_meta.json

Where last argument is the path to your JSON file with new metadata.

[leofang]

@jessewiles This is my rule of thumb based on observations. I could be wrong but so far it seems I am right 🙂 If the package info (license, license url, etc) is correctly configured at the very first upload of the package to anaconda.org, regardless if it's staging or conda-forge (i.e. it's uploaded or copied), the info would be correct. Otherwise, the info would be incorrect.

[leofang]

Otherwise, the info would be incorrect.

And cannot be fixed by any subsequent package update.

[jessewiles]

The --replace and --update flags on the copy anaconda client command can be used to trigger requests that result in metadata change. The --replace flag causes the client to issue a PUT request and --update cause the client to issue a PATCH request.

Also of note, it's possible to update the metadata using the API directly. For example, I used the following request to update the metadata in a copy of zfp I created:

curl -X 'PATCH' \ 
        'https://api.anaconda.org/release/jwixel/zfp/1.0.0' \
        -H 'accept: application/json' \
        -H 'Content-Type: application/json' \
        -H 'Authorization: token <REDACT-TOKEN> \
        -d '{"public_attrs": {"doc_url": "https://zfp.readthedocs.io/", "dev_url": "https://github.com/LLNL/zfp", "home": "https://zfp.llnl.gov"}}'

[beckermr]

Ahhhh. I think we need to set replace=True here then: https://github.com/conda-forge/conda-forge-webservices/blob/main/conda_forge_webservices/feedstock_outputs.py#L126

[jessewiles]

And cannot be fixed by any subsequent package update.

As @yshmatov-anaconda (and I) noted, the info can be fixed with an API request that is independent of the package upload or copy.

[beckermr]

Nope. I tried multiple iterations of uploading, deleting etc. Here is how to reproduce the bug:

1. Copy a new package to your channel. I used

ac.copy(
    "conda-forge",
    "zfp",
    "0.5.5",
    basename=urllib.parse.quote("linux-64/zfp-0.5.5-he1b5a44_1.tar.bz2", safe=""),
    to_owner="beckermr",
    update=False,
    replace=False,
)

2. Delete this package from your personal channel. I did this in the UI.

3. Then copy a newer version

ac.copy(
    "conda-forge",
    "zfp",
    "1.0.0",
    basename=urllib.parse.quote("osx-64/zfp-1.0.0-ha894c9a_3.tar.bz2", safe=""),
    to_owner="beckermr",
    update=False,
    replace=True,
)

This new version has in its about.json

"home": "https://zfp.llnl.gov",
  "identifiers": [],
  "keywords": [],
  "license": "BSD-3-Clause",
  "license_family": "BSD",
  "license_file": "LICENSE",

and yet the UI still shows the info for version 0.5.5

I tried with and without the replace=True keyword and it doesn't change anything. I tried multiple replace=True calls and that didn't do anything either.

As @leofang eluded to, it appears the info in the UI is only set once on the first upload and can be added to but not replaced.

[jessewiles]

@jakirkham --update will result in a patch in which null or missing values in the target record are updated with the value from the source record. --replace will completely replace the target record with the source record. In this case, the package record has a null license, so you could probably get away with an --update? you don't need both though. --replace is more dependable unless you have a good reason to only partially update the target data.

[jakirkham]

Thanks Jesse! 🙏

Let's go with replace then. Would rather make sure metadata is getting updated when new packages go up (even and especially if that means overwriting something)

This is now in PR ( conda-forge/conda-forge-webservices#559 )

[beckermr]

With this flag, would attempting to upload a duplicate package end up overwriting the existing package itself? Conda-forge packages are strictly immutable except for repodata patches.

[jakirkham]

What we learned in the other thread is replace does more than replace metadata, it will overwrite the package itself, which is too much for our use case. So we are thinking update is a better choice for now

That said, it would be nice if there was a way to update metadata that is non-null (without overwriting the package). Not sure what that would be called, --update-all ?

[leofang]

Update: Thanks for help from @jakirkham @beckermr, we merged the above PR to use update on copy, and it does fix the missing license for us.

However, as both Matt and John noted above and elsewhere, this patch is only applicable to missing metadata, not for overwriting/correcting existing (wrong or outdated) metadata. So, I think John's suggestion

That said, it would be nice if there was a way to update metadata that is non-null (without overwriting the package). Not sure what that would be called, --update-all?

is still needed in case of such events, which does happen (ex: #739, where the license hyperlink needs to be fixed).