Retrospectively assign group attribution for existing GO-CAMs

[RLovering]

Hi
I have raised this issue a couple of times, obviously I should have created a ticket so that I can be kept in the loop about progress in this area.

Currently there is no incentive for my team at UCL to use Noctua to create annotation because the annotations we create will not appear in AmiGO or QuickGO as attributed to one of the UCL teams, therefore I will not be able to create a report for my grant funding bodies on the annotations created and demonstrate that I have met my funded targets.

In addition, It is important that any SynGO Noctua annotations that we make that are exported to the GOC annotation files are attributed to SynGO-UCL as SynGO VU curators want to be able to distinguish their 'expert created' annotations from those UCL is creating.

In addition, Tony has pointed out that currently he is unable to pick up the Noctua annotations. Consequently the GO annotations displayed by QuickGO, NCBI Gene, Ensembl, UniProt and the files incorporated into many functional analysis tools are out of sync with the data displayed in AmiGO.

Best

Ruth

@cmungall @tonysawfordebi @BarbaraCzub @thomaspd

[cmungall]

Obviously this is not working as intended.

Let's break this down. The first thing to check is the model itself, let's see if your work is attributed to the relevant groups in the model - can you post a link to a model?

[cmungall]

• retrospective: we need a way to retrospectively go back to annotations done by people in your group and add the provided_by assertions
• prospective: we need to make sure moving forward people know how attribution works, and that they need to select a group for it not to be attributed to GO_Noctua. We will go over this on an annotation call.
• Translation from models to GPAD. We will also go over this and ensure it works according to people's expectations. There are some known issues to be discussed - e.g. what if >1 group is assigned to a piece of evidence (see providedBy should be used in GPAD export #502)

[BarbaraCzub]

Hello @cmungall

You're welcome to use the model entitled 'SynGO no. 1a SLC1A1_PMID:7914198' as an example:
http://noctua.berkeleybop.org/editor/graph/gomodel:586fc17a00001440

It falls under SynGO-UCL.

Thanks,
Barbara

[cmungall]

Thanks!

OK, so in this model there is no providedBy at the level of the evidence assertions. That's as expected as you didn't know about the groups ability when making these in March. Now don't worry we're not going to make you go back and retrospectively manually tag every evidence instance with your project, we'll figure out a solution here.

One idea we kicked around today was this one:
geneontology/minerva#119
In this case, you already have a model-level providedBy:

so the idea would be that 'orphan' evidences would inherit from the model level (prospectively, you would try and make sure you wear the correct 'hat' whenever making new assertions). However, we need to discuss this further.

Thanks for your patience!

[BarbaraCzub]

I think a model-level providedBy could be a suitable solution.

[kltm]

For looking at the scope of the providedBy issues, here are some numbers:

Of the 595 non-SYNGO models...

• has 0 providedBy: 470
  • has 0 ORCIDs: 51
  • has 1 ORCID: 73
  • has more than 1 ORCID: 346
• has at least 1 providedBy: 125
  • has 0 ORCIDs: 1
  • has 1 ORCID: 89
  • has more than 1 ORCID: 35

(Also, we have 98 models that include some kind of GOC xref for the user; I believe we should map those forward: geneontology/go-site#453 (comment))

Given that this problem is unlikely to be manually tractable, I would suggest the following approach once the metadata is updated, but would like feedback and agreement from all parties:

• All GOC xrefs are replaced with the full user URIs.
• For all models that do not have a model-level providedBy
  • For every contributor statement that does not already have an analogous providedBy statement we add one to the user's default group.

The middle step there is for hoping that groups that are using the groups to their full effect (to track different funding sources, etc.) can give us a list of those at the model level so we can add those with a quick script.

This is more a @cmungall or @balhoff question, but I'm not actually sure of the state of our use of model-level providedBys--I can see they are used by SYNGO as a default, but not much elsewhere. I think that unless there will be more model-level defaults used in the future it might simplify things to just make sure that those are pushed down into the model as much as possible (as the current meaning is more like a "touch" than a default and we will have cases where there is a mix).

For those who want to look at getting these numbers out of the files on the command line:

grep -c provided 0*.ttl 5*.ttl  | grep "\:0" | cut -d ':' -f 1 | xargs -d '\n' bash -c 'for filename; do grep -H orcid "$filename" | sort | uniq; done;' | cut -d ':' -f 1 | sort | uniq -c | sort -nr | grep "1 " | wc

[kltm]

Moving forward:

• script for
  • if noctua perms, check at least one group
  • if noctua perms, check has ORCID

For now, make report public; later, make sanity check.
Once completed, can migrate all GOC:xzys forward.

All people in users.yaml who are Noctua/GO-CAM editors need to understand that the top group is the default one. If we cannot make this clear, we should add a new required field: group-default.

Once the above migration is done, create a spreadsheet has:

• col1 model name
• col2 model ID
• col3 contributor names/ids
• col4: for users to add their preferred providedBy for all activity in a model that is different than their current default.

(We may tag @dougli1sqrd to help out with that spreadsheet generation.)

Once that is filled in, it will be back to the software group to figure out the implementation strategy.

[kltm]

@vanaukenk looking at the output of the scripts, we have 177 violations. The script will run in a public place tonight; I'll try and add the link soon.

[vanaukenk]

@kltm -thanks. If you have the link for the script output, I can add it to the agenda for the annotation call tomorrow:
http://wiki.geneontology.org/index.php/Annotation_Conf._Call_2017-11-28#users.yaml_file

[kltm]

@vanaukenk
This should now update nightly, assuming the pipeline is successful:
http://skyhook.berkeleybop.org/snapshot/metadata/users-and-groups-report.txt

[vanaukenk]

Okay, thanks. Right now I'm getting this when I click on the link above:

Not Found

The requested URL /snapshot/metadata/users-and-groups-report.txt was not found on this server.

[kltm]

Clicking on the above works for me (now). It is possible that you looked at the snapshot while the data products were rebuilding.

[vanaukenk]

Okay, thanks @kltm I can see the report now, and will add the link to the 2017-11-28 annotation call minutes on the wiki.

[kltm]

Please keep in mind that the report is generated as part of the new build and a new attempt is made every night. This means that 1) if the load fails the report may not be there (which is a good thing) and 2) there will times every day during regeneration when the report is not yet there.

[cmungall]

Some additional notes for my benefit

to run the report, from g-site:

python3 ./scripts/sanity-check-users-and-groups.py --users metadata/users.yaml --groups metadata/groups.yaml

looks like we have a lot of people missing groups. Many may no longer be active so it would be good to indicate these

[balhoff]

@cmungall @kltm there are also group IRIs in Noctua models for which there is no entry in groups.yaml. For example http://purl.obolibrary.org/go/groups/UniProt in 59dc728000000205. I'm not sure how these have gotten in.

[kltm]

Okay, these are getting in by two of our workbenches.

#459 (comment)

Essentially,

                        reqs.use_groups([
                            // WARNING: We're minting money that we
                            // might not honor here.
                            'http://purl.obolibrary.org/go/groups/' + assby
                        ]);

This also relates to #539

@cmungall

[kltm]

@cmungall
From #458 (comment)
I would like to add a check such that if there are noctua edit rights then a group must be assigned.
I would implement this as a first check for progress.

[balhoff]

List of production models without model-level providedBy: http://yasgui.org/short/HJSFxyvFM

I tried a similar query that dug into providedBy on particular statements and got the same number of models, so I think it's fine to use this simpler query.

61 models—maybe not too many to just look at individually?

[balhoff]

Once users.ttl and groups.ttl are added to the triplestore, I could make a query that provides an example of what an automatically proposed group for each would be.

[kltm]

A bit of a slog, but not impossible. If we make a spreadsheet or something that curators can get and and update, I have a tool that we can use to just inject those into the models.

Just to confirm, currently, if there is no contributes/provided by annotation "locally" present, the exporter grabs it from the model level annotations?

[balhoff]

Just to confirm, currently, if there is no contributes/provided by annotation "locally" present, the exporter grabs it from the model level annotations?

@kltm yes that is true.

[balhoff]

This query adds contributor ID and name: http://yasgui.org/short/Hy0bwkwtG

It only creates 4 additional rows, so nearly all of these models have only one contributor. It's a very short list of folks; I think this will be easy to sort out.

[balhoff]

Query for groups used in production models: http://yasgui.org/short/SJvdsyvFz

[cmungall]

@balhoff @dougli1sqrd - when was that triplestore last updated? Will we be able to sparql the source store (or something closely synced to it) soon?

[cmungall]

I would like to add a check such that if there are noctua edit rights then a group must be assigned.

Yes, let's do this.

[kltm]

@cmungall I updated the triplestores earlier today.

[kltm]

As a step in this process (and discussion today), @dougli1sqrd will be looking at fixing up the users.yaml and groups.yaml. As part of this change, any Noctua editors who do not have an ORCID as their uri will temporarily lose permissions to edit in Noctua until they are re-added. I think most editors at this point should be clear, but if anybody wants to double check the current report can be found here:
http://snapshot.geneontology.org/reports/users-and-groups-report.txt
The end goal of @dougli1sqrd 's work will be that report is empty (and we've added it to our travis checks).

[cmungall]

covered by geneontology/noctua-models#84, and done