Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test adding and removing annotator from projects part way & check utility for repeat annotation #366

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from

Conversation

twinkarma
Copy link
Collaborator

Resolves #322, resolves #323

I've downloaded a db backup from the production site and tried to reproduce the error from #322 and #323 but can't seem to reproduced it. I've contacted Mughda about this and we may be able to get the annotator to check if they're still getting the same errors.

Changes

  • Added backend tests for:
    • Removing an annotator from a project they've partially annotated in and adding them to another one
    • Rejecting an annotator from a project they've partially annotated in and adding them to another one
  • Added CLI command check_util repeat_annotation to check whether there are any repeat annotations, i.e. a document has multiple annotations from the same annotator (status COMPLETED, PENDING or REJECTED only)
  • When checking the test score, only completed annotation should be checked

@twinkarma
Copy link
Collaborator Author

Failing test not related to this PR, will be resolved in #372

@twinkarma
Copy link
Collaborator Author

I'm now getting a different error, will be looking into this:

=================================== FAILURES ===================================
______________ TestProjectModel.test_get_annotator_document_score ______________

self = <backend.tests.test_models.TestProjectModel testMethod=test_get_annotator_document_score>

    def test_get_annotator_document_score(self):
        annotator = self.annotators[0]
        # No annotations == 0
        self.assertEqual(0, self.project.get_annotator_document_score(annotator, DocumentType.TEST))
    
        incorrect_data = {
            "label1": "Incorrect",
            "label2": "Incorrect"
        }
    
        # All incorrect
        for doc in self.test_docs:
            anno = Annotation.objects.create(user=annotator, document=doc)
            anno.data = incorrect_data
    
        self.assertEqual(0, self.project.get_annotator_document_score(annotator, DocumentType.TEST))
    
        # All correct
        annotator2 = self.annotators[1]
        for doc in self.test_docs:
            correct_annotation_data = {
                "label1": doc.data["gold"]["label1"]["value"],
                "label2": doc.data["gold"]["label2"]["value"],
            }
            anno = Annotation.objects.create(user=annotator2, document=doc)
            anno.data = correct_annotation_data
    
>       self.assertEqual(self.num_test_docs, self.project.get_annotator_document_score(annotator2, DocumentType.TEST))
E       AssertionError: 7 != 0

backend/tests/test_models.py:696: AssertionError

get_annotation_score was fixed to check only for COMPLETED annotation in the last commit. The test has been changed to reflect this.
@twinkarma
Copy link
Collaborator Author

The issue should have been fixed with previous commits, this is then just an extra tests to check for the exact conditions described in the issues.

The current failing test is due to be fixed in #372

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant