Implement POST /v1/collections

[Bento007]

Description: Opens a new submission with a unique UUID. If no project UUID is given, then this call will also create a new folder with the generated project UUID in the data portal S3 bucket. On success, a message will be returned with the project’s UUID.

Parameters:

1. name [REQUIRED]: Name of the collection to be created.
2. description [REQUIRED]: A description of the collection to be created.
3. user_id [REQUIRED]: the OAuth id of the user creating the new collection.
4. links [OPTIONAL]: a mapping of link names to their link type and URL that should be associated with the collection.

UPDATE (11/2/2020):

Example JSON payload from FE:

"{"name":"collection name","description":"collection description","contact_name":"collection contact name","contact_email":"contact@gmail.com","links":[{"url":"http://doi.org/10.1016","type":"DOI"},{"url":"http://doi.org/10.1017","type":"DOI"}]}"

Responses:

1. 201 Created: {collection_uuid}
2. 401 Unauthorized user

Behavior to implement

• The backend needs to generate a new UUID.
• Create a row in the database with the created UUID, given name, owner, and description.
• Populate the ProjectLink table in the database with the inputted links if any.
• Return

Definition of Done:

• API is implemented with both unit tests and E2E tests
• High level architecture is updated to reflect the exact API definition that was implemented.

[Bento007]

We could update this API to only create a new submission for now and not update an existing. We will also need an API to upload files to s3 for the dataset assets.

I think it would actually be better if we had a separate endpoint for uploading datasets, and dataset assets.

This ticket still needs a definition of done.

[Bento007]

Definition of Done

• Use POST to create a new collections

• No updating existing collections.

• No deleting of existing collections even if owner.

• Project, name description, links attributes are included in the api request

• User must authenticate to create a submission

• collection is assigned the owner who creates it

• Stretch: The project page loads performantly when 1000 projects exist in project table.

Q: Check with @tihuan about feature flagging this.

[tihuan]

Ooh yes, feature flag!

@brianraymor for incrementally rolling out the submission feature, we were thinking about using a feature flag to control who get access to this feature, so we get to QA this in prod and share it with a few internal users to collect feedback before a full launch.

Some ideas I have in mind are:

1. Easiest way: put submission feature on a separate page/url, and only people who know the url can access the page. E.g., cellxgene.cziscience.com/projects/new or cellxgene.cziscience.com/collections/new (since I remember we want to move from project to collection?) And then full feature launch would just be adding links to the URL at relevant places on the website

Upside: straightforward to implement + low engineering overhead
Downside: Launching the feature requires code change and a deployment

2. Still quite easy way: Put submission feature on a separate page/url like Adding some initial structure and basic files to repo. #1, but show/hide links to the URL based on URL param. E.g., (cellxgene.cziscience.com?submission=1)

Upside: straightforward to implement + low engineering overhead
Downside: Launching the feature requires minimal code change and a deployment

3. Use Split.io to control which user gets to see the submission feature

Upside: launching the feature only requires flipping a switch in Split.io's website. Anyone can do it!
Downside: Would probably take two sprints to design, get approval, and implement Split.io for feature flag management. Will need to implement AuthN for Split.io to know who get to see the feature

My recommendation is #2 for rolling out submission feature, and then once we have AuthN for DP, spend time to do #3. But would love to hear your thoughts on this!

Thank you!

[maniarathi]

Mind adding "Update High Level Architecture document with implemented API definition" as part of the list of tasks for this to be done?

[tihuan]

Hi @MDunitz ! Thanks so much for working on the POST endpoint 🎉🤩

A few things I think we need to sync on:

1. user_id: This one is sent back to the BE via httpOnly Auth0 cookie, which means FE can't read it via JavaScript. So BE will need to parse the cookie to get the user id 🙏 @bmccandless should be able to help parse the cookie!

2. The design has added contact_name and contact_email as required fields too, so we might need to add that to the DB as well

3. For collection_links, should we go with a JSON string like [{ "name": "link_name", "type": "link_type", "url": "link_url" }, ...]?

4. I'm assuming the BE expects the payload to be JSON string, is that right?

5. TEMP SOLUTION: Since Data Portal doesn't have AuthN yet, maybe FE can send a hard coded value as user_id, so BE can use that to create a new collection? If that sounds like, what hard coded value do you prefer for BE? Any string, like test-id would do?

CC: @Bento007 for keeping everyone in sync!

Thanks all!

[tihuan]

OH and also, I think FE is gonna send attestation version number to BE, what field key should I use? attestation_version? Thank you!

UPDATE: I see it's data_submission_policy_version now! Thanks!

[MDunitz]

I've run into issues creating a functional test due to an inability to get a token from auth0.
Useful links/documentation

• [stack overflow example] (https://stackoverflow.com/questions/48552474/auth0-obtain-access-token-for-unit-tests-in-python/48554119#48554119)
• [auth0 python] (https://auth0.com/docs/quickstart/backend/python/02-using)
• [auth0 webauth] (https://auth0.com/docs/libraries/auth0js?_ga=2.152784511.792956135.1604942109-633642578.1604942109#webauth-login-)
• [auth0 settings] (https://manage.auth0.com/dashboard/us/czi-cellxgene-dev/applications/c3O5vw6lTeuL5uhRwb5FuK1Jwq22mR5R/settings)
  This is what I've tried

 def getUserToken(userName):
        # testing user password database:
        testingUsers = {
            'testingUser2@funnymail.com': 'BuQ3tUS3 :jbFAL',
            'testingUser3w@funnymail.com': 'y(1726854(b(-KY'
        }

        # client id and secret come from LogIn (Test Client)! which has password enabled under "Client > Advanced > Grant Types > Tick Password"
        url = 'https://YOUR_AUTH0_DOMAIN/oauth/token'
        headers = {'content-type': 'application/json'}
        password = testingUsers[userName]
        parameter = {"client_id": "XXX",
                     "client_secret": "XXXX",
                     "audience": 'AUTH0_AUDIENCE',
                     "grant_type": "password",
                     "username": userName,
                     "password": password, "scope": "openid"}
        # do the equivalent of a CURL request from https://auth0.com/docs/quickstart/backend/python/02-using#obtaining-an-access-token-for-testing
        responseDICT = json.loads(requests.post(url, json=parameter, headers=headers).text)
        return responseDICT['access_token']

       
 def get_bearer_token_from_auth_0(self):
        import http.client

        conn = http.client.HTTPSConnection("czi-cellxgene-dev.us.auth0.com")

        payload = "{\"client_id\":\"XXXX\",\"client_secret\":\"XXXX\",\"audience\":\"https://czi-cellxgene-dev.us.auth0.com/api/v2/\",\"grant_type\":\"client_credentials\"}"

        headers = {'content-type': "application/json"}

        conn.request("POST", "/oauth/token", payload, headers)

        res = conn.getresponse()
        data = json.loads(res.read())
        return f"{data['token_type']} {data['access_token']}"

    def get_token_from_auth0(self):
        import requests

        # conn = http.client.HTTPConnection("https://czi-cellxgene-dev.us.auth0.com/oauth/token")

        headers = {
            'authorization': "Bearer XXXX"}
        res = requests.get("https://czi-cellxgene-dev.us.auth0.com/authorize", headers=headers)
        return res
    def test_create_and_retrieve_collection(self):
        # token = self.get_token_from_auth0()

        data = json.dumps(dict(
            name="test collection",
            description="This is a test collection",
            contact_name="person human",
            contact_email="person@human.com",
            data_submission_policy_version="0.0.1",
            links=[
                {"link_name": "DOI Link", "link_url": "http://doi.org/10.1016", "link_type": "DOI"},
                {"link_name": "DOI Link 2", "link_url": "http://doi.org/10.1017", "link_type": "DOI"}]
        ))
        res = requests.post(
            f"{self.api}/dp/v1/collections",
            headers={"host": "localhost", "Content-Type": "application/json", "Cookie": "Bearer XXX"},
            data=data
        )
        self.assertEqual(res.status_code, requests.codes.created)

But I'm still getting a 401. For now I'm going to push the code without the functional test