Skip to content

Commit

Permalink
Merge branch 'main' into za/2402-design-review
Browse files Browse the repository at this point in the history
  • Loading branch information
zandercymatics committed Oct 3, 2024
2 parents f67c4e0 + 9aa3504 commit 2f8408e
Show file tree
Hide file tree
Showing 32 changed files with 1,800 additions and 400 deletions.
2 changes: 1 addition & 1 deletion docs/developer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ You can change the logging verbosity, if needed. Do a web search for "django log

## Mock data

[load.py](../../src/registrar/management/commands/load.py) called from docker-compose (locally) and reset-db.yml (upper) loads the fixtures from [fixtures_user.py](../../src/registrar/fixtures_users.py) and [fixtures_domain_requests.py](../../src/registrar/fixtures_domain_requests.py), giving you some test data to play with while developing.
[load.py](../../src/registrar/management/commands/load.py) called from docker-compose (locally) and reset-db.yml (upper) loads the fixtures from [fixtures_user.py](../../src/registrar/fixtures/fixtures_users.py) and the rest of the data-loading fixtures in that fixtures folder, giving you some test data to play with while developing.

See the [database-access README](./database-access.md) for information on how to pull data to update these fixtures.

Expand Down
55 changes: 29 additions & 26 deletions docs/operations/import_export.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,16 @@ Simple scripts are provided as detailed below.
### Export

To export from the source environment, run the following command from src directory:
manage.py export_tables

Connect to the source sandbox and run the command:
cf ssh {source-app}
/tmp/lifecycle/shell
./manage.py export_tables
`cf ssh {source-app}`
`/tmp/lifecycle/shell`
`./manage.py export_tables`

example exporting from getgov-stable:
cf ssh getgov-stable
/tmp/lifecycle/shell
./manage.py export_tables
`cf ssh getgov-stable`
`/tmp/lifecycle/shell`
`./manage.py export_tables`

This exports a file, exported_tables.zip, to the tmp directory

Expand All @@ -42,14 +41,16 @@ After exporting the file from the target environment, scp the exported_tables.zi
file from the target environment to local. Run the below commands from local.

Get passcode by running:
cf ssh-code
`cf ssh-code`

scp file from source app to local file:
scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app {source-app} --guid)/processes | jq -r '.resources[] | select(.type=="web") | .guid')/0 ssh.fr.cloud.gov:app/tmp/exported_tables.zip {local_file_path}
`scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app {source-app} --guid)/processes | jq -r '.resources[] | select(.type=="web") | .guid')/0 ssh.fr.cloud.gov:app/tmp/exported_tables.zip {local_file_path}`
when prompted, supply the passcode retrieved in the 'cf ssh-code' command

example copying from stable to local cwd:
scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app getgov-stable --guid)/processes | jq -r '.resources[] | select(.type=="web") | .guid')/0 ssh.fr.cloud.gov:app/tmp/exported_tables.zip .
`scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app getgov-stable --guid)/processes | jq -r '.resources[] | select(.type=="web") | .guid')/0 ssh.fr.cloud.gov:app/tmp/exported_tables.zip .`

`scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app getgov-stable --guid)/processes | jq -r '.resources[] | select(.type=="web") | .guid')/0 ssh.fr.cloud.gov:app/tmp/exported_tables.zip .`


### Import
Expand All @@ -63,14 +64,14 @@ that there are no database conflicts on import.

In order to delete all rows from the appropriate tables, run the following
command:
cf ssh {target-app}
/tmp/lifecycle/shell
./manage.py clean_tables
`cf ssh {target-app}`
`/tmp/lifecycle/shell`
`./manage.py clean_tables`

example cleaning getgov-backup:
cf ssh getgov-backup
/tmp/lifecycle/backup
./manage.py clean_tables
`cf ssh getgov-backup`
`/tmp/lifecycle/shell`
`./manage.py clean_tables`

For reference, this deletes all rows from the following tables:

Expand All @@ -96,28 +97,30 @@ with --skipEppSave option set to False. If you set to False, it will attempt to
records to the registry on load. If this is unset, or set to True, it will load the database and not
attempt to update the registry on load.

Please note that there is currently a bug (missing batch importing, see #2862) so this may not work
smoothly right now currently.

To scp the exported_tables.zip file from local to the sandbox, run the following:

Get passcode by running:
cf ssh-code
`cf ssh-code`

scp file from local to target app:
scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app {target-app} --guid)/processes | jq -r '.resources[] | select(.type=="web") | .guid')/0 {local_file_path} ssh.fr.cloud.gov:app/tmp/exported_tables.zip
`scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app {target-app} --guid)/processes | jq -r '.resources[] | select(.type=="web") | .guid')/0 {local_file_path} ssh.fr.cloud.gov:app/tmp/exported_tables.zip`
when prompted, supply the passcode retrieved in the 'cf ssh-code' command

example copy of local file in tmp to getgov-backup:
scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app getgov-backup --guid)/processes | jq -r '.resources[] | select(.type=="web") | .guid')/0 tmp/exported_tables.zip ssh.fr.cloud.gov:app/tmp/exported_tables.zip

`scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app getgov-backup --guid)/processes | jq -r '.resources[] | select(.type=="web") | .guid')/0 exported_tables.zip ssh.fr.cloud.gov:app/tmp/exported_tables.zip`

Then connect to a shell in the target environment, and run the following import command:
cf ssh {target-app}
/tmp/lifecycle/shell
./manage.py import_tables
`cf ssh {target-app}`
`/tmp/lifecycle/shell`
`./manage.py import_tables`

example cleaning getgov-backup:
cf ssh getgov-backup
/tmp/lifecycle/backup
./manage.py import_tables --no-skipEppSave
`cf ssh getgov-backup`
`/tmp/lifecycle/shell`
`./manage.py import_tables --no-skipEppSave`

For reference, this imports tables in the following order:

Expand Down
66 changes: 66 additions & 0 deletions src/api/tests/test_rdap.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
"""Test the domain rdap lookup API."""

import json

from django.contrib.auth import get_user_model
from django.test import RequestFactory
from django.test import TestCase

from ..views import rdap

API_BASE_PATH = "/api/v1/rdap/?domain="


class RdapViewTest(TestCase):
"""Test that the RDAP view function works as expected"""

def setUp(self):
super().setUp()
self.user = get_user_model().objects.create(username="username")
self.factory = RequestFactory()

def test_rdap_get_no_tld(self):
"""RDAP API successfully fetches RDAP for domain without a TLD"""
request = self.factory.get(API_BASE_PATH + "whitehouse")
request.user = self.user
response = rdap(request, domain="whitehouse")
# contains the right text
self.assertContains(response, "rdap")
# can be parsed into JSON with appropriate keys
response_object = json.loads(response.content)
self.assertIn("rdapConformance", response_object)

def test_rdap_invalid_domain(self):
"""RDAP API accepts invalid domain queries and returns JSON response
with appropriate error codes"""
request = self.factory.get(API_BASE_PATH + "whitehouse.com")
request.user = self.user
response = rdap(request, domain="whitehouse.com")

self.assertContains(response, "errorCode")
response_object = json.loads(response.content)
self.assertIn("errorCode", response_object)


class RdapAPITest(TestCase):
"""Test that the API can be called as expected."""

def setUp(self):
super().setUp()
username = "test_user"
first_name = "First"
last_name = "Last"
email = "info@example.com"
title = "title"
phone = "8080102431"
self.user = get_user_model().objects.create(
username=username, title=title, first_name=first_name, last_name=last_name, email=email, phone=phone
)

def test_rdap_get(self):
"""Can call RDAP API"""
self.client.force_login(self.user)
response = self.client.get(API_BASE_PATH + "whitehouse.gov")
self.assertContains(response, "rdap")
response_object = json.loads(response.content)
self.assertIn("rdapConformance", response_object)
44 changes: 18 additions & 26 deletions src/api/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

from django.apps import apps
from django.views.decorators.http import require_http_methods
from django.http import HttpResponse
from django.http import HttpResponse, JsonResponse
from django.utils.safestring import mark_safe

from registrar.templatetags.url_helpers import public_site_url
Expand All @@ -18,7 +18,7 @@
from registrar.utility.s3_bucket import S3ClientError, S3ClientHelper


DOMAIN_FILE_URL = "https://raw.githubusercontent.com/cisagov/dotgov-data/main/current-full.csv"
RDAP_URL = "https://rdap.cloudflareregistry.com/rdap/domain/{domain}"


DOMAIN_API_MESSAGES = {
Expand All @@ -41,30 +41,6 @@
}


# this file doesn't change that often, nor is it that big, so cache the result
# in memory for ten minutes
@ttl_cache(ttl=600)
def _domains():
"""Return a list of the current .gov domains.
Fetch a file from DOMAIN_FILE_URL, parse the CSV for the domain,
lowercase everything and return the list.
"""
DraftDomain = apps.get_model("registrar.DraftDomain")
# 5 second timeout
file_contents = requests.get(DOMAIN_FILE_URL, timeout=5).text
domains = set()
# skip the first line
for line in file_contents.splitlines()[1:]:
# get the domain before the first comma
domain = line.split(",", 1)[0]
# sanity-check the string we got from the file here
if DraftDomain.string_could_be_domain(domain):
# lowercase everything when we put it in domains
domains.add(domain.lower())
return domains


def check_domain_available(domain):
"""Return true if the given domain is available.
Expand Down Expand Up @@ -99,6 +75,22 @@ def available(request, domain=""):
return json_response


@require_http_methods(["GET"])
@login_not_required
# Since we cache domain RDAP data, cache time may need to be re-evaluated this if we encounter any memory issues
@ttl_cache(ttl=600)
def rdap(request, domain=""):
"""Returns JSON dictionary of a domain's RDAP data from Cloudflare API"""
domain = request.GET.get("domain", "")

# If inputted domain doesn't have a TLD, append .gov to it
if "." not in domain:
domain = f"{domain}.gov"

rdap_data = requests.get(RDAP_URL.format(domain=domain), timeout=5).json()
return JsonResponse(rdap_data)


@require_http_methods(["GET"])
@login_not_required
def get_current_full(request, file_name="current-full.csv"):
Expand Down
Loading

0 comments on commit 2f8408e

Please sign in to comment.