Correct the Search URL setting (#525)

* Incrementing version number for next set of minor updates.

* Pin dependencies (#461)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* Update dependency @types/jest to v28.1.8 (#462)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* Support backend settings in config (#482)

* enable settings on the backend for urls

* add the settings for workflow

* add missing import

* add newline

* Update react monorepo to v18 (#478)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* Pin dependencies (#477)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* Fixed tiny flake8 complaint

* hotfix to the last update so that local containers have the required environment variables. Otherwise local build of backend will fail.

* Update .django

* Updating a few dependencies in the backend

* Updating the Flake8 pre-commit config file to see if that resolves pre-commit issues...

* Fixed typo in .coveragerc which caused migrations to be included in tests. Removed unneccessary type checking in the import headers of the test models. Updated coverage for some files that didn't have 100% coverage. Updated some django dependencies after testing to make sure nothing broke. Updated some deprecated imports to remove warnings. Updated pytest and coverage libraries.

* Upgraded to Django 4.1.7, along with some other python packages.

* Upgraded more python packages while testing to make sure nothing breaks.

* Reverted dj-rest-auth as it caused issues with tests failing (although the application from users perspective was working fine)

* Updated dj-rest-auth to latest (before breaking changes)

* Updating front-end to use the latest react-router-dom major version 6

* Removed some comments and updated some tests. Tests still failing, need to troubleshoot.

* Updated tests to use memory router to account for the useNavigate changes that came with react-router-dom v6. Tests are passing now.

* Updated some more frontend packages, however a few updates are breaking and will cause errors, so ignored those.

* Updated yarn.lock file to see if tests pass.

* Feature/500 info notifications (#520)

* Refactored the joyrideTutorials in order to reduce the chances for errors with misspelling target names. Created unique target calssnames to simplify the process of adding and using targets. Removed the chance of duplicate target names causing errors etc. Added a test startup component that should show when first opening metagrid. Startup window still needs some work to improve visuals.

* Added a welcome and changelog template component to use with thh startup display. Made the startup display change based on existing version so that a new changelog will appear when version is changed. Modified the startup component behavior to hadle a welcome and changelog template based on whether its first-time startup or not.

* Updated django-all-auth to 0.54.0

* Upgraded the antd library to latest version. Updated components to work with the latests antd library and added some styling to correct some undesired changes. Added a temporary drawer component which can be opened by clicking a new message button in the top-right navbar. Fixed some styling issues with window resizing to make the top navbar work better with smaller windows and improve visual behavior. Removed redundant components that are no longer needed with the new antd library (which is now working with typescript well). Created a popup window that shows at the start for new users. Created a startup template system that allows different templates to be used for displaying popup messages. More fixed needed for tests and the welcome popup tutorials need work.

* Worked to correct issues with tests hanging by reverting back the antd library. Also updated to node version 18. Restored previously deleted files and will remove them later when all tests are passing. Still need to correct tests to pass. Still need to complete messaging feature and update tutorial.

* Updated tests so that they pass, coverage will still be needed.

* Updated the Startup template functionality and added actions that can be triggered by the templates. Updated the welcome message buttons to work properly and select between pages to start tutorials.

* Removed redundant and unneccessary components as they are now directly taken from antd library. Removed commented out import statements.

* Refactored some names and added messages for the right drawer. Added ability to create messages for the message bar and provide content from markdown files. Updated the changelog to have useful information on latest changes. Need to troubleshoot tests and complete coverage.

* Added markdown file support for the messages. Did refactoring so messages on the right drawer and popups will show the markdown. Created new markdown files with version info and example message. Added flexibility to popup so that styles can be modified on a per message basis. Modified the changelog template and types.

* Created a new welcome tour which will show users the help buttons to encourage them to use them if they have issues. Updated the navbar tour to include the new news button. Created new test files and updated tests to bring back 100% text coverage. Removed obsolete tour target related files, as they've been updated with target object class. Troubleshooted and ran tests to fix some bugs and added more functionality for jest tests. Deleted test markdown files and updated message markdown slightly.

* modify the messages for release (#522)

---------

Co-authored-by: Sasha Ames <sashakames@users.noreply.github.com>

* Feature/500 info notifications (#523)

* Refactored the joyrideTutorials in order to reduce the chances for errors with misspelling target names. Created unique target calssnames to simplify the process of adding and using targets. Removed the chance of duplicate target names causing errors etc. Added a test startup component that should show when first opening metagrid. Startup window still needs some work to improve visuals.

* Added a welcome and changelog template component to use with thh startup display. Made the startup display change based on existing version so that a new changelog will appear when version is changed. Modified the startup component behavior to hadle a welcome and changelog template based on whether its first-time startup or not.

* Updated django-all-auth to 0.54.0

* Upgraded the antd library to latest version. Updated components to work with the latests antd library and added some styling to correct some undesired changes. Added a temporary drawer component which can be opened by clicking a new message button in the top-right navbar. Fixed some styling issues with window resizing to make the top navbar work better with smaller windows and improve visual behavior. Removed redundant components that are no longer needed with the new antd library (which is now working with typescript well). Created a popup window that shows at the start for new users. Created a startup template system that allows different templates to be used for displaying popup messages. More fixed needed for tests and the welcome popup tutorials need work.

* Worked to correct issues with tests hanging by reverting back the antd library. Also updated to node version 18. Restored previously deleted files and will remove them later when all tests are passing. Still need to correct tests to pass. Still need to complete messaging feature and update tutorial.

* Updated tests so that they pass, coverage will still be needed.

* Updated the Startup template functionality and added actions that can be triggered by the templates. Updated the welcome message buttons to work properly and select between pages to start tutorials.

* Removed redundant and unneccessary components as they are now directly taken from antd library. Removed commented out import statements.

* Refactored some names and added messages for the right drawer. Added ability to create messages for the message bar and provide content from markdown files. Updated the changelog to have useful information on latest changes. Need to troubleshoot tests and complete coverage.

* Added markdown file support for the messages. Did refactoring so messages on the right drawer and popups will show the markdown. Created new markdown files with version info and example message. Added flexibility to popup so that styles can be modified on a per message basis. Modified the changelog template and types.

* Created a new welcome tour which will show users the help buttons to encourage them to use them if they have issues. Updated the navbar tour to include the new news button. Created new test files and updated tests to bring back 100% text coverage. Removed obsolete tour target related files, as they've been updated with target object class. Troubleshooted and ran tests to fix some bugs and added more functionality for jest tests. Deleted test markdown files and updated message markdown slightly.

* modify the messages for release (#522)

* Updated the node status information alert so that text displays if there is an api error.

* Updated Django to 4.1.9

* Fixed a test regarding the changes to the error message.

---------

Co-authored-by: Sasha Ames <sashakames@users.noreply.github.com>

* Added mip_era and data_node facets to the input4mips project.

* Updated the general facet order for input 4 mips

* update naming, config template

* fix settings, update local config

* Update backend.yml - fix setting for CI to use correct name

* Delete test_message.md

* Added some fixes to the tests to make it passing.

* Removed unecessary file.

---------

Co-authored-by: Carlos Downie <42552189+downiec@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

.github/workflows/backend.yml

@@ -17,10 +17,9 @@ env:
   KEYCLOAK_REALM: metagrid
   KEYCLOAK_CLIENT_ID: backend
   DATABASE_URL: pgsql://postgres:postgres@localhost:5432/postgres
-  REACT_APP_ESGF_NODE_URL: https://esgf-node.llnl.gov/esg-search/search
-  REACT_APP_WGET_API_URL: https://greyworm1-rh7.llnl.gov/wget
-  REACT_APP_ESGF_NODE_STATUS_URL: https://aims4.llnl.gov/prometheus/api/v1/query?query=probe_success%7Bjob%3D%22http_2xx%22%2C+target%3D~%22.%2Athredds.%2A%22%7D
-
+  REACT_APP_SEARCH_URL: https://esgf-node.llnl.gov/esg-search/search
+  REACT_APP_WGET_API_URL: https://esgf-node.llnl.gov/esg-search/wget
+  REACT_APP_ESGF_NODE_STATUS_URL: https://aims2.llnl.gov/metagrid-backend/proxy/status
 
 jobs:
   build:

backend/.envs/.django

@@ -15,11 +15,11 @@ KEYCLOAK_CLIENT_ID=backend
 
 # ESGF wget API
 # https://github.com/ESGF/esgf-wget
-REACT_APP_WGET_API_URL=https://nimbus3.llnl.gov/wget
+REACT_APP_WGET_API_URL=https://esgf-node.llnl.gov/esg-search/wget
 
 # ESGF Search API
 # https://esgf.github.io/esg-search/ESGF_Search_RESTful_API.html
-REACT_APP_ESGF_NODE_URL=https://esgf-node.llnl.gov/esg-search/search
+REACT_APP_SEARCH_URL=https://esgf-node.llnl.gov/esg-search/search
 
 # ESGF Node Status API
 # https://github.com/ESGF/esgf-utils/blob/master/node_status/query_prom.py

backend/config/settings/base.py

@@ -298,6 +298,6 @@
 CORS_ORIGIN_WHITELIST = env.list("CORS_ORIGIN_WHITELIST")
 
 
-SEARCH_URL = env("REACT_APP_ESGF_NODE_URL")
+SEARCH_URL = env("REACT_APP_SEARCH_URL")
 WGET_URL = env("REACT_APP_WGET_API_URL")
 STATUS_URL = env("REACT_APP_ESGF_NODE_STATUS_URL")

backend/metagrid/api_proxy/tests/test_views.py

@@ -9,7 +9,7 @@ def test_wget(self):
         response = self.client.get(
             url,
             {
-                "dataset_id": " CMIP6.CMIP.IPSL.IPSL-CM6A-LR.abrupt-4xCO2.r12i1p1f1.Amon.n2oglobal.gr.v20191003|esgf-data1.llnl.gov"
+                "dataset_id": "CMIP6.CMIP.IPSL.IPSL-CM6A-LR.abrupt-4xCO2.r12i1p1f1.Amon.n2oglobal.gr.v20191003|esgf-data1.llnl.gov"
             },
         )
         assert response.status_code == status.HTTP_200_OK
@@ -23,7 +23,7 @@ def test_search(self):
     def test_status(self):
         url = reverse("do-status")
         response = self.client.get(url)
-        assert response.status_code == status.HTTP_400_BAD_REQUEST
+        assert response.status_code == status.HTTP_200_OK
 
     def test_citation(self):
         url = reverse("do-citation")

backend/metagrid/api_proxy/views.py

@@ -3,7 +3,11 @@
 
 import requests
 from django.conf import settings
-from django.http import HttpResponse, HttpResponseBadRequest
+from django.http import (
+    HttpResponse,
+    HttpResponseBadRequest,
+    HttpResponseServerError,
+)
 from django.views.decorators.csrf import csrf_exempt
 from django.views.decorators.http import require_http_methods
 
@@ -13,8 +17,8 @@
 def do_search(request):
     esgf_host = getattr(
         settings,
-        "REACT_APP_SEARCH_URL",
-        "https://esgf-node.llnl.gov/esg-search/search",
+        "SEARCH_URL",
+        "",
     )
     return do_request(request, esgf_host)
 
@@ -56,7 +60,7 @@ def do_citation(request):
 def do_status(request):
     status_url = getattr(
         settings,
-        "REACT_APP_ESGF_NODE_STATUS_URL",
+        "STATUS_URL",
         "https://aims4.llnl.gov/prometheus/api/v1/query?query=probe_success%7Bjob%3D%22http_2xx%22%2C+target%3D~%22.%2Athredds.%2A%22%7D",
     )
     resp = requests.get(status_url)
@@ -73,15 +77,22 @@ def do_wget(request):
         request,
         getattr(
             settings,
-            "REACT_APP_WGET_API_URL",
-            "https://esgf-node.llnl.gov/esg-search/wget",
+            "WGET_URL",
+            "",
         ),
     )
 
 
 def do_request(request, urlbase):
     resp = None
 
+    if len(urlbase) < 1:  # pragma: no cover
+        print(
+            "ERROR:  urlbase string empty, ensure you have the settings loaded"
+        )
+        return HttpResponseServerError(
+            "ERROR: missing url configuration for request"
+        )
     if request:
         if request.method == "POST":
             url_params = request.POST.copy()

frontend/src/api/routes.ts

@@ -1,4 +1,4 @@
-import { esgfNodeURL, metagridApiURL } from '../env';
+import { esgfSearchURL, metagridApiURL } from '../env';
 
 export type HTTPCodeType = 400 | 401 | 403 | 404 | 405 | 'generic';
 
@@ -47,7 +47,7 @@ type ApiRoutes = {
  * served as a clickable link within the browser.
  */
 export const clickableRoute = (route: string): string =>
-  route.replace(`${metagridApiURL}/proxy/search`, `${esgfNodeURL}`);
+  route.replace(`${metagridApiURL}/proxy/search`, `${esgfSearchURL}`);
 
 // Any path with parameters (e.g. '/:datasetID/') must be in camelCase
 // https://mswjs.io/docs/basics/path-matching#path-with-parameters

frontend/src/env.ts

@@ -23,8 +23,7 @@ export const wgetApiURL = process.env.REACT_APP_WGET_API_URL as string;
 // ESGF Search API
 // ------------------------------------------------------------------------------
 // https://esgf.github.io/esg-search/ESGF_Search_RESTful_API.html
-export const esgfNodeURL = `${process.env.REACT_APP_ESGF_NODE_URL as string}`;
-export const esgfNodeURLNoProtocol = esgfNodeURL.split('//')[1];
+export const esgfSearchURL = `${process.env.REACT_APP_SEARCH_URL as string}`;
 
 // ESGF Node Status API
 // ------------------------------------------------------------------------------

metagrid_configs/metagrid_config

@@ -41,15 +41,15 @@ REACT_APP_METAGRID_API_URL=
 
 # ESGF wget API
 # https://github.com/ESGF/esgf-wget
-REACT_APP_WGET_API_URL=https://pcmdi8vm.llnl.gov/wget
+REACT_APP_WGET_API_URL=
 
 # ESGF Search API
 # https://esgf.github.io/esg-search/ESGF_Search_RESTful_API.html
-REACT_APP_ESGF_NODE_URL=https://esgf-node.llnl.gov/esg-search/search
+REACT_APP_SEARCH_URL=
 
 # ESGF Node Status API
 # https://github.com/ESGF/esgf-utils/blob/master/node_status/query_prom.py
-REACT_APP_ESGF_NODE_STATUS_URL=https://aims4.llnl.gov/prometheus/api/v1/query?query=probe_success%7Bjob%3D%22http_2xx%22%2C+target%3D~%22.%2Athredds.%2A%22%7D
+REACT_APP_ESGF_NODE_STATUS_URL=
 
 # Keycloak
 # https://github.com/keycloak/keycloak