NO2, SPM and RPM don't load

[olafveerman]

The NO2, SPM and RPM products are not loading in production.

May this be related to the raster-opacity fix we did earlier today?

Frontend breaks with:

Error: Failed to fetch

API call results in an Internal Server Error:

https://8ib71h0627.execute-api.us-east-1.amazonaws.com/v1/4/7/9@1x?url=s3://covid-eo-data/OMNO2d_HRMBaseline/OMI_trno2_0.10x0.10_Baseline_201507_Col3_V4.nc.tif&esampling_method=bilinear&bidx=1&rescale=0%2C1.5e16&color_map=custom_no2&color_formula=gamma%20r%201.05

[leothomas]

NO2

The issue is coming from the tiles under the compare section of the dataset. The front end end is requesting a URL with a {date} string in the filename, in the format YYYYMM (eg: 201507 in the URL you posted), whereas the comparison tiles stored under s3://covid-eo-data/OMNO2d_HRMBaseline just have a number 01 through 12 in their filename.

Am I correct in saying that the idea behind the "Compare" tiles is to compare a given N02 tile with the 5 year average for that month (ie: 202001 should be compared with the average(201901, 201801, 201701, 201601, 201501)which is stored under a single tile with just 01 in its filename?

If that's the case we need to figure out a way for the frontend to be able to request something like 07 instead of 201507 when requesting a "Compare" tile.

Taking the url you posted, replacing 201507 with just 07 loads correctly: https://8ib71h0627.execute-api.us-east-1.amazonaws.com/v1/4/7/9@1x?url=s3://covid-eo-data/OMNO2d_HRMBaseline/OMI_trno2_0.10x0.10_Baseline_07_Col3_V4.nc.tif&esampling_method=bilinear&bidx=1&rescale=0%2C1.5e16&color_map=custom_no2&color_formula=gamma%20r%201.05

SPM

I'm seeing errors in the console when loading SPM, but the actual dataset seems to be loading:

@olafveerman is the above image correct? Or should there be more data loaded into the view?

My suspicion for the error messages in the console is that if the zoom level is high enough it will query tiles that are within the view of the map, but not within the tile itself, which causes a (harmless) error message in the console. @drewbo is this correct? If so, is there a way to squash these errors?

RPM

This one is weird! When I load the production dashboard and request RPM over SF
Request (url: https://8ib71h0627.execute-api.us-east-1.amazonaws.com/v1/10/163/396@1x?url=s3://covid-eo-data/rpm/rpm-sf.cog.tif&resampling_method=bilinear&bidx=1%2C2%2C3%24):

The "Preview" tab of the console shows data:

(You can see the data more clearly if you just follow the requested link directly).

However the "Response" tab shows:

I'm not 100% sure about what this might be. Perhaps an incorrect mimetype?

[drewbo]

Yes (to plenty of error messages on missing tiles, should be harmless). There should be a TileOutsideBounds error on latest rio-tiler we can catch if we upgrade (note we need to version pin carefully because of many breaking changes) cc: @vincentsarago

[vincentsarago]

@drewbo if you are not using mosaic, TileOutsideBounds should be raised by default for old rio-tiler version but thoses needs to be caught by fastapi (I don't think you need to change rio-tiler version)

from typing import Callable, Dict, Type

from rio_tiler.errors import TileOutsideBounds
from starlette import status
from starlette.requests import Request
from starlette.responses import JSONResponse
from fastapi import FastAPI

DEFAULT_STATUS_CODES = {
    TileOutsideBounds: status.HTTP_404_NOT_FOUND,
    Exception: status.HTTP_500_INTERNAL_SERVER_ERROR,
}

def exception_handler_factory(status_code: int) -> Callable:
    """
    Create a FastAPI exception handler from a status code.
    """

    def handler(request: Request, exc: Exception):
        logger.error(exc, exc_info=True)
        return JSONResponse(content={"detail": str(exc)}, status_code=status_code)

    return handler

def add_exception_handlers(
    app: FastAPI, status_codes: Dict[Type[Exception], int]
) -> None:
    """
    Add exception handlers to the FastAPI app.
    """
    for (exc, code) in status_codes.items():
        app.add_exception_handler(exc, exception_handler_factory(code))


add_exception_handlers(app, DEFAULT_STATUS_CODES)

ref:

• https://github.com/developmentseed/titiler/blob/master/titiler/main.py#L33
• https://github.com/developmentseed/titiler/blob/master/titiler/errors.py
• https://github.com/cogeotiff/rio-tiler/blob/4e0c7a2dceda7080c38c4ffa023324ac5f013235/rio_tiler/reader.py#L592

[olafveerman]

Cool, thanks @leothomas .

NO2

Split the NO2 compare bug into a separate issue ^.

SPM

Regarding SPM. I should have been more clear. This is the Slowdown Proxy Map, not turbidity (confusingly, both products have the same acronym: Suspended Particulate Matter)

[leothomas]

This is the Slowdown Proxy Map, not turbidity (confusingly, both products have the same acronym: Suspended Particulate Matter)

Ahh! Should have caught that. I'll take a quick look. I suspect it will be the same issue as with recovery

[leothomas]

Confirming it's the same issue as with recovery.

Looking it the recovery tif for SF:

The data seems quite sparse (and light in colour). In the image above you can barely even see the data over SF itself. The top right blob (Oakland/Richmond area) is more visible.

Is it possible that the data is being loaded correctly, but not very visibly against the darker/more contrasted background image?

[leothomas]

The slowdown tifs for la (slowdown_proxy_map/la.tif) is basically empty, however there are some other files under slowdown_proxy_map/ that are not empty. eg: slowdown_proxy_map/LosAngeles_A2_SPM_10m_20200101-20200228_20200401-20200430_th-0.35.cog.tif which looks like:

@olafveerman Do you know what those other files correspond to? It looks like the filename of the one I posted contains 4 days, each spaced by 30 days)

[leothomas]

Still pushing on the RPM / SPM not showing up in the front end. Tiles load from the API and don't raise errors in the "Network Tab" of the developer's console. eg: rpm example, spm example, yet they don't show up in the map view. I'm inclined to say that the front end may be mis-handling these files. @danielfdsilva @ricardoduplos any thoughts?

[olafveerman]

@leothomas Got it. The API response for NO2, SPM and RPM contains:

      "paint": {
        "raster-opacity": 0
      }

Which explains why they don't error, but also don't show up.

[olafveerman]

This was fixed in the latest release of the API.