First call failiure using transform but then it works? - unsure if it's a bug or something else

[4andy]

Hi I am seeing the below issue where the first call to transform fails but after that it seems to return the expected values. Does someone know what is going on here and how I can avoid it? Thanks!

from pyproj import Transformer
import numpy as np

t1 =Transformer.from_crs("EPSG:4326", "ESRI:102008", always_xy=True)
coords = np.array([[-90.166259, 32.546644]])
t1.transform(coords[:, 0], coords[:, 1])  # fails
t1.transform(coords[:, 0], coords[:, 1])  # works?

Python 3.12.4 | packaged by conda-forge | (main, Jun 17 2024, 10:04:44) [MSC v.1940 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyproj import Transformer
>>> import numpy as np
>>> t1 =Transformer.from_crs("EPSG:4326", "ESRI:102008", always_xy=True)
>>> coords = np.array([[-90.166259, 32.546644]])
>>> t1.transform(coords[:, 0], coords[:, 1])  # fails
(array([inf]), array([inf]))
>>> t1.transform(coords[:, 0], coords[:, 1])  # works?
(array([522352.71447431]), array([-857684.1764726]))
>>>

[snowman2]

Here are some suggestions for debugging:

• https://pyproj4.github.io/pyproj/stable/advanced_examples.html#debugging-internal-proj
• Set errcheck=True in transform

[4andy]

Here's the logging.

[...]
>>> coords = np.array([[-90.166259, 32.546644]])
>>> t1.transform(coords[:, 0], coords[:, 1], errcheck=True)
DEBUG:PROJ_ERROR: Cannot open https://cdn.proj.org/us_noaa_mshpgn.tif: schannel: next InitializeSecurityContext failed: CRYPT_E_NO_REVOCATION_CHECK (0x80092012) - The revocation function was unable to check revocation for the certificate.
DEBUG:PROJ_ERROR: Cannot open https://cdn.proj.org/us_noaa_mshpgn.tif: schannel: next InitializeSecurityContext failed: CRYPT_E_NO_REVOCATION_CHECK (0x80092012) - The revocation function was unable to check revocation for the certificate.    
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\miniforge3\envs\hazardreports\Lib\site-packages\pyproj\transformer.py", line 843, in transform
    self._transformer._transform(
  File "pyproj\\_transformer.pyx", line 726, in pyproj._transformer._Transformer._transform
pyproj.exceptions.ProjError: transform error: Network error when accessing a remote resource: (Internal Proj Error: Cannot open https://cdn.proj.org/us_noaa_mshpgn.tif: schannel: next InitializeSecurityContext failed: CRYPT_E_NO_REVOCATION_CHECK (0x80092012) - The revocation function was unable to check revocation for the certificate.)
>>> t1.transform(coords[:, 0], coords[:, 1], errcheck=True)
(array([522352.71447431]), array([-857684.1764726]))
>>>

It looks like it fails to reach some external file and fails the first time. But works fine the second.

[snowman2]

Pre-downloading the grids may help: https://pyproj4.github.io/pyproj/stable/transformation_grids.html

[4andy]

I can confirm that pre-downloading the grids resolves the issue. However, I have also seen the issue in multiple systems and environments. As the issue happens only on the first use and fails blindly it does seem like a bug @snowman2. It would be great if it at least failed with an error. It also cascades through other packages such as shapely.

[snowman2]

It would be great if it at least failed with an error.

I agree. However, that one is tricky. There are a lot of errors that some people would like to ignore. That is why the errcheck option exists. It may be worthwhile to check for network errors and always raise those errors.

[snowman2]

Related #705

[4andy]

Interesting. Would it make sense for errorcheck to default to True? Especially as interaction with it is often through wrapping packages such as shapely?

[jjimenezshaw]

Does it fail using PROJ command line cs2cs, with the environment variable PROJ_NETWORK=ON and PROJ_DEBUG=3?

[4andy]

Using a container without internet access and PROJ 9.1.0, I tried with PROJ_NETWORK ON and OFF.

cs2cs EPSG:4326 ESRI:102008 <<EOF
32.546644 -90.166259
EOF

With network ON the logging runs over in the terminal and hangs forever at this point.

[...]
pipeline:     ellps=GRS80
aea: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
aea: pj_ellipsoid - final:    ellps=GRS80
pipeline: Pipeline: Step 3 (proj=aea) at 0x55bcae54cd60
pipeline: Pipeline at [0x55bcae54a9d0]:    step at [0x55bcae54cd60] (proj=aea) done
pipeline: Pipeline: 4 steps built. Determining i/o characteristics
Using coordinate operation Inverse of NAD83 to WGS 84 (23) + North_America_Albers_Equal_Area_Conic
pj_open_lib(us_noaa_lahpgn.tif): call fopen(/mnt/imported/data/geospatial-conda-envs/convectivestorms/share/proj/us_noaa_lahpgn.tif) - failed
pj_open_lib(lahpgn.gsb): call fopen(/mnt/imported/data/geospatial-conda-envs/convectivestorms/share/proj/lahpgn.gsb) - failed

With network OFF it works without issue.

[...]
aea: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
aea: pj_ellipsoid - final:    ellps=GRS80
pipeline: Pipeline: Step 7 (proj=aea) at 0x55f23e0ba310
pipeline: Pipeline at [0x55f23e0b67f0]:    step at [0x55f23e0ba310] (proj=aea) done
pipeline: Pipeline: 8 steps built. Determining i/o characteristics
Using coordinate operation Inverse of NAD83 to WGS 84 (1) + North_America_Albers_Equal_Area_Conic
522352.71       -857684.18 0.00

I also ran the same thing in the same container but with the environment where I downloaded the grid files and PROJ 9.4.1. It works fine with network ON as the grids are there. Slightly different coordinates returned. I assume the transformation is more precise.

[...]
aea: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
aea: pj_ellipsoid - final:    ellps=GRS80
pipeline: Pipeline: Step 3 (proj=aea) at 0x563cf19bf300
pipeline: Pipeline at [0x563cf19bca90]:    step at [0x563cf19bf300] (proj=aea) done
pipeline: Pipeline: 4 steps built. Determining i/o characteristics
Using coordinate operation Inverse of NAD83 to WGS 84 (27) + North_America_Albers_Equal_Area_Conic
pj_open_lib(us_noaa_mshpgn.tif): call fopen(/mnt/imported/data/geospatial-conda-envs/cafairplan/share/proj/us_noaa_mshpgn.tif) - failed
pj_open_lib(mshpgn.gsb): call fopen(/mnt/imported/data/geospatial-conda-envs/cafairplan/share/proj/mshpgn.gsb) - succeeded
NTv2 ms       25x29: LL=(-92,29) UR=(-86,36)
522352.78       -857683.70 0.00

[jjimenezshaw]

As mentioned in #705 "I believe the problem may be caused by some incorrect CA on my machine " , it seems to be a network issue. You should see something similar using curl to open a file from PROJ-data like https://cdn.proj.org/us_noaa_mshpgn.tif. In that case it is more a local configuration problem than a bug in pyproj.

[4andy]

I'm often working in offline environments. So failing to get the grids from the internet with any number of errors is expected. How pyproj handles that environment is something to consider however as I am sure there are others in similar environments.

I'm just guessing here but it seems like pyproj/PROJ transform looks for the online grids and there is an error (request hangs or is otherwise blocked) and pyproj blindly fails on that transform. The second time around, pyproj tries without the network support and provides a transformation. It would be great if that catch/switch would happen within the first transform call - ideally throwing a warning letting users know that the network grid support failed and the transformation may be less precise than with the grids.

[snowman2]

It would be great if that catch/switch would happen within the first transform call

Yes, ideally this would happen in PROJ and not pyproj.