-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Largefire projection shifted in V3 outputs #61
Comments
Hmm, so @ychenzgithub and I had a similar issue a while back and it was related to a faulty projected coordinate system, which we then fixed. I specifically remember vetting the fix and being satisfied with the resolution, which led to the data being recreated, hence the "REDO" in the path: There is always the chance I missed something, but this is worth checking out again, especially because I'm nearly done with a draft of a paper that uses the data. One of the best ways I verified this was to overlay input VIIRS data with the @zebbecker I'm going to whip up a notebook for this real quick and I'll invite you to add in the v3 data once I have the v2 data displayed. |
Sorry, I reread the above comments again:
Seems like the pixel data are OK then? If it's related to Issue 32, I'm wondering if the false easting/northing are being incorrectly inferred when interpreting a crs. |
@zebbecker so check out the example at this path: I overlaid the VIIRS pixels (red, top image) with the FEDS pixel data (black, bottom image): Aside from a few extra pixels in the FEDS data (I didn't load in all the VIIRS data), there appears to be a close overlap because all the red VIIRS pixels are sitting right on top of the FEDS data (shown in the top image). This leads me to think that it's definitely a v3 issue, as I think you've already been able to confirm yourself. |
Thanks for the VIIRS example @eorland! I used your notebook to compare VIIRS pixels against pixels from both allpixels and Largefire/nfplist.fgb files generated by v3, and neither has this projection shift issue. This is interesting because the nfplist.fgb outputs are generated with |
Yep, I just took a look at those functions. It seems that |
OK yep, this is pretty similar to what @ychenzgithub and I were working on. In short, the CRS we used was somehow both deprecated and being misinterpreted as the correct one, so the crs info looked identical, but it actually had a different northing/easting, which is what it looks like is going on here. Are you able to pull up where the CRS info in v3 is being set? I'd be curious to see what it is saying. |
@eorland here you go! It is a bit ugly, but I went and pulled out the CRS for a largefire perimeter immediately after it was set in postprocess.py. That is the first JSON object below. The second is the CRS for allfires, pulled out at the same point. They look to be the same at this point, and the built in equals methods agrees. The only thing that happens after this point is the data gdf being saved to fgb format. I'm worried something might be going on with the encoding/decoding of the CRS during the file write or read. Example:
|
The issue may actually be related to how we read the fgb files in with geopandas. [Edit to clarify: how we read the fgb files when writing new code to do things with FEDS output.] For context, geopandas uses either Fiona or Pyogrio as its read/write engine. Until geopandas v1.0.0, Fiona was used by default any time we call Fiona does not read the CRS in correctly from our fgb files; Pyogrio does. I tested using geopandas v0.14.2 and v1.0.0, and it does not matter which geopandas version we use or which engine we specify for writing the fgbs in v3. It just matters which engine is used to load the files. As long as we read the fgb using Pyogrio, the resulting GeoDataFrame will have the correct CRS, and therefore will align with v2 perimeters. So, going forward, anyone using geopandas >= 1.0.0 will not encounter this projection issue. Anyone still using an earlier version can read in fgb files like so: |
Amazing work @zebbecker ✨ Feel free to put in a PR off |
@ranchodeluxe a quick chat tomorrow would be great! Because the issue occurs when reading our fgb outputs, I don't think we can address it in our codebase directly. As far as I can tell, the files are being written correctly. The workaround above is something that users would need in any code they write that tries to load our products, until they upgrade to geopandas>=1.0.0. So, would like your insight on how to move forward, since its definitely an issue that will pop up for people but might not be something we can fix with a PR in this project. |
What I was after is that maybe we should upgrade to benefit all places we read from (our notebooks mostly). I see we don't use a pinned version of Though however now I'm wondering if we're gonna see bugs b/c of any breaking changes 🤔 Maybe we should read about the "Backward Incompatible Changes" https://github.com/geopandas/geopandas/releases |
@jsignell and I chatted. Safe thing to do here is pin geopandas<1.0.0 and manually remember to set the |
Going forward, recent outputs generated with either v2 or v3 can be read correctly using the Pyogrio engine, either by default when using geopandas>=1.0.0, or by specifying However, I have encountered at least one historical v2 run where the fgb outputs are projected incorrectly when doing this. The issue was resolved by using Fiona to read in the files. I suspect that this has something to do with the respective versions of Fiona used to generate the original files vs read them in now, but have no way of knowing which version was used in the older run. Essentially, this shouldn't be a problem going forward, but anyone who runs into trouble reading older fgbs with geopandas should try using both Pyogrio and Fiona as a troubleshooting step. Even if the CRS is read incorrectly (leading to the projection being shifted) no warnings are generated, so you will need to map your results and check manually which is correct. |
@ranchodeluxe @eorland
When loading fire perimeters generated with V3, it looks like they are offset from V2 perimeters. The V3 Largefire perimeters are the ones that are wrong, between the two- they can be seen going over water in places where the V2 perimeters correctly follow the shoreline.
This only occurs with
Largefire
andCombinedLargefire
fgb files generated with V3.allfires
andallpixels
data are projected correctly.May be related to Issue 32
Example w/ Creek fire:
V3 Largefire perimeters shifted over water
The text was updated successfully, but these errors were encountered: