[WIP] add Rasters.jl benchmarks #18
Conversation
avoid the `-` splitting
This allows exactextractr to be installed in a separate environment, so it doesn't provoke a version conflict with sf and stars via gdal. This also segments each language and creates dependency trees so life is a bit easier.
conforming with all the other benchmarks
|
Edited the first post now that the PR is functionally complete from the Rasters.jl end. |
| Rasters.checkmem!(false) # Avoid memory check bug on some machines | ||
| raster = Raster(RasterStack(raster_files; name=band_names)) | ||
| # Downsample from 30m to 90m (1/3 of the original resolution) | ||
| benchmark = @be Rasters.resample($raster; res=90) seconds=60 |
There was a problem hiding this comment.
would it be possible to use Rasters.aggregate here instead? That would work if we are resampling to exactly 1/3 of the resolution
There was a problem hiding this comment.
Yes, I think this was just copied from python code and we left it as resample. But for the actual problem I'm not sure why they would use resample when you can just aggregate X/Y by 3.
|
@asinghvi17, let's wait with this, because it uses a different dataset. Separate issue is that depending on the dataset, {fasterize} can be significantly slower than GDAL. I have another question -- In task "zonal" are exact zone statistics like in {exactextractr} used or approximate like in the other packages? |
|
Its approximate like the other packages. Now I've seen exactextractr is exact I'll add that option in a few months! But I think it will be even faster after that in most cases because it would be best with switching to an online stats approach (at least where that is possible for sum/prod/mean etc). Currently So for #15 it would be really nice to have (It would also be good for Another thing is there are things Rasters.jl and gdal can do that fasterize can't do, and actually a lot Rasters can do that gdal can't do either - like arbitrary functions (even functions like median that need to sort) and custom objects/number types. it would be good to cover some of those - at least things gdal can do that fasterize cant) |
| Rasters.checkmem!(false) # avoid a memory size bug on some machines | ||
| ras = Raster(RasterStack(raster_files; name=band_names)) | ||
| stack_file = joinpath(data_dir, "stack.tif") | ||
| benchmark = @be write($stack_file, $ras; force=true) seconds=60 |
There was a problem hiding this comment.
Rasters compression default is "ZSTD" but the others are all using "LZW".
We don't have a compress keyword yet to handle this so it uses the ugly
options=Dict("COMPRESS"=>"ZSTD"). Maybe we should change that
|
FWIW: Ideally, it would be useful to compare the performance of the current GDAL algorithm and scan line algorithm at the C++ level, since what I presented #15 is quite limited. If the latter algorithm turned out to be significantly better, it would be worth implementing it directly in GDAL so that all packages could benefit from it. And as you mentioned, {fasterize} has some limitations, e.g. it only works with {raster} objects and polygons, and fewer options compared to GDAL. Here is related issue in the GDAL repository: OSGeo/gdal#7200 CC: @mdsumner because is also involved in this topic. |
|
I was working on fasterize today ... Btw, see this related effort here, and discussion in six hours from now: https://github.com/developmentseed/warp-resample-profiling |
|
Well, Rasters wont benefit from faster gdal rasterize as we only use gdal for i/o and for gdalwarp. But it would be good to have detailed comparisons of these algorithms. I put a bunch of work into optimising the scanline in Rasters but there will be places it will be slower than fasterize - a few nice diagrams of the performance space would really help understand where things are at. |
|
I almost forgot about this, but for exact zonal statistics we have |
|
I imagined we can do that in the line burning phase and get coverage for each pixel the line touches instead of burning. But We also have subsampling coverage in Rasters |
This PR adds benchmarks for Rasters.jl, and adds a
pixienvironment file to manage the installations of R, Python, Julia, and packages.I added the pixi file because there's a version conflict if you try to install exactextractr and stars in the same environment, they require different versions of gdal. So now each package has a separate environment.