JP2kak is not lossless when there is more than one band

[IdanAviv89]

What is the bug?

When converting an image with multiple bands using the jp2kak driver, with the lossless options, the band checksum after the conversion is different. While when there is only one band the checksum is the same before and after the conversion.

Steps to reproduce the issue

Taking an image with multiple bands (for example rgb) and using the quality= 100 option.

Versions and provenance

ubuntu 20, gdal 3.6.1 (I look at newer version and it doesn't seems there is a fix there).

Additional context

No response

[rouault]

@IdanAviv89 Please provide an image to reproduce, as I don't reproduce the issue with the autotest/gdrivers/data/small_world.tif of our test suite and Kakadu 8.4.1 and GDAL master

$ gdal_translate autotest/gdrivers/data/small_world.tif out.jp2 -of jp2kak -co quality=100
$ gdalinfo -checksum autotest/gdrivers/data/small_world.tif | grep Checksum
  Checksum=30111
  Checksum=32302
  Checksum=40026
$ gdalinfo -checksum out.jp2 | grep Checksum
  Checksum=30111
  Checksum=32302
  Checksum=40026

[IdanAviv89]

For me it happens when the raster is 16bits per band.

[rouault]

I still can't reproduce any issue:

Unsigned 16 bit:

$ gdal_translate autotest/gdrivers/data/small_world.tif in.tif -ot uint16 -scale 0 255 0 65535
$ gdal_translate in.tif out.jp2 -of jp2kak -co quality=100
$ gdalinfo -checksum in.tif| grep Checksum
  Checksum=15817
  Checksum=53722
  Checksum=41363
$ gdalinfo -checksum out.jp2 | grep Checksum
  Checksum=15817
  Checksum=53722
  Checksum=41363

Signed 16 bit:

$ gdal_translate autotest/gdrivers/data/small_world.tif in.tif -ot int16 -scale 0 255 -32768 32767
$ gdal_translate in.tif out.jp2 -of jp2kak -co quality=100
$ gdalinfo -checksum in.tif| grep Checksum
  Checksum=46758
  Checksum=52208
  Checksum=57041
$ gdalinfo -checksum out.jp2 | grep Checksum
  Checksum=46758
  Checksum=52208
  Checksum=57041

Please provide a reproducer input image (as small as possible, so use potentially gdal_translate to subset it first), as guessing in the dark isn't fun

[IdanAviv89]

Hi,
Here is example of an image that can show the problem: https://drive.google.com/file/d/1onis1SJFAICUTyIcArSvwuNxRG5wtygu/view?usp=sharing

gdalinfo la_16.tif -checksum | grep Checksum

  Checksum=20649

  Checksum=21583

  Checksum=10472

  Checksum=47643

I converted it to jpeg 2000 using the following command:
gdal_translate la_16.tif la_16.jp2 -of jp2kak -co quality=100 -co Clevels=5 -co LAYERS=10 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co Corder=RPCL -co ORGen_plt=yes -co ORGtparts=R -co ORGgen_tlm=6 -co FLUSH=FALSE

I got the following checksum:

gdalinfo la_16.jp2 -checksum | grep Checksum

  Checksum=20649

  Checksum=49868

  Checksum=36270

  Checksum=40148

[IdanAviv89]

gdal/frmts/jp2kak/jp2kakdataset.cpp

Lines 2887 to 2911 in 27a73be

	std::vector<int> recommended_stripe_heights(num_components);
	compressor.get_recommended_stripe_heights(
	1,
	std::max(1, static_cast<int>(
	std::min(nMaxBufferSize, nNeededBufferSize) /
	nLineBufferSize)),
	&recommended_stripe_heights[0], nullptr);
	const int stripe_height = recommended_stripe_heights[0];
	CPLDebug("JP2KAK", "stripe_height = %d", stripe_height);
	GByte *pBuffer = static_cast<GByte *>(VSI_MALLOC3_VERBOSE(
	num_components * nDataTypeSizeBytes, nXSize, stripe_height));
	if (pBuffer == nullptr)
	bOK = false;
	std::vector<kdu_byte *> stripe_bufs(num_components);
	std::vector<int> stripe_heights = recommended_stripe_heights;
	std::vector<vector_safe_bool> is_signed(num_components);
	std::vector<int> precisions(num_components);
	for (int i = 0; i < num_components; ++i)
	{
	stripe_bufs[i] = pBuffer + nXSize * nDataTypeSizeBytes * i;
	is_signed[i] = CPL_TO_BOOL(GDALDataTypeIsSigned(eType));
	precisions[i] = nBits;
	}
	const int flush_period = bFlushEnabled ? TILE_CHUNK_SIZE : 0;
	int nHeight = stripe_height;

I think the problem is somewhere over here.
It might be that the stripe_height is set specifically to the first band?

const int stripe_height = recommended_stripe_heights[0];

[rouault]

Fix in #10615

You may workaround the bug by adding --config JP2KAK_USE_STRIPE_COMPRESSOR NO

[IdanAviv89]

Hi,
I am trying to use gdal_translate with large image. I get that the warning during the translate, and then the checksum is not the same.

command:
gdal_translate -of JP2KAK -co QUALITY=100 -co FLUSH=TRUE -co Corder=RPCL -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co ORGgen_plt=yes -co ORGgen_tlm=7 -co Creversible=yes -co ORGtparts=R -co Clevels=6 input_image.tif output_image.jp2

I get the following warning:

Warning 1: Kakadu Core Warning:
Calls to 'kdu_sample_allocator::release_permit' do not match successful 'kdu_sample_allocator::seek_permit' calls. Memory leak seems likely!

And when I check the checksum of the input and the output I get different values. Am I missing any argument ?

[jratike80]

How big is large image? What GDAL and Kakudu SDK versions do you have?

[IdanAviv89]

Hi,
I am using the tiff image from here, using gdal master.
https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/raster/HYP_HR_SR_OB_DR.zip

I am getting that the checksum of the tif:

gdalinfo HYP_HR_SR_OB_DR.tif -checksum | grep Checksum

  Checksum=46247

  Checksum=57583

  Checksum=62266

And after converting it to jp2kak using the following command:
gdal_translate HYP_HR_SR_OB_DR.tif HYP_HR_SR_OB_DR.jp2 -of jp2kak -co quality=100 -co Clevels=6 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co Corder=RPCL -co ORGen_plt=yes -co ORGtparts=R -co ORGgen_tlm=7 -co FLUSH=FALSE -co Creversible=yes

gdalinfo HYP_HR_SR_OB_DR.jp2 -checksum | grep Checksum

  Checksum=813

  Checksum=8223

  Checksum=27255

[jratike80]

Does gdalcompare https://gdal.org/en/latest/programs/gdalcompare.html find differences from pixels as well?

[IdanAviv89]

Yes

[rouault]

@IdanAviv89 I don't reproduce the issue with GDAL master using Kakadu v8.4.1:

$ gdal_translate HYP_HR_SR_OB_DR.tif  HYP_HR_SR_OB_DR.jp2 -of jp2kak -co quality=100 -co Clevels=6  -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co Corder=RPCL -co ORGgen_plt=yes -co ORGtparts=R -co ORGgen_tlm=7 -co FLUSH=FALSE  -co Creversible=yes  
Input file size is 21600, 10800
0...10...20...30...40...50...60...70...80...90...100 - done.
$ gdalinfo HYP_HR_SR_OB_DR.jp2 -checksum | grep Checksum
  Checksum=46247
  Checksum=57583
  Checksum=62266