Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better length handling for JSON fields required. #3103

Closed
rPraml opened this issue Jun 13, 2023 · 1 comment
Closed

Better length handling for JSON fields required. #3103

rPraml opened this issue Jun 13, 2023 · 1 comment

Comments

@rPraml
Copy link
Contributor

rPraml commented Jun 13, 2023

I have to go back a bit here as to why I think the extension in Ebean makes sense.

Expected behavior

  • What ebean writes, it should also be able to read again

Actual behavior

  • With the update to Jackson 2.15.0 the following code fails:

Steps to reproduce

    String s = new String(new char[5_000_001]).replace('\0', 'x');

    EBasicJsonMap bean = new EBasicJsonMap();
    bean.setName("b1");
    bean.setContent(Map.of("string", s));
    DB.save(bean); // Expected: Throw an error that data is too long here.

    DB.find(EBasicJsonMap.class, bean.getId()); // Actual: throws LoadError, because String is too long

See Discussion FasterXML/jackson-core#863 and FasterXML/jackson-core#1014
Long story short: Jackson disallows the deserialization of strings > 5M (2.15.1 >20M)

The bug/change in jackson crashes our application
All in all, writing so long strings in JSON is
a) a bad idea
b) was caused in our situation by a programming error, that has written too big JSON string.

We have used @DbJson for now without specifying a length attribute, which means, that this defaults effectively to 2G on most platforms - we've learned our lession, that this might bring us further trouble, if some code produces too big JSONs, so we tend to to specify JSON lengths to these properties.

Unfortunately, this causes further issues:

  • while @DbJson(length = 5000) works fine on H2, it fails on SqlServer when creating the column as nvarchar(5000) with The size (5000) given to the parameter 'content' exceeds the maximum allowed (4000)
  • Similar limits for other DBMS (MariaDB supports up to 64K bytes per row. Depending on the charset etc. this will eat up more bytes)
  • unfortunately, we often have very big limits (16K or even 1M for complex configuration documents), so varchar is not always the best option and we need to fall back to "clob"

What I want to change now is mainly in this PR #3102, but I think I have to break up this in several things

@rbygrave I know, the causing issue is mainly jackson related, but it shows up, that length checks for JSON is quite limited in ebean (at least if you have to deal with JSON lengths > 4000), so it would be great, if I get some feedback here.

@rob-bygrave
Copy link
Contributor

Hi @rPraml I think we can close this issue now right? (As Fixed in 13.20.0 via #3107)

@rPraml rPraml closed this as completed Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants