Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't allocate memory when ensuring capacity #102

Merged
merged 1 commit into from
Nov 17, 2018

Conversation

felixbarny
Copy link
Contributor

If you are having bad luck and the buffer is full while serializing a number, the buffer size doubles - even when the JsonWriter is underlying a target OutputStream. This has the potential to grow the buffer size indefinitely, leading to OOMEs.

See also this JMC screenshot from a flight recording which shows a huge allocation when calling NumberConverter#serialize(int, JsonWriter) ->
JsonWriter#ensureCapacity. I hope this can be fixed soon :)

screen shot 2018-11-17 at 09 56 19

I didn't get the tests to run so I couldn't test the change.

@zapov zapov merged commit 115b219 into ngs-doo:master Nov 17, 2018
@zapov
Copy link
Member

zapov commented Nov 17, 2018

Tnx. What error do you get when running tests?

@felixbarny
Copy link
Contributor Author

Wow, that was quick 😄

I get

java.lang.AssertionError: expected:<NOTE> but was:<ERROR>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at com.dslplatform.json.DslValidationTest.checkNonNull(DslValidationTest.java:129)

My maven setup:

$ mvn -v
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T20:33:14+02:00)
Maven home: /usr/local/Cellar/maven/3.5.4/libexec
Java version: 1.8.0_162, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.14.1", arch: "x86_64", family: "mac"

When do you plan to do the next release?

@felixbarny felixbarny deleted the ensure-capacity-allocation-free branch November 17, 2018 09:18
@zapov
Copy link
Member

zapov commented Nov 17, 2018

I was planing around months end. Waiting for #99 and maybe I'll look in some map stuff.

I've managed to reproduce the test issue on another machine. Tnx

@felixbarny
Copy link
Contributor Author

Would it be possible to do a bugfix release a bit earlier? We just went GA with the Elastic APM Java agent (and are big fans of DSL-JSON). But I'm a bit concerned that this could cause OOMEs in user's applications.

@zapov
Copy link
Member

zapov commented Nov 17, 2018

Hm... ok. I'll look into releasing a new version early next week

@felixbarny
Copy link
Contributor Author

That would be really great! Thank you!

@zapov
Copy link
Member

zapov commented Nov 20, 2018

I've released a new version just now. Please let me know if tests are working for you now.

@felixbarny
Copy link
Contributor Author

Thanks! I'll have a look and release a bugfix for our agent.

felixbarny added a commit to felixbarny/apm-agent-java that referenced this pull request Nov 20, 2018
felixbarny added a commit to elastic/apm-agent-java that referenced this pull request Nov 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants