-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rust: Support writing without chunks #1201
Conversation
Topic: nochunk On very memory-constrained systems like microcontrollers, the overhead of storing all the chunk offset data in memory until the end of the file can be significant. To avoid that, we instead write all Channel, Schema, and Message data directly as records rather than using chunks, and rely on the reader (or an intermediate ingestion process) to generate the index information.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good, and the rationale makes a lot of sense. Just one small change needed.
Topic: nochunk
Issue was resolved, still want @james-rms to have a look before approving this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems good with a test that demonstrates that this mode produces valid MCAP.
Topic: nochunk
Done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding the test, this looks good to me
On very memory-constrained systems like microcontrollers, the overhead of storing all the chunk offset data in memory until the end of the file can be significant. To avoid that, we instead write all Channel, Schema, and Message data directly as records rather than using chunks, and rely on the reader (or an intermediate ingestion process) to generate the index information.
Changelog
The rust library now (optionally) supports writing unchunked MCAP files for decreased memory footprint.
Docs
None
Description
In our internal testing, storing chunk data required >40KB of memory for several-minute logs, which is a lot for an embedded system with only 300KB of RAM total. Notably, this grows without bound during the test, since the offsets and data about all chunks needs to be stored until the end of the log when it will be written to the index.
For our usecase, it's acceptable to disable chunking entirely (along with compression and indexing), since we can add those things later during the log upload process.
We're using this internally on an ESP32-S3 and it is working well.