Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize memory footprint of resources #151

Merged
merged 8 commits into from
Sep 10, 2019

Conversation

adrienball
Copy link
Collaborator

Description

  • Use hashing trick for HashSetGazetteer, HashMapStemmer and HashMapWordClusterer and replace raw string keys by i32 keys
  • HashMapWordClusterer: try to load word clusters as u16 with a fallback to String for all values as soon as one value can't be converted to u16

In English, with the current word clusters included in the resources, this results in a constant 25MB gain in memory.
For other languages without word clusters, the expected gain is between 0.5MB and 1MB.

Backward compatibility
The new implementation is backward compatible. Old word clusters, which typically are stored like hierarchical binary paths of the form "10001011001", can still be loaded. In this case, clusters will be loaded as strings.
New word clusters, introduced in snipsco/snips-nlu-language-resources#33, will benefit from this improved implementation, as all clusters are u16-like.

Copy link
Collaborator

@ClemDoum ClemDoum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏

@adrienball adrienball merged commit 75f1258 into master Sep 10, 2019
@adrienball adrienball deleted the task/optimize-resources-loading branch September 10, 2019 10:06
@adrienball adrienball mentioned this pull request Sep 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants