Cache masks computed during regex guidance #239
Labels
enhancement
optimization
Related to performance optimizations
structured generation
Linked to structured generation
This has come up a few times, but we apparently haven't created an issue for it: we can keep the masks in the "index" we build instead of the vocab indices. That will save on allocation time.
The text was updated successfully, but these errors were encountered: