vocab: refactor tokenizer to reduce the overhead of creating multi times tokenizer #9449
+238
−138
We went looking everywhere, but couldn’t find those commits.
Sometimes commits can disappear after a force-push. Head back to the latest changes here.