v0.9.0

What's Changed

• feat: add option to retrieve offsets from tokenizer by @riccardopinosio in #21
• Update to huggingface/tokenizers v0.20.0 by @daulet in #23

New Contributors

• @riccardopinosio made their first contribution in #21

Full Changelog: v0.8.0...v0.9.0

v0.8.0

Breaking change:

Path to compiled rust library needs to be specified via -ldflags. I found it most convenient to use CGO_LDFLAGS env variable to avoid always setting it. See #18 for more details.

What's Changed

• Update to allow for platform dependent libs in CGO by @jmoney in #18

New Contributors

• @jmoney made their first contribution in #18

Full Changelog: v0.7.1...v0.8.0

v0.7.1

• Update core tokenizers library to latest: v0.15.2;
• Expose init time parameter to encode special tokens (or not);

Full Changelog: v0.7.0...v0.7.1

v0.7.0

What's Changed

• support more attributes from the Encoding structure by @clems4ever in #5

Full Changelog: v0.6.1...v0.7.0

v0.6.1

• Simply changing bazel target names

v0.6.0

• Update underlying core library to v0.14.1 (latest at the moment);
• Support bazel build system so downstream projects can easily consume this;
• Artifacts are smaller too since we lost dependency on openssl;

v0.5.1

• fix tokenizer memory leak
• fix panic in encode/decode with invalid utf8 string

v0.5.0

• Encode now returns token string representations;
• Proper free of Rust strings in Decode;

v0.4.3

• Release artifact for darwin-x86_64

v0.4.2

• Make libtokenizers.a location relative to the source location;
• Handle empty inputs to encode/decode;