v0.6.8

Highlights

• Produce builds for Python 3.12 (#236)
• Add a simple configuration API
• Add surface projections (#230)

Surface Projections

• For chiTra compatibility SudachiPy can now directly produce different tokens in the surface field.
• Original surface is accessible via Morheme.raw_surface() method
• It is possible to customize projection dictionary-wise, via Config object, passing it on a dictionary creation, or for a single pre-tokenizer.
  • Config API
  • Pretokenizer API

0.6.7

Highlights

• Provide binary wheels for Python 3.11
• Add Dictionary.lookup() method which allows you to enumerate morphemes from the dictionary without performing analysis.

0.6.6

Highlights

• Add boundary matching mode to regex oov handler
• macOS binary builds are now unversal2 (arm+x64)

MacOS

• Binary builds are universal2
• Caveat: we don't run tests on arm because there are no public arm instances, so builds may be broken without any warning

0.6.5

Highlights

• Fixed invalid POS tags which appeared when using user-defined POS tags both in user dictionaries and OOV handlers. You are not affected by this bug if you did not use user-defined POS in OOV handlers.

Version 0.6.4

Highlights

• Remove Python 3.6 support which reached end-of-life status on 2021-12-23
• OOV handler plugins support user-defined POS, similar to Java version
• Added Regex OOV handler

Regex OOV Handler

• For details, see Java version changelog
• In Rust/Python Regexes do not support backtracking and backreferences
• maxLength setting defines maximum length in unicode codepoints, not in utf-8 bytes as in Java (will be changed to codepoints later)

0.6.3

Highlights

• Fixed path resolution algorithm for resources. They are now resolved in the following order (first existing file wins):
  1. Absolute paths stay as they are
  2. Relative to "path" value of the config file
  3. Relative to "resource_dir" parameter of the config object during creation
     • For SudachiPy it is the parameter of Dictionary constructor
  4. Relative to the location of the configuration file
  5. Relative to the current directory

Python

• Dictionary now has __repr__() function which displays absolute paths to dictionaries in use.
• Dictionary now has pos_of() function which returns a POS tuple for a given POS id.
• PosMatcher supports set operations
  • union (m1 | m2)
  • intersection (m1 & m2)
  • difference (m1 - m2)
  • negation (~m1)

0.6.2

Highlights

• Fixed analysis differences from 0.5.4
  • Central dot ・ is handled correctly
  • Catch-all OOV handler was used even if other OOV handlers could produces some results

0.6.1

Highlights

• Added Fuzzing (see sudachi-fuzz subdirectory), Sudachi.rs seems to be pretty robust towards arbitrary inputs (no crashes and panics)
  • Issues like #182 should never occur more
• ~5% analysis speed improvement over 0.6.0
• Added support for Unicode combining symbols, now Sudachi.rs/py should be much better with emoji (🎅🏾) and more complex Unicode (İstanbul)

Rust

• Added partial dictionary read functionality, it is now possible to skip reading certain fields if they are not needed
• Improved startup times, especially for debug builds

Python

• Morpheme.part_of_speech method now returns Tuple of POS components instead of a list.
• Partial Dictionary Read
  • It is possible to ask for a subset of morpheme fields instead of all fields
  • Supported API: Dictionary.create(), Dictionary.pre_tokenizer()
• HuggingFace PreTokenizer support
  • We provide a built-in HuggingFace-compatible pre-tokenizer
  • API: Dictionary.pre_tokenizer()
  • It is multithreading-compatible and supports customization
• Memory allocation reuse
  • It is possible to reduce re-allocation overhead by using out parameters which accept MorphemeLists
  • Supported API: Tokenizer.tokenize(), Morpheme.split()
  • It is now a recommended way to use both those APIs
• PosMatcher
  • New API for checking if a morpheme has a POS tag from a set
  • Strongly prefer using it instead of string comparison of POS components
• Performance
  • Greatly decreased cost of accessing POS components
• len(Morpheme) now returns the length of the morpheme in Unicode codepoints. Use it instead of len(m.surface())
• Morpheme.split() has new add_single parameter, which can be used to check whether the split has produced anything
  • E.g. with if m.split(SplitMode.A, out=res, add_single=False): handle_splits(res)
  • add_single=True, returning the list with the current morpheme is the current behavior
• Morpheme/MorphemeList now have readable __repr__ and __str__
  • #187

0.6.0

Full Changelog

Highlights

• Full feature parity with Java version
• ~15% analysis speed improvement over 0.6.0-rc1
• SudachiPy compatible Python bindings
• ~30x speed improvement over original SudachiPy

Rust

• No public API at the moment (contact us if you want to use Rust version directly, internals will significantly change and names are not finalized)
• Added dictionary build functionality
  • #143
• Added an option to perform analysis without sentence splitting
  • Use it with --split-sentences=no

Python

• Added bindings for dictionary build (undocumented and not supported as API).
  • See #157
• sudachipy build and sudachipy ubuild should work once more
  • Report on build times and dictionary part sizes can differ from the original SudachiPy

0.6.0-rc1

Highlights

• First release of Sudachi.rs
• SudachiPy compatible Python bindings
• ~30x speed improvement over original SudachiPy
• Dictionary build mode will be done before 0.6.0 final (See #13)

Rust

• Analysis: feature parity with Python and Java version
• Dictionary build is not supported in rc1
• ~2x faster than Java version (with sentence splitting)
• No public API at the moment (contact us if you want to use Rust version directly, internals will significantly change and names are not finalized)

Python

• Mostly compatible with SudachiPy 0.5.4
• We provide binary wheels for popular platforms
• ~30x faster than 0.5.4
• IgnoreYomigana input text plugin is now supported (and enabled by default)
• We provide binary wheels for convenience (and additional speed on Linux)

Known Issues

• List of deprecated SudachiPy API:
  • MorphemeList.empty(dict: Dictionary)
    • This also needs a dictionary as an argument.
  • Morpheme.split(mode: SplitMode)
  • Morpheme.get_word_info()
  • Most of instance attributes are not exported: e.g. Dictionary.grammar, Dictionary.lexicon.
    • See API reference page for supported APIs.
• Dictionary Build is not supported: sudachipy build and sudachipy ubuild will not work, please use 0.5.3 in another virtual environment for the time being until the feature is implemented: #13