Add an ability to convert result of surface() method to normalized variants by specifying a projection

[eiennohito]

Projection can be normalization transforms as specified by ChiTra or a user-passed callable.

Original surface will always be available as raw_surface().

Specification

• Add an projection argument to Dictionary.create which can accept either a string or a callable.
• When projection is passed, instead of usual Morpheme.surface() result, Sudachi.rs will produce a projected string.
• Projected strings are not cached, projection is computed each time on call to surface()
• Original string is available as raw_surface() method on Morpheme

[hiroshi-matsuda-rit]

@eiennohito We need to consider the way to pass the projection type without modifying BertJapaneseTokenizer in transformers repo.

• sudachi_config_path
• sudachi_resource_dir
• sudachi_dict_type

These three arguments are the candidates for specifying projection type.

https://github.com/huggingface/transformers/blob/v4.29.0/src/transformers/models/bert_japanese/tokenization_bert_japanese.py#L533-L546
https://github.com/huggingface/transformers/blob/v4.29.0/src/transformers/models/bert_japanese/tokenization_bert_japanese.py#L533-L546
https://github.com/huggingface/transformers/blob/v4.29.0/src/transformers/models/bert_japanese/tokenization_bert_japanese.py#L590-L592

[eiennohito]

In the long term it would be better to submit a patch to huggingface transformers to provide **kwargs with options to all Japanese tokenizers though

[eiennohito]

@hiroshi-matsuda-rit see https://github.com/WorksApplications/sudachi.rs/blob/da5aca62e3cef8892ceaf64e7ac4f9ef25c2f8d1/python/tests/test_projection.py, any comments on the functionality?
Also see #234

[hiroshi-matsuda-rit]

@eiennohito Thank you for all your efforts in this issue. It seems new implementations are very nice.
In WordFormTypes of SudachiTra, some of the word form types have the fallback to the surface field.
I think we do not need all the fallbacking logics in WordFormTypes.
But normalized_and_surface, normalized_nouns, and dictionary_and_surface, which are used in SudahiTra or Megagon Labs' ELECTRA model, I think it would be better from a compatibility point of view if these were supported.
(I'm sorry that I do not have time to review entire source codes to identify the appropriate implementation points.)

[eiennohito]

No problem, I missed point on requirements about supporting chiTra forms. I will add that implementation as well.

[eiennohito]

@hiroshi-matsuda-rit I will add support only for normalized_and_surface, normalized_nouns, and dictionary_and_surface, please say if other types are also needed (they require more work)

[hiroshi-matsuda-rit]

Excellent! Thanks for additional implementations. (I think those three options are enough for us.) @eiennohito