Support non-english text

[Samyak2]

What and why?

Currently, the word selection algorithm in toipe only supports English ASCII characters. Any other characters (including ASCII special characters will be ignored).

This is not good because the world doesn't only write in English :). Especially for a typing test app which can be used to measure and improve speeds in other languages too.

Once this is implemented, it would also be good to have built-in word lists for more languages (maybe even auto-detect the user's locale and change word list based on it?)

How?

The changes need to be done in RawWordSelector (source code).

• Currently, it uses raw bytes (b'a', b'z', etc.) to compare and store characters. This needs to be changed to use the char type.
• It tracks starting positions of each character in a fixed length array of 26 characters. This needs to be made more general.
• Use unicode lowercase instead of ASCII -

  toipe/src/textgen.rs

  Line 91 in cfba937

  let line = buffer.to_ascii_lowercase();

• Do not use is_ascii_alphabetic here -

  toipe/src/textgen.rs

  Line 247 in cfba937

  while word.len() < 2 || word.len() > 8 || !word.chars().all(|c| c.is_ascii_alphabetic()) {

This may require some more changes that I'm not yet aware of.

Please drop a comment if you find any inconsistencies or problems with this idea :)

[Samyak2]

Some suggestions from the author of terminal-typeracer: https://www.reddit.com/r/rust/comments/tvamfz/comment/i3eva1h/