Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix llama tokenizer #22402

Merged
merged 23 commits into from
Apr 3, 2023
Merged

Fix llama tokenizer #22402

merged 23 commits into from
Apr 3, 2023
+480 −62

Commits on Mar 24, 2023

  1. draft

    @ArthurZucker
    ArthurZucker committed Mar 24, 2023
    Configuration menu
    Copy the full SHA
    d3b2050 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2023

  1. update tokenization limma and conversion script

    @ArthurZucker
    ArthurZucker committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    0da5717 View commit details
    Browse the repository at this point in the history

  2. more udpates

    @ArthurZucker
    ArthurZucker committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    a2f2b19 View commit details
    Browse the repository at this point in the history

  3. initial commit

    @ArthurZucker
    ArthurZucker committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    7cf369c View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' of https://github.com/huggingface/transformers in… 

    …to fix-llama-tokenizer
    @ArthurZucker
    ArthurZucker committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    ff70e2c View commit details
    Browse the repository at this point in the history

  5. style

    @ArthurZucker
    ArthurZucker committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    0cf62ec View commit details
    Browse the repository at this point in the history

  6. default pad to None

    @ArthurZucker
    ArthurZucker committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    1f1a4bf View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2023

  1. draft tokenization tests

    @ArthurZucker
    ArthurZucker committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    7e3ea59 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2023

  1. Merge branch 'main' of https://github.com/huggingface/transformers in… 

    …to fix-llama-tokenizer
    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    33dfe79 View commit details
    Browse the repository at this point in the history

  2. update test

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    ea0c834 View commit details
    Browse the repository at this point in the history

  3. update tokenization tests

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    3bd23e6 View commit details
    Browse the repository at this point in the history

  4. nits

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    8edf9b1 View commit details
    Browse the repository at this point in the history

  5. update

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    a334056 View commit details
    Browse the repository at this point in the history

  6. versioning test

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    5b9be49 View commit details
    Browse the repository at this point in the history

  7. major fix

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    1b6532a View commit details
    Browse the repository at this point in the history

  8. fix more testst

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    d988805 View commit details
    Browse the repository at this point in the history

  9. finish fixing special masks

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    6bdf1e0 View commit details
    Browse the repository at this point in the history

  10. last nit

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    1ab5d7c View commit details
    Browse the repository at this point in the history

  11. more nits

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    512f77f View commit details
    Browse the repository at this point in the history

  12. add encode decode tests

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    4c3c2e2 View commit details
    Browse the repository at this point in the history

  13. add more

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    3175977 View commit details
    Browse the repository at this point in the history

  14. fix token type ids

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    1871030 View commit details
    Browse the repository at this point in the history

  15. style

    @ArthurZucker
    ArthurZucker committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    fd60fb6 View commit details
    Browse the repository at this point in the history