Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: split PreProcessor #3557

Closed
wants to merge 54 commits into from
Closed

feat: split PreProcessor #3557

wants to merge 54 commits into from

Commits on Nov 8, 2022

  1. test and improve header/footer detection

    @ZanSara
    ZanSara committed Nov 8, 2022
    Configuration menu
    Copy the full SHA
    e4794d2 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2022

  1. improve preprocessor testing

    @ZanSara
    ZanSara committed Nov 9, 2022
    Configuration menu
    Copy the full SHA
    b503a1e View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2022

  1. extracted headline alignment algorithm

    @ZanSara
    ZanSara committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    3a02ef0 View commit details
    Browse the repository at this point in the history

  2. reuse remove_substrings to remove headers and footers

    @ZanSara
    ZanSara committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    ad44cc2 View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2022

  1. clean functions tested

    @ZanSara
    ZanSara committed Nov 11, 2022
    Configuration menu
    Copy the full SHA
    6627196 View commit details
    Browse the repository at this point in the history

  2. add options to split

    @ZanSara
    ZanSara committed Nov 11, 2022
    Configuration menu
    Copy the full SHA
    81e3e69 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2022

  1. implement regex cleaning and splitting, tested

    @ZanSara
    ZanSara committed Nov 15, 2022
    Configuration menu
    Copy the full SHA
    e5bb35b View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2022

  1. testing regex splitting

    @ZanSara
    ZanSara committed Nov 16, 2022
    Configuration menu
    Copy the full SHA
    9ddc82b View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2022

  1. one stubborn test failing

    @ZanSara
    ZanSara committed Nov 17, 2022
    Configuration menu
    Copy the full SHA
    f335c8b View commit details
    Browse the repository at this point in the history

  2. stub word tokenization

    @ZanSara
    ZanSara committed Nov 17, 2022
    Configuration menu
    Copy the full SHA
    e8e8f5c View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2022

  1. simplify testing slightly

    @ZanSara
    ZanSara committed Nov 18, 2022
    Configuration menu
    Copy the full SHA
    9aee20d View commit details
    Browse the repository at this point in the history

  2. all tests seems to be passing

    @ZanSara
    ZanSara committed Nov 18, 2022
    Configuration menu
    Copy the full SHA
    cbfcbd9 View commit details
    Browse the repository at this point in the history

  3. few more tests failing

    @ZanSara
    ZanSara committed Nov 18, 2022
    Configuration menu
    Copy the full SHA
    0030e67 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2022

  1. Remove base class

    @ZanSara
    ZanSara committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    a0d95b4 View commit details
    Browse the repository at this point in the history

  2. improving the merger to use in preprocessor

    @ZanSara
    ZanSara committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    b9ea09c View commit details
    Browse the repository at this point in the history

  3. split into splitter, cleaner and merger. Tests not passing

    @ZanSara
    ZanSara committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    1e12a36 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2022

  1. Merge branch 'main' into preprocessor-max-length

    @ZanSara
    ZanSara committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    07b3677 View commit details
    Browse the repository at this point in the history

  2. merger and splitter pass all tests

    @ZanSara
    ZanSara committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    c4b1e65 View commit details
    Browse the repository at this point in the history

  3. all tests are passing

    @ZanSara
    ZanSara committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    59833cd View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2022

  1. add char based split

    @ZanSara
    ZanSara committed Nov 23, 2022
    Configuration menu
    Copy the full SHA
    ae3fad7 View commit details
    Browse the repository at this point in the history

  2. expose split_max_chars in preProcessor

    @ZanSara
    ZanSara committed Nov 23, 2022
    Configuration menu
    Copy the full SHA
    217228f View commit details
    Browse the repository at this point in the history

  3. separate split_by qord and token

    @ZanSara
    ZanSara committed Nov 23, 2022
    Configuration menu
    Copy the full SHA
    f848c3b View commit details
    Browse the repository at this point in the history

  4. more tests

    @ZanSara
    ZanSara committed Nov 23, 2022
    Configuration menu
    Copy the full SHA
    65b01c2 View commit details
    Browse the repository at this point in the history

  5. proper split_by token

    @ZanSara
    ZanSara committed Nov 23, 2022
    Configuration menu
    Copy the full SHA
    6c209fd View commit details
    Browse the repository at this point in the history

  6. openapi

    @ZanSara
    ZanSara committed Nov 23, 2022
    Configuration menu
    Copy the full SHA
    46af584 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2022

  1. some mypy fixes

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    bc482e5 View commit details
    Browse the repository at this point in the history

  2. integrate reviewer feedback

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    603ba46 View commit details
    Browse the repository at this point in the history

  3. reintroduce old preprocessor with deprecation notice

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    b79d7f5 View commit details
    Browse the repository at this point in the history

  4. implement and lightly test max_tokens

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    ec7e218 View commit details
    Browse the repository at this point in the history

  5. Expose max_tokens to the NewPreprocessor and add docstrings

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    ff87cb8 View commit details
    Browse the repository at this point in the history

  6. add error log

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    ce4be7d View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into preprocessor-max-length

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    fa8432f View commit details
    Browse the repository at this point in the history

  8. typo

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    4a857c8 View commit details
    Browse the repository at this point in the history

  9. mypy

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    f226f83 View commit details
    Browse the repository at this point in the history

  10. pylint

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    6047e9b View commit details
    Browse the repository at this point in the history

  11. mypy + pylint

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    d0e8c02 View commit details
    Browse the repository at this point in the history

  12. schema fix

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    163400d View commit details
    Browse the repository at this point in the history

  13. typing

    @ZanSara
    ZanSara committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    2b20564 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2022

  1. investigating weaviate

    @ZanSara
    ZanSara committed Nov 28, 2022
    Configuration menu
    Copy the full SHA
    1242fa4 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2022

  1. implement most of the reviewer feedback

    @ZanSara
    ZanSara committed Nov 29, 2022
    Configuration menu
    Copy the full SHA
    d7a435c View commit details
    Browse the repository at this point in the history

  2. simplify split_respect_sentence_boundary warning message

    @ZanSara
    ZanSara committed Nov 29, 2022
    Configuration menu
    Copy the full SHA
    5f9dd7d View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2022

  1. fix a few bug with max_tokens

    @ZanSara
    ZanSara committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    6c918ff View commit details
    Browse the repository at this point in the history

  2. fixing headlines

    @ZanSara
    ZanSara committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    c6bf667 View commit details
    Browse the repository at this point in the history

  3. add new tokenization option to speed up the tokens count

    @ZanSara
    ZanSara committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    89c1c80 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2022

  1. default to word as in the old preprocessor

    @ZanSara
    ZanSara committed Dec 5, 2022
    Configuration menu
    Copy the full SHA
    3fb14e6 View commit details
    Browse the repository at this point in the history

  2. rename into DocumentPreprocessor

    @ZanSara
    ZanSara committed Dec 5, 2022
    Configuration menu
    Copy the full SHA
    674d0ff View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2022

  1. less deepcopy

    @ZanSara
    ZanSara committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    3c4d676 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2022

  1. improving merger

    @ZanSara
    ZanSara committed Dec 9, 2022
    Configuration menu
    Copy the full SHA
    d4e8332 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2022

  1. improve merger, drastic speedup of splitter - wip

    @ZanSara
    ZanSara committed Dec 12, 2022
    Configuration menu
    Copy the full SHA
    f3cc2cc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into preprocessor-max-length

    @ZanSara
    ZanSara committed Dec 12, 2022
    Configuration menu
    Copy the full SHA
    0a979b6 View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2022

  1. extracting helpers

    @ZanSara
    ZanSara committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    02efb06 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2022

  1. making extracted helpers work again

    @ZanSara
    ZanSara committed Dec 14, 2022
    Configuration menu
    Copy the full SHA
    b6af119 View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2022

  1. testing merging alg

    @ZanSara
    ZanSara committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    6e1ed66 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2022

  1. another approach, not working

    @ZanSara
    ZanSara committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    c39a6bf View commit details
    Browse the repository at this point in the history