Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

probe: assess 🤗 repos for potentially malicious files #767

Merged
merged 67 commits into from
Aug 2, 2024
Merged

probe: assess 🤗 repos for potentially malicious files #767

merged 67 commits into from
Aug 2, 2024

Commits on Jun 23, 2024

  1. draft probe for scanning file formats

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    4703799 View commit details
    Browse the repository at this point in the history

  2. draft probe for scanning file formats

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    ca26631 View commit details
    Browse the repository at this point in the history

  3. accept generators in detector results, allowing 'yield'

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    0723974 View commit details
    Browse the repository at this point in the history

  4. correctly clear up detector result dict in attempt

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    f216d69 View commit details
    Browse the repository at this point in the history

  5. relax some constraints around detector detect() return type

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    f4c0938 View commit details
    Browse the repository at this point in the history

  6. remove dead code

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    8e1dc44 View commit details
    Browse the repository at this point in the history

  7. list can be set successfully from within iteration, generators less so

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    b28ce1a View commit details
    Browse the repository at this point in the history

  8. add detector for possible pickle file extensions

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    29a8de8 View commit details
    Browse the repository at this point in the history

  9. add probe, detector, and doc stubs for checking for pickle extensions

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    7b1d711 View commit details
    Browse the repository at this point in the history

  10. add possiblepickle test, fix multiple yield bug

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    28c6e58 View commit details
    Browse the repository at this point in the history

  11. add links to docs

    @leondz
    leondz committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    efc852d View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. add hub requirement

    @leondz
    leondz committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c41fc2d View commit details
    Browse the repository at this point in the history

  2. rollback

    @leondz
    leondz committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    83eea54 View commit details
    Browse the repository at this point in the history

  3. overwrite openai options

    @leondz
    leondz committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    30d602f View commit details
    Browse the repository at this point in the history

  4. Revert "overwrite openai options" 

    This reverts commit c9aab98.

Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
    @jmartin-tech @leondz
    jmartin-tech authored and leondz committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    10b17c2 View commit details
    Browse the repository at this point in the history

  5. add deps, prune import

    @leondz
    leondz committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    72ef666 View commit details
    Browse the repository at this point in the history

  6. scan hf repo contents for pickles

    @leondz
    leondz committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    e528be3 View commit details
    Browse the repository at this point in the history

  7. shift responsibility for file fetching to probe, making detector more… 

    … generic
    @leondz
    leondz committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    24a568f View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. check attempt format, check if path is file

    @leondz
    leondz committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    8c3fe95 View commit details
    Browse the repository at this point in the history

  2. tighten data validation

    @leondz
    leondz committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    e848d58 View commit details
    Browse the repository at this point in the history

  3. add FileIsPickle test for: format; non-pickled data; default protocol… 

    … pickled data; fixed protocol pickled data
    @leondz
    leondz committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    cddaab1 View commit details
    Browse the repository at this point in the history

  4. refactor fileformat detector, add rudimentary executable checking

    @leondz
    leondz committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    131a646 View commit details
    Browse the repository at this point in the history

  5. refactor fileformat detector, add rudimentary executable checking

    @leondz
    leondz committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    c0505f2 View commit details
    Browse the repository at this point in the history

  6. fileformats test executable file excerpts

    @leondz
    leondz committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    33c9f39 View commit details
    Browse the repository at this point in the history

  7. add python-magic dep

    @leondz
    leondz committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    943818e View commit details
    Browse the repository at this point in the history

  8. fix dep name

    @leondz
    leondz committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    c9e6687 View commit details
    Browse the repository at this point in the history

  9. add type sigs, allow format check skipping

    @leondz
    leondz committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    e01ee0a View commit details
    Browse the repository at this point in the history

  10. possiblepickledetector is now a filedetector

    @leondz
    leondz committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    98055a9 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Merge branch 'main' into feature/hf_formats

    @leondz
    leondz committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    dc01b6d View commit details
    Browse the repository at this point in the history

  2. rm stray debug line 

    Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz @jmartin-tech
    leondz and jmartin-tech committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    74e56e2 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. log non-isfile() entries in filedetector

    @leondz
    leondz committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    1640d95 View commit details
    Browse the repository at this point in the history

  2. test skipping behaviour for FileDetector

    @leondz
    leondz committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    175418a View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into feature/hf_formats

    @leondz
    leondz committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    816344d View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. Merge branch 'main' into feature/hf_formats 

    Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz
    leondz committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    b615df0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feature/hf_formats

    @leondz
    leondz committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    6b50682 View commit details
    Browse the repository at this point in the history

  3. fix missing import

    @leondz
    leondz committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    b2410c6 View commit details
    Browse the repository at this point in the history

  4. detectors can return generators

    @leondz
    leondz committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    50c6ab4 View commit details
    Browse the repository at this point in the history

  5. move FileDetector over to all_outputs; pause extended testing of all_… 

    …outputs detection result for FileDetectors
    @leondz
    leondz committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    f62381a View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into feature/hf_formats

    @leondz
    leondz committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    19df2e8 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into feature/hf_formats

    @leondz
    leondz committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    5e3ac08 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. rm debug print 

    Co-authored-by: Erick Galinkin <egalinkin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz @erickgalinkin
    leondz and erickgalinkin committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    f46d5b4 View commit details
    Browse the repository at this point in the history

  2. rm gcg doc link

    @leondz
    leondz committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    c786de3 View commit details
    Browse the repository at this point in the history

  3. test list len

    @leondz
    leondz committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    0a7b240 View commit details
    Browse the repository at this point in the history

  4. distribute base64 versions of truncated bins

    @leondz
    leondz committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    82cc431 View commit details
    Browse the repository at this point in the history

  5. convert string path to a Path before unlink()ing

    @leondz
    leondz committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    9159fab View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into feature/hf_formats

    @leondz
    leondz committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    b518560 View commit details
    Browse the repository at this point in the history

  7. avoid attempt w/ empty prompt; use centralised colours; add fileforma… 

    …ts tests
    @leondz
    leondz committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    42a1304 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. Merge branch 'main' into feature/hf_formats

    @leondz
    leondz committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    8c4b224 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. ensure decoded binary cleanup in teardown

    @leondz
    leondz committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    0deee76 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. Merge branch 'main' into feature/hf_formats 

    Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz
    leondz committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    79c0105 View commit details
    Browse the repository at this point in the history

  2. restore comma lost in merge

    @leondz
    leondz committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    e7d85eb View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. merge w main

    @leondz
    leondz committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    12e44af View commit details
    Browse the repository at this point in the history

  2. convert tempfile name to path before calling unlink()

    @leondz
    leondz committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    4717118 View commit details
    Browse the repository at this point in the history

  3. add exe mimetype found in osx libmagic impl

    @leondz
    leondz committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    dec93ea View commit details
    Browse the repository at this point in the history

  4. use bin-including magic on win, osx

    @leondz
    leondz committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    f82fd8a View commit details
    Browse the repository at this point in the history

  5. python-magic-bin only compatible with intel osx. add libmagic install… 

    … to macos test deps; only use python-magic-bin for win
    @leondz
    leondz committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    4d616fb View commit details
    Browse the repository at this point in the history

  6. make sure testing model doesn't go onto MPS (insufficient memory on g… 

    …h's shared setup)
    @leondz
    leondz committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    5a2f100 View commit details
    Browse the repository at this point in the history

  7. handle case where brew/other external dep is required (reqs should in… 

    …stall a bin on win, linux is ok)
    @leondz
    leondz committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    6415732 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'main' into feature/hf_formats

    @leondz
    leondz committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    fb42d71 View commit details
    Browse the repository at this point in the history

  9. strip space 

    Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz @jmartin-tech
    leondz and jmartin-tech committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    c6e5e6d View commit details
    Browse the repository at this point in the history

  10. update goal from default generic value 

    Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz @jmartin-tech
    leondz and jmartin-tech committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    fb207ca View commit details
    Browse the repository at this point in the history

  11. trim whitespace 

    Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz @jmartin-tech
    leondz and jmartin-tech committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    3f0f855 View commit details
    Browse the repository at this point in the history

  12. gate testing on correctly-loaded magic lib to workaround libmagic i… 

    …nstall portability

Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz @jmartin-tech
    leondz and jmartin-tech committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    13b8274 View commit details
    Browse the repository at this point in the history

  13. == -> = for assignment 

    Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz @jmartin-tech
    leondz and jmartin-tech committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    71230e4 View commit details
    Browse the repository at this point in the history

  14. defer handling of import

    @leondz
    leondz committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    36b10df View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. set version restriction for python-magic-bin 

    Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz @jmartin-tech
    leondz and jmartin-tech committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    a0800c4 View commit details
    Browse the repository at this point in the history

  2. version restr. for python-magic-bin in requirements.txt 

    Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
    @leondz @jmartin-tech
    leondz and jmartin-tech committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    fdb9f07 View commit details
    Browse the repository at this point in the history