Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

probe: assess 🤗 repos for potentially malicious files #767

Merged
merged 67 commits into from
Aug 2, 2024
Merged

probe: assess 🤗 repos for potentially malicious files #767

merged 67 commits into from
Aug 2, 2024

Conversation

leondz
Copy link
Owner

@leondz leondz commented Jul 1, 2024

recreating this one, merged too early, sorry

leondz and others added 30 commits June 23, 2024 15:27
@leondz
draft probe for scanning file formats
4703799
@leondz
draft probe for scanning file formats
ca26631
@leondz
accept generators in detector results, allowing 'yield'
0723974
@leondz
correctly clear up detector result dict in attempt
f216d69
@leondz
relax some constraints around detector detect() return type
f4c0938
@leondz
remove dead code
8e1dc44
@leondz
list can be set successfully from within iteration, generators less so
b28ce1a
@leondz
add detector for possible pickle file extensions
29a8de8
@leondz
add probe, detector, and doc stubs for checking for pickle extensions
7b1d711
@leondz
add possiblepickle test, fix multiple yield bug
28c6e58
@leondz
add links to docs
efc852d
@leondz
add hub requirement
c41fc2d
@leondz
rollback
83eea54
@leondz
overwrite openai options
30d602f
@jmartin-tech @leondz
Revert "overwrite openai options"
10b17c2 
This reverts commit c9aab98.

Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
@leondz
add deps, prune import
72ef666
@leondz
scan hf repo contents for pickles
e528be3
@leondz
shift responsibility for file fetching to probe, making detector more…
24a568f 
… generic
@leondz
check attempt format, check if path is file
8c3fe95
@leondz
tighten data validation
e848d58
@leondz
add FileIsPickle test for: format; non-pickled data; default protocol…
cddaab1 
… pickled data; fixed protocol pickled data
@leondz
refactor fileformat detector, add rudimentary executable checking
131a646
@leondz
refactor fileformat detector, add rudimentary executable checking
c0505f2
@leondz
fileformats test executable file excerpts
33c9f39
@leondz
add python-magic dep
943818e
@leondz
fix dep name
c9e6687
@leondz
add type sigs, allow format check skipping
e01ee0a
@leondz
possiblepickledetector is now a filedetector
98055a9
@leondz
Merge branch 'main' into feature/hf_formats
dc01b6d
@leondz @jmartin-tech
rm stray debug line
74e56e2 
Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
@leondz leondz added the probes Content & activity of LLM probes label Jul 5, 2024
@leondz leondz removed the request for review from erickgalinkin August 1, 2024 18:23
jmartin-tech
jmartin-tech requested changes Aug 1, 2024
View reviewed changes
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor syntax issue, and some recommendations.

garak/detectors/fileformats.py Outdated Show resolved Hide resolved
garak/detectors/fileformats.py Outdated Show resolved Hide resolved
garak/detectors/fileformats.py Show resolved Hide resolved
garak/probes/fileformats.py Outdated Show resolved Hide resolved
tests/detectors/test_detectors_fileformats.py Show resolved Hide resolved
garak/detectors/fileformats.py Outdated Show resolved Hide resolved
leondz and others added 6 commits August 1, 2024 21:08
@leondz @jmartin-tech
strip space
c6e5e6d 
Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
@leondz @jmartin-tech
update goal from default generic value
fb207ca 
Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
@leondz @jmartin-tech
trim whitespace
3f0f855 
Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
@leondz @jmartin-tech
gate testing on correctly-loaded magic lib to workaround libmagic i…
13b8274 
…nstall portability

Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
@leondz @jmartin-tech
== -> = for assignment
71230e4 
Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
@leondz
defer handling of import
36b10df
jmartin-tech
jmartin-tech reviewed Aug 1, 2024
View reviewed changes
pyproject.toml Outdated Show resolved Hide resolved
jmartin-tech
jmartin-tech reviewed Aug 1, 2024
View reviewed changes
requirements.txt Outdated Show resolved Hide resolved
leondz and others added 2 commits August 2, 2024 07:31
@leondz @jmartin-tech
set version restriction for python-magic-bin
a0800c4 
Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
@leondz @jmartin-tech
version restr. for python-magic-bin in requirements.txt
fdb9f07 
Co-authored-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
@leondz leondz merged commit 52441af into main Aug 2, 2024
10 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Aug 2, 2024
@leondz leondz deleted the feature/hf_formats branch August 15, 2024 15:04
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@jmartin-tech jmartin-tech jmartin-tech requested changes

@erickgalinkin erickgalinkin erickgalinkin requested changes

Assignees
No one assigned
Labels
probes Content & activity of LLM probes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@leondz @jmartin-tech @erickgalinkin