Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revisit ignores and "junk" files tagging #252

Closed
pombredanne opened this issue Jul 23, 2021 · 1 comment
Closed

Revisit ignores and "junk" files tagging #252

pombredanne opened this issue Jul 23, 2021 · 1 comment

Comments

@pombredanne
Copy link
Contributor

We should consider reusing commoncode ignores https://github.com/nexB/commoncode/blob/430cf8362d5055a72595410a5e10e61c1b492266/src/commoncode/ignore.py#L67

In many case we should also attach a reason message on why we ignore a certain file
JonoYang added a commit that referenced this issue Jul 27, 2021
@JonoYang
Create pipeline step to tag ignorable files #252
c8c690c 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 28, 2021
@JonoYang
Create pipeline step to tag ignorable files #252
275cdd6 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 28, 2021
@JonoYang
Create pipeline step to tag ignorable files #252
439a7fe 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 2, 2021
@JonoYang
Create pipeline step to tag ignorable files #252
4636576 
Signed-off-by: Jono Yang <jyang@nexb.com>
tdruez added a commit that referenced this issue Aug 4, 2021
@pombredanne @JonoYang @tdruez
Add support for Windows containers (#181)
a7e3897 
* Use newer version of container libraries

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Use new container-inspector structures

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add minimal support for Windows containers

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Update Windows package getter

    * The windows_helper module from scancode is not available on pypi

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use newer version of container libraries

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Update call to windows_helper to win_reg

Signed-off-by: Jono Yang <jyang@nexb.com>

* Create new pipeline for Windows Docker images

    * Create Windows specific tag_uninteresting_windows_codebase_resources function

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add function to find packages at well-known paths

    * Update tests

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add step to tag known software in pipeline

    * Change name of Docker step from "find_images_linux_distro" to "find_images_os_and_distro"

Signed-off-by: Jono Yang <jyang@nexb.com>

* Get version from path in tag_known_software #238

    * Update docstrings
    * Pin fetchcode dep

Signed-off-by: Jono Yang <jyang@nexb.com>

* Troubleshoot regex patterns #238

Signed-off-by: Jono Yang <jyang@nexb.com>

* Report Program File contents as packages #238

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update Windows-specific regex

    * Add more file names and file extensions to be ignored
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Do not ignore .mui files #238

Signed-off-by: Jono Yang <jyang@nexb.com>

* Filter using extension field rather than path #238

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update scanpipe/pipes/docker.py

Create issue to track extraction issue

See #251

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Fix scancode-toolkit pinned version in base.txt #238

Signed-off-by: Jono Yang <jyang@nexb.com>

* Create pipeline step to tag ignorable files #252

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update formatting #238

Signed-off-by: Jono Yang <jyang@nexb.com>

* Generalize regex expressions #238

    * Modify regex used for Windows container analysis so it can be used outside the context of a Windows Docker image
    * Update tests

Signed-off-by: Jono Yang <jyang@nexb.com>

* Create new pipes for ignoring files #238

    * Create pipes that ignore media files and data files with no clues
    * Update test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add more file extensions to ignore #238

Signed-off-by: Jono Yang <jyang@nexb.com>

* Bump dep versions #238

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update docstring #238

    * Use InstalledWindowsProgram object instead of Package

Signed-off-by: Jono Yang <jyang@nexb.com>

* Improve regex used in tag_known_software #238

    * Update tests with more paths to test regex patterns

Signed-off-by: Jono Yang <jyang@nexb.com>

* Adjust code for consistency across the codebase #181

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Address PR comments #238

    * Use re.match instead of re.split
    * Rename WindowsDocker pipeline to DockerWindows
    * Set the default value of the q_objects argument for tag_installed_package_files to be a tuple

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add is_media field to CodebaseResource #238

    * Update test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Simplify tag_media_files_as_unintersting() #238

    * Update test

Signed-off-by: Jono Yang <jyang@nexb.com>

* Refine windows pipes #238

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Co-authored-by: Jono Yang <jyang@nexb.com>
Co-authored-by: Thomas Druez <tdruez@nexb.com>
@JonoYang
Copy link
Contributor

JonoYang commented Aug 5, 2021

This has been added to scancode.io in scanpipe.pipes.rootfs

@JonoYang JonoYang closed this as completed Aug 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pombredanne @JonoYang