Skip to content

Commit

Permalink
refactor(OrtResult)!: Apply curations on-the-fly
Browse files Browse the repository at this point in the history 
The `OrtResult` does not store the uncurated packages as part of the
analyzer result, but only the curated packages along with the applied
package curation data.

This tightly couples the curations with the analyzer without need, because
the analyzer does not need (to consume) any curations at all. Also,
computing the respective uncurated package from each curated package is
not always possible due to missing data [1]. So, curations currently
cannot properly be (re-applied) without re-running the analyzer [2].
Furthermore, the current representation stores package curation data
redundantly in case the curation applies to multiple packages.

Given that, it makes sense to store the curations separately from the
uncurated package. So, utilize the new toplevel `resolvedConfiguration`
to store the package curations and change the analyzer result to contain
uncurated instead of curated packages.

Note that this partially implements [1] and [2]. Adjusting the logic
which turns curated into uncurated packages, e.g.
`toUncuratedPackage()`, is left for a future change to limit the size of
this change. Apart from that [3] can be implemented by relatively easily
without redundantly encoding the provider (for each curation data).

[1] #5637
[2] #6188
[3] #5668

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
  • Loading branch information
@fviernau
fviernau committed Jan 27, 2023
1 parent 8420b6b commit 205bc89
Show file tree
Hide file tree
Showing 44 changed files with 15,517 additions and 16,414 deletions.
325 changes: 158 additions & 167 deletions advisor/src/test/assets/ort-analyzer-result.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -410,174 +410,165 @@ analyzer:
\ are defined."
severity: "ERROR"
packages:
- metadata:
id: "Maven:junit:junit:4.12"
purl: "pkg:maven/junit/junit@4.12"
declared_licenses:
- "Eclipse Public License 1.0"
declared_licenses_processed:
spdx_expression: "EPL-1.0"
mapped:
Eclipse Public License 1.0: "EPL-1.0"
description: "JUnit is a unit testing framework for Java, created by Erich\
\ Gamma and Kent Beck."
homepage_url: "http://junit.org"
binary_artifact:
url: "https://repo.maven.apache.org/maven2/junit/junit/4.12/junit-4.12.jar"
hash:
value: "2973d150c0dc1fefe998f834810d68f278ea58ec"
algorithm: "SHA-1"
source_artifact:
url: "https://repo.maven.apache.org/maven2/junit/junit/4.12/junit-4.12-sources.jar"
hash:
value: "a6c32b40bf3d76eca54e3c601e5d1470c86fcdfa"
algorithm: "SHA-1"
vcs:
type: "Git"
url: "git://github.com/junit-team/junit.git"
revision: "r4.12"
path: ""
vcs_processed:
type: "Git"
url: "https://github.com/junit-team/junit.git"
revision: "r4.12"
path: ""
curations: []
- metadata:
id: "Maven:org.apache.commons:commons-lang3:3.5"
purl: "pkg:maven/org.apache.commons/commons-lang3@3.5"
declared_licenses:
- "Apache License, Version 2.0"
declared_licenses_processed:
spdx_expression: "Apache-2.0"
mapped:
Apache License, Version 2.0: "Apache-2.0"
description: "Apache Commons Lang, a package of Java utility classes for the\n\
\ classes that are in java.lang's hierarchy, or are considered to be so\n\
\ standard as to justify existence in java.lang."
homepage_url: "http://commons.apache.org/proper/commons-lang/"
binary_artifact:
url: "https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar"
hash:
value: "6c6c702c89bfff3cd9e80b04d668c5e190d588c6"
algorithm: "SHA-1"
source_artifact:
url: "https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5-sources.jar"
hash:
value: "f7d878153e86a1cdddf6b37850e00a9f8bff726f"
algorithm: "SHA-1"
vcs:
type: "Git"
url: "http://git-wip-us.apache.org/repos/asf/commons-lang.git"
revision: "LANG_3_5"
path: ""
vcs_processed:
type: "Git"
url: "http://git-wip-us.apache.org/repos/asf/commons-lang.git"
revision: "LANG_3_5"
path: ""
curations: []
- metadata:
id: "Maven:org.apache.commons:commons-text:1.1"
purl: "pkg:maven/org.apache.commons/commons-text@1.1"
declared_licenses:
- "Apache License, Version 2.0"
declared_licenses_processed:
spdx_expression: "Apache-2.0"
mapped:
Apache License, Version 2.0: "Apache-2.0"
description: "Apache Commons Text is a library focused on algorithms working\
\ on strings."
homepage_url: "http://commons.apache.org/proper/commons-text/"
binary_artifact:
url: "https://repo.maven.apache.org/maven2/org/apache/commons/commons-text/1.1/commons-text-1.1.jar"
hash:
value: "c336bf600f44b88af356c8a85eef4af822b06a4d"
algorithm: "SHA-1"
source_artifact:
url: "https://repo.maven.apache.org/maven2/org/apache/commons/commons-text/1.1/commons-text-1.1-sources.jar"
hash:
value: "f0770f7f0472bf120ada47beecadce4056fbd20a"
algorithm: "SHA-1"
vcs:
type: "Git"
url: "http://git-wip-us.apache.org/repos/asf/commons-text.git"
revision: ""
path: ""
vcs_processed:
type: "Git"
url: "http://git-wip-us.apache.org/repos/asf/commons-text.git"
revision: ""
path: ""
curations: []
- metadata:
id: "Maven:org.apache.struts:struts2-assembly:2.5.14.1"
purl: "pkg:maven/org.apache.struts/struts2-assembly@2.5.14.1"
declared_licenses:
- "The Apache Software License, Version 2.0"
declared_licenses_processed:
spdx_expression: "Apache-2.0"
mapped:
The Apache Software License, Version 2.0: "Apache-2.0"
description: "Apache Struts 2"
homepage_url: "http://struts.apache.org/struts2-assembly/"
binary_artifact:
url: "https://repo.maven.apache.org/maven2/org/apache/struts/struts2-assembly/2.5.14.1/struts2-assembly-2.5.14.1-min-lib.zip"
hash:
value: "8e75a38e3b8ceb01e007c5899d8d29e7a075cb7d"
algorithm: "SHA-1"
source_artifact:
url: ""
hash:
value: ""
algorithm: ""
vcs:
type: "Git"
url: "https://gitbox.apache.org/repos/asf/struts.git"
revision: "STRUTS_2_5_14_1"
path: ""
vcs_processed:
type: "Git"
url: "https://gitbox.apache.org/repos/asf/struts.git"
revision: "STRUTS_2_5_14_1"
path: ""
is_metadata_only: true
curations: []
- metadata:
id: "Maven:org.hamcrest:hamcrest-core:1.3"
purl: "pkg:maven/org.hamcrest/hamcrest-core@1.3"
declared_licenses:
- "New BSD License"
declared_licenses_processed:
spdx_expression: "BSD-3-Clause"
mapped:
New BSD License: "BSD-3-Clause"
description: "This is the core API of hamcrest matcher framework to be used\
\ by third-party framework providers. This includes the a foundation set\
\ of matcher implementations for common operations."
homepage_url: "https://github.com/hamcrest/JavaHamcrest/hamcrest-core"
binary_artifact:
url: "https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar"
hash:
value: "42a25dc3219429f0e5d060061f71acb49bf010a0"
algorithm: "SHA-1"
source_artifact:
url: "https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3-sources.jar"
hash:
value: "1dc37250fbc78e23a65a67fbbaf71d2e9cbc3c0b"
algorithm: "SHA-1"
vcs:
type: "Git"
url: "git@github.com:hamcrest/JavaHamcrest.git"
revision: ""
path: ""
vcs_processed:
type: "Git"
url: "ssh://git@github.com/hamcrest/JavaHamcrest.git"
revision: ""
path: ""
curations: []
- id: "Maven:junit:junit:4.12"
purl: "pkg:maven/junit/junit@4.12"
declared_licenses:
- "Eclipse Public License 1.0"
declared_licenses_processed:
spdx_expression: "EPL-1.0"
mapped:
Eclipse Public License 1.0: "EPL-1.0"
description: "JUnit is a unit testing framework for Java, created by Erich Gamma\
\ and Kent Beck."
homepage_url: "http://junit.org"
binary_artifact:
url: "https://repo.maven.apache.org/maven2/junit/junit/4.12/junit-4.12.jar"
hash:
value: "2973d150c0dc1fefe998f834810d68f278ea58ec"
algorithm: "SHA-1"
source_artifact:
url: "https://repo.maven.apache.org/maven2/junit/junit/4.12/junit-4.12-sources.jar"
hash:
value: "a6c32b40bf3d76eca54e3c601e5d1470c86fcdfa"
algorithm: "SHA-1"
vcs:
type: "Git"
url: "git://github.com/junit-team/junit.git"
revision: "r4.12"
path: ""
vcs_processed:
type: "Git"
url: "https://github.com/junit-team/junit.git"
revision: "r4.12"
path: ""
- id: "Maven:org.apache.commons:commons-lang3:3.5"
purl: "pkg:maven/org.apache.commons/commons-lang3@3.5"
declared_licenses:
- "Apache License, Version 2.0"
declared_licenses_processed:
spdx_expression: "Apache-2.0"
mapped:
Apache License, Version 2.0: "Apache-2.0"
description: "Apache Commons Lang, a package of Java utility classes for the\n\
\ classes that are in java.lang's hierarchy, or are considered to be so\n\
\ standard as to justify existence in java.lang."
homepage_url: "http://commons.apache.org/proper/commons-lang/"
binary_artifact:
url: "https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar"
hash:
value: "6c6c702c89bfff3cd9e80b04d668c5e190d588c6"
algorithm: "SHA-1"
source_artifact:
url: "https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5-sources.jar"
hash:
value: "f7d878153e86a1cdddf6b37850e00a9f8bff726f"
algorithm: "SHA-1"
vcs:
type: "Git"
url: "http://git-wip-us.apache.org/repos/asf/commons-lang.git"
revision: "LANG_3_5"
path: ""
vcs_processed:
type: "Git"
url: "http://git-wip-us.apache.org/repos/asf/commons-lang.git"
revision: "LANG_3_5"
path: ""
- id: "Maven:org.apache.commons:commons-text:1.1"
purl: "pkg:maven/org.apache.commons/commons-text@1.1"
declared_licenses:
- "Apache License, Version 2.0"
declared_licenses_processed:
spdx_expression: "Apache-2.0"
mapped:
Apache License, Version 2.0: "Apache-2.0"
description: "Apache Commons Text is a library focused on algorithms working\
\ on strings."
homepage_url: "http://commons.apache.org/proper/commons-text/"
binary_artifact:
url: "https://repo.maven.apache.org/maven2/org/apache/commons/commons-text/1.1/commons-text-1.1.jar"
hash:
value: "c336bf600f44b88af356c8a85eef4af822b06a4d"
algorithm: "SHA-1"
source_artifact:
url: "https://repo.maven.apache.org/maven2/org/apache/commons/commons-text/1.1/commons-text-1.1-sources.jar"
hash:
value: "f0770f7f0472bf120ada47beecadce4056fbd20a"
algorithm: "SHA-1"
vcs:
type: "Git"
url: "http://git-wip-us.apache.org/repos/asf/commons-text.git"
revision: ""
path: ""
vcs_processed:
type: "Git"
url: "http://git-wip-us.apache.org/repos/asf/commons-text.git"
revision: ""
path: ""
- id: "Maven:org.apache.struts:struts2-assembly:2.5.14.1"
purl: "pkg:maven/org.apache.struts/struts2-assembly@2.5.14.1"
declared_licenses:
- "The Apache Software License, Version 2.0"
declared_licenses_processed:
spdx_expression: "Apache-2.0"
mapped:
The Apache Software License, Version 2.0: "Apache-2.0"
description: "Apache Struts 2"
homepage_url: "http://struts.apache.org/struts2-assembly/"
binary_artifact:
url: "https://repo.maven.apache.org/maven2/org/apache/struts/struts2-assembly/2.5.14.1/struts2-assembly-2.5.14.1-min-lib.zip"
hash:
value: "8e75a38e3b8ceb01e007c5899d8d29e7a075cb7d"
algorithm: "SHA-1"
source_artifact:
url: ""
hash:
value: ""
algorithm: ""
vcs:
type: "Git"
url: "https://gitbox.apache.org/repos/asf/struts.git"
revision: "STRUTS_2_5_14_1"
path: ""
vcs_processed:
type: "Git"
url: "https://gitbox.apache.org/repos/asf/struts.git"
revision: "STRUTS_2_5_14_1"
path: ""
is_metadata_only: true
- id: "Maven:org.hamcrest:hamcrest-core:1.3"
purl: "pkg:maven/org.hamcrest/hamcrest-core@1.3"
declared_licenses:
- "New BSD License"
declared_licenses_processed:
spdx_expression: "BSD-3-Clause"
mapped:
New BSD License: "BSD-3-Clause"
description: "This is the core API of hamcrest matcher framework to be used\
\ by third-party framework providers. This includes the a foundation set of\
\ matcher implementations for common operations."
homepage_url: "https://github.com/hamcrest/JavaHamcrest/hamcrest-core"
binary_artifact:
url: "https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar"
hash:
value: "42a25dc3219429f0e5d060061f71acb49bf010a0"
algorithm: "SHA-1"
source_artifact:
url: "https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3-sources.jar"
hash:
value: "1dc37250fbc78e23a65a67fbbaf71d2e9cbc3c0b"
algorithm: "SHA-1"
vcs:
type: "Git"
url: "git@github.com:hamcrest/JavaHamcrest.git"
revision: ""
path: ""
vcs_processed:
type: "Git"
url: "ssh://git@github.com/hamcrest/JavaHamcrest.git"
revision: ""
path: ""
has_issues: true
scanner: null
advisor: null
evaluator: null
resolved_configuration: {}
3 changes: 1 addition & 2 deletions advisor/src/test/kotlin/AdvisorTest.kt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ import io.mockk.mockk
import org.ossreviewtoolkit.model.AdvisorResult
import org.ossreviewtoolkit.model.AnalyzerResult
import org.ossreviewtoolkit.model.AnalyzerRun
import org.ossreviewtoolkit.model.CuratedPackage
import org.ossreviewtoolkit.model.Identifier
import org.ossreviewtoolkit.model.OrtResult
import org.ossreviewtoolkit.model.Package
Expand Down Expand Up @@ -133,7 +132,7 @@ private fun createOrtResultWithPackages(packages: Set<Package>): OrtResult =
analyzer = AnalyzerRun.EMPTY.copy(
result = AnalyzerResult(
projects = setOf(Project.EMPTY.copy(id = Identifier.EMPTY.copy(name = "test-project"))),
packages = packages.mapTo(mutableSetOf()) { CuratedPackage(it) }
packages = packages
)
)
)
Expand Down
Loading

0 comments on commit 205bc89

Please sign in to comment.