ScanResult: Add a list of all files with checksum and directories

[fviernau]

A complete list of all file and directory names in the repository along with the checksum can be valueable for various use cases.
For example:

1. Compute the package verification code on-the-fly
2. Implement basic file / dir existence checks without the need for cloning the source code
3. Auto generate excludes based on directory names

Moreover, the serialized scan result is redundant in terms of the file path. Each file path can be listed zero to multiple times, as it is listed once per license and copyright finding. That redundancy can be removed by changing the way license and copyright findings are serialized.

Goal:

A scan result should:

1. contain a full list of all files with checksums
2. A full list of all (empty) dirs, so that directory existance checks can be implemented on top
3. have the redundancy in terms of file paths eliminated

[sschuberth]

Some remarks from my side:

• How do we know what "all" files are? Scanners might only report files with findings (this is also true for ScanCode when run with the --only-findings option), so we cannot generally just use the files from the scanner's raw result.
  • If ORT should itself determine the list of "all" files, then ORT should probably not also calculate the hashes for all these files, as e.g. ScanCode already does that for files contained in its raw result. So as a performance optimization, ORT should only calculate the hashes of the remaining files.
• We should be prepared to calculate multiple different hashes per file. SHA1 obviously makes sense, but so does SHA1GIT (for lookup at Software Heritage).
• In general, starting to list all scanned files in the scan result, including those without findings, will make the scan result grow quite a bit. Does it still make sense to only add files with findings to the result, and only the hashes for those?

[fviernau]

We've discussed my proposal with the core devs, here's the proposed changed to the OrtResult model,
and some notes from the meeting.

Planned changes in ORT result model

The following things I believe are somewhat on the agenda and will impact
the model, so these should be considered to avoid conflicts or going into the wrong direction:

1. Have scan results per provenance to drop redundancy, e.g. map package IDs to provenances.
2. Have a file list with hashes for all files for each resolved provenance
3. Have sniplet findings

Current data structure

OrtResult {
  - scanner: ScannerRun { 
    - startTime
    - endTime
    - environment
    - config
    - results: ScanRecord {
      - storageStats
      - scanResults: Map<Id, List<ScanResult>>
    }
  }     
}    
   
ScanResult {
  - provenance
  - scanner: ScannerDetails
  - additionalData
  - summary: ScanSummary {
    - startTime:
    - endTime:
    - packageVerificationCode
    - licenseFindings []
    - copyrightFindings []
    - issues []
  }
}   

Target data structure

OrtResult {
  - scanner: ScannerRun { 
    - startTime
    - endTime
    - environment
    - config
    - provenances: Map<Id, Map<String, Provenance>> <-- resolved provenances
    - storageStats
    - scanResults: Map<Provenance (resolved), ScanResult>  <-- if no scan result, then no entry.
  }     
}

ScanResult {
  - provenance  (with resolved revision)
  - file listing of all files with hashes   
  - summaries[]: ScanSummary {
    - startTime:
    - endTime:
    - licenseFindings []     <--- references file listing
    - copyrightFindings []   <--- references file listing
    - snipletFindings []     <--- references file listing
    - issues []
    - scanner: ScannerDetails
    - additionalData
  }
}   

some conclusions around above data structure:

• Check if there is a reason in code why we have the ScannerRecord class, e.g. tests.
• File listing must be stored in storage in MVP.
• idea: do not require complete file lists in the first steps.
• per provenance: implement only after removal of classic scanner
• storage data structures should be considered separately, and can be de-coupled. Different models can be used.

[sschuberth]

FYI @mmurto, this is what would also enable "delta-scans" with ScanCode of only the modified files.

[fviernau]

Implementation will be covered by this epic: #6945

[sschuberth]

Implementation will be covered by this epic: #6945

As the epic was just closed, can this be closed now, too?

[sschuberth]

Implementation will be covered by this epic: #6945

As the epic was just closed, can this be closed now, too?

Ping @fviernau.

[fviernau]

Closing as done, as now all files are listed for all provenances in OrtResult, which are within the respective VCS path.
Note: empty directories are not listed (mentioned in ticket description). But that in most of the cases doesn't work anyway because Git does not have the concept of directories.