Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

First commits to create cyhy-db #5

Draft
wants to merge 101 commits into
base: develop
Choose a base branch
from
Draft

First commits to create cyhy-db #5

wants to merge 101 commits into from
+2,403 −303

Conversation

felddy
Copy link
Member

@felddy felddy commented Sep 25, 2024 •
edited
Loading

🗣 Description

This PR contains the initial functionality of the cyhy-db module, which provides a object model for accessing the various document collections in a Cyber Hygiene database.

This PR also includes an assortment of unit tests, though more testing will likely be added later.

💭 Motivation and context

We have two main motivations here:

  1. To migrate from our legacy data model which was built on outdated Python 2 libraries to a more modern model that uses Python 3 (hello Beanie!)
  2. To break the CyHy database model out into its own repository and module thus making it more testable and maintainable.

🧪 Testing

All unit tests currently pass in pytest. This code has also been partially tested by its use in the cyhy-kevsync module (initial PR coming soon).

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • All future TODOs are captured in issues, which are referenced
    in code comments.
  • All relevant type-of-change labels have been added.
  • I have read the CONTRIBUTING document.
  • These code changes follow cisagov code standards.
  • All relevant repo and/or project documentation has been updated
    to reflect the changes in this PR.
  • Tests have been added and/or modified to cover the changes in this PR.
  • All new and existing tests pass.

✅ Pre-merge checklist

  • Finalize version.

✅ Post-merge checklist

  • Create a release.
  • Create (or move) v1 tag that points to new release.

@felddy @dav3r
Add cyhy_db package and models
53be3c2 
Delete example package and files

Update Python version to 3.12 in build workflows

Update package name and description in setup.py

Update dependencies in setup.py
@felddy @dav3r
Apply changes made by linters
904692b 
Add comments to severity calculation.
@felddy @dav3r
Add doc strings where needed
6593b87
@felddy @dav3r
Simplify field validation
312752b
@felddy @dav3r
Add test case for invalid CVSS score
250d6f6
@felddy @dav3r
Switch to MongoEngine
3f39aae
@felddy @dav3r
Add utcnow function to utils module
8999e26
@felddy @dav3r
Update MongoDB connection and add support for Mongo Express
3a291cd
@felddy @dav3r
Refactor CVE model and test
97359b9 
This commit refactors the CVE model and test in the `cyhy_db` package. The changes include:
- Replacing the `odmantic` library with `mongoengine` for the model definition.
- Updating the field types and constraints in the CVE model.
- Removing the `calculate_severity` method and integrating the severity calculation directly in the `save` method.
- Updating the test cases to reflect the changes in the model.
@felddy @dav3r
Add IPAddressField and test cases for IP address validation and conve…
a568454 
…rsion
@felddy @dav3r
Add ScanDoc and SnapshotDoc models
9c97a24
@felddy @dav3r
Sort imports
09fd02c
@felddy @dav3r
Implement CVE, ScanDoc with beanie
c29996e
@felddy @dav3r
Add cyhy_db package and initialize_db function
08920dd
@felddy @dav3r
Update conftest.py with Docker and mongo express fixtures
69117b4
@felddy @dav3r
Add test for database connection using motor and beanie libraries
4d620e6
@felddy @dav3r
Refactor imports and update test cases in test_cve.py
c7959a9
@felddy @dav3r
Remove test_ip_address.py
d551473
@felddy @dav3r
Add tests for ScanDoc model functionality
577ee7d
@felddy @dav3r
Add asyncio_mode to pytest.ini
f2e99a3
@felddy @dav3r
Switch to beanie ORM as a dependency
89e8303
@felddy @dav3r
Add enum classes for various types and statuses
2cf4587
@felddy @dav3r
Add RequestDoc model and test
43da94b
@felddy @dav3r
Refactor CVE model and add model validator for calculating severity
4162bdb
@felddy @dav3r
Rename ScanDoc test
c45dd48
@felddy @dav3r
Add RequestDoc model to ALL_MODELS list
a26b5c2
@felddy @dav3r
Add new dependencies to setup.py
11b5bb2
@felddy @dav3r
Add test case to reset latest flag by owner
1b1d518
@felddy @dav3r
Sort imports
2433f21
@felddy @dav3r
Refactor ScanDoc model and add new tests
7adf2d1
@dav3r
Add a whole bunch of docstrings to placate flake8
757caa8
@dav3r
Tell bandit to ignore warnings about our use of random numbers
7b6e816 
We aren't using Random() for the purposes of cryptography here, so we can safely ignore these warnings.
@felddy felddy added documentation This issue or pull request improves or adds to documentation hacktoberfest-accepted Pull request that should count toward Hacktoberfest participation improvement This issue or pull request will add or improve functionality, maintainability, or ease of use labels Sep 25, 2024
@felddy felddy self-assigned this Sep 25, 2024
@coveralls
Copy link

coveralls commented Sep 25, 2024 •
edited
Loading

Pull Request Test Coverage Report for Build 11164912521

Details

  • 546 of 546 (100.0%) changed or added relevant lines in 22 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 100.0%
Totals Coverage Status
Change from base Build 10273838208: 0.0%
Covered Lines: 547
Relevant Lines: 547
💛 - Coveralls

dav3r and others added 16 commits October 1, 2024 16:33
@dav3r @felddy
Explicitly set the pytest log level to INFO
030607c 
This is primarily because pymongo DEBUG level logging is extremely verbose and should only be enabled when needed.

Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r @felddy
Restore a useful comment and import from the skeleton
9aad896 
Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r @felddy
Remove a commented out line that is no longer needed
86d8c38 
Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r @felddy
Add two unit tests for the initialize_db function
0afb2ed 
Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r
Correctly set state in HostDoc
ddab65e
@dav3r
Add unit tests for HostDoc.set_state()
9983150
@dav3r
Add unit tests for SystemControlDoc
b510c09
@dav3r
Add additional unit tests for ScanDoc.tag_latest()
5642a7c
@dav3r
Add a "no op" unit test for HostDoc.set_state()
2870a9a
@dav3r
Add unit tests for utils/decorators.py
41fefe8
@dav3r @felddy
Improve field validator for start window in RequestDoc
323666e 
This field now accepts either a string representing the time (%H:%M:%S) or a datetime.time object, which is stored in the database as a string in %H:%M:%S format.

This also improves our input checking so that invalid times (e.g. 34:45:56) can no longer be added.

Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r @felddy
Add unit tests for parse_time in Window (request_doc.py)
3b76e7b 
Also, remove some commented-out code that is no longer needed.

Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r @felddy
Avoid using the reserved "_id" field name in TallyDoc
22ed790 
Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r @felddy
Add unit tests for TallyDoc
6fbef45 
Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r @felddy
Output mongodb URI at end of pytest run when used with --mongo-expres…
058a0c5 
…s option

Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r @felddy
Update README from skeleton to include useful information
d3e4e03 
Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
dav3r and others added 4 commits October 2, 2024 17:31
@dav3r @felddy
Remove a TODO that is no longer relevant
c2157df 
Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r @felddy
Make all children of ScanDoc (HostScanDoc, PortScanDoc, VulnScanDoc) …
7fec381 
…include the same indexes as ScanDoc

Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@dav3r @felddy
Update a comment
aca07b4 
Co-authored-by: Mark Feldhousen <mark.feldhousen@gwe.cisa.dhs.gov>
@felddy
Remove duplicate note text
9e06b47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dav3r dav3r Awaiting requested review from dav3r dav3r will be requested when the pull request is marked ready for review dav3r is a code owner

@jsf9k jsf9k Awaiting requested review from jsf9k jsf9k will be requested when the pull request is marked ready for review jsf9k is a code owner

@mcdonnnj mcdonnnj Awaiting requested review from mcdonnnj mcdonnnj will be requested when the pull request is marked ready for review mcdonnnj is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

@felddy felddy

Labels
documentation This issue or pull request improves or adds to documentation hacktoberfest-accepted Pull request that should count toward Hacktoberfest participation improvement This issue or pull request will add or improve functionality, maintainability, or ease of use
Projects
CyHy Cloud
Status: In Progress
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@felddy @coveralls @dav3r