Tests fail during feed downloads and others finished with setup errors in the VD suite #4370

roronoasins · 2023-07-28T09:27:53Z

Description

During the 4.5.0 Alpha 1 Integration testing, some Vulnerability tests failed:

test_vulnerability_detector/test_feeds/test_download_feeds.py
- SUSE Linux Enterprise Server 12
- SUSE Linux Enterprise Server 15
- SUSE Linux Enterprise Desktop 15

The error reported is a TimeoutError when monitoring the log (.*) feed finished log, whereas the expected values are:

SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Desktop 15

This issue needs to be investigated and addressed to ensure the proper functioning of the Vulnerability tests for the mentioned distributions. It's possible that the timeout duration does not align with the download time required for the mentioned feeds.

The text was updated successfully, but these errors were encountered:

mauromalara · 2023-07-28T20:09:31Z

Evidence of the test

HTML-report.zip
TXT-report.txt

Analysis of errors in the test's setup

There are also some tests with errors:

ERROR test_vulnerability_detector/test_feeds/test_validate_feed_content.py::test_validate_xml_feed_content[Red Hat Enterprise Linux0]
ERROR test_vulnerability_detector/test_feeds/test_validate_feed_content.py::test_validate_xml_feed_content[Canonical Focal]
ERROR test_vulnerability_detector/test_feeds/test_validate_feed_content.py::test_validate_xml_feed_content[Canonical Bionic]
ERROR test_vulnerability_detector/test_feeds/test_validate_feed_content.py::test_validate_xml_feed_content[Canonical Xenial]
ERROR test_vulnerability_detector/test_feeds/test_validate_feed_content.py::test_validate_xml_feed_content[Canonical Trusty]

The test has a fixture to download the feed before the test starts and then it decompress the data from the feed. But, when the provider has changed the URL or the file is not available, the test will fail as it tries to decompress HTML data, for instance:

data = b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>N...server.</p>\n<hr>\n<address>Apache/2.4.29 (Ubuntu) Server at people.canonical.com Port 443</address>\n</body></html>\n'

    def decompress(data):
        """Decompress a block of data.
    
        For incremental decompression, use a BZ2Decompressor object instead.
        """
        results = []
        while data:
            decomp = BZ2Decompressor()
            try:
>               res = decomp.decompress(data)
E               OSError: Invalid data stream

Conclusion

That test is flaky because it does not handle this kind of error at startup preventing the test from being executed. The test should control this kind of scenario and let the test continue its normal execution.

⚠️ Anyway, I don't think we should download the feed here. I think that this is not something related to our product and we are trying to verify Wazuh's capabilities.

mauromalara · 2023-07-28T20:29:31Z

Evidence of the test

HTML-report.zip
TXT-report.txt

Analysis of failures in the test

FAILED test_vulnerability_detector/test_feeds/test_download_feeds.py::test_download_feeds[SUSE Linux Enterprise Server 12]
FAILED test_vulnerability_detector/test_feeds/test_download_feeds.py::test_download_feeds[SUSE Linux Enterprise Server 15]
FAILED test_vulnerability_detector/test_feeds/test_download_feeds.py::test_download_feeds[SUSE Linux Enterprise Desktop 15]

Test timeout: 6 minutes
Wazuh manager timeout: 5 minutes

So, when I tried to get the feed through wget:

$ wget https://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.12.xml
--2023-07-28 16:54:35--  https://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.12.xml
Resolving ftp.suse.com (ftp.suse.com)... 195.135.221.132
Connecting to ftp.suse.com (ftp.suse.com)|195.135.221.132|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 185753599 (177M) [text/xml]
Saving to: ‘suse.linux.enterprise.server.12.xml’

      suse.linux.enterprise.s   0%[                                                 ] 447.49K  70.4KB/s    eta 42m 49s

The ETA is above the test and manager timeouts. So, the test is failing as expected. Also, the manager is detecting this timeout and reporting it in the logs:

DEBUG: CURL ERROR: Operation timed out after 300000 milliseconds with 72121841 out of 185753599 bytes received

Conclusion

The failures are not related to the test itself or the product but to SUSE's servers.

⚠️ Anyway, I don't think this test is well designed as we cannot infinitely wait for the vendor to return their vulnerability information. I think that this test should verify that Wazuh is able to download a feed, but not a production one, and also the test should verify that Wazuh can handle timeouts or format errors.

We cannot say that a test fails because a provider has a problem. Because we want to test our product.

davidjiglesias · 2023-08-01T12:05:03Z

Based on the conclusions above from @mauromalara I will move this issue to @wazuh/core for them to fix or refactor accordingly.

Deblintrake09 · 2023-08-07T22:19:41Z

This issue errorss where present in 4.5.1 - Alpha 1 wazuh/wazuh#18286

MarcelKemp · 2023-09-06T16:30:34Z

This issue is related to the following development:

SUSE OVAL feeds download are too slow wazuh#18593

So they should be fixed after completion.

MarcelKemp · 2023-09-14T11:26:17Z

Done in PR: Update SUSE OVAL URL to download compressed feeds wazuh#18783 (4.5.3)

roronoasins added level/task Task issue type/bug labels Jul 28, 2023

roronoasins mentioned this issue Jul 28, 2023

Release 4.5.0 - Alpha 1 - Integration tests wazuh/wazuh#18114

Closed

2 tasks

juliamagan added the test/integration label Jul 28, 2023

mauromalara self-assigned this Jul 28, 2023

mauromalara changed the title ~~SUSE Download Feed tests failing with TimeoutError~~ Some tests fail with TimeoutError and other finished with errors in the VD suite Jul 28, 2023

mauromalara changed the title ~~Some tests fail with TimeoutError and other finished with errors in the VD suite~~ Tests fail with TimeoutError and other finished with errors in the VD suite Jul 28, 2023

mauromalara changed the title ~~Tests fail with TimeoutError and other finished with errors in the VD suite~~ Tests fail while downloading feeds and other finished with setup errors in the VD suite Jul 28, 2023

davidjiglesias changed the title ~~Tests fail while downloading feeds and other finished with setup errors in the VD suite~~ Tests fail during feed downloads and others finished with setup errors in the VD suite Jul 31, 2023

juliamagan mentioned this issue Aug 2, 2023

Release 4.5.0 - Alpha 1 wazuh/wazuh#18058

Closed

Deblintrake09 mentioned this issue Aug 4, 2023

Release 4.5.0 - RC 1 - Integration tests wazuh/wazuh#18259

Closed

1 task

Deblintrake09 mentioned this issue Aug 7, 2023

Release 4.5.1 - Alpha 1 - Integration tests wazuh/wazuh#18286

Closed

1 task

damarisg mentioned this issue Aug 9, 2023

Release 4.5.1 - Alpha 1 wazuh/wazuh#18224

Closed

Rebits mentioned this issue Aug 14, 2023

Release 4.5.1 - RC 1 - Integration tests wazuh/wazuh#18383

Closed

1 task

damarisg mentioned this issue Aug 15, 2023

Release 4.5.1 - RC 1 wazuh/wazuh#18366

Closed

Deblintrake09 mentioned this issue Aug 18, 2023

Release 4.5.1 - RC 2 - Integration tests wazuh/wazuh#18492

Closed

1 task

damarisg mentioned this issue Aug 18, 2023

Release 4.5.1 - RC 2 wazuh/wazuh#18475

Closed

mateocervilla mentioned this issue Aug 24, 2023

SUSE OVAL feeds download are too slow wazuh/wazuh#18593

Closed

BelenValdivia mentioned this issue Aug 28, 2023

Release 4.5.2 - RC 1 - Integration tests wazuh/wazuh#18622

Closed

1 task

mauromalara added the qa_known Issues that are already known by the QA team label Aug 29, 2023

mateocervilla self-assigned this Sep 6, 2023

This was linked to pull requests Sep 6, 2023

Update SUSE OVAL URL for offline update wazuh/wazuh-documentation#6435

Merged

Update ITs URL for SUSE OVAL #4496

Merged

Update SUSE OVAL URL to download compressed feeds wazuh/wazuh#18783

Merged

rauldpm mentioned this issue Sep 11, 2023

Release 4.6.0 - Alpha 1 - E2E UX tests - Demo environment wazuh/wazuh#18843

Closed

10 tasks

mauromalara mentioned this issue Sep 12, 2023

Release 4.6.0 - Alpha 1 - Integration tests wazuh/wazuh#18941

Closed

2 tasks

MarcelKemp closed this as completed Sep 14, 2023

fcaffieri mentioned this issue Sep 26, 2023

Release 4.5.3 - RC 1 - E2E UX tests - Demo environment wazuh/wazuh#19109

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests fail during feed downloads and others finished with setup errors in the VD suite #4370

Tests fail during feed downloads and others finished with setup errors in the VD suite #4370

roronoasins commented Jul 28, 2023 •

edited by mauromalara

Loading

mauromalara commented Jul 28, 2023 •

edited

Loading

mauromalara commented Jul 28, 2023 •

edited

Loading

davidjiglesias commented Aug 1, 2023

Deblintrake09 commented Aug 7, 2023

MarcelKemp commented Sep 6, 2023

MarcelKemp commented Sep 14, 2023

Tests fail during feed downloads and others finished with setup errors in the VD suite #4370

Tests fail during feed downloads and others finished with setup errors in the VD suite #4370

Comments

roronoasins commented Jul 28, 2023 • edited by mauromalara Loading

Description

mauromalara commented Jul 28, 2023 • edited Loading

Evidence of the test

Analysis of errors in the test's setup

Conclusion

mauromalara commented Jul 28, 2023 • edited Loading

Evidence of the test

Analysis of failures in the test

Conclusion

davidjiglesias commented Aug 1, 2023

Deblintrake09 commented Aug 7, 2023

MarcelKemp commented Sep 6, 2023

MarcelKemp commented Sep 14, 2023

roronoasins commented Jul 28, 2023 •

edited by mauromalara

Loading

mauromalara commented Jul 28, 2023 •

edited

Loading

mauromalara commented Jul 28, 2023 •

edited

Loading