NVD API request failures are occurring; retrying request for the n time #6515

tbattisti · 2024-03-14T08:54:14Z

Hi,
I'm having problems updating the database with the new CVEs.
The version is 9.0.9.
I'm using NVD api key.

Below the errors:

2024-03-14 09:13:04.616 DEBUG 35944 --- [ck-scheduling-1] i.g.j.o.client.nvd.NvdCveClient          : requesting URI: https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-01-25T12%3A15%3A46Z&lastModEndDate=2024-05-24T12%3A15%3A46Z&resultsPerPage=2000&startIndex=0
2024-03-14 09:13:06.827  WARN 35944 --- [ient-dispatch-5] i.g.j.o.client.nvd.NvdApiRetryStrategy   : NVD API request failures are occurring; retrying request for the 5 time
2024-03-14 09:13:07.167  WARN 35944 --- [ient-dispatch-6] i.g.j.o.client.nvd.NvdApiRetryStrategy   : NVD API request failures are occurring; retrying request for the 6 time
2024-03-14 09:13:07.536  WARN 35944 --- [ient-dispatch-7] i.g.j.o.client.nvd.NvdApiRetryStrategy   : NVD API request failures are occurring; retrying request for the 7 time
2024-03-14 09:13:07.911  WARN 35944 --- [ient-dispatch-8] i.g.j.o.client.nvd.NvdApiRetryStrategy   : NVD API request failures are occurring; retrying request for the 8 time
2024-03-14 09:13:08.264  WARN 35944 --- [ient-dispatch-9] i.g.j.o.client.nvd.NvdApiRetryStrategy   : NVD API request failures are occurring; retrying request for the 9 time
2024-03-14 09:13:08.593  WARN 35944 --- [ent-dispatch-10] i.g.j.o.client.nvd.NvdApiRetryStrategy   : NVD API request failures are occurring; retrying request for the 10 time
2024-03-14 09:13:08.947  WARN 35944 --- [ent-dispatch-11] i.g.j.o.client.nvd.NvdApiRetryStrategy   : NVD API request failures are occurring; retrying request for the 11 time
2024-03-14 09:13:09.135 DEBUG 35944 --- [ck-scheduling-1] i.g.j.o.client.nvd.NvdCveClient          : Error retrieving the NVD data

java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.getCompletedFuture(NvdCveClient.java:412)
	at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:321)
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349)
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116)
	at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
	at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
	at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
	at xxx.analyze(DependencyCheckService.java:350)
	at xxx.analyze(DependencyCheckService.java:160)
	at xxx.analysisJob(QueueService.java:72)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.ExecutionException: org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
	at org.apache.hc.core5.concurrent.BasicFuture.getResult(BasicFuture.java:72)
	at org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:85)
	at io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient.delayedExecute(RateLimitedClient.java:201)
	at io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient.lambda$execute$0(RateLimitedClient.java:172)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	... 3 common frames omitted
Caused by: org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
	at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:350)
	at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
	at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
	at org.apache.hc.core5.reactor.ssl.SSLIOSession.decryptData(SSLIOSession.java:575)
	at org.apache.hc.core5.reactor.ssl.SSLIOSession.access$400(SSLIOSession.java:72)
	at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.inputReady(SSLIOSession.java:172)
	at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:133)
	at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
	at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178)
	at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127)
	at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:85)
	at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
	... 1 common frames omitted

When i try to get informations with curl, no problems arise:

curl -L 'https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-01-25T12%3A15%3A46Z&lastModEndDate=2024-05-24T12%3A15%3A46Z&resultsPerPage=2000&startIndex=0' \
-H 'apiKey: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
-H 'User-Agent: vulnz'

Is anyone else experiencing this problem?

The text was updated successfully, but these errors were encountered:

jeremylong · 2024-03-14T10:50:40Z

Try increasing the delay. Also after you get the initial download successful - if you keep the data directory around you only have to download a small subset of the data (just the updated/new entries) so it'll be much faster/stable after the first download.

tbattisti · 2024-03-14T11:41:11Z

Hi Jeremy, thank you.
I tried with:

settings.setIntIfNotNull(Settings.KEYS.NVD_API_DELAY, 60000);

but seems that problems persist.

Any other advice?

wei-qiang · 2024-03-20T08:30:42Z

Hi,
Have you found a solution yet? I am encountering the same issue.

jeremylong · 2024-03-20T10:11:40Z

@tbattisti you appear to be using the internal API. No clue what other settings you are using or how your env is setup - so I have no advice.

jeremylong · 2024-03-20T10:13:19Z

@wei-qiang use an API key, make sure the API key is valid, make sure you can hit the API from whatever machine is running ODC (use curl/wget). Increase the delay. Once you've downloaded the data the first - don't throw it away. Persist the data directory somehow. See https://jeremylong.github.io/DependencyCheck/data/cacheh2.html

tbattisti · 2024-03-20T10:25:30Z

Hi @wei-qiang,
it seems that in my case it was a temporary problem.
I relaunched the application today, with the same configurations, and the problem did not recur.
To continue using the application last week i used this flag:

settings.setIntIfNotNull(Settings.KEYS.NVD_API_VALID_FOR_HOURS, xxx);

for temporary skip updates.

OrangeDog · 2024-03-20T15:41:52Z

NVD seems to be having API problems again today.
Returning 503 for everything: https://services.nvd.nist.gov/

AntoineJT · 2024-03-20T16:16:36Z

Hi,
I have the same problem: the NVD NIST API return 503 so that's not on dependency check side.
To clarify, I have no stacktrace, only "retrying request for the n time" things.

The fact is I'm using this plugin in a CI/CD pipeline and it just burned nearly 30 minutes of credit before I realized the pipeline has still not finished (for reference, the whole pipeline takes under 5 minutes to complete).

Maybe you could reduce the default timeout duration?
I don't see any common use-case needing to wait for the API response for 30 minutes. The edge cases could always configure the plugin to change it.

THausherr · 2024-03-21T07:46:29Z

It seems to be dead.
https://ci-builds.apache.org/job/PDFBox/job/PDFBox-2.0.x/1174/console
[INFO] --- dependency-check:9.0.10:check (default) @ pdfbox-parent ---
[INFO] Checking for updates
[WARNING] An NVD API Key was not provided - it is highly recommended to use an NVD API key as the update can take a VERY long time without an API Key
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] NVD API request failures are occurring; retrying request for the 10 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
Build timed out (after 90 minutes). Marking the build as aborted.

jeremylong · 2024-03-21T10:12:06Z

I haven't been having any issues with the API as I have been keeping a cache/datafeed up-to-date. You can see the job running here: https://github.com/dependency-check/DependencyCheck_Builder/actions/runs/8357974490

Following either of these should help:

jesperronn · 2024-03-21T11:02:13Z

I have the same problem now, and add that when testing with curl I actually get a 503 response.

It is unclear for me if my API key is not valid or if this is the same for everybody

MY_API_KEY=xxxx-xxxx-xxxx;
curl -L 'https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-01-25T12%3A15%3A46Z&lastModEndDate=2024-05-24T12%3A15%3A46Z&resultsPerPage=2000&startIndex=0' \
-H 'apiKey:  $MY_API_KEY' \
-H 'User-Agent: vulnz'

returns:

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

OrangeDog · 2024-03-21T11:20:12Z

It's the same for everybody. The NVD API has been down since yesterday. They appear to have run out of money or something.

NIST is currently working to establish a consortium to address challenges in the NVD program and develop improved tools and methods. You will temporarily see delays in analysis efforts during this transition. We apologize for the inconvenience and ask for your patience as we work to improve the NVD program.

jeremylong · 2024-03-21T11:27:22Z

This was refreshed last night: https://dependency-check.github.io/DependencyCheck_Builder/

OrangeDog · 2024-03-21T12:03:26Z

How do you actually get it to continue with stale data? It says it will, but then it doesn't.

[WARNING] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
[ERROR] Unable to continue dependency-check analysis.

jeremylong · 2024-03-21T12:27:10Z

Use --noupdate; however, you have to have a database. In this case it might not exist?

OrangeDog · 2024-03-21T12:31:04Z

I don't want no update though. I want it to try to update all sources, and then continue with what it's got. Like the warning implies is going to happen.

The database was fully updated two days ago, and up-to-date on Sonatype and CISA thanks to overriding nvdValidForHours today. But that's an ad-hoc solution that won't work in CI.

jeremylong · 2024-03-21T12:36:07Z

The error Unable to continue dependency-check analysis only occurs when there is no data in the database. If you think there should be then maybe something is setup wrong?

DependencyCheck/core/src/main/java/org/owasp/dependencycheck/Engine.java

Lines 641 to 643 in 0e183da

    
               ensureDataExists(); 
        
           } catch (NoDataException ex) { 
        
               throwFatalExceptionCollection("Unable to continue dependency-check analysis.", ex, exceptions);

jeremylong · 2024-03-21T12:36:51Z

You shouldn't need the --noupdate it should just continue like the warning said - but in this case it can't continue as there is no data. Not even old data.

OrangeDog · 2024-03-21T12:53:04Z

there is no data. Not even old data.

That is not true. If I add nvdValidForHours=300 or autoUpdate=false then it runs successfully with the existing data.

jeremylong · 2024-03-21T12:59:02Z

All I'm saying is look at the code I linked. I'm not sure how it isn't true... Could be a bug...

OrangeDog · 2024-03-21T13:06:37Z

It is not logging this, so something you're not expecting must be happening.

https://github.com/jeremylong/DependencyCheck/blob/v9.0.10/core/src/main/java/org/owasp/dependencycheck/data/nvdcve/CveDB.java#L1412-L1416

I'll go drop some debug logs on a new issue.

TIBCOrkrajews · 2024-03-21T17:28:49Z

Should this issue be used to request improvement of the retry message? It is not clearly or idiomatically worded. For example (if it means what I think it means):

[WARNING] NVD API request failures are occurring; retrying request with attempt 1 of 5.
[WARNING] NVD API request failures are occurring; retrying request with attempt 2 of 5.
[WARNING] NVD API request failures are occurring; retrying request with attempt 3 of 5.
[WARNING] NVD API request failures are occurring; retrying request with attempt 4 of 5.
[WARNING] NVD API request failures are occurring; retrying request with attempt 5 of 5.

…then fail if the last attempt fails.

I assume that a nvdMaxRetryCount of n means that up to n + 1 requests could be made before giving up.

usergithhubtest · 2024-03-21T18:04:33Z

anyone is seeing this exception: ConnectionClosedException: Connection closed by peer?

taylorbasso · 2024-03-21T18:58:57Z

+1 on improving the retry logic, there is a bug somewhere

[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time

I understand that the NVD api is undergoing maintenance but there should be better retry/timeout/failure flags to continue with whats in the database if updates fail instead of just hanging for hours.

jesli96 · 2024-03-22T18:21:42Z

Hi,
I also experience the same issue. I noticed when using java 11 the NVD API request proceeds, but in java 17 the requests fail.

jesli96 · 2024-03-22T19:38:17Z

It seems, upgrading the open-vulnerability-clients fixed it for me. (The current version in dependency-check-maven:9.0.9 was 5.1.1)

    <dependency>
        <groupId>io.github.jeremylong</groupId>
        <artifactId>open-vulnerability-clients</artifactId>
        <version>6.0.0</version>
        <scope>runtime</scope>
    </dependency>

OrangeDog · 2024-03-22T20:00:51Z

@jesli96 nothing you did fixed anything. The NVD service was struggling heavily from Wednesday, but was mostly fine today.

jeremylong · 2024-03-23T11:04:04Z

NVD API appears to be working okay again. Closing this ticket.

usergithhubtest · 2024-04-01T20:24:20Z

I'm seeing the issue again.
Anyone is facing it?

pavliczandris · 2024-07-01T08:02:01Z

Same for me today.

[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 7 time
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 5 time

Too long with no output (exceeded 10m0s): context deadline exceeded

chadlwilson · 2024-07-01T08:11:26Z

Different root cause, see #6746 (comment)

For this one, ODC itself needs to be released with a new version which everyone needs to upgrade to.

pavliczandris · 2024-07-01T08:14:16Z

Thank you! Looking forward to the new release then!

paul-redwood · 2024-07-02T13:18:26Z

Doesn't the 10.0.0 release fix this?

From https://github.com/jeremylong/DependencyCheck/blob/main/CHANGELOG.md#version-1000-2024-07-01

feat: fix the NVD API related errors by adding cvssV4 support (#6756)

OrangeDog · 2024-07-02T13:34:40Z

@paul-redwood it fixed #6746, yes. That's why it's closed now.

tbattisti added the question label Mar 14, 2024

OrangeDog mentioned this issue Mar 21, 2024

Analysis does not continue after update failure #6535

Open

jeremylong closed this as completed Mar 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NVD API request failures are occurring; retrying request for the n time #6515

NVD API request failures are occurring; retrying request for the n time #6515

tbattisti commented Mar 14, 2024

jeremylong commented Mar 14, 2024

tbattisti commented Mar 14, 2024

wei-qiang commented Mar 20, 2024

jeremylong commented Mar 20, 2024

jeremylong commented Mar 20, 2024

tbattisti commented Mar 20, 2024

OrangeDog commented Mar 20, 2024

AntoineJT commented Mar 20, 2024 •

edited

Loading

THausherr commented Mar 21, 2024

jeremylong commented Mar 21, 2024

jesperronn commented Mar 21, 2024

OrangeDog commented Mar 21, 2024

jeremylong commented Mar 21, 2024

OrangeDog commented Mar 21, 2024

jeremylong commented Mar 21, 2024

OrangeDog commented Mar 21, 2024 •

edited

Loading

jeremylong commented Mar 21, 2024

jeremylong commented Mar 21, 2024

OrangeDog commented Mar 21, 2024 •

edited

Loading

jeremylong commented Mar 21, 2024

OrangeDog commented Mar 21, 2024

TIBCOrkrajews commented Mar 21, 2024

usergithhubtest commented Mar 21, 2024

taylorbasso commented Mar 21, 2024 •

edited

Loading

jesli96 commented Mar 22, 2024

jesli96 commented Mar 22, 2024

OrangeDog commented Mar 22, 2024

jeremylong commented Mar 23, 2024

usergithhubtest commented Apr 1, 2024

pavliczandris commented Jul 1, 2024

chadlwilson commented Jul 1, 2024

pavliczandris commented Jul 1, 2024

paul-redwood commented Jul 2, 2024

OrangeDog commented Jul 2, 2024

NVD API request failures are occurring; retrying request for the n time #6515

NVD API request failures are occurring; retrying request for the n time #6515

Comments

tbattisti commented Mar 14, 2024

jeremylong commented Mar 14, 2024

tbattisti commented Mar 14, 2024

wei-qiang commented Mar 20, 2024

jeremylong commented Mar 20, 2024

jeremylong commented Mar 20, 2024

tbattisti commented Mar 20, 2024

OrangeDog commented Mar 20, 2024

AntoineJT commented Mar 20, 2024 • edited Loading

THausherr commented Mar 21, 2024

jeremylong commented Mar 21, 2024

jesperronn commented Mar 21, 2024

OrangeDog commented Mar 21, 2024

jeremylong commented Mar 21, 2024

OrangeDog commented Mar 21, 2024

jeremylong commented Mar 21, 2024

OrangeDog commented Mar 21, 2024 • edited Loading

jeremylong commented Mar 21, 2024

jeremylong commented Mar 21, 2024

OrangeDog commented Mar 21, 2024 • edited Loading

jeremylong commented Mar 21, 2024

OrangeDog commented Mar 21, 2024

TIBCOrkrajews commented Mar 21, 2024

usergithhubtest commented Mar 21, 2024

taylorbasso commented Mar 21, 2024 • edited Loading

jesli96 commented Mar 22, 2024

jesli96 commented Mar 22, 2024

OrangeDog commented Mar 22, 2024

jeremylong commented Mar 23, 2024

usergithhubtest commented Apr 1, 2024

pavliczandris commented Jul 1, 2024

chadlwilson commented Jul 1, 2024

pavliczandris commented Jul 1, 2024

paul-redwood commented Jul 2, 2024

OrangeDog commented Jul 2, 2024

AntoineJT commented Mar 20, 2024 •

edited

Loading

OrangeDog commented Mar 21, 2024 •

edited

Loading

OrangeDog commented Mar 21, 2024 •

edited

Loading

taylorbasso commented Mar 21, 2024 •

edited

Loading