adding rate limiting for command engine

[stevenwinship]

What this PR does / why we need it: Adds rate limiting to command APIs in order to prevent over taxing of the system

Which issue(s) this PR closes: #9356

• Closes Add execution rate metering to the command engine #9356

Special notes for your reviewer: Creates a redid service in docker but may need install/deploy work for production. Someone with more experience in this please chime in.

Suggestions on how to test this: Test with both superuser and non-superuser as well as guest account.

set up guest (tier 0) as 1 call per hour and authenticated user (tier 1) as 3 calls per hour.
curl http://localhost:8080/api/admin/settings/:RateLimitingDefaultCapacityTiers -X PUT -d '1,3'

PLSQL
INSERT INTO setting (name,content) VALUES (':RateLimitingDefaultCapacityTiers','1,3');
INSERT INTO setting (name,content) VALUES (':RateLimitingCapacityByTierAndAction','{"rateLimits":[{"tier": 0, "limitPerHour": 10, "actions": ["GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand"]}, {"tier": 0, "limitPerHour": 1, "actions": ["CreateGuestbookResponseCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}, {"tier": 1, "limitPerHour": 2, "actions": ["CreateGuestbookResponseCommand", "GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}]}');

make more than 3 calls to a command API to see the exception (create dataset and delete dataset). Wait 30 minutes to see that the call now works.

after testing delete :RateLimitingDefaultCapacityTiers and : RateLimitingCapacityByTierAndAction from setting to turn off/ reset rate limiting to no limit
Note: settings are only read at startup so a restart will be needed to test turning off rate limiting

Does this PR introduce a user interface change? If mockups are available, please link/include them here: Not now

Is there a release notes update needed for this change?: in 9356-rate-limiting.md

Additional documentation:

[coveralls]

coverage: 20.661% (+0.09%) from 20.57%
when pulling a9b2514 on 9356-rate-limiting-command-engine
into 4f46d15 on develop.

[landreev]

I've been educating myself about Redis; I've never used it before. Had a few questions, but was able to figure things out on my own so far. But I will be asking more questions/adding comments today.

[pdurbin]

I've only looked at docs so far but I love the granularity, that each command can have it's own limits.

[pdurbin]

I jumped on the branch and ran this to limit to 3 calls per hour:

curl http://localhost:8080/api/admin/settings/:RateLimitingDefaultCapacityTiers -X PUT -d '1,3'

Then I ran mvn test -Dtest=DatasetsIT figuring that it'll more than three collections, datasets etc.

I got what is perhaps an expected error (though it repeats itself):

{
    "status": "ERROR",
    "message": "Error creating dataverse: Error creating dataverse."
}

However, I'm seeing 500 errors: Expected status code <201> but was <500>. I imagine we shouldn't be throwing 500 errors.

I'm also seeing this in server.log: dev_dataverse> Caused by: jakarta.ejb.TransactionRolledbackLocalException: Exception thrown from bean: java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because "index" is null

Anyway, this was a quick test.

[landreev]

@stevenwinship @poikilotherm What was decided for a multi-server case, like ours? Do we want to invest into figuring out how to share the same hazelcast cache between independent Payara instances - or are we going to be content with each node using its own?

[poikilotherm]

@landreev Payara autoclusters via Hazelcast if the instances can reach each other.

[landreev]

Payara autoclusters via Hazelcast if the instances can reach each other.

@poikilotherm But, "clusters" appears to be the key word here - from spending some time looking into this on Monday, my takeaway was that it would involve changing our config, from independent instances to a payara cluster. And that's kind of a big change, no?
Unless I'm missing some straightforward simpler alternative.

[landreev]

I'd like to figure out, and experiment with a multi-instance setup sharing a cache eventually. I don't think this needs to be done as part of this PR (I started looking into it, but concluded it should be a separate effort) so I don't think we need anything in the guide on this case either at this point.

I have a very practical and simple request though: I want to be able to use this functionality to throttle access to the main, workhorse pages - dataset.xhtml and dataverse.xhtml. So I'd like to have 2 dedicated commands - something like CheckRateLimitForDatasetPage and CheckRateLImitForCollectionPage commands - that don't do anything, just to be executed as the first thing in the init() methods of the respective pages. If the command throws a rate limit exception there, I don't even want the page to do anything special/create any new error pages necessarily - return permissionsWrapper.notAuthorized(); would be perfectly acceptable.

Aside from this, I believe we are ready to merge this, finally. Good-looking code.

[landreev]

For the record, this is the exception that's thrown on consecutive attempts to deploy (without restarting Payara):
Caused by: javax.cache.CacheException: A cache named 'rateLimitCache' already exists

[landreev]

@stevenwinship

For the record, this is the exception that's thrown on consecutive attempts to deploy (without restarting Payara): Caused by: javax.cache.CacheException: A cache named 'rateLimitCache' already exists

Does it just need a
manager.destroyCache(RATE_LIMIT_CACHE);
line in CacheFactoryBean, before trying to create it?

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:9356-rate-limiting-command-engine

ghcr.io/gdcc/configbaker:9356-rate-limiting-command-engine

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:9356-rate-limiting-command-engine

ghcr.io/gdcc/configbaker:9356-rate-limiting-command-engine

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[landreev]

This is not looking good, in that I'm not seeing any traces of my commits that I made yesterday (fixing the 500/internal server errors from the Dataset and Dataverse pages, and renaming the commands). @stevenwinship did you revert my changes by any chance? - I'm looking at:

Not the end of the world, we may just need to make a followup pr.