[Project] Optimize database queries

[kmcginnes]

We can trace many reliability and performance issues to sub-optimal database queries. Here, we attempt to tackle some of the most impactful database queries in the app.

Plan

The general approach is to reduce the complexity of the database queries we are performing. At the same time, we can defer performing some of the queries until the user interacts with the UI that requires that data.

Also, by using TanStack Query for the database queries we can regain caching of the queries. This will also provide good options for automatic and manual retries.

Related Issues

Tasks

• [Project] Update libraries #357
• [Project] Database queries should use React Query #304
• [Project] Avoid loading information prematurely #369
• [Project] Optimize Gremlin queries #370
• [Project] Optimize SPARQL queries #371
• [Project] Optimize openCypher queries #372

Addresses

• [Bug] RDF: Adding resource to the canvas is very slow or fails with out of memory #263
• [Bug] Neighbor expansion is slow and unreliable #324
• [Feature Request] Improve SPARQL query performance #240
• [Bug] Gremlin Connector Timeout when Fetching Vertex Schema upon Database Synchronizing #225
• [Feature Request] Improve data load performance targeting 20% (2024 Roadmap item) #301
• [Feature Request] Retry failed requests #269
• [Feature Request] Prefer HTTP GET over POST #401

Consider

These issues may be partially or completely solved by the effort in this issue.

• [Bug] Using the air-routes data set the DFW airport does not expand correctly #314
• [Bug] Value in "Limit returned neighbors to" is exceeded (SPARQL ENDPOINT) #184
• [Bug] 413 Payload Too Large #410