[Project] Handle thousands of node or edge types

[kmcginnes]

There have been a couple of reports of errors due to queries simply being too large. This is most likely caused by the queries in the system that list out every single node type or edge type to fulfill its function. When you have thousands of node or edge types, these queries can become quite large.

There are also some UI elements that list all the node or edge types. Thes likely have performance issues.

Queries that enumerate all node or edge types

• Fetch node properties for each node type
• Fetch edge properties for each edge type
• Keyword search for all node types

Related issues

• Bug QueryTooLargeException #566
• Related [Project] Optimize database queries #358
  • [Project] Optimize openCypher queries #372
  • [Project] Optimize SPARQL queries #371
  • [Project] Optimize Gremlin queries #370
• Part 2 [Project] Handle thousands of node or edge types (Part 2) #601

Tasks

• [Task] Create a database with thousands of nodes #591
• [Task] Gremlin - Split node property fetch in to batches #592
• [Task] Gremlin - Split edge property fetch in to batches #593
• [Task] Gremlin - Remove search filter for node type when all types selected #594
• [Task] openCypher - Remove search filter for node type when all types selected #595
• [Task] SPARQL - Remove search filter for node type when all types selected #596
• [Task] Simplify "All" node types search query #533

[kmcginnes]

This video shows some of the lag from a database with 3,000 node types and 1,800 edge types.

• Apple M1 Max
• 64 GB

CleanShot.2024-09-09.at.10.34.26.mp4