scraper: Scraper global statistics cache optimization

[jamescowens]

This PR timplements a significant global statistics cache memory optimization for the scraper. Note that this cache is used on all nodes, whether they are scrapers or normal nodes.

The current scraper global ConvergedScraperStatsCache, in the interest of simplicity, was designed to copy the CScraperManifest parts into the ConvergedManifestPartsMap indexed by project to make the ConvergedScraperStatsCache relatively independent of the scraper_net global manifest and parts map. This was to simplify object lifecycle management.

Review of the scraper shows this is an area ripe for optimization, since the number of unique parts is actually quite low in practice (even smaller than the number of manifests that reference them, because of the overlapping nature of the manifests!).

Given that, I have determined that it is worth the effort to convert the ConvergedScraperStatsCache to use pointers to the parts in CSplitBlob::mapParts via ConvergedManifestPartPtrsMap, which is a replacement for ConvergedManifestPartsMap, which actually contained copies of the part data. Note that the index is still the projects, but the element is of type CPart* instead of CSerializeData. This map holds the CPart* pointers directly, while the shared pointer to the CScraperManifest Convergences (both present via ConvergedManifest Convergence and past via std::map<uint32_t, std::pair<NN::QuorumHash, ConvergedManifest>> PastConvergences) hold the "references" to the CParts, indirectly through the vPart vector in the manifests. This safeguards their deletion and ensures the pointers are valid for a read only pattern (which is what is used by both the node side of the scraper AND the quorum/superblock classes) until both the Convergences are purged from the global cache AND the CScraperManifest::mapManifest global cache in scraper_net.cpp (with is the manifest level map that holds inventory system manifests and parts), thereby calling the relevant destructors and allowing removal of the CPart objects from the CSplitBlob::mapParts (and invalidation of the pointers) when the last reference to a given CPart is removed. Adding parts to the global scraper_net parts map CSplitBlob::mapParts is ALWAYS done through CSplitBlob::addPart with a lock held on both CScraperManifest::cs_mapManifest and CSplitBlob::cs_mapParts.

The quorum and superblock classes operate on manifests and parts in the ConvergedScraperStatsCache or directly on the CScraperManifest::mapManifest and CSplitBlob::mapParts via appropriate locks, which are also respected by the scraper thread and the scraper_net thread, so this is thread-safe.

This optimization is completely transparent to the operation of a scraper or a node, except it will lower memory usage.

This also includes the scraperreport RPC to gain visibility into the scraper object internals that are relevant to this PR.

I also slipped in a trivial fix to eliminate a possible uninitialized warning message during compilation.

[jamescowens]

@cyrossignol This appears to be working well on one of my test nodes. I have one failing test in the superblock tests that I just don't understand. It is failing with

Running 437 test cases...
unknown location(0): fatal error: in "Superblock/it_initializes_from_a_fallback_by_project_scraper_convergnce": memory access violation at address: 0x00000012: no mapping at fault address
test/neuralnet/superblock_tests.cpp(609): last checkpoint: "it_initializes_from_a_fallback_by_project_scraper_convergnce" test entry

*** 1 failure is detected in the test module "Gridcoin Test Suite"

[jamescowens]

I think I may know what is going on here. There are no global caches for the parts and manifests in the test suite version, and so the pointers may be invalid in that test that is failing. I may have to populate the global scraper_net maps CScraperManifest::mapManifest and CSplitBlob::mapParts via manifest creation and calling of addPart to properly hold the references to the CPart objects in the test suite and ensure the pointers remain valid.

[jamescowens]

Making progress on the tests... only two test failures left... and I know why. Just have to fix up the expected serialization...

Running 437 test cases...
test/neuralnet/superblock_tests.cpp(1148): error: in "Superblock/it_serializes_to_a_stream_for_fallback_convergences": check { stream.begin(), stream.end() } == { expected.begin(), expected.end() } has failed.
Mismatch at position 106: '=' != 0xffffffd0
Mismatch at position 0x6b: 0xffffff88 != 0xffffff8b
Mismatch at position 0x6c: 0xffffff9a != 0x1f
Mismatch at position 0x6d: 'R' != '*'
test/neuralnet/superblock_tests.cpp(1931): error: in "Superblock__ProjectIndex/it_serializes_to_a_stream_for_fallback_convergences": check output == expected has failed

*** 2 failures are detected in the test module "Gridcoin Test Suite"

[jamescowens]

Tests fixed.

[jamescowens]

Here is the optimization results about an hour into a scraper node run on testnet...

jco@jco-linux:~/.GridcoinResearchTest/testnet> gridcoinresearchd-4.0.6.42-integrated_scraper_2-62 -testnet -datadir=/home/jco/.GridcoinResearchTest scraperreport
{
"global_scraper_net": {
"manifest_map_size": 270,
"parts_map_size": 83
},
"converged_scraper_stats_cache": {
"current_convergence_publishing_scrapers": 8,
"current_convergence_part_pointer_map_size": 19,
"past_convergence_map_size": 7,
"total_convergences_part_pointer_maps_size": 152,
"total_convergences_part_unique_pointer_maps_size": 42,
"part_objects_reduced": 110
}
}

The total_convergences_part_unique_pointer_maps_size will not exceed the parts_map_size since the parts_map_size is the actual number of unique parts, which corresponds to the number of unique pointers. The total_convergences_part_pointer_maps_size is the number of parts that would be stored in the global stats cache, so the difference is the number of parts saved from being in memory. This is an excellent optimization!

The scraper is part of the convergence on testnet, testnewsb is passing, and the node is stable. Very happy about this.

[jamescowens]

Here are the statistics after the complete scraper run...

jco@jco-linux:~/.GridcoinResearchTest/testnet> gridcoinresearchd-4.0.6.42-integrated_scraper_2-62 -testnet -datadir=/home/jco/.GridcoinResearchTest scraperreport
{
"global_scraper_net": {
"manifest_map_size": 265,
"parts_map_size": 67
},
"converged_scraper_stats_cache": {
"current_convergence_publishing_scrapers": 8,
"current_convergence_part_pointer_map_size": 19,
"past_convergence_map_size": 12,
"total_convergences_part_pointer_maps_size": 247,
"total_convergences_part_unique_pointer_maps_size": 49,
"part_objects_reduced": 198
}
}
Actually, I should change the parts_objects_reduced to be the same as total_convergences_part_pointer_maps_size, because the unique pointer maps POINT TO elements in the global scraper net map, so they use NO additional space.

Notice that after one 4 hour scraper run, the number of unique parts pointed to by the global cache is 49. If the scraper node is not rebooted between runs, then after the second run, the number of unique pointers should be approximately the same, or the same as the parts_map_size.

[jamescowens]

@cyrossignol and I are running this and it is running very smoothly and saves quite a bit of memory already, and will be even more important as the network grows. Given that we are going to put this in Fern, rather than waiting for Gladys.

[jamescowens]

I am still working on the locking on this.

[jamescowens]

Locks like the locking is good. Seeing stablility across all nodes now. Will see through the SB cycle today (8/22).

[jamescowens]

I squashed the commits for merging.