Port of the Bitcoin Logger to Gridcoin

[jamescowens]

This is the initial commit to port Bitcoin's Logger class to
Gridcoin. Templating has been used to handle the legacy string categories.

[jamescowens]

Ok. This works and actually is outputting as expected. Currently have the category mask nailed up to uint32_t ~0 which is "ALL" in the InitLogging(), because I haven't done the category mapping yet. [I have added our categories now and have the proper stuff in init to pick them up as args and also the new rpc logging function.]

Note that I had to put in a \n in LogInstance().LogPrintStr(log_msg + "\n");` because Gridcoin LogPrintf's don't have LF's in the format string, but Bitcoin's now do, and I want to avoid a whole host of changes all over the place.

[jamescowens]

Our init code is radically different (and more primitive than Bitcoin's more modern approach), so I think I have put the initialization code in the right places. Would certainly like a double check on that.

[jamescowens]

You will notice I went ahead and pulled in threadnames.* and time.* , because the logger needed those, and we need to port them and rerationalize util.* anyway. I had to dumb down a few of the functions because our code is more primitive.

[jamescowens]

Ok. I ported the scraper's log archiver over to the main logger class. Next will be to retire the scraper logger.

Note that the uncategorized LogPrintf's that are used all over Gridcoin will remain that way. I will be figuring out the best way to introduce the BCLog::LogFlags category enum which will eventually replace the debug flags, but it unavoidably will require a search and replace of the if (fDebugX) LogPrintf(...., ....); with LogPrintf(BCLog::LogFlags category, ...., ....);

[jamescowens]

This is an example of running the archivelog for the debug.log and scraper.log...

jowens@jco-linux4:~/.GridcoinResearch/logarchive$ ls -al
total 80
drwx------ 2 jowens jowens 4096 Nov 17 16:30 .
drwxrwxr-x 6 jowens jowens 4096 Nov 17 16:30 ..
-rw------- 1 jowens jowens 53948 Nov 17 16:24 debug-20191117212447.log.gz
-rw------- 1 jowens jowens 1881 Nov 17 16:30 debug-20191117213023.log.gz
-rw------- 1 jowens jowens 6585 Nov 17 16:24 scraper-20191117212411.log.gz
-rw------- 1 jowens jowens 493 Nov 17 16:30 scraper-20191117213002.log.gz

[jamescowens]

With the porting of the scraper archiver, which compresses and rolls logs to a logarchive directory, this BCLog Bitcoin logger port addresses the non-UI part of #571.

[jamescowens]

Scheduler works for archiving main debug.log at the next trigger after 0:00Z.

[nathanielcwm]

Just curious what improvements does the ported logger bring?

[jamescowens]

@nathanielcwm it has a much better categorization system, allowing you to turn on and off catagories of logging much easier. The main logger is a singleton object, so it allows me to implement the log rotation and archiving that I implemented on the scraper log already for the main log. (No more log truncations unless you want it.)

[jamescowens]

This is ready to go. There is a slight conflict in the scheduler with @iFoggz's update PR, but it is easily resolved.

[jamescowens]

These are the only categorized entries we actually have in the codebase...

jco@jco-linux:~/builds/Gridcoin-Research/src> grep -Rn -E "LogPrint\(\".*\",.*,.*\)"
qt/bitcoin.cpp:149: LogPrint("qt", "GUI: %s\n", msg);
qt/bitcoin.cpp:159: LogPrint("qt", "GUI: %s\n", msg.toStdString());
db.cpp:444: LogPrint("db", "Flush(%s)%s", fShutdown ? "true" : "false", fDbEnvInit ? "" : " db not started");
db.cpp:454: LogPrint("db", "%s refcount=%d", strFile, nRefCount);
db.cpp:459: LogPrint("db", "%s checkpoint", strFile);
db.cpp:461: LogPrint("db", "%s detach", strFile);
db.cpp:464: LogPrint("db", "%s closed", strFile);
db.cpp:470: LogPrint("db", "DBFlush(%s)%s ended %15" PRId64 "ms", fShutdown ? "true" : "false", fDbEnvInit ? "" : " db not started", GetTimeMillis() - nStart);
banman.cpp:211: LogPrint("net", "%s: Removed banned node ip/subnet from banlist.dat: %s\n", func, sub_net.ToString());
addrman.h:406: LogPrint("addrman", "ADDRMAN CONSISTENCY CHECK FAILED!!! err=%i", err);
addrman.h:422: LogPrint("addrman","Added %s from %s: %i tried, %i new", addr.ToStringIPPort(), source.ToString(), nTried, nNew);
addrman.h:438: LogPrint("addrman","Added %i addresses from %s: %i tried, %i new", nAdd, source.ToString(), nTried, nNew);
scraper_net.cpp:55: LogPrint("manifest", "received part %s %u refs", hash.GetHex(),(unsigned)part.refs.size());
scraper_net.cpp:69: LogPrint("manifest", "received duplicate part %s", hash.GetHex());
scraper_net.cpp:202: if(pfrom) LogPrint("manifest", "new manifest %s from %s", inv.hash.GetHex(), pfrom->addrName);
scraper_net.cpp:420: LogPrint("manifest", "CScraperManifest::UnserializeCheck: hash of signature = %s", Hash(signature.begin(), signature.end()).GetHex());
scraper_net.cpp:540: LogPrint("manifest", "invalid manifest %s received", hash.GetHex());
scraper_net.cpp:550: LogPrint("manifest", "invalid manifest %s received", hash.GetHex());
scraper_net.cpp:570: LogPrint("manifest", "received manifest %s with %u / %u parts", hash.GetHex(),(unsigned)manifest.cntPartsRcvd,(unsigned)manifest.vParts.size());
scraper_net.cpp:647: LogPrint("manifest", "manifest %s complete with %u parts", phash->GetHex(),(unsigned)vParts.size());
addrman.cpp:305: LogPrint("addrman", "Moving %s to tried", addr.ToString());
util.cpp:200: if (fDebug10) LogPrint("rand", "RandAddSeed() %lu bytes", nSize);
main.cpp:778: LogPrint("mempool", "ignoring large orphan tx (size: %" PRIszu ", hash: %s)", nSize, hash.ToString().substr(0,10));
main.cpp:786: LogPrint("mempool", "stored orphan tx %s (mapsz %" PRIszu ")", hash.ToString().substr(0,10), mapOrphanTransactions.size());
main.cpp:1333: LogPrint("mempool", "Rate limit dFreeCount: %g => %g", dFreeCount, dFreeCount+nSize);
main.cpp:1358: LogPrint("mempool", "AcceptToMemoryPool : replacing tx %s with new version", ptxOld->GetHash().ToString());
main.cpp:1368: if (fDebug) LogPrint("mempool", "AcceptToMemoryPool : accepted %s (poolsz %" PRIszu ")", hash.ToString(), pool.mapTx.size());
alert.cpp:216: LogPrint("alert", "cancelling alert %d", alert.nID);
alert.cpp:222: LogPrint("alert", "expiring alert %d", alert.nID);
alert.cpp:236: LogPrint("alert", "alert already cancelled by %d", alert.nID);
alert.cpp:274: LogPrint("alert", "accepted alert %d, AppliesToMe()=%d", nID, AppliesToMe());

[jamescowens]

Note that I have left them as strings, but they can be changed to the enums because of the templating I did. We will do this, and convert a lot of the fDebug protected LogPrintf's to categorized LogPrints after Fern to minimize a gigantic global PR that would create conflicts for just about every branch.

[jamescowens]

Rebased to the head of development to resolve the conflict. Ready for review.

[jamescowens]

Some of the categories in the enum that are sourced from Bitcoin we may want to get rid off, because we will never? use them. This is a leisure thing, because the categories are a purely local structure.

[cyrossignol]

Looks like this is working well.

@jamescowens Do you have an inventory of the remaining bits we'll need to do to complete the transition?

[jamescowens]

The only thing we have to do is make decisions on the categories we want to implement, and then ensure those are in the enum. We then replace the strings in the few existing categories with the enums for those. The intent is to replace the debug2, 3,4,5, 10 with categories as well, and then those flags (and corresponding rpc functions that set them) will be retired.