Overhaul the core tally and accrual system #1583

cyrossignol · 2019-10-30T05:30:27Z

This refactors the tally system used to calculate research rewards. It adds a new set of classes used to track research age accrual and replaces a significant amount of legacy code. As part of the changes, this PR hooks up the new superblock class to the rest of the application.

This work prepares the accrual system for future updates that improve the accuracy of reward calculations based on superblock windows (#768, #1578, #1579). However, I intentionally avoided implementing modifications to these rules in this PR to simplify review and so we can focus on those changes on their own--the new tally system proposed here must be compatible with the current protocol. These changes introduce an IAccrualComputer interface with implementations for the existing research age calculations and provide structure for upcoming improvements.

The design of the new tally system attempts to improve the performance and scalability of research age data structures by eliminating almost all of the unneeded fields present in the old StructCPID objects. It removes duplicated containers and the sets of block index pointers used by the old GetLifetimeCPID() functionality in favor of a running tally that's updated when blocks are connected and disconnected from the main chain. This saves about 200 MB of memory on mainnet.

Deserialization of claims in version 11 blocks will automatically fail if the client version or organization fields are larger than allowed.

This RPC method calls a function that fills staging data structures with superblock data. That data will never be used--the next tally cycle will overwrite it before the accrual code ever sees it.

denravonska · 2019-10-30T08:47:52Z

I get this on my machine:

../../src/neuralnet/tally.cpp: In member function ‘NN::BlockPtrOption NN::ResearchAccount::LastRewardBlock() const’:
../../src/neuralnet/tally.cpp:729:75: error: no matching function for call to ‘make_optional<const CBlockIndex* const>(const CBlockIndex* const&)’
     return boost::make_optional<const CBlockIndex* const>(m_last_block_ptr);
                                                                           ^
In file included from /usr/include/boost/optional.hpp:15,
                 from /usr/include/boost/thread/future.hpp:42,
                 from /usr/include/boost/thread.hpp:24,
                 from ../../src/util.h:29,
                 from ../../src/beacon.h:8,
                 from ../../src/neuralnet/tally.cpp:1:
/usr/include/boost/optional/optional.hpp:1367:56: note: candidate: ‘template<class T> boost::optional<typename boost::decay<T>::type> boost::make_optional(T&&)’
 optional<BOOST_DEDUCED_TYPENAME boost::decay<T>::type> make_optional ( T && v  )
                                                        ^~~~~~~~~~~~~
/usr/include/boost/optional/optional.hpp:1367:56: note:   template argument deduction/substitution failed:
../../src/neuralnet/tally.cpp:729:59: note:   cannot convert ‘((const NN::ResearchAccount*)this)->NN::ResearchAccount::m_last_block_ptr’ (type ‘const CBlockIndex* const’) to type ‘const CBlockIndex* const&&’
     return boost::make_optional<const CBlockIndex* const>(m_last_block_ptr);
                                                           ^~~~~~~~~~~~~~~~

jamescowens

Awesome work on this!

src/gridcoin.cpp

src/neuralnet/neuralnet_native.cpp

src/neuralnet/tally.cpp

src/scraper/scraper.cpp

jamescowens · 2019-10-30T16:32:30Z

@denravonska, what version of boost are you using? I didn't get that error and I am compiling against the same version that depends uses.... 1.65.1

jamescowens · 2019-10-30T16:42:11Z

I have this running on my Linux testnet node. Syncing from zero now.

src/rpcblockchain.cpp

denravonska · 2019-10-30T16:56:47Z

@denravonska, what version of boost are you using? I didn't get that error and I am compiling against the same version that depends uses.... 1.65.1

It looks like 1.67.0.

jamescowens · 2019-10-30T17:36:00Z

Maybe your compiler version?

jamescowens · 2019-10-30T18:39:42Z

Hmm...

jamescowens · 2019-10-30T18:45:27Z

It appears testnet wallets try to stake long before the wallet is actually in sync. This is probably not related to this PR, I suppose it could be....

Legacy fallback research age validation masked these blocks with a short block span.

This check skips the miner loop while the tally runs. Because the tally runs quickly on main thread now, this check is unnecessary. The lock on cs_main will prevent the miner from pulling stale accrual data.

cyrossignol · 2019-10-31T07:42:23Z

@denravonska Please try building again. Looks like Boost removed the overload for make_optional() that takes an l-value reference (the const pointer was impossible to convert to an r-value reference). I removed the const qualifier.

jamescowens · 2019-10-31T15:08:39Z

Hmm.... syncing towards the top of testnet has gotten really slow,, and the UI is non-responsive. (Correction.... extremely slow in response... takes 30 seconds to respond to a button.)

jamescowens · 2019-10-31T15:09:02Z

Here is a gdb trace after a control-c

jamescowens · 2019-10-31T15:09:35Z

(gdb) thread apply all bt

Thread 24 (Thread 0x7fff7f49b700 (LWP 13736)):
#0 0x00007ffff484b964 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000055555597297b in BGThread (this=0x7fffbf804fd0) at leveldb/util/env_posix.cc:649
#2 leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7fffbf804fd0) at leveldb/util/env_posix.cc:564
#3 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#4 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7fff98c54700 (LWP 13715)):
#0 0x00007ffff484bd11 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000555555650ad3 in do_timed_wait (mut=..., abs_time=..., this=0x7fffe80dd058)
at /home/jco/local/boost-gcc5/include/boost/interprocess/sync/posix/condition.hpp:184
#2 boost::interprocess::ipcdetail::posix_condition::timed_wait<boost::interprocess::ipcdetail::internal_mutex_lock<boost::interprocess::scoped_lockboost::interprocess::interprocess_mutex > > (this=this@entry=0x7fffe80dd058, lock=..., abs_time=...)
at /home/jco/local/boost-gcc5/include/boost/interprocess/sync/posix/condition.hpp:97
#3 0x0000555555650dc6 in timed_wait<boost::interprocess::scoped_lockboost::interprocess::interprocess_mutex > (abs_time=..., lock=..., this=
0x7fffe80dd058) at /home/jco/local/boost-gcc5/include/boost/interprocess/sync/interprocess_condition.hpp:128
#4 boost::interprocess::message_queue_t<boost::interprocess::offset_ptr<void, long, unsigned long, 0ul> >::do_receive (this=this@entry=
0x555556ef81a0, block=block@entry=boost::interprocess::message_queue_t<boost::interprocess::offset_ptr<void, long, unsigned long, 0> >::timed,
buffer=buffer@entry=0x7fff98c53d00, buffer_size=buffer_size@entry=256, recvd_size=@0x7fff98c53c70: 0, priority=@0x7fff98c53c64: 0, abs_time=...)
at /home/jco/local/boost-gcc5/include/boost/interprocess/ipc/message_queue.hpp:887
#5 0x000055555564de89 in timed_receive (abs_time=..., priority=@0x7fff98c53c64: 0, recvd_size=@0x7fff98c53c70: 0, buffer_size=256, buffer=
0x7fff98c53d00, this=0x555556ef81a0) at /home/jco/local/boost-gcc5/include/boost/interprocess/ipc/message_queue.hpp:841
#6 ipcThread2 (pArg=pArg@entry=0x555556ef81a0) at qt/qtipcserver.cpp:104
#7 0x000055555564e25c in ipcThread (pArg=0x555556ef81a0) at qt/qtipcserver.cpp:82
#8 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#9 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fff997fa700 (LWP 13708)):
#0 0x00007ffff484bd11 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff6f82071 in boost::this_thread::hidden::sleep_for(timespec const&) () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#2 0x00005555558ce463 in sleep_for (ns=) at /home/jco/local/boost-gcc5/include/boost/thread/pthread/thread_data.hpp:252
#3 sleep_for<long, boost::ratio<1, 1000> > (d=...) at /home/jco/local/boost-gcc5/include/boost/thread/v2/thread.hpp:124
#4 MilliSleep (n=8000) at util.cpp:132
#5 0x0000555555878e69 in Scraper (bSingleShot=) at scraper/scraper.cpp:748
#6 0x00005555557acec2 in ThreadScraper (parg=) at net.cpp:1711
#7 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#8 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#9 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fff99ffb700 (LWP 13707)):
#0 0x00007ffff484bd11 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff6f82071 in boost::this_thread::hidden::sleep_for(timespec const&) () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#2 0x00005555558ce463 in sleep_for (ns=) at /home/jco/local/boost-gcc5/include/boost/thread/pthread/thread_data.hpp:252
#3 sleep_for<long, boost::ratio<1, 1000> > (d=...) at /home/jco/local/boost-gcc5/include/boost/thread/v2/thread.hpp:124
#4 MilliSleep (n=8000) at util.cpp:132
#5 0x000055555577ca33 in StakeMiner (pwallet=0x7fffb52ba360) at miner.cpp:1219
#6 0x00005555557acdbc in ThreadStakeMiner (parg=0x7fffb52ba360) at net.cpp:1687
---Type to continue, or q to quit---
#7 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#8 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#9 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7fff9a7fc700 (LWP 13706)):
#0 0x00007ffff484b964 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000055555581ca48 in wait<boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex > (lock=..., this=0x7fff9c2cd5a8)
at /home/jco/local/boost-gcc5/include/boost/asio/detail/posix_event.hpp:106
#2 do_run_one (ec=..., this_thread=..., lock=..., this=0x7fff9c2cd550)
at /home/jco/local/boost-gcc5/include/boost/asio/detail/impl/task_io_service.ipp:380
#3 run (ec=..., this=0x7fff9c2cd550) at /home/jco/local/boost-gcc5/include/boost/asio/detail/impl/task_io_service.ipp:149
#4 boost::asio::io_service::run (this=) at /home/jco/local/boost-gcc5/include/boost/asio/impl/io_service.ipp:59
#5 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#6 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fff9affd700 (LWP 13705)):
#0 0x00007ffff484bd11 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff6f82071 in boost::this_thread::hidden::sleep_for(timespec const&) () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#2 0x00005555558ce463 in sleep_for (ns=) at /home/jco/local/boost-gcc5/include/boost/thread/pthread/thread_data.hpp:252
#3 sleep_for<long, boost::ratio<1, 1000> > (d=...) at /home/jco/local/boost-gcc5/include/boost/thread/v2/thread.hpp:124
#4 MilliSleep (n=600000) at util.cpp:132
#5 0x00005555557b2a6f in ThreadDumpAddress2 (parg=) at net.cpp:1608
#6 ThreadDumpAddress (parg=) at net.cpp:1619
#7 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#8 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#9 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7fff9b7fe700 (LWP 13704)):
#0 0x00007ffff484bd11 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff6f82071 in boost::this_thread::hidden::sleep_for(timespec const&) () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#2 0x00005555558ce463 in sleep_for (ns=) at /home/jco/local/boost-gcc5/include/boost/thread/pthread/thread_data.hpp:252
#3 sleep_for<long, boost::ratio<1, 1000> > (d=...) at /home/jco/local/boost-gcc5/include/boost/thread/v2/thread.hpp:124
#4 MilliSleep (n=100) at util.cpp:132
#5 0x00005555557b05a8 in ThreadMessageHandler2 (parg=parg@entry=0x0) at net.cpp:2093
#6 0x00005555557b06cc in ThreadMessageHandler (parg=0x0) at net.cpp:2022
#7 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#8 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#9 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fffa88b0700 (LWP 13701)):
#0 0x00007ffff484b964 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000055555581ca48 in wait<boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex > (lock=..., this=0x7fff9c2cd5a8)
at /home/jco/local/boost-gcc5/include/boost/asio/detail/posix_event.hpp:106
#2 do_run_one (ec=..., this_thread=..., lock=..., this=0x7fff9c2cd550)
at /home/jco/local/boost-gcc5/include/boost/asio/detail/impl/task_io_service.ipp:380
#3 run (ec=..., this=0x7fff9c2cd550) at /home/jco/local/boost-gcc5/include/boost/asio/detail/impl/task_io_service.ipp:149
#4 boost::asio::io_service::run (this=) at /home/jco/local/boost-gcc5/include/boost/asio/impl/io_service.ipp:59
#5 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
---Type to continue, or q to quit---
#6 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fff9bfff700 (LWP 13703)):
#0 0x00007ffff261af23 in select () from /lib64/libc.so.6
#1 0x00005555557a9d79 in ConnectSocketDirectly (addrConnect=..., hSocketRet=@0x7fff9bffeb24: 4294967295, nTimeout=nTimeout@entry=5000)
at netbase.cpp:371
#2 0x00005555557aa29e in ConnectSocket (addrDest=..., hSocketRet=@0x7fff9bffeb24: 4294967295, nTimeout=5000) at netbase.cpp:490
#3 0x00005555557b4a68 in ConnectNode (addrConnect=..., pszDest=pszDest@entry=0x0) at net.cpp:620
#4 0x00005555557b4cb9 in OpenNetworkConnection (addrConnect=..., grantOutbound=grantOutbound@entry=0x7fff9bffed30, strDest=strDest@entry=0x0,
fOneShot=fOneShot@entry=false) at net.cpp:2001
#5 0x00005555557b55c2 in ThreadOpenConnections2 (parg=parg@entry=0x0) at net.cpp:1889
#6 0x00005555557b5afc in ThreadOpenConnections (parg=0x0) at net.cpp:1644
#7 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#8 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#9 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fffa90b1700 (LWP 13698)):
#0 0x00007ffff484bd11 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff6f82071 in boost::this_thread::hidden::sleep_for(timespec const&) () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#2 0x00005555558ce463 in sleep_for (ns=) at /home/jco/local/boost-gcc5/include/boost/thread/pthread/thread_data.hpp:252
#3 sleep_for<long, boost::ratio<1, 1000> > (d=...) at /home/jco/local/boost-gcc5/include/boost/thread/v2/thread.hpp:124
#4 MilliSleep (n=120000) at util.cpp:132
#5 0x00005555557b6479 in ThreadOpenAddedConnections2 (parg=parg@entry=0x0) at net.cpp:1979
#6 0x00005555557b666c in ThreadOpenAddedConnections (parg=0x0) at net.cpp:1900
#7 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#8 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#9 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fffa98b2700 (LWP 13697)):
#0 0x00007ffff484bd11 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff6f82071 in boost::this_thread::hidden::sleep_for(timespec const&) () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#2 0x00005555558ce463 in sleep_for (ns=) at /home/jco/local/boost-gcc5/include/boost/thread/pthread/thread_data.hpp:252
#3 sleep_for<long, boost::ratio<1, 1000> > (d=...) at /home/jco/local/boost-gcc5/include/boost/thread/v2/thread.hpp:124
#4 MilliSleep (n=10) at util.cpp:132
#5 0x00005555557b74d2 in ThreadSocketHandler2 (parg=parg@entry=0x0) at net.cpp:1348
#6 0x00005555557b8bfc in ThreadSocketHandler (parg=0x0) at net.cpp:991
#7 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#8 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#9 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fffaa0b3700 (LWP 13696)):
#0 0x00007ffff484b964 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000055555581ca48 in wait<boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex > (lock=..., this=0x7fff9c2cd5a8)
at /home/jco/local/boost-gcc5/include/boost/asio/detail/posix_event.hpp:106
#2 do_run_one (ec=..., this_thread=..., lock=..., this=0x7fff9c2cd550)
at /home/jco/local/boost-gcc5/include/boost/asio/detail/impl/task_io_service.ipp:380
#3 run (ec=..., this=0x7fff9c2cd550) at /home/jco/local/boost-gcc5/include/boost/asio/detail/impl/task_io_service.ipp:149
#4 boost::asio::io_service::run (this=) at /home/jco/local/boost-gcc5/include/boost/asio/impl/io_service.ipp:59
---Type to continue, or q to quit---
#5 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#6 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fffab0b5700 (LWP 13694)):
#0 0x00007ffff26226c3 in epoll_wait () from /lib64/libc.so.6
#1 0x00005555557e5c6b in boost::asio::detail::epoll_reactor::run (this=0x7fff9c2f0020, block=block@entry=true, ops=...)
at /home/jco/local/boost-gcc5/include/boost/asio/detail/impl/epoll_reactor.ipp:438
#2 0x000055555581c94b in do_run_one (ec=..., this_thread=..., lock=..., this=0x7fff9c2cd550)
at /home/jco/local/boost-gcc5/include/boost/asio/detail/impl/task_io_service.ipp:356
#3 run (ec=..., this=0x7fff9c2cd550) at /home/jco/local/boost-gcc5/include/boost/asio/detail/impl/task_io_service.ipp:149
#4 boost::asio::io_service::run (this=) at /home/jco/local/boost-gcc5/include/boost/asio/impl/io_service.ipp:59
#5 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#6 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fffc522b700 (LWP 13691)):
#0 0x00007ffff484bd11 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff6f82071 in boost::this_thread::hidden::sleep_for(timespec const&) () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#2 0x00005555558ce463 in sleep_for (ns=) at /home/jco/local/boost-gcc5/include/boost/thread/pthread/thread_data.hpp:252
#3 sleep_for<long, boost::ratio<1, 1000> > (d=...) at /home/jco/local/boost-gcc5/include/boost/thread/v2/thread.hpp:124
#4 MilliSleep (n=500) at util.cpp:132
#5 0x000055555590059f in ThreadFlushWalletDB (parg=0x7fffb52ba498) at walletdb.cpp:630
#6 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#7 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fffe0ca0700 (LWP 13684)):
#0 0x00007ffff484bd11 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00005555558bab12 in do_wait_until (timeout=..., m=..., this=0x555555f92cb0 <scheduler+48>)
at /home/jco/local/boost-gcc5/include/boost/thread/pthread/condition_variable.hpp:115
#2 wait_until (tp=..., lk=..., this=0x555555f92cb0 <scheduler+48>)
at /home/jco/local/boost-gcc5/include/boost/thread/pthread/condition_variable_fwd.hpp:266
#3 wait_until<boost::chrono::duration<long, boost::ratio<1, 1000000000> > > (t=, lock=..., this=0x555555f92cb0 <scheduler+48>)
at /home/jco/local/boost-gcc5/include/boost/thread/pthread/condition_variable_fwd.hpp:223
#4 CScheduler::serviceQueue (this=0x555555f92c80 ) at scheduler.cpp:68
#5 0x00005555557177b4 in operator() (this=0x7fffe0c9fe20) at /usr/gcc73/include/c++/7.3.1/bits/std_function.h:706
#6 TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=, func=...) at util.h:533
#7 0x0000555555718411 in __invoke_impl<void, void (&)(char const, std::function<void()>), char const*&, std::function<void()>&> (
__f=) at /usr/gcc73/include/c++/7.3.1/bits/invoke.h:60
#8 __invoke<void (&)(char const, std::function<void()>), char const*&, std::function<void()>&> (__fn=)
at /usr/gcc73/include/c++/7.3.1/bits/invoke.h:95
#9 __call<void, 0, 1> (__args=, this=) at /usr/gcc73/include/c++/7.3.1/functional:467
#10 operator()<> (this=) at /usr/gcc73/include/c++/7.3.1/functional:551
#11 boost::detail::thread_data<std::_Bind<void ((char const, std::function<void ()>))(char const*, std::function<void ()>)> >::run() (this=
0x7fffbc001ab0) at /home/jco/local/boost-gcc5/include/boost/thread/detail/thread.hpp:116
#12 0x00007ffff6f80865 in thread_proxy () from /home/jco/local/boost-gcc5/lib/libboost_thread.so.1.65.1
#13 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#14 0x00007ffff26220dd in clone () from /lib64/libc.so.6
---Type to continue, or q to quit---

Thread 5 (Thread 0x7fffe232b700 (LWP 13682)):
#0 0x00007ffff261917d in poll () from /lib64/libc.so.6
#1 0x00007fffef328aa4 in ?? () from /usr/lib64/libglib-2.0.so.0
#2 0x00007fffef328bc4 in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#3 0x00007ffff4d2ebaf in QEventDispatcherGlib::processEvents(QFlagsQEventLoop::ProcessEventsFlag) ()
from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#4 0x00007ffff4cd926a in QEventLoop::exec(QFlagsQEventLoop::ProcessEventsFlag) () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#5 0x00007ffff4b036dc in QThread::exec() () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#6 0x00007ffff4b0860c in ?? () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#7 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fffe8919700 (LWP 13681)):
#0 0x00007ffff261917d in poll () from /lib64/libc.so.6
#1 0x00007fffef328aa4 in ?? () from /usr/lib64/libglib-2.0.so.0
#2 0x00007fffef328bc4 in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#3 0x00007ffff4d2ebaf in QEventDispatcherGlib::processEvents(QFlagsQEventLoop::ProcessEventsFlag) ()
from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#4 0x00007ffff4cd926a in QEventLoop::exec(QFlagsQEventLoop::ProcessEventsFlag) () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#5 0x00007ffff4b036dc in QThread::exec() () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#6 0x00007ffff4b0860c in ?? () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#7 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fffe9ad2700 (LWP 13677)):
#0 0x00007ffff261917d in poll () from /lib64/libc.so.6
#1 0x00007fffef328aa4 in ?? () from /usr/lib64/libglib-2.0.so.0
#2 0x00007fffef328bc4 in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#3 0x00007ffff4d2ebcb in QEventDispatcherGlib::processEvents(QFlagsQEventLoop::ProcessEventsFlag) ()
from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#4 0x00007ffff4cd926a in QEventLoop::exec(QFlagsQEventLoop::ProcessEventsFlag) () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#5 0x00007ffff4b036dc in QThread::exec() () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#6 0x00007ffff51c06a5 in ?? () from /opt/Qt/5.10.0/gcc_64/lib/libQt5DBus.so.5
#7 0x00007ffff4b0860c in ?? () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#8 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#9 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fffec3a5700 (LWP 13676)):
#0 0x00007ffff261917d in poll () from /lib64/libc.so.6
#1 0x00007fffed8dc0e2 in ?? () from /usr/lib64/libxcb.so.1
#2 0x00007fffed8dd86f in xcb_wait_for_event () from /usr/lib64/libxcb.so.1
#3 0x00007fffed15c079 in ?? () from /opt/Qt/5.10.0/gcc_64/lib/libQt5XcbQpa.so.5
#4 0x00007ffff4b0860c in ?? () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#5 0x00007ffff4847e0f in start_thread () from /lib64/libpthread.so.0
#6 0x00007ffff26220dd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7f78740 (LWP 13671)):
#0 0x00005555556cd046 in _M_find_before_node (__code=, __k=..., __n=1551214, this=)
---Type to continue, or q to quit---
at /usr/gcc73/include/c++/7.3.1/bits/hashtable.h:1545
#1 _M_find_node (__c=, __key=..., __bkt=1551214, this=) at /usr/gcc73/include/c++/7.3.1/bits/hashtable.h:642
#2 std::_Hashtable<uint256, std::pair<uint256 const, CBlockIndex*>, std::allocator<std::pair<uint256 const, CBlockIndex*> >, std::__detail::_Select1st, std::equal_to, BlockHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find (this=, __k=...) at /usr/gcc73/include/c++/7.3.1/bits/hashtable.h:1422
#3 0x0000555555735e44 in find (__x=..., this=0x555555f936e0 ) at /usr/gcc73/include/c++/7.3.1/bits/unordered_map.h:923
#4 CMerkleTx::GetDepthInMainChainINTERNAL (this=0x7fffb75a1930, pindexRet=@0x7fffffffc6b0: 0x7fff3e63f240) at main.cpp:1470
#5 0x000055555573615e in CMerkleTx::GetDepthInMainChain (this=0x7fffb75a1930, pindexRet=) at main.cpp:1492
#6 0x00005555558dbcef in GetDepthInMainChain (this=0x7fffb75a1930) at main.h:932
#7 IsConfirmed (this=0x7fffb75a1930) at wallet.h:724
#8 CWallet::GetUnconfirmedBalance (this=0x7fffb52ba360) at wallet.cpp:1174
#9 0x0000555555681448 in WalletModel::checkBalanceChanged (this=0x7fffffffced0) at qt/walletmodel.cpp:114
#10 0x0000555555681dbe in WalletModel::updateTransaction (this=0x7fffffffced0, hash=..., status=) at qt/walletmodel.cpp:133
#11 0x00007ffff4d06b21 in QObject::event(QEvent*) () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#12 0x00007ffff5d7254c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Widgets.so.5
#13 0x00007ffff5d79897 in QApplication::notify(QObject*, QEvent*) () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Widgets.so.5
#14 0x00007ffff4cda9f8 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#15 0x00007ffff4cdd48b in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#16 0x00007ffff4d2f563 in ?? () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#17 0x00007fffef3287d5 in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#18 0x00007fffef328b08 in ?? () from /usr/lib64/libglib-2.0.so.0
#19 0x00007fffef328bc4 in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#20 0x00007ffff4d2ebaf in QEventDispatcherGlib::processEvents(QFlagsQEventLoop::ProcessEventsFlag) ()
from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#21 0x00007fffed1f19a1 in ?? () from /opt/Qt/5.10.0/gcc_64/lib/libQt5XcbQpa.so.5
#22 0x00007ffff4cd926a in QEventLoop::exec(QFlagsQEventLoop::ProcessEventsFlag) () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#23 0x00007ffff4ce1da4 in QCoreApplication::exec() () from /opt/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
#24 0x00005555555e4b08 in main (argc=, argv=0x7fffffffd8c8) at qt/bitcoin.cpp:369

jamescowens · 2019-10-31T15:11:14Z

Section of log for syncing...

10/31/2019 14:24:07 Tally::LegacyRecount(1017970)
10/31/2019 14:24:07 {SBC} new best {dae4462f7cc26c042f994e300699f6c5462d8ea9913044c95983b2dfdc478320 1017970} ;
10/31/2019 14:24:07 {SBC} new best {9c108e7eb078889a40600a98af1ef6c2037b638e40b89b8e9ce8fb779f543f5e 1017971} ;
10/31/2019 14:24:07 .11.
10/31/2019 14:24:07 {SBC} new best {4699734e20e07a156b3c525edd29bba2d1864c2be9591d567b91a6b549e8939f 1017972} ;
10/31/2019 14:24:07 {SBC} new best {68df9e931ff7a41ba65176557001703f9a038824bfdad4cf205f4c859727da98 1017973} ;
10/31/2019 14:24:07 {SBC} new best {e6f97c5803fcd01bea0c391faef68b6164ce2390c5e3e2be8528aa295a372d7d 1017974} ;
10/31/2019 14:24:07 .11.
10/31/2019 14:24:07 {SBC} new best {7227be7a0ee9b8a53702f32b38c6882aedf2105ee544875e64a849c975382af2 1017975} ;
10/31/2019 14:24:07 {SBC} new best {5f5d98e75b0ac19298202dc1c54f7f4b16872b0e182f422c02d198fde9433cee 1017976} ;
10/31/2019 14:24:08 {SBC} new best {98fae0cac26225db2f9447d0a1fa151c8d6c280658ee29b2506cc85f720e0520 1017977} ;
10/31/2019 14:24:08 .11.
10/31/2019 14:24:08 {SBC} new best {370a0f239d060b35711b9bb307032f89d6aae1e843f51aa578f75dc5ee28eaff 1017978} ;
10/31/2019 14:24:08 {SBC} new best {45e6a88505476aae81f104a13316da29f72ddc0d40a73a59a04af3ab06ea515f 1017979} ;
10/31/2019 14:24:08 Tally::LegacyRecount(1017980)
10/31/2019 14:24:08 {SBC} new best {091098f1febbf809f60c214fe68e086b93f78a0298b7163e378c8ef3ee4a1a09 1017980} ;
10/31/2019 14:24:08 .11.
10/31/2019 14:24:08 {SBC} new best {0d08756e461e81bab8a5801fc4127964aee479326c4759f6d9f68d7312b454f2 1017981} ;
10/31/2019 14:24:08 {SBC} new best {0ed539b03af2b39096b3bf5bcce6141265e9e149717e653135456d0ce967a4cb 1017982} ;
10/31/2019 14:24:08 {SBC} new best {0e5dde3282539cd4f295c9432c312eeb2647c41271cc554593c521544174f200 1017983} ;
10/31/2019 14:24:08 .11.
10/31/2019 14:24:08 {SBC} new best {ac4d4fb1fcd22a5c7a4821be3de07d6fa8c75b485cc3450a7448dec2f88acf37 1017984} ;
10/31/2019 14:24:08 {SBC} new best {5a00bc62a200655b1a54f890f499ac611a14c2283df9db0ed7bbf982798f32a8 1017985} ;
10/31/2019 14:24:08 {SBC} new best {85f35078529980d71f517e79468fb7ded214b64166a31c32b3c8f59e2a2f3c8a 1017986} ;
10/31/2019 14:24:08 .11.
10/31/2019 14:24:08 {SBC} new best {3535c55fa52931dce4faafdae77f29a248004f7024ee04a69dd4210b312752e8 1017987} ;
10/31/2019 14:24:08 {SBC} new best {703c6deea91f2a2d6c5f4db6855ec6452c8e235790d342f4e7789629c5e2df87 1017988} ;
10/31/2019 14:24:08 {SBC} new best {af8e53ea9bacb6ffc415030aa19a19d2248c6e3f6613e034fdd24b981383bc63 1017989} ;
10/31/2019 14:24:08 Tally::LegacyRecount(1017990)
10/31/2019 14:24:08 .11.
10/31/2019 14:24:08 {SBC} new best {6498da38cbc7886040be9aab349193479c573c505292cae6f5b0818a922bc308 1017990} ;
10/31/2019 14:24:08 {SBC} new best {739a09c8871e3b654ab69f1c9c04042ab2d89640a7be2cde35497980e2500176 1017991} ;
10/31/2019 14:24:08 {SBC} new best {c2b8ee125f564e24b1416cb0dd3703b6943e8001ba02f5d353b370366bcebd64 1017992} ;
10/31/2019 14:24:08 .11.
10/31/2019 14:24:08 {SBC} new best {4c9f0247faba24d374240ecae7c3246dd25bed29f3eac923b13baec2d92e45e2 1017993} ;
10/31/2019 14:24:08 {SBC} new best {4851df38ce5567875a9571094f7e520c8b777371f7461ea14d0832b81838ca96 1017994} ;
10/31/2019 14:24:08 {SBC} new best {dbabb46202b98080722f182cb20b91cca8ce24b26456d9e8c443854b768400ee 1017995} ;
10/31/2019 14:24:08 .11.
10/31/2019 14:24:08 {SBC} new best {9a2a3c64ac022ff8cd14a29a659aff472901d9cd8fca65d363d39f05fd757357 1017996} ;
10/31/2019 14:24:08 {SBC} new best {5c8b21f6a5c230703ec03be8fb0d7eb393cb2e52943a1787c20d812386ec27cd 1017997} ;
10/31/2019 14:24:08 {SBC} new best {510f62c137766446b2848611521d1785cdea1e9b99358413dea2889e159e3df6 1017998} ;
10/31/2019 14:24:09 .11.
10/31/2019 14:24:09 {SBC} new best {fddbf0fa682a7d914086fc57be6a32ba1ba6f6ad8fced9aeeec551b9f92e1ec9 1017999} ;
10/31/2019 14:24:09 Tally::LegacyRecount(1018000)

jamescowens · 2019-10-31T15:22:30Z

You can see it is pretty slow...

jamescowens · 2019-10-31T16:25:59Z

I think this is just due to the number of incoming sidestake transactions on my wallet later in the chain... The UI model updates are horribly slow and blocking, and we see this same effect with consolidateunspent and a lot of built up UTXO's when a large input limit is specified.

jamescowens

Looks good to me!

jamescowens · 2019-11-03T00:37:05Z

@denravonska I am not waiting for your response on this. If you still have a compilation problem, we can merge a true-up.

@cyrossignol

Added - Backport newer uint256 types from Bitcoin #1570 (@cyrossignol) - Implement project level rain for rainbymagnitude #1580 (@jamescowens) - Upgrade utilities (Update checker and snapshot downloader/application) #1576 (@iFoggz) - Provide fees collected in the block by the miner #1601 (@iFoggz) - Add support for generating legacy superblocks from scraper stats #1603 (@cyrossignol) - Port of the Bitcoin Logger to Gridcoin #1600 (@jamescowens) - Implement zapwallettxes #1605 (@jamescowens) - Implements a global event filter to suppress help question mark #1609 (@jamescowens) - Add next target difficulty to RPC output #1615 (@cyrossignol) - Add caching for block hashes to CBlock #1624 (@cyrossignol) - Make toolbars and tray icon red for testnet #1637 (@jamescowens) - Add an rpc call convergencereport #1643 (@jamescowens) - Implement newline filter on config file read in #1645 (@jamescowens) - Implement beacon status icon/button #1646 (@jamescowens) - Add gridcointestnet.png #1649 (@caraka) - Add precision to support magnitudes less than 1 #1651 (@cyrossignol) - Replace research accrual calculations with superblock snapshots #1657 (@cyrossignol) - Publish example gridcoinresearch.conf as a md document to the doc directory #1662 (@jamescowens) - Add options checkbox to disable transaction notifications #1666 (@jamescowens) - Add support for self-service beacon deletion #1695 (@cyrossignol) - Add support for type-specific contract fee amounts #1698 (@cyrossignol) - Add verifiedbeaconreport and pendingbeaconreport #1696 (@jamescowens) - Add preliminary testing option for block v11 height on testnet #1706 (@cyrossignol) - Add verified beacons manifest part to superblock validator #1711 (@cyrossignol) - Implement beacon, vote, and superblock display categories/icons in UI transaction model #1717 (@jamescowens) - neuralnet: Add integrity checking to researcher accrual snapshot registry #1727 (@jamescowens) - Add workaround for scrypt assembly on macOS #1740 (@cyrossignol) - gui: Build onboarding/beacon wizard #1739 (@cyrossignol) - doc: Add CONTRIBUTING.md from bitcoin #1723 (@div72) - rpc: Implement inspectaccrualsnapshot and parseaccrualsnapshotfile #1744 (@jamescowens) - scraper: Add disk based state backing for verified beacon list in scraper #1751 (@jamescowens) - Add ability to recover beacon in block version 11+ #1768 (@cyrossignol) - refactor: Add transaction context to contract handlers #1777 (@cyrossignol) - gui: Add context for when BOINC is attached to a pool #1775 (@cyrossignol) - doc: Clarify what to do if PR in multiple categories (for CONTRIBUTING.md) #1798 (@RoboticMind) - qt: Add option to choose not to start the wallet minimized #1804 (@jamescowens) - superblock: Add check for OutOfSyncByAge to SuperblockValidator::Validate #1806 (@jamescowens) - contract: Standardize contract validation and add block context #1808 (@cyrossignol) - add seed.gridcoin.pl to default config #1812 (@wilkart) - gui: Implement sidestake send display #1813 (@jamescowens) - gui: Add pool/investor pages to researcher wizard #1819 (@cyrossignol) - ci: Port lint scripts from Bitcoin #1823 (@div72) - doc: Create basic readme in contrib #1826 (@RoboticMind) - gui: Implement TransactionRecord::Message #1829 (@jamescowens) - rpc: Add private_key_available to beaconstatus #1833 (@a123b) - gui: Validate email address in researcher wizard #1840 (@a123b) - rpc: Add "getrawwallettransaction" RPC function #1842 (@cyrossignol) - consensus: Set block version 11 threshold height for mainnet #1862 (@cyrossignol) Changed - Upgrade LevelDB from v1.17 to v1.20 #1562 (@cyrossignol) - Re-enable scrypt optimizations #1450 (@denravonska) - Derive CScript from prevector type (optimization) #1554 (@cyrossignol) - Disable quorum for grandfathered blocks to speed up sync #1568 (@cyrossignol) - Refactor hashBoinc for binary claim contexts #1558 (@cyrossignol) - integrated_scraper_2 branch tracking PR #1559 (@jamescowens) - Upgrade depends - OpenSSL to 1.1.1d #1581 (@jamescowens) - Ubuntu 19.10 fixes #1590 (@denravonska) - Force a re-parse of legacy claims in generated blocks #1592 (@cyrossignol) - Improve the "versionreport" RPC output #1595 (@cyrossignol) - Overhaul the core tally and accrual system #1583 (@cyrossignol) - Overhaul the superblock quorum system #1597 (@cyrossignol) - Add more data to the "superblocks" RPC output #1599 (@cyrossignol) - Update Windows Build doc #1606 (@barton2526) - Change the order of calls in gridcoinresearchd.cpp to optimize rpc shunt path #1610 (@jamescowens) - Change staking tooltip to display frequency #1611 (@jamescowens) - Enhancements to ETTS #1442 (@jamescowens) - Standardize money values as integers #1614 (@cyrossignol) - Clean up and optimize legacy coin age code #1616 (@cyrossignol) - Some scraper cleanups #1620 (@jamescowens) - Reorganize accrual code and fix 6-month cutoff #1630 (@cyrossignol) - Update Copyright years #1633 (@barton2526) - Change team whitelist delimiter to <> for CPID detection #1634 (@cyrossignol) - Change team whitelist separator to <> to accomodate more team names #1632 (@jamescowens) - Change Curl download speed type to support older environments #1640 (@cyrossignol) - Optimize logo SVGs used for tray icons #1638 (@cyrossignol) - Tweak consolidateunspent rpc function #1644 (@jamescowens) - ETTS and staking icon enhancements #1650 (@jamescowens) - Implement new transaction fees for block version 11 #1652 (@jamescowens) - Optimize in-memory storage of superblock data #1653 (@cyrossignol) - Miscellaneous superblock API improvements and housekeeping #1654 (@cyrossignol) - Update openssl to 1.1.1f compatibility #1660 (@jamescowens) - Optimize bdb to avoid synchronous flush of database #1659 (@jamescowens) - Add support for CPID input to "lifetime" RPC function #1668 (@cyrossignol) - Overhaul the contract handling system #1669 (@cyrossignol) - Make the autostart mainnet/testnet aware #1671 (@jamescowens) - Remove slashes from User Agent in peers tab #1674 (@div72) - Refactor contracts for polymorphic binary payloads #1676 (@cyrossignol) - Overhaul the beacon system #1678 (@cyrossignol) - Replace boost::optional<T&> with non-owning pointers #1680 (@cyrossignol) - Optimize proof-of-stake validation #1681 (@cyrossignol) - Updated Slack link #1683 (@NeuralMiner) - Update build-unix.md #1686 (@Quezacoatl1) - Replace deprecated QT methods #1693 (@Pythonix) - Made protocol.h more similar to bitcoin #1688 (@Pythonix) - Touch up some details for block version 11 #1697 (@cyrossignol) - More tweaks for block version 11 #1700 (@cyrossignol) - Finish the conversion to the BCLog class based logger #1699 (@jamescowens) - Move claim version transitional code in miner for proper signature #1712 (@cyrossignol) - doc: Update threads in coding.txt #1730 (@div72) - qt: Include QPainterPath in trafficgraphwidget.cpp #1733 (@div72) - doc: Update doc/build-unix.md #1731 (@div72) - gui: Show peers tab on connections icon click #1734 (@div72) - refactor: Change return type of IsMine to isminetype && move wallet files to wallet directory #1722 (@div72) - build: Updates boost to 1.73.0 for depends #1673 (@jamescowens) - doc: Update Unit Test Readme #1743 (@RoboticMind) - wallet: Change Assert To Error Message In kernel.cpp #1748 (@RoboticMind) - scraper: Shorten display representation of verification codes #1754 (@cyrossignol) - log: Change ".B." to Clear Message #1758 (@RoboticMind) - util: Fix braindamage in GetDefaultDataDir() #1737 (@jamescowens) - scraper: Improve scraper processing of beacon verifications #1760 (@jamescowens) - scraper: Add instrumentation to convergencereport #1763 (@jamescowens) - rpc: Improve rpc stress test script #1767 (@tunisiano187) - Generalize enum serialization #1770 (@cyrossignol) - scraper: Improve handling of ETags in http class and tweak verified beacon logic #1776 (@jamescowens) - scraper: Improve ProcessNetworkWideFromProjectStats and other tweaks #1778 (@jamescowens) - researcher: Automate beacon advertisement for renewals only #1781 (@cyrossignol) - gui: Tweak behavior of beacon page in researcher wizard #1784 (@cyrossignol) - Prepare for block version 11 hard-fork on testnet #1787 (@cyrossignol) - scraper: Modify UpdateVerifiedBeaconsFromConsensus #1791 (@jamescowens) - gui: Optimize OverviewPage::updateTransactions() #1794 (@jamescowens) - ci: Adopt ci changes from Bitcoin #1795 (@div72) - consensus: switch snapshot accrual calculation to integer arithmetic #1799 (@cyrossignol) - voting: Overhaul the voting system #1809 (@cyrossignol) - contract: Optimize contract replay after chain reorganization #1815 (@cyrossignol) - contract: Reimplement transaction messages as contracts #1816 (@cyrossignol) - staking: Sign claim contracts with coinstake transaction #1817 (@cyrossignol) - gui: Change research wizard text #1820 (@div72) - net: Update protocol version and clean up net messaging #1824 (@cyrossignol) - rpc, wallet: Corrections to GetAmounts #1825 (@jamescowens) - gui: Tweak some minor researcher wizard details #1830 (@cyrossignol) - gui: Change GetEstimatedStakingFrequency text #1836 (@jamescowens) - scraper: Scraper global statistics cache optimization #1837 (@jamescowens) - doc: Update Vulnerability Response Process #1843 (@RoboticMind) - scraper: Optimization of manifest and parts sharing between ConvergedScraperStatsCache, mapManifest, and mapParts #1851 (@jamescowens) - consensus: Update Checkpoints #1855 (@barton2526) - docs: Update docs to build off master #1856 (@barton2526) - gui: Fix and improve GUI combo box styles #1858 (@cyrossignol) - build: Tweak Gridcoin installer for Fern release #1863 (@jamescowens) Removed - Remove old research age checks (rebase #1365) #1572 (@cyrossignol) - Remove PrimaryCPID check from diagnostics dialog #1586 (@cyrossignol) - Remove missed label for PrimaryCPID from diagnostics #1588 (@cyrossignol) - Remove legacy quorum messaging system (@neural network) #1589 (@cyrossignol) - Remove old remnants of legacy smart contract experiments #1594 (@cyrossignol) - Remove block nonce for version 11 #1622 (@cyrossignol) - Delete obsolete contrib/Installer and Upgrader directories #1623 (@jamescowens) - Remove redundant LoadAdminMessages() calls #1625 (@cyrossignol) - Remove some legacy informational RPC commands #1658 (@cyrossignol) - Remove informational magnitude field from binary claims #1661 (@cyrossignol) - Remove fDebug3,4, and net and convert to BCLog::LogFlags #1663 (@jamescowens) - Remove qt5.7.1 depends support build System #1665 (@iFoggz) - Remove unused jQuery library #1679 (@cyrossignol) - Remove unused NetworkTimer() function and global state #1701 (@cyrossignol) - Refactor claim context objects into contracts #1704 (@cyrossignol) - Clean old assets up #1718 (@div72) - Remove legacy "rain" RPC (not by-project rain) #1742 (@cyrossignol) - Temporarily disable voting system on testnet #1769 (@cyrossignol) - gui: Remove legacy GUI transaction description for contracts #1772 (@cyrossignol) - gui: Remove transaction fee setting #1780 (@cyrossignol) - trivial: Cleanup unused legacy functions #1793 (@cyrossignol) - mining, rpc: Remove kernel-diff-best and kernel-diff-sum #1796 (@jamescowens) - refactor: Remove libs subdirectory #1802 (@div72) - scraper: cleanup unused/unnecessary functions #1803 (@jamescowens) - gui: Remove useless "Detach databases at shutdown" #1810 (@jamescowens) - test: Remove testnet condition for standard transactions #1814 (@cyrossignol) - consensus: Remove transitional testnet code #1854 (@cyrossignol) Fixed - Fix "Owed" amount in output of "magnitude" RPC method #1569 (@cyrossignol) - Add support for paths with special characters on Windows #1571 (@cyrossignol) - Fix lingering peers.dat temp files and clean up remaining paths #1582 (@cyrossignol) - Fix incorrect beacon length warning in GUI transaction list #1585 (@cyrossignol) - Fix default config file line endings on Windows #1587 (@cyrossignol) - Reenable Travis builds for MacOS #1591 (@jamescowens) - Correct peer detail info background color #1593 (@jamescowens) - Fix exception in debug3 mode #1598 (@cyrossignol) - Fix deadlock in "getmininginfo" RPC function #1596 (@cyrossignol) - Fix accuracy of statistics in "network" RPC output #1602 (@cyrossignol) - Fix heights for quorum vote weight calculations #1604 (@cyrossignol) - Fix deadlock in log archiver when rename fails #1607 (@cyrossignol) - Fix a spurious segmentation fault during client load on Windows with fast CPUs #1608 (@jamescowens) - Fix lock order debugging and potential deadlocks #1612 (@jamescowens) - Add dependencies #1613 (@Scalextrix) - Fix std namespace pollution #1617 (@denravonska) - Add missing condition for newbie accrual computer #1618 (@cyrossignol) - Track first reward blocks in research accounts #1619 (@cyrossignol) - Fix lingering beacon warning after advertisement #1627 (@cyrossignol) - Fix accrual calculation for new, zero-magnitude CPIDs #1636 (@cyrossignol) - Fix diagnostics, add ETTS test, fix tooltipcolor, add missing lock, and add email=investor check #1647 (@jamescowens) - Fix help message of two RPC methods #1656 (@div72) - Fix legacy accrual for newbie with non-zero past reward #1667 (@cyrossignol) - Fix GUI autostart on Windows for paths with wide characters #1670 (@cyrossignol) - Qualify boost bind placeholders with their full namespace #1672 (@Ponce) - Fix suffix when copying txids #1677 (@div72) - Unnecessary if-statement removed #1685 (@Pythonix) - Fix consolidatemsunspent Help Message #1687 (@Pythonix) - Fix gettransaction help message #1691 (@Pythonix) - Fix GetNewMint To Look for Stakes #1692 (@RoboticMind) - Suppress deprecated copy warnings for Qt with GCC 9+ #1702 (@cyrossignol) - Fix exclusion error on stats processing and misplaced ENDLOCK logging entry #1710 (@jamescowens) - Removed unnecessary comparison #1708 (@Pythonix) - Fixed typo #1707 (@Pythonix) - Fix out-of-bounds exception for peers tab version slashes #1713 (@cyrossignol) - Fix transition for v1 superblocks when reorganizing #1714 (@cyrossignol) - Touch up transition to version 2 transactions #1715 (@cyrossignol) - Avoid mutating transactions in ConnectBlock() #1716 (@cyrossignol) - Skip beacon advertisement when already pending #1726 (@cyrossignol) - Fix Windows cross-compilation in newer environments #1728 (@cyrossignol) - Fix out-of-bounds access in IsMineInner() #1736 (@cyrossignol) - Fix a couple of block version 11 issues #1738 (@cyrossignol) - Fix null pointer dereference in GUI researcher model #1741 (@cyrossignol) - accrual: Reset research accounts when rebuilding accrual snapshots #1745 (@cyrossignol) - scraper: Correct update for verified beacons #1747 (@jamescowens) - accrual: Refactor tally initialization for snapshot rebuild #1749 (@cyrossignol) - rpc: Fix "cpid" field in "beaconconvergence" RPC output #1750 (@cyrossignol) - accrual: Fix snapshot accrual superblock state transitions #1752 (@cyrossignol) - scraper: Correct stale verified beacon logic #1753 (@jamescowens) - rpc: Correct possible divide by zero in getblockstats #1755 (@jamescowens) - gui: Fix issues with researcher wizard flow #1756 (@cyrossignol) - wallet: Stop Error When Starting From Zero #1759 (@RoboticMind) - Don't count empty email as explicit investor #1761 (@cyrossignol) - accrual: Fix snapshot accrual superblock state transitions #1764 (@cyrossignol) - rpc: Cleanup Help Message and Fix Typo #1771 (@RoboticMind) - scraper: Fix scraper etag header case sensitivity #1773 (@cyrossignol) - consensus: Use explicit time to check if superblock needed #1774 (@cyrossignol) - gui: Fix scroll area dark theme styles #1785 (@cyrossignol) - rpc, gui: Fix three divide by zero possibilities #1789 (@jamescowens) - rpc: Fix balance pre-check in "rainbymagnitude" RPC #1792 (@cyrossignol) - accrual: Fix outdated comment and correct grammar #1800 (@RoboticMind) - gui: Fix stuck cursor on labels #1801 (@div72) - beacon: Fix research wizard beacon renewal status #1805 (@cyrossignol) - gui: Fix translations for port numbers #1818 (@cyrossignol) - util: Create parent directory #1821 (@div72) - mining: Fix coinstake/claim signature order #1828 (@cyrossignol) - voting: Remove double increment in loop #1831 (@cyrossignol) - neuralnet, scraper: Fix compilation with gcc5 and older libcurl #1832 (@a123b) - wallet: Fix smallest coin selection for contracts #1841 (@cyrossignol) - gui: Fix display of polls with no votes yet #1844 (@cyrossignol) - gui: add indentation to diagnostic status bar labels #1849 (@jamescowens) - voting, gui: Fix formatting and alignment of vote shares and percent #1850 (@jamescowens) - wallet, rpc: Fix for self-transactions in listtransactions #1852 (@jamescowens) - accrual: Clear any accrual snapshots when syncing from pre-v11 #1853 (@cyrossignol) - accrual: Fix reset of accrual directory if starting sync below research age height #1857 (@jamescowens) - gui: Fix researcher wizard layout on macOS with native theme #1860 (@cyrossignol)

cyrossignol added 10 commits October 29, 2019 23:13

Serialize claim client version and org with LIMITED_STRING

11ac1c1

Deserialization of claims in version 11 blocks will automatically fail if the client version or organization fields are larger than allowed.

Remove unnecessary pre-v9 tally for superblock

011f865

Remove legacy tally interval in ConnectBlock()

c33e845

Remove reference from NN::CpidOption

b81a65f

Remove now unused GetMagnitudeMultiplier() function

828b515

Remove unneeded bZeroOutAfterPOR parameter from BeaconTimeStamp()

661846d

Remove now unused GetLastPORBlockHash() function

0059174

Remove unused SuccessFail() function

2d2e272

Remove old "testnewcontract" RPC method

0114a4e

Remove useless "dportally" RPC method

9a922e1

This RPC method calls a function that fills staging data structures with superblock data. That data will never be used--the next tally cycle will overwrite it before the accrual code ever sees it.

cyrossignol force-pushed the tally branch from 678d296 to e5e6faf Compare October 30, 2019 05:33

jamescowens requested review from jamescowens, denravonska and iFoggz October 30, 2019 11:36

jamescowens assigned cyrossignol Oct 30, 2019

jamescowens added this to the Fern milestone Oct 30, 2019

jamescowens reviewed Oct 30, 2019

View reviewed changes

src/gridcoin.cpp Show resolved Hide resolved

src/neuralnet/neuralnet_native.cpp Show resolved Hide resolved

src/neuralnet/tally.cpp Show resolved Hide resolved

src/neuralnet/tally.cpp Show resolved Hide resolved

src/scraper/scraper.cpp Show resolved Hide resolved

jamescowens reviewed Oct 30, 2019

View reviewed changes

src/rpcblockchain.cpp Outdated Show resolved Hide resolved

cyrossignol added 5 commits October 31, 2019 02:25

Create new NN::Tally classes to replace legacy tally/accrual code

2d54032

Refactor accrual code for new Tally system

263c0d0

Update bad testnet blocks in GetBadBlocks(), return const reference

a2037a0

Legacy fallback research age validation masked these blocks with a short block span.

Remove bNetAveragesLoaded miner check

09ebdd4

This check skips the miner loop while the tally runs. Because the tally runs quickly on main thread now, this check is unnecessary. The lock on cs_main will prevent the miner from pulling stale accrual data.

Remove unused parameters from GetProofOfStakeReward()

a20cd1f

cyrossignol added 2 commits October 31, 2019 02:25

Remove legacy string superblock contracts from scraper

d332604

Remove the SUPERBLOCK AppCache section

8ef8990

cyrossignol force-pushed the tally branch from e5e6faf to 8ef8990 Compare October 31, 2019 07:27

jamescowens approved these changes Oct 31, 2019

View reviewed changes

jamescowens removed request for denravonska and iFoggz November 2, 2019 22:47

cyrossignol marked this pull request as ready for review November 2, 2019 22:49

jamescowens merged commit 4910488 into gridcoin-community:development Nov 3, 2019

This was referenced Nov 4, 2019

Consolidate internal caches #230

Closed

Consider removing RSAWeight and StructCPID::ResearchSubsidy #919

Closed

cyrossignol mentioned this pull request Nov 23, 2019

Add support for generating legacy superblocks from scraper stats #1603

Merged

cyrossignol mentioned this pull request Jan 27, 2020

Reorganize accrual code and fix 6-month cutoff #1630

Merged

cyrossignol mentioned this pull request Feb 7, 2020

Fix accrual calculation for new, zero-magnitude CPIDs #1636

Merged

cyrossignol mentioned this pull request Apr 1, 2020

Replace research accrual calculations with superblock snapshots #1657

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overhaul the core tally and accrual system #1583

Overhaul the core tally and accrual system #1583

cyrossignol commented Oct 30, 2019 •

edited

Loading

denravonska commented Oct 30, 2019 •

edited

Loading

jamescowens left a comment

jamescowens commented Oct 30, 2019 •

edited

Loading

jamescowens commented Oct 30, 2019

denravonska commented Oct 30, 2019

jamescowens commented Oct 30, 2019

jamescowens commented Oct 30, 2019

jamescowens commented Oct 30, 2019

cyrossignol commented Oct 31, 2019

jamescowens commented Oct 31, 2019 •

edited

Loading

jamescowens commented Oct 31, 2019

jamescowens commented Oct 31, 2019

jamescowens commented Oct 31, 2019

jamescowens commented Oct 31, 2019

jamescowens commented Oct 31, 2019

jamescowens left a comment

jamescowens commented Nov 3, 2019

Overhaul the core tally and accrual system #1583

Overhaul the core tally and accrual system #1583

Conversation

cyrossignol commented Oct 30, 2019 • edited Loading

denravonska commented Oct 30, 2019 • edited Loading

jamescowens left a comment

Choose a reason for hiding this comment

jamescowens commented Oct 30, 2019 • edited Loading

jamescowens commented Oct 30, 2019

denravonska commented Oct 30, 2019

jamescowens commented Oct 30, 2019

jamescowens commented Oct 30, 2019

jamescowens commented Oct 30, 2019

cyrossignol commented Oct 31, 2019

jamescowens commented Oct 31, 2019 • edited Loading

jamescowens commented Oct 31, 2019

jamescowens commented Oct 31, 2019

jamescowens commented Oct 31, 2019

jamescowens commented Oct 31, 2019

jamescowens commented Oct 31, 2019

jamescowens left a comment

Choose a reason for hiding this comment

jamescowens commented Nov 3, 2019

cyrossignol commented Oct 30, 2019 •

edited

Loading

denravonska commented Oct 30, 2019 •

edited

Loading

jamescowens commented Oct 30, 2019 •

edited

Loading

jamescowens commented Oct 31, 2019 •

edited

Loading