Improve adjust_balance performance #1083

abitmore · 2018-06-23T11:44:17Z

Replay profiling data:

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls   s/call   s/call  name
 22.90    574.86   574.86 948058596     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_object ...  >::find(graphene::db::object_id_type) const
 10.99    850.85   275.99 391901309     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_balance_object, ...  >::modify(graphene::db::object const&, std::function<void (graphene::db::object&)> const&)
  6.41   1011.89   161.04 20731743387     0.00     0.00  graphene::chain::operator>(graphene::chain::price const&, graphene::chain::price const&)
  4.31   1119.98   108.09 390785433     0.00     0.00  graphene::chain::database::adjust_balance(graphene::db::object_id<(unsigned char)1, (unsigned char)2, graphene::chain::account_object>, graphene::chain::asset)
  4.18   1224.80   104.82    22671     0.00     0.01  graphene::chain::database::perform_chain_maintenance(graphene::chain::signed_block const&, graphene::chain::global_property_object const&)
  3.56   1314.29    89.50 22967573183     0.00     0.00  graphene::chain::operator<(graphene::chain::price const&, graphene::chain::price const&)
  3.40   1399.71    85.42 307443552     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_statistics_object, ...  >::find(graphene::db::object_id_type) const

Note:

the first entry (account_index::find(object_id_type)) will be discussed in Improve vote tally performance #803. It's not in scope of this issue.
The 2nd and 4th entries are related to adjust_balance.

Background: in order to implement top_n_authorities asset committee mechanism (for STEALTH), we added by_asset_balance into indices of account_balance_index, which impacted overall performance:

most assets don't need by_asset_balance for consensus;
even for assets enabled top_n_authorities (e.g. STEALTH), since the asset committee only gets updated in maintenance interval, it's an overkill to maintain by_asset_balance for consensus.

Another use of by_asset_balance is asset_api::get_asset_holders, which can be used to fetch holders of specified asset in descending order. Since it is not related to consensus, in addition, asset_api is not enabled by default due to potential performance issues (#312 (comment)), I propose we move by_asset_balance to a plugin and refactor asset_api.

This is a sub-task of #982.

The text was updated successfully, but these errors were encountered:

abitmore · 2018-06-25T15:27:19Z

Updated profiling data after #1085:

---------------------- first 27 M blocks ----------------------------
1486171ms th_a db_management.cpp:124 reindex ] Done reindexing, elapsed time: 5633.52932400000008784 sec

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls   s/call   s/call  name
 13.89    382.25   382.25 391901309     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_balance_object, ... >::modify(graphene::db::object const&, std::function<void (graphene::db::object&)> const&)
  8.92    627.59   245.34 448458421     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_object, ... >::find(graphene::db::object_id_type) const
  8.00    847.80   220.21 20731743387     0.00     0.00  graphene::chain::operator>(graphene::chain::price const&, graphene::chain::price const&)
  5.43    997.08   149.28 390785433     0.00     0.00  graphene::chain::database::adjust_balance(graphene::db::object_id<(unsigned char)1, (unsigned char)2, graphene::chain::account_object>, graphene::chain::asset)
  4.39   1117.86   120.78 22967573183     0.00     0.00  graphene::chain::operator<(graphene::chain::price const&, graphene::chain::price const&)
  4.28   1235.73   117.87 307448392     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_statistics_object, ... >::find(graphene::db::object_id_type) const
  3.74   1338.50   102.77                             sha256_block_data_order_avx
  3.36   1431.04    92.54 3183637020     0.00     0.00  graphene::chain::generic_index<graphene::chain::asset_bitasset_data_object, ... >::find(graphene::db::object_id_type) const
  3.04   1514.78    83.74 805850110     0.00     0.00  graphene::chain::generic_index<graphene::chain::asset_object, ... >::find(graphene::db::object_id_type) const
  2.56   1585.22    70.44 245765371     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_statistics_object, ... >::modify(graphene::db::object const&, std::function<void (graphene::db::object&)> const&)
  2.48   1653.40    68.18 27000000     0.00     0.00  graphene::chain::database::update_expired_feeds()
  2.10   1711.27    57.87 72953692     0.00     0.00  boost::multi_index::detail::ordered_index<boost::multi_index::composite_key<graphene::chain::limit_order_object, ... >::link_point( ... )
  1.94   1764.72    53.45 72953692     0.00     0.00  graphene::chain::generic_index<graphene::chain::limit_order_object, ... >::create(std::function<void (graphene::db::object&)> const&)
  1.87   1816.27    51.55 78522251     0.00     0.00  graphene::chain::database::get_balance(graphene::chain::account_object const&, graphene::chain::asset_object const&) const
  1.60   1860.23    43.96                             ripemd160_block_data_order

index<account_balance_object>::modify() became the first entry.

After first attempt to optimize, new profile:

---------------------- first 27 M blocks ----------------------------
764718ms th_a db_management.cpp:124 reindex ] Done reindexing, elapsed time: 5063.73962899999969522 sec

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls   s/call   s/call  name
  9.62    215.94   215.94 448458421     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_object, ... >::find(graphene::db::object_id_type) const
  8.47    406.11   190.17 20731743387     0.00     0.00  graphene::chain::operator>(graphene::chain::price const&, graphene::chain::price const&)
  7.18    567.18   161.07 390785433     0.00     0.00  graphene::chain::database::adjust_balance(graphene::db::object_id<(unsigned char)1, (unsigned char)2, graphene::chain::account_object>, graphene::chain::asset)
  4.68    672.30   105.12 307448392     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_statistics_object, ... >::find(graphene::db::object_id_type) const
  4.64    776.50   104.20 22967573183     0.00     0.00  graphene::chain::operator<(graphene::chain::price const&, graphene::chain::price const&)
  4.39    874.95    98.45                             sha256_block_data_order_avx
  3.92    962.82    87.87 3183637020     0.00     0.00  graphene::chain::generic_index<graphene::chain::asset_bitasset_data_object, ... >::find(graphene::db::object_id_type) const
  3.62   1044.04    81.22 386682199     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_balance_object, ... >::modify(graphene::db::object const&, std::function<void (graphene::db::object&)> const&)
  3.32   1118.50    74.46 805850110     0.00     0.00  graphene::chain::generic_index<graphene::chain::asset_object, ... >::find(graphene::db::object_id_type) const
  2.85   1182.43    63.93 245765371     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_statistics_object, ... >::modify(graphene::db::object const&, std::function<void (graphene::db::object&)> const&)
  2.76   1244.32    61.89 27000000     0.00     0.00  graphene::chain::database::update_expired_feeds()
  2.16   1292.90    48.58 72953692     0.00     0.00  graphene::chain::generic_index<graphene::chain::limit_order_object, ... >::create(std::function<void (graphene::db::object&)> const&)

Total replay time reduced by 1 - 5063/5633 ~= 10%.

index<account_balance_object>::modify() became the 8th entry, time used on it became around 25% of the original value ((81.22/190.17) / (382.25/220.21)).

I'll create a new PR after done cleanup.

From the profile, we can see that the position of adjust_balance call didn't change, so there is something else which need to optimize.

Note: asset_api::get_asset_holders(...) and asset_api::get_all_asset_holders() will return unordered results after this change.

abitmore · 2018-09-05T22:28:53Z

This is still needed. Latest data profiling shows more than 10% of CPU time is used to (unnecessarily) sort balances. #1099 attempted to improve it by copying some data to a new "table". A new idea is to use a secondary index and do partial sorting, E.G. store the balances in a binary heap (and this link is useful: updating data in a heap takes O(log(n)), so a heap is still not perfect for us).

pmconrad · 2019-03-01T16:34:27Z

@abitmore can you run your profiling with the latest codebase, i. e. including #1462?

abitmore · 2019-03-04T20:30:22Z

Will do. Recently I moved my dev env to a new machine, need to get comparable data. In October 2018 when reviewing #1359, I've run profiling on my old machine, to replay 27M blocks,

171404ms th_a db_management.cpp:144 reindex ] Done reindexing, elapsed time: 3672.15432099999998172 sec

In my new machine:

527258ms th_a db_management.cpp:144 reindex ] Done reindexing, elapsed time: 3879.96790899999996327 sec

1259883ms th_a db_management.cpp:144 reindex ] Done reindexing, elapsed time: 3678.45521700000017518 sec

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  Ks/call  Ks/call  name    
 16.92    332.01   332.01 391901327     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_balance_object, ... >::modify(graphene::db::object const&, std::function<void (graphene::db::object&)> const&)
 11.89    565.21   233.20 38721699988     0.00     0.00  graphene::chain::operator<(graphene::chain::price const&, graphene::chain::price const&)
 11.52    791.21   226.00 448147437     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_object, ... >::find(graphene::db::object_id_type) const
  6.70    922.56   131.35 390785470     0.00     0.00  graphene::chain::database::adjust_balance(graphene::db::object_id<(unsigned char)1, (unsigned char)2, graphene::chain::account_object>, graphene::chain::asset)
  5.78   1035.89   113.33 307142799     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_statistics_object, ... >::find(graphene::db::object_id_type) const
  3.74   1109.30    73.41 679895295     0.00     0.00  graphene::chain::generic_index<graphene::chain::asset_object, ... >::find(graphene::db::object_id_type) const
  3.39   1175.82    66.52 245765406     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_statistics_object, ... >::modify(graphene::db::object const&, std::function<void (graphene::db::object&)> const&)
  2.57   1226.25    50.43 72953692     0.00     0.00  graphene::chain::generic_index<graphene::chain::limit_order_object, ... >::create(std::function<void (graphene::db::object&)> const&)
  2.52   1275.68    49.43 72953699     0.00     0.00  boost::multi_index::detail::ordered_index<boost::multi_index::composite_key<graphene::chain::limit_order_object, ... >::link_point(...)
  2.22   1319.17    43.49 78096073     0.00     0.00  graphene::chain::database::get_balance(graphene::chain::account_object const&, graphene::chain::asset_object const&) const
  1.52   1349.05    29.88 496424860     0.00     0.00  boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >::rebalance_for_erase(boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >*, boost::multi_index::detail::ordered_index_node_compressed_base<std::allocator<char> >::parent_ref, boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >*&, boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >*&)
  1.45   1377.55    28.50 75169948     0.00     0.00  graphene::chain::database::_apply_transaction(graphene::chain::signed_transaction const&) 
  1.45   1405.90    28.35 125177353     0.00     0.00  graphene::chain::generic_index<graphene::chain::limit_order_object, boost::multi_index::multi_index_container<graphene::chain::limit_order_object, ... >::find(graphene::db::object_id_type) const
  1.44   1434.22    28.32    22671     0.00     0.00  graphene::chain::database::perform_chain_maintenance(graphene::chain::signed_block const&, graphene::chain::global_property_object const&)

abitmore · 2019-03-07T00:00:09Z

When replaying the first 27M blocks, current develop branch is significantly slower than #1359. More data will come later.

1382392ms th_a db_management.cpp:155 reindex ] Done reindexing, elapsed time: 7334.77768699999978708 sec

abitmore · 2019-03-07T01:42:32Z

I figured out why current develop branch is slower: default plugins including account_history, grouped_orders and etc are loaded. It seems the "plugins" option specified in config.ini is ignored, also, when generating a new config.ini, there is no longer a "plugins" option inside. I think it's a bug, will submit a new issue. In the meanwhile, I'll run profiling again with "plugins" option as a command line argument.

abitmore · 2019-03-07T07:22:28Z

When loading only witness plugin when running current develop branch, total execution time is slightly shorter:

3151056ms th_a db_management.cpp:155 reindex ] Done reindexing, elapsed time: 3631.68949900000006892 sec

Percentages are a bit different:

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  Ks/call  Ks/call  name    
 21.99    322.86   322.86 391901327     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_balance_object, ... >::modify(graphene::db::object const&, std::function<void (graphene::db::object&)> const&)
 15.58    551.62   228.76 38721699988     0.00     0.00  graphene::chain::operator<(graphene::chain::price const&, graphene::chain::price const&)
  4.83    622.51    70.89 193397119     0.00     0.00  graphene::chain::generic_index<graphene::chain::account_statistics_object, ... >::modify(graphene::db::object const&, std::function<void (graphene::db::object&)> const&)
  4.35    686.32    63.81 466869217     0.00     0.00  graphene::chain::balances_by_account_index::get_account_balance(graphene::db::object_id<(unsigned char)1, (unsigned char)2, graphene::chain::account_object> const&, graphene::db::object_id<(unsigned char)1, (unsigned char)3, graphene::chain::asset_object> const&) const
  3.39    736.14    49.83 72953699     0.00     0.00  boost::multi_index::detail::ordered_index<boost::multi_index::composite_key<graphene::chain::limit_order_object, ... >::link_point( ... )
  2.86    778.11    41.97 72953692     0.00     0.00  graphene::chain::generic_index<graphene::chain::limit_order_object, ... >::create(std::function<void (graphene::db::object&)> const&)
  2.36    812.69    34.58    22671     0.00     0.00  graphene::chain::database::perform_chain_maintenance(graphene::chain::signed_block const&, graphene::chain::global_property_object const&)
  1.96    841.42    28.73 125177353     0.00     0.00  graphene::db::primary_index<graphene::chain::generic_index<graphene::chain::limit_order_object, ... >::find(graphene::db::object_id_type) const
  1.78    867.50    26.08 496424851     0.00     0.00  boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >::rebalance_for_erase(boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >*, boost::multi_index::detail::ordered_index_node_compressed_base<std::allocator<char> >::parent_ref, boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >*&, boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >*&)
  1.70    892.44    24.94 55122778     0.00     0.00  graphene::db::primary_index<graphene::chain::generic_index<graphene::chain::vesting_balance_object, ... >::find(graphene::db::object_id_type) const
  1.17    909.57    17.13 8568177285     0.00     0.00  fc::impl::dynamic_storage::data() const
  1.16    926.55    16.98 238024223     0.00     0.00  graphene::chain::database::check_for_blackswan(graphene::chain::asset_object const&, bool, graphene::chain::asset_bitasset_data_object const*)
  1.10    942.76    16.21 448147437     0.00     0.00  graphene::db::primary_index<graphene::chain::generic_index<graphene::chain::account_object, ... >::find(graphene::db::object_id_type) const
  1.06    958.25    15.50 508108258     0.00     0.00  boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >::link(boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >*, boost::multi_index::detail::ordered_index_side, boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >*, boost::multi_index::detail::ordered_index_node_impl<std::allocator<char> >*)
  1.02    973.30    15.05 403072179     0.00     0.00  graphene::chain::database::check_call_orders(graphene::chain::asset_object const&, bool, bool, graphene::chain::asset_bitasset_data_object const*)
  0.87    986.02    12.72 45537860     0.00     0.00  graphene::chain::operator==(graphene::chain::price const&, graphene::chain::price const&)
  0.86    998.68    12.66 4088831657     0.00     0.00  graphene::db::object_database::get_index(unsigned char, unsigned char) const
  0.84   1011.07    12.39 390785470     0.00     0.00  graphene::chain::database::adjust_balance(graphene::db::object_id<(unsigned char)1, (unsigned char)2, graphene::chain::account_object>, graphene::chain::asset)
  0.83   1023.28    12.21 27000000     0.00     0.00  graphene::chain::database::update_last_irreversible_block()
  0.78   1034.77    11.49 20664885     0.00     0.00  graphene::chain::asset_bitasset_data_object::update_median_feeds(fc::time_point_sec)
  0.75   1045.80    11.03 41836313     0.00     0.00  graphene::chain::generic_index<graphene::chain::vesting_balance_object, ... >::modify(graphene::db::object const&, std::function<void (graphene::db::object&)> const&)
  0.72   1056.39    10.59  5524715     0.00     0.00  graphene::chain::database::get_account_stats_by_owner(graphene::db::object_id<(unsigned char)1, (unsigned char)2, graphene::chain::account_object>) const
  0.66   1066.04     9.65 157397010     0.00     0.00  graphene::chain::generic_evaluator::prepare_fee(graphene::db::object_id<(unsigned char)1, (unsigned char)2, graphene::chain::account_object>, graphene::chain::asset)
  0.61   1075.04     9.00 254774512     0.00     0.00  graphene::db::primary_index<graphene::chain::generic_index<graphene::chain::account_statistics_object, ... >::find(graphene::db::object_id_type) const

index<account_balance_object>::modify() is still the first entry. Queries are faster.

pmconrad · 2019-03-07T13:20:50Z

Thanks.
I agree with the analysis in the OP. The by_asset_balance index is expensive to maintain and mostly useless.

pmconrad · 2019-03-13T17:48:20Z

FTR: I have made various attempts to optimize operator<(price const&, price const&) - and failed miserably.
One conclusion I have drawn is that boost::multiprecision performs significantly better than fc::uint128, which is another reason to work on #1584 .

abitmore · 2019-03-13T19:57:19Z

@pmconrad I had made some efforts trying to optimize operator<(price, price) with little success, see #1094.

abitmore · 2019-03-18T22:09:45Z

AFAICT fc::uint128 is in the code for serialization. For better performance, we could/should avoid using it for routine computation. E.G.

bitshares-core/libraries/chain/protocol/fee_schedule.cpp

Lines 132 to 134 in 319d7b9

    
           auto base_value = op.visit( calc_fee_visitor( *this, op ) ); 
        
           auto scaled = fc::uint128(base_value) * scale; 
        
           scaled /= GRAPHENE_100_PERCENT;

,

bitshares-core/libraries/chain/protocol/operations.cpp

Lines 28 to 30 in 319d7b9

    
           uint64_t base_operation::calculate_data_fee( uint64_t bytes, uint64_t price_per_kbyte ) 
        
           { 
        
              auto result = (fc::uint128(bytes) * price_per_kbyte) / 1024;

,

bitshares-core/libraries/chain/asset_object.cpp

Lines 40 to 44 in 58c0c49

    
              fc::uint128 volume = current_supply.value + force_settled_volume.value; 
        
              volume *= options.maximum_force_settlement_volume; 
        
              volume /= GRAPHENE_100_PERCENT; 
        
              return volume.to_uint64(); 
        
           }

and etc.

I guess it worth a new issue. (Update: created #1660)

abitmore added performance 6 Performance Impacts flag identifying system/user efficiency, performance, etc. labels Jun 23, 2018

abitmore mentioned this issue Jun 23, 2018

Improve vote tally performance #803

Closed

2 tasks

abitmore added this to the 201807 - Next Non-Consensus-Changing Release milestone Jun 23, 2018

abitmore modified the milestones: 201807 - Next Non-Consensus-Changing Release, Future Non-Consensus-Changing Release Jun 23, 2018

abitmore mentioned this issue Jun 23, 2018

Improve replay performance #982

Open

6 tasks

abitmore self-assigned this Jun 25, 2018

abitmore modified the milestones: Future Non-Consensus-Changing Release, 201807 - Next Non-Consensus-Changing Release Jun 25, 2018

abitmore mentioned this issue Jun 25, 2018

Improve performance of database::update_expired_feeds() #1093

Closed

4 tasks

abitmore added a commit that referenced this issue Jun 26, 2018

Improve adjust_balance performance #1083

3a654f2

Note: asset_api::get_asset_holders(...) and asset_api::get_all_asset_holders() will return unordered results after this change.

abitmore mentioned this issue Jun 26, 2018

Improve adjust_balance performance #1083 #1099

Closed

abitmore mentioned this issue Jul 22, 2018

Ability to airdrop/sharedrop/pay dividends of one asset to all holders of another. #1172

Open

6 tasks

abitmore removed this from the 201807 - Next Non-Consensus-Changing Release milestone Jul 25, 2018

abitmore added this to the Future Non-Consensus-Changing Release milestone Sep 5, 2018

abitmore mentioned this issue Dec 8, 2018

Add new direct index type #1462

Merged

abitmore mentioned this issue Mar 7, 2019

"plugins" option in config.ini is ignored #1637

Closed

17 tasks

abitmore mentioned this issue Mar 18, 2019

Replace/rewrite fc::uint128 with boost::multiprecision for better performance #1660

Closed

17 tasks

abitmore removed the 2e Ready for Testing Status indicating the solution is sufficiently developed to begin testing label Mar 18, 2019

abitmore mentioned this issue Mar 19, 2019

Improve fee_schedule::calculate_fee() performance #1662

Merged

abitmore mentioned this issue Jul 11, 2019

Rich list data bitshares/bitshares-ui#2784

Open

abitmore mentioned this issue Nov 28, 2019

New BSIP: voting with non-core asset, and asset-specific committee bitshares/bsips#81

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve adjust_balance performance #1083

Improve adjust_balance performance #1083

abitmore commented Jun 23, 2018 •

edited

Loading

abitmore commented Jun 25, 2018

abitmore commented Sep 5, 2018 •

edited

Loading

pmconrad commented Mar 1, 2019

abitmore commented Mar 4, 2019 •

edited

Loading

abitmore commented Mar 7, 2019 •

edited

Loading

abitmore commented Mar 7, 2019

abitmore commented Mar 7, 2019

pmconrad commented Mar 7, 2019

pmconrad commented Mar 13, 2019

abitmore commented Mar 13, 2019

abitmore commented Mar 18, 2019 •

edited

Loading

Improve adjust_balance performance #1083

Improve adjust_balance performance #1083

Comments

abitmore commented Jun 23, 2018 • edited Loading

abitmore commented Jun 25, 2018

abitmore commented Sep 5, 2018 • edited Loading

pmconrad commented Mar 1, 2019

abitmore commented Mar 4, 2019 • edited Loading

abitmore commented Mar 7, 2019 • edited Loading

abitmore commented Mar 7, 2019

abitmore commented Mar 7, 2019

pmconrad commented Mar 7, 2019

pmconrad commented Mar 13, 2019

abitmore commented Mar 13, 2019

abitmore commented Mar 18, 2019 • edited Loading

abitmore commented Jun 23, 2018 •

edited

Loading

abitmore commented Sep 5, 2018 •

edited

Loading

abitmore commented Mar 4, 2019 •

edited

Loading

abitmore commented Mar 7, 2019 •

edited

Loading

abitmore commented Mar 18, 2019 •

edited

Loading