Skip to content

Commit

Permalink
LID benchmarking tool (#1276)
Browse files Browse the repository at this point in the history 
* feat: LID benchmarking tool

* fix: bench thread safety

* refactor: structured logging

* refactor: postgres bulk insert

* lid bench: Add foundationdb impl

* lid fdb: Fix Tx sizing, parallel chunk puts

* lid fdb: More efficient sample generation

* feat: array of piece count / blocks per piece (#1314)

* lid bench: print add rate

* lid bench: Add retry to postgres put (#1316)

* lid bench: Make cassandra put much more robust (#1318)

* instrumentation for bench tool (#1337)

* instrument postgres

* more instrumentation

* check for err getoffsetsize

* emit metrics every 10sec

* ignore errors

* add postgres-drop

* use directly tables

* fix: go mod tidy

* use INSERT INTO instead of tmp tables

* try to catch sig

* remove transaction commit

* fixup

* add postgres-init

* fixuop

* split create and init

* fixup

* remove if not exist

---------

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* feat: batch insert queries for postgres

* feat: add flag to insert into postgres using tmp table

* refactor: merge changes from nonsense/lid-bench

* refactor: just use one database (dont create bench database)

* refactor: remove unused params

* refactor: command structure

* fix: cassandra - dont use batch insert for PayloadToPieces

* fix: create tables CQL

* fix: increase payload to pieces insert parallelism

* fix: use simple replication strategy

* feat: use yugabyte cassandra driver

* fix: remove bench binary

* update metrics endpoint

* fix random generated piece cid

* fixup

* fix: cassandra bitswap benchmark

* remove foundationdb

---------

Co-authored-by: Łukasz Magiera <magik6k@gmail.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>
  • Loading branch information
@dirkmc @magik6k @nonsense
4 people authored and LexLuthr committed Jul 20, 2023
1 parent 6bc7eb9 commit f14a081
Show file tree
Hide file tree
Showing 4 changed files with 100 additions and 34 deletions.
27 changes: 17 additions & 10 deletions extern/boostd-data/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,21 @@ require (
github.com/davecgh/go-spew v1.1.1
github.com/docker/docker v20.10.7+incompatible
github.com/docker/go-connections v0.4.0
github.com/ethereum/go-ethereum v1.11.5
github.com/filecoin-project/boost v1.4.0
github.com/filecoin-project/go-address v1.0.0
github.com/filecoin-project/go-jsonrpc v0.1.8
github.com/filecoin-project/go-state-types v0.1.10
github.com/google/uuid v1.3.0
github.com/gorilla/mux v1.8.0
github.com/influxdata/influxdb v1.8.3
github.com/ipfs/go-cid v0.3.2
github.com/ipfs/go-datastore v0.6.0
github.com/ipfs/go-ds-leveldb v0.5.0
github.com/ipfs/go-log/v2 v2.5.1
github.com/ipld/go-car/v2 v2.4.2-0.20220707083113-89de8134e58e
github.com/ipld/go-car/v2 v2.5.1
github.com/jackc/pgtype v1.10.0
github.com/lib/pq v1.10.4
github.com/mitchellh/go-homedir v1.1.0
github.com/multiformats/go-multicodec v0.6.0
github.com/multiformats/go-multihash v0.2.1
Expand All @@ -31,12 +34,13 @@ require (
go.opentelemetry.io/otel/exporters/jaeger v1.13.0
go.opentelemetry.io/otel/sdk v1.13.0
go.opentelemetry.io/otel/trace v1.13.0
golang.org/x/net v0.0.0-20220920183852-bf014ff85ad5
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
golang.org/x/net v0.4.0
golang.org/x/sync v0.1.0
)

require (
github.com/Microsoft/go-winio v0.5.2 // indirect
github.com/StackExchange/wmi v1.2.1 // indirect
github.com/btcsuite/btcd v0.22.1 // indirect
github.com/btcsuite/btcd/btcec/v2 v2.2.0 // indirect
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.1 // indirect
Expand All @@ -49,6 +53,8 @@ require (
github.com/docker/go-units v0.5.0 // indirect
github.com/go-logr/logr v1.2.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.5 // indirect
github.com/go-stack/stack v1.8.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.2 // indirect
Expand Down Expand Up @@ -90,9 +96,7 @@ require (
github.com/jbenet/goprocess v0.1.4 // indirect
github.com/klauspost/cpuid/v2 v2.1.1 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
github.com/libp2p/go-libp2p v0.23.2 // indirect
github.com/libp2p/go-libp2p-core v0.16.1 // indirect
github.com/libp2p/go-libp2p-testing v0.12.0 // indirect
github.com/libp2p/go-openssl v0.1.0 // indirect
github.com/mattn/go-isatty v0.0.16 // indirect
github.com/mattn/go-pointer v0.0.1 // indirect
Expand All @@ -112,9 +116,12 @@ require (
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/polydawn/refmt v0.0.0-20201211092308-30ac6d18308e // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible // indirect
github.com/sirupsen/logrus v1.8.1 // indirect
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/tklauser/go-sysconf v0.3.5 // indirect
github.com/tklauser/numcpus v0.2.2 // indirect
github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11 // indirect
github.com/whyrusleeping/cbor-gen v0.0.0-20220323183124-98fa8256a799 // indirect
github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f // indirect
Expand All @@ -123,11 +130,11 @@ require (
go.uber.org/atomic v1.10.0 // indirect
go.uber.org/multierr v1.8.0 // indirect
go.uber.org/zap v1.23.0 // indirect
golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e // indirect
golang.org/x/exp v0.0.0-20220916125017-b168a2c6b86b // indirect
golang.org/x/sys v0.0.0-20220919091848-fb04ddd9f9c8 // indirect
golang.org/x/text v0.3.7 // indirect
golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f // indirect
golang.org/x/crypto v0.1.0 // indirect
golang.org/x/exp v0.0.0-20230206171751-46f607a40771 // indirect
golang.org/x/sys v0.5.0 // indirect
golang.org/x/text v0.7.0 // indirect
golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df // indirect
google.golang.org/genproto v0.0.0-20220502173005-c8bf987b8c21 // indirect
google.golang.org/grpc v1.47.0 // indirect
google.golang.org/protobuf v1.28.1 // indirect
Expand Down
Loading

0 comments on commit f14a081

Please sign in to comment.