Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: simple performance bench for foyer-memory #336

Open
wants to merge 260 commits into
base: main
Choose a base branch
from
Open

feat: simple performance bench for foyer-memory #336

wants to merge 260 commits into from

Conversation

xiaguan
Copy link
Contributor

@xiaguan xiaguan commented Apr 14, 2024

What's changed and what's your intention?

Please explain IN DETAIL what the changes are in this PR and why they are needed. :D

As title

Checklist

  • I have written the necessary rustdoc comments
  • I have added the necessary unit tests and integration tests
  • I have passed make all (or make fast instead if the old tests are not modified) in my local environment.

Related issues or PRs (optional)

MrCroxx added 30 commits May 25, 2023 11:23
@MrCroxx
fix: add simple test for read only file store and fix bugs (foyer-rs#10)
7a2467c 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: add foyer bench, reorg workspace (foyer-rs#11)
d6a7a28 
* feat: add foyer bench, reorg workspace

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* sort cargo file

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* make fmt and clippy happy

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix bug

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
fix: fix bug with bench (foyer-rs#12)
3e9cc5f 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
ci: add asan test (foyer-rs#13)
4d3a8e5 
* ci: add asan test

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix Cargo.toml and CI

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* regen ci

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* rename CI step, test asan fail

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* regen CI

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: add license checker (foyer-rs#14)
5c09e60 
* chore: add license checker

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix license checker config

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: wrapper cache type, reorg utils, add metrics framework (foyer-…
e38ad6a 
…rs#15)

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
fix: fix read only store recovery, impl random drop (foyer-rs#16)
f44aec1 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: add metrics (foyer-rs#17)
8e80c6c 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: add log support (foyer-rs#19)
79610aa 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: add write stall config for read only file store (foyer-rs#20)
23e2612 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: introduce instrusive indexers and collections (foyer-rs#21)
506f3ef 
* feat: introduce instrusive indexers and collections

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: introduce FTL-like storage engine (foyer-rs#22)
e9714c5 
* feat: introduce FTL-like storage engine

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* update ci

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* sort cargo file

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix memory leak

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: remove unused deps (foyer-rs#28)
64b72b4 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: enable direct i/o on linux target (foyer-rs#29)
8388de6 
* feat: enable direct i/o on linux target

- enable direct i/o on linux target
- refine flusher and reclaimer

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix unit test

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: remove unused old storage engien and other components (foyer-r…
7a5624e 
…s#30)

* chore: remove unused old storage engien and other components

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* update ci

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: foyer storage bench support flexible writers and readers (foyer…
85d8341 
…-rs#31)

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: impl storage recovery (foyer-rs#32)
7c81272 
* feat: impl storage recovery

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: introduce segment fifo eviction policy (foyer-rs#35)
ad3f125 
* feat: add segment fifo eviction policy

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* export fifo fs store

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: export mods (foyer-rs#37)
98a0af7 
* feat: export mods

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* make cargo sort happy

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
fix: export extern crate (foyer-rs#38)
a8a56cc 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
Revert "fix: export extern crate (foyer-rs#38)" (foyer-rs#39)
09bf57a 
This reverts commit a8a56cc.
@MrCroxx
fix: disable O_DIRECT on non-linux targets (foyer-rs#40)
691054a 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
fix: deadlock (foyer-rs#43)
f067d1c 
* fix: deadlock

fix deadlocks:

1. exclusive lock was not `Send` but unsafe impled.
2. submit flush task await blocks `set_region_evictable`
3. no new reclamation task created after `set_region_evctable` if all
   write process are waiting

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* update ci

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: ignore Cargo.lock and bump toolchain (foyer-rs#46)
7fc8542 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
fix: gracefullly shutdown runners (foyer-rs#47)
8979084 
* fix: gracefullly shutdown runners

Use `Store::shutdown_runners()` to gracefully shutdown runners.`

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* log warn if send failed instead of panic

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: export storage config (foyer-rs#50)
1018bca 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
fix: AdmitAll & ReinsertNone no longer requires KV default (foyer-rs#51)
011a401 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
fix: region advance & recovery & seal (foyer-rs#53)
e61eb5d 
- fix region advance
- fix recovery slice destroy
- seal last dirty region when shutdown

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: introduce prometheus metrics (foyer-rs#54)
f9548e6 
* feat: introduce prometheus metrics

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* calc metrics

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix registry metrics

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
fix: recovery test (foyer-rs#55)
3c0a0ca 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
MrCroxx and others added 29 commits March 12, 2024 18:41
@MrCroxx
fix: fix build with trace future (foyer-rs#291)
12eff65 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: bump bench tools version (foyer-rs#292)
bd3fc1a 
* chore: bump bench tools version

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore: update change log, restore unreleased version

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
refactor: impl Clone for config for simplification (foyer-rs#293)
4db04a2 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: bump foyer-memory to 0.1.1 (foyer-rs#294)
fabed44 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
refactor: export DefaultCacheEventListener (foyer-rs#296)
050a3d0 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: bump foyer-memory version (foyer-rs#297)
524b645 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@xiaguan
feat: add eviction alg hit ratio bench (foyer-rs#299)
e76a499 
* feat: add eviction alg hit ratio bench

* refactor: use enum cache instead of generic cache

The enum cache is more easy to use since it only has two generic parameters. The generic cache is more like an internal interface.

Signed-off-by: xiaguan <751080330@qq.com>

* chore : sort foyer_memory's dependencies

Signed-off-by: xiaguan <751080330@qq.com>

* chore: use hakari to gen better dependencies

Signed-off-by: xiaguan <751080330@qq.com>

---------

Signed-off-by: xiaguan <751080330@qq.com>
@MrCroxx
chore: update license (foyer-rs#295)
6bea810 
* chore: update license

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore: fix license checker

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore: update license header

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@Ishiihara @MrCroxx
feat: add S3Fifo eviction for memory (foyer-rs#303)
41eebba 
* Add S3Fifo eviction for memory

* fix: refine s3fifo, fix some bugs

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: fix license

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* refactor: expose s3fifo, fix hakari, add s3fifo fuzzy test

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* bench: add s3fifo to hit ratio bench

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* test: add s3fifo uts

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
Co-authored-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: bump foyer-intrusive to 0.3.1, foyer-memory to 0.1.3 (foyer-rs…
cd612f6 
…#307)

* chore: bump foyer-memory to 0.1.3

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore: bump foyer-intrusive to 0.3.1

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: bump foyer-storage to 0.5.1 (foyer-rs#309)
5899ee4 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
fix: fix state() for s3fifo (foyer-rs#312)
b50813b 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: bump foyer-memory to 0.1.4 (foyer-rs#313)
5a410c8 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
refactor: use generic type for cursor instead of associated type (foy…
823a879 
…er-rs#314)

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
refactor: remove unstable feature let_chains and lint_reasons (foyer-…
3aa689d 
…rs#315)

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
refactor: remove usage of unstable features (foyer-rs#316)
2693669 
* refactor: remove usage of unstable features

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* refactor: some more

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
refactor: use crate allocator_api2 to make foyer build on stable (foy…
e423dc4 
…er-rs#317)

* refactor: use crate allocator_api2 to make foyer build on stable

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: use stable pipeline for ci tests, use nightly for sanitizer

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: update ci prefix key

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: try fix ci

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: update ci nightly version

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore: try resolve github issue

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
test: run CI on both linux and macos (foyer-rs#318)
3e8f334 
* test: run CI on both linux and macos

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: fix build on macos

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: try fix ci run on targets

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
doc: update README and add a simple example (foyer-rs#319)
475aee0 
* doc: update README and add a simple example

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore: update Cargo.toml

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: add msrv for foyer, update bager in readme (foyer-rs#321)
2abe3df 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
fix: do not truncate when open wal (foyer-rs#322)
cb2616e 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: bump foyer to 0.7.0 (foyer-rs#323)
48c3990 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
refactor: loose query key bound for in-memory cache (foyer-rs#324)
cc64fc9 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
feat: impl contains for in-memory cache (foyer-rs#325)
e47c090 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
perf: add dynamic dispatch bench (foyer-rs#326)
818e3f9 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
chore: fix ci coverage (foyer-rs#331)
3fda8cc 
* chore: fix ci coverage

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore: fix check

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
refactor: refine generic handle type (foyer-rs#330)
6b820fa 
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@MrCroxx
refactor: loose bound of in-memory eviction related traits (foyer-rs#334
f590f09 
)

* refactor: remove Handle bound from Eviction

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* refactor: loose bound of in-memory eviction related traits

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

---------

Signed-off-by: MrCroxx <mrcroxx@outlook.com>
@xiaguan
feat: performance becnh for foyer-memory
1c7381c 
Signed-off-by: xiaguan <751080330@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
foyer - Development Roadmap
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@xiaguan @MrCroxx @wenym1 @xxchan @TennyZhuang @Ishiihara