Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compact in-memory epoch-history in logger #19

Open
hkimura opened this issue May 28, 2015 · 0 comments
Open

Compact in-memory epoch-history in logger #19

hkimura opened this issue May 28, 2015 · 0 comments
Assignees

Comments

@hkimura
Copy link
Collaborator

hkimura commented May 28, 2015

Currently we keep every single history even though they are thousands of epochs earlier.
We don't need to keep all of them in-memory. We can discard 9/10 of them, for instance.
If a gleaner is lagging behind, it might have to scan more logs due to the history compaction, so we have to find a right balance between wasted memory and wasted work.

Or, write the history out to a file.

related code:

void LoggerRef::add_epoch_history(const EpochMarkerLogType& epoch_marker) {
...
if (control_block_->epoch_history_count_ >= LoggerControlBlock::kMaxEpochHistory) {
// TASK(Hideaki) To avoid this, we should maintain sparse history, like one history per
// tens of epochs. So far kMaxEpochHistory is enough big, so it won't happen.
LOG(FATAL) << "Exceeded kMaxEpochHistory. Unexpected.";
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant