Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bitfield optimization of phase2 #120

Merged
merged 11 commits into from
Nov 9, 2020
Merged

bitfield optimization of phase2 #120

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
03da699
use bitfield
arvidn Oct 15, 2020
a362f87
test bitfield
arvidn Oct 23, 2020
c238ec5
Add BufferedFile, that incorporates read and write cache. This simpli…
arvidn Oct 26, 2020
aecf3e3
speed up bitfield, or at least make it work on compilers that don't o…
arvidn Oct 27, 2020
6c46607
introduce abstraction between phase2 and phase3 to allow lazy computa…
arvidn Oct 28, 2020
178135e
heap allocate sort buffers in SortManager, to support passing it from…
arvidn Nov 2, 2020
2d72602
heap allocate bitfield
arvidn Nov 2, 2020
397ebb3
additional tests for bitfield_index
arvidn Nov 5, 2020
bac1629
save read/write pass of table 1 by compacting it lazily
arvidn Nov 2, 2020
bea9a63
fix build with disk logging enabled on Linux/GCC
arvidn Nov 4, 2020
af2bee9
print test output on failure on CI
arvidn Nov 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 28 additions & 4 deletions tests/test.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -744,7 +744,6 @@ TEST_CASE("Sort on disk")
uint32_t iters = 250000;
uint32_t size = 32;
vector<Bits> input;
uint32_t begin = 0;
const uint32_t memory_len = 1000000;
uint8_t* memory = new uint8_t[memory_len];
SortManager manager(memory, memory_len, 16, 4, size, ".", "test-files", 0, 1);
Expand All @@ -763,7 +762,7 @@ TEST_CASE("Sort on disk")
sort(input.begin(), input.end());
uint8_t* buf3;
for (uint32_t i = 0; i < iters; i++) {
buf3 = manager.ReadEntry(begin + i * size);
buf3 = manager.ReadEntry(i * size);
input[i].ToBytes(buf);
REQUIRE(memcmp(buf, buf3, size) == 0);
}
Expand All @@ -775,7 +774,6 @@ TEST_CASE("Sort on disk")
uint32_t iters = 120000;
uint32_t size = 32;
vector<Bits> input;
uint32_t begin = 0;
const uint32_t memory_len = 1000000;
uint8_t* memory = new uint8_t[memory_len];
SortManager manager(memory, memory_len, 16, 4, size, ".", "test-files", 0, 1);
Expand All @@ -794,7 +792,7 @@ TEST_CASE("Sort on disk")
sort(input.begin(), input.end());
uint8_t* buf3;
for (uint32_t i = 0; i < iters; i++) {
buf3 = manager.ReadEntry(begin + i * size);
buf3 = manager.ReadEntry(i * size);
input[i].ToBytes(buf);
REQUIRE(memcmp(buf, buf3, size) == 0);
}
Expand Down Expand Up @@ -832,3 +830,29 @@ TEST_CASE("Sort on disk")
delete[] memory;
}
}

TEST_CASE("bitfield_index-simple")
{
std::vector<bool> const bitfield{true, true, false, true};
bitfield_index const idx(bitfield);
CHECK(idx.lookup(0, 0) == std::pair<uint64_t, uint64_t>{0,0});
CHECK(idx.lookup(0, 1) == std::pair<uint64_t, uint64_t>{0,1});

CHECK(idx.lookup(0, 3) == std::pair<uint64_t, uint64_t>{0,2});

CHECK(idx.lookup(1, 0) == std::pair<uint64_t, uint64_t>{1,0});
CHECK(idx.lookup(1, 2) == std::pair<uint64_t, uint64_t>{1,1});
CHECK(idx.lookup(3, 0) == std::pair<uint64_t, uint64_t>{2,0});
}

TEST_CASE("bitfield_index-use index")
{
std::vector<bool> bitfield(1024 * 1024, false);
CHECK(bitfield.size() == 1048576);
bitfield[1048576 - 3] = true;
bitfield[1048576 - 2] = true;
bitfield[1048576 - 1] = true;
bitfield_index const idx(bitfield);
CHECK(idx.lookup(1048576 - 3, 1) == std::pair<uint64_t, uint64_t>{0,1});
CHECK(idx.lookup(1048576 - 2, 1) == std::pair<uint64_t, uint64_t>{1,1});
}
Copy link
Contributor

@aqk aqk Nov 5, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest boundary condition tests for bitfield_index:

// One past bucket
idx.lookup(pos=0, offset=kIndexBucket)

// One past bitfield.size()

// Init a bitfield_index with a bitfield that has .size == kIndexBucket-1, kIndexBucket, kIndexBucket+1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used to have tests for unset bits, that they would throw. But this is a pretty hot function so I backed out that check. Now it's up to the caller to only use valid positions and offsets (hence the asserts)