Skip to content

Commit

Permalink
test(storage): more tweaks to ranged reads (#9400)
Browse files Browse the repository at this point in the history
The benchmark would perform a range read with 50% probability, but some
ranged reads were "full" as the range was larger than the object. I
found it hard to reason about this, so now only the range size controls
whether the read is full or not.

I also rediscovered a quirk of the JSON API, that it is better avoided:
reading the last 0 bytes actually returns all the bytes. Maybe we should
fix this in the client library itself, for now just avoid running into
it.
  • Loading branch information
coryan committed Jul 1, 2022
1 parent d52655a commit 36396e4
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 6 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -369,15 +369,19 @@ void RunThread(ThroughputOptions const& options, std::string const& bucket_name,
options.minimum_read_size, options.maximum_read_size,
options.read_size_quantum);

auto const read_range_enabled =
options.minimum_read_size != options.maximum_read_size;
auto read_range_generator = [&](auto& g, std::int64_t object_size)
-> absl::optional<std::pair<std::int64_t, std::int64_t>> {
if (!read_range_enabled || !std::bernoulli_distribution{}(g)) {
return absl::nullopt;
}
auto offset = (std::min)(object_size, read_offset_generator(g));
auto size = (std::min)(object_size - offset, read_size_generator(g));
// This makes it easier to control the ratio of ranged vs. full reads from
// the command-line. To make more full reads happen set the read range size
// to be larger than the object sizes. The larger this read range size is,
// the higher the proportion of full range reads.
if (offset == 0 && size == object_size) return absl::nullopt;
// The REST API has a quirk: reading the last 0 bytes returns all the bytes.
// Just read the *first* 0 bytes in that case. Note that `size == 0` is
// implied by the initialization to `min(object_size - offset, ...)`.
if (offset == object_size) return std::make_pair(0, 0);
return std::make_pair(offset, size);
};

Expand Down
2 changes: 1 addition & 1 deletion google/cloud/storage/benchmarks/throughput_options.cc
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ namespace {
Status ValidateQuantizedRange(std::string const& name, std::int64_t minimum,
std::int64_t maximum, std::int64_t quantum) {
using ::google::cloud::StatusCode;
if (minimum > maximum) {
if (minimum > maximum || minimum < 0 || maximum < 0) {
std::ostringstream os;
os << "Invalid range for " << name << " [" << minimum << ',' << maximum
<< "]";
Expand Down

0 comments on commit 36396e4

Please sign in to comment.