Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/experiment scaling #375

Draft
wants to merge 14 commits into
base: develop
Choose a base branch
from
Draft

Conversation

rfhaque
Copy link
Collaborator

@rfhaque rfhaque commented Sep 25, 2024

Draft implementation of the scaling functionality for benchmarks. Currently only strong scaling is implemented. Works for amg2023/openmp
bin/benchpark experiment init --dest ./experiments kripke programming_model=openmp scaling=strong strong-scaling-factor=2 strong-scaling-num-exprs=3

and kripke/openmp
bin/benchpark experiment init --dest ./experiments amg2023 programming_model=openmp experiment=strong strong-scaling-factor=2 strong-scaling-num-exprs=4

@github-actions github-actions bot added the experiment New or modified experiment label Sep 25, 2024
Comment on lines 5 to 27
variant(
"strong-scaling-factor",
default="2",
description="Strong-scaling factor (factor by which to increase resources)",
)

variant(
"strong-scaling-num-exprs",
default="4",
description="Number of strong-scaling experiments",
)

variant(
"weak-scaling-factor",
default="2",
description="Weak-scaling factor (factor by which to increase resources and problem sizes)",
)

variant(
"weak-scaling-num-exprs",
default="4",
description="Number of weak-scaling experiments",
)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to implement conditional variants so we can make this clearer. This could be

variant("scaling", default="none", values=("none", "strong", "weak"), description="...")

variant("scaling-factor", default=2, values=lambda x: x >= 2, when="scaling=weak", description="...")
variant("scaling-factor", default=2, values=lambda x: x >= 2, when="scaling=strong", description="...")

variant("scaling-iterations", default=4, values=lambda x: x > 1, when="scaling=weak", description="...")
variant("scaling-iterations", default=4, values=lambda x: x > 1, when="scaling=strong", description="...")

And with a little additional syntax beyond that we could get rid of the duplicate lines.

from benchpark.directives import variant


class Scaling(object):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would call this ScalingExperiment instead of just Scaling, I think it's a bit clearer that way

Comment on lines 35 to 48
def generate_strong_scaling_parameters(self, initial_resource_list: list):
scaling_factor = int(self.spec.variants["strong-scaling-factor"][0])
num_exprs = int(self.spec.variants["strong-scaling-num-exprs"][0]) - 1
round_robin_order = self.compute_round_robin_order(initial_resource_list)
resource_list = [[x] for x in initial_resource_list]

while num_exprs > 0:
for idx in round_robin_order:
for i, r in enumerate(resource_list):
r.append(r[-1]*scaling_factor if i == idx else r[-1])
num_exprs=num_exprs-1
if not num_exprs:
break
return resource_list
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this method can be simplified slightly

Suggested change
def generate_strong_scaling_parameters(self, initial_resource_list: list):
scaling_factor = int(self.spec.variants["strong-scaling-factor"][0])
num_exprs = int(self.spec.variants["strong-scaling-num-exprs"][0]) - 1
round_robin_order = self.compute_round_robin_order(initial_resource_list)
resource_list = [[x] for x in initial_resource_list]
while num_exprs > 0:
for idx in round_robin_order:
for i, r in enumerate(resource_list):
r.append(r[-1]*scaling_factor if i == idx else r[-1])
num_exprs=num_exprs-1
if not num_exprs:
break
return resource_list
def generate_strong_scaling_parameters(self, initial_resource_list: list):
scaling_factor = int(self.spec.variants["strong-scaling-factor"][0])
num_exprs = int(self.spec.variants["strong-scaling-num-exprs"][0])
round_robin_order = self.compute_round_robin_order(initial_resource_list)
resource_list = [[x] for x in initial_resource_list]
for i in range(num_exprs - 1):
idx = (i + round_robin_order[0]) * len(initial_resource_list)
for r_idx, resource in enumerate(resource_list):
next_value = resource[-1] * scaling_value if r_idx == idx else resource[-1]
resource.append(next_value)
return resource_list

break
return resource_list

def generate_weak_scaling_parameters(self, initial_resource_list: list, initial_problem_size_list: list):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's necessarily safe for us to assume that the resource requirements and the problem size are expressed in such neatly compatible ways. I think we should probably assume these could be different dimensions (e.g. resource list is a scalar, problem size is 3D) and round-robin them each separately.

@becker33
Copy link
Collaborator

The variant logic can be simplified once we merge #379

@pearce8 pearce8 marked this pull request as draft September 27, 2024 19:47
@pearce8 pearce8 added the WIP A work-in-progress not yet ready to commit label Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
experiment New or modified experiment WIP A work-in-progress not yet ready to commit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants