Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor benchmark suite to collect statistically significant results #3237

Closed
lucaswoj opened this issue Sep 21, 2016 · 3 comments
Closed

Comments

@lucaswoj
Copy link
Contributor

Note that map-load benchmarks pretty much always has the branch time much lower than the master one, making this particular benchmark not very useful. Maybe it's because V8 warms up on the first run.

#3224 (comment)

cc @mourner @jfirebaugh

@jfirebaugh
Copy link
Contributor

Can we use Benchmark.js? It handles repeating the benchmarked code until it's able to collect a statistically significant result.

@lucaswoj
Copy link
Contributor Author

@jfirebaugh Sounds like a good fit! I'll look into it.

@lucaswoj lucaswoj added this to the Denver milestone Sep 21, 2016
@lucaswoj lucaswoj changed the title Make "map-load" benchmark consistent Refactor benchmark suite to use "benchmark.js" Sep 22, 2016
@lucaswoj lucaswoj modified the milestone: Denver Sep 27, 2016
@lucaswoj lucaswoj self-assigned this Oct 6, 2016
@lucaswoj lucaswoj changed the title Refactor benchmark suite to use "benchmark.js" Refactor benchmark suite to use collect statistically significant results Oct 19, 2016
@lucaswoj
Copy link
Contributor Author

After toying with this a little, I think using benchmark.js specifically is more trouble than its worth. benchmark.js does not work well with asynchronous setup / teardown or non-time-based tests. We should be able to implement our own stat sig metrics.

@lucaswoj lucaswoj removed their assignment Oct 19, 2016
@lucaswoj lucaswoj changed the title Refactor benchmark suite to use collect statistically significant results Refactor benchmark suite to collect statistically significant results Jan 5, 2017
@lucaswoj lucaswoj assigned lucaswoj and unassigned lucaswoj Mar 27, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants