Run octane benchmarks as part of nightly benmarks

[mhdawson]

We had a a recent case where one of our floating patches on v8 caused a performance issue. We believe this could have been caught with octane runs.

I'm thinking this means we should run octane nightly on master and publish the results.

@CurryKitten I know we run octane internally, would you be able to help putting together something that we can use to run in the community builds ?

[CurryKitten]

@mhdawson Absolutely, I'll check how we're doing the Octane currently and see how this can be adapted for the community builds

[bnoordhuis]

If it helps, here is the script I use to run Octane locally:

'use strict';
const fs = require('fs');
const vm = require('vm');

global.load = load;
global.read = read;
global.print = console.log;

load('run.js');

function load(filename) {
  vm.runInThisContext(read(filename));
}

function read(filename) {
  return fs.readFileSync(filename, 'utf8');
}

I suppose we want something that writes a file (JSON?) with benchmark numbers?

[bnoordhuis]

As a small aside, it's interesting how much slower a new VM context is because of how global property loads/stores are intercepted. Code:

'use strict';
const fs = require('fs');
const vm = require('vm');

const context = vm.createContext({ load, read, print: console.log });

load('run.js');

function load(filename) {
  vm.runInContext(read(filename), context);
}

function read(filename) {
  return fs.readFileSync(filename, 'utf8');
}

Baseline, running in the main context with the code from #75 (comment):

Richards: 16977
DeltaBlue: 30242
Crypto: 18320
RayTrace: 40033
EarleyBoyer: 5395
RegExp: 2965
Splay: 10561
SplayLatency: 17355
NavierStokes: 20545
PdfJS: 9680
Mandreel: 14538
MandreelLatency: 35356
Gameboy: 32829
CodeLoad: 9013
Box2D: 13528
zlib: 45012
Typescript: 19904
----
Score (version 9): 16300

Running in a new context:

Richards: 288
DeltaBlue: 635
Crypto: 3192
RayTrace: 1813
EarleyBoyer: 765
RegExp: 2901
Splay: 2714
SplayLatency: 9947
NavierStokes: 19886
PdfJS: 2873
Mandreel: 71.8
MandreelLatency: 377
Gameboy: 11405
CodeLoad: 7358
Box2D: 3300
zlib: 41335
Typescript: 16536
----
Score (version 9): 2676

Performance relative to the baseline:

 0.49 Mandreel
 1.07 MandreelLatency
 1.70 Richards
 2.10 DeltaBlue
 4.53 RayTrace
14.18 EarleyBoyer
17.42 Crypto
24.39 Box2D
25.70 Splay
29.68 PdfJS
34.74 Gameboy
57.31 SplayLatency
81.64 CodeLoad
83.08 Typescript
91.83 zlib
96.79 NavierStokes
97.84 RegExp

IOW, RegExp is almost as fast while Mandreel runs 200x slower.

[mhdawson]

@CurryKitten provided scripts to run through #79 which has now landed.

I've updated the jobs to run and publish results

[mhdawson]

Kicked off jobs to validate across the versions, when we do manual chart generation run

[mhdawson]

Seems like it does not run on 0.12.x:

HEAD is now at 4852334... Merge pull request #41 from chromium/revert-40-revert-39-fix-richards-box
+ cd ..
+ cp /home/iojs/build/workspace/benchmark-footprint-experimental-0.12.x/benchmarking/benchmarks/octane/octane.js .
+ node octane.js+ 
tee octane.log
/home/iojs/build/workspace/benchmark-footprint-experimental-0.12.x/octane.js:2
const fs = require('fs');
^^^^^
SyntaxError: Use of const in strict mode.
    at exports.runInThisContext (vm.js:73:16)
    at Module._compile (module.js:443:25)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)
    at startup (node.js:129:16)
    at node.js:814:3

@CurryKitten @bnoordhuis

Seems like the issue is in our script. Can you submit a PR to change it to use var instead and validate it runs ok on 0.12.X

[mhdawson]

Everything should be in place now, but the www for benchmarking is not being mirrored as expected. Raised this issue: nodejs/build#556.

You can see the chart directly at: https://benchmarking.nodejs.org/charts/octane.png

[mhdawson]

Still need help to get it showing up on bechmarking.nodejs.org

[mhdawson]

Ok octane benchmarks now showing up on benchmarking.nodejs.org