Cannot render large documents with white-space suppression disabled

[shakyShane]

If you attempt to render this file with white-space suppression disabled, you end up with "RangeError: Maximum call stack size exceeded"

var dust = require("dustjs-helpers");
var fs   = require("fs");
var md   = fs.readFileSync("./docs/index.md", "utf-8");

dust.optimizers.format = function (ctx, node) {
    return node;
};

dust.loadSource(dust.compile(md, "test"));

dust.render("test", {}, function (err, out) {

    // RangeError: Maximum call stack size exceeded
});

I'm aware Dust was never intended for such use-cases, but following this speed boost #497 I am attempting to create a site-generator that is comprised mostly of partials, but will also need to crunch through large documents too.

[shakyShane]

Looking through the compiled source, it seems this can be avoided simply adding format sections in the same way you do with consecutive buffers.

For example, that file linked above will currently result in 3496 calls to .write()

But with something like this, that would be reduced to 2

// in compiler.js
  // Compacts consecutive buffer nodes into a single node
  function compactBuffers(context, node) {
    var out = [node[0]],
        memo, i, len, res;
    for (i=1, len=node.length; i<len; i++) {
      res = compiler.filterNode(context, node[i]);
      if (res) {
//        if (res[0] === 'buffer') {
        if (res[0] === 'buffer' || res[0] === "format") {
          if (memo) {
            memo[1] += getString(res)
//            memo[1] += res[1]
          } else {
            memo = res;
            out.push(res);
          }
        } else {
          memo = null;
          out.push(res);
        }
      }
    }
    return out;
  }


  function getString(node) {
    return node[0] === "buffer"
      ? node[1]
      : node[1] + node[2]
  }

This example just provided as context to my issue - but hopefully it shows what I mean.

A change along these lines, combined with the 90% reduction in compile time from #497 would make this project more than suitable for my type of application & any other that love Dust, but also need to parse large docs.

[shakyShane]

... and here's what is possible when you combine my change above, with #497

http://quick.as/oo0t157

[sethkinast]

The root of this is really the whitespace handling. Nuking the formatter from orbit is a hacky workaround that gets recommended too much as an official solution.

I tentatively agree with the grouping of buffer and format. The underlying grammar actually treats them both as buffers. They are split out to give you the ability to drop whitespace.

I will look into this more as part of not making you monkey-patch the optimizer to change the way whitespace is handled.

[shakyShane]

thanks for the reply, @sethkinast

My snippet above, as you probably guessed, does not actually work correctly. :(

As a work around for now, (as I'm not sure of the best way to do it) I'm simply changing the grammar slightly here https://github.com/linkedin/dustjs/blob/master/src/dust.pegjs#L157

instead of:

buffer "buffer"
  = e:eol w:ws*
  { return ["format", e, w.join('')].concat([['line', line], ['col', column]]) }
  / b:(!tag !raw !comment !eol c:. {return c})+
  { return ["buffer", b.join('')].concat([['line', line], ['col', column]]) }

I simply changed the format section to be

buffer "buffer"
  = e:eol w:ws*
  { return ["buffer", e + w.join('')].concat([['line', line], ['col', column]]) }
  / b:(!tag !raw !comment !eol c:. {return c})+
  { return ["buffer", b.join('')].concat([['line', line], ['col', column]]) }

And it works exactly how I want it to (without touching the buffer compacting) :0

I know it's a hack though, so it'd be nice to hear your thoughts on a better solution