[Question] Extending esbuild-style logs in TypeScript

[zaydek]

I really enjoy how esbuild logs are printed. I have a very clear sense of what went wrong and why, and the colors help draw attention to the important bits.

I’m poking around in internal and found:

// ...
if terminalInfo.UseColorEscapes {
  if d.Suggestion != "" {
    return fmt.Sprintf("%s%s%s:%d:%d: %s%s: %s%s\n%s%s%s%s%s%s\n%s%s%s%s%s\n%s%s%s%s%s%s%s\n",
      textColor, textIndent, d.Path, d.Line, d.Column,
      kindColor, kind.String(),
      textResetColor, d.Message,
      colorResetDim, d.SourceBefore, colorGreen, d.SourceMarked, colorResetDim, d.SourceAfter,
      emptyMarginText(maxMargin, false), d.Indent, colorGreen, d.Marker, colorResetDim,
      emptyMarginText(maxMargin, true), d.Indent, colorGreen, d.Suggestion, colorResetDim,
      d.ContentAfter, colorReset)
  }

  return fmt.Sprintf("%s%s%s:%d:%d: %s%s: %s%s\n%s%s%s%s%s%s\n%s%s%s%s%s%s%s\n",
    textColor, textIndent, d.Path, d.Line, d.Column,
    kindColor, kind.String(),
    textResetColor, d.Message,
    colorResetDim, d.SourceBefore, colorGreen, d.SourceMarked, colorResetDim, d.SourceAfter,
    emptyMarginText(maxMargin, true), d.Indent, colorGreen, d.Marker, colorResetDim,
    d.ContentAfter, colorReset)
}
// ...

My question is, does this cover all esbuild-style logs? It would be helpful to better understand this so as I attack this in JS, I have a complete picture of how logs should appear generally.

I want to faithfully recreate these logs for JavaScript since esbuild cannot be used to run JavaScript (as talked about in #903). So I know I can use logLevel to get esbuild to emit errors for bundling / building; I simply want to extend this terminal UI to errors generally as I’ve found them to be so useful.

Previously I drafted something like this:

export function format(msg: esbuild.Message, color: (...args: unknown[]) => void): string {
  const meta = msg.location!

  const namespace = `${meta.file}:${meta.line}:${meta.column}`
  const error = `esbuild: ${msg.text}`

  let code = ""
  code += `${meta.lineText.slice(0, meta.column)}`
  code += `${color(meta.lineText.slice(meta.column, meta.column + meta.length))}`
  code += `${meta.lineText.slice(meta.column + meta.length)}`

  return `${namespace}: ${error}

  ${meta.line} ${terminal.dim("|")} ${code}
  ${" ".repeat(String(meta.line).length)} \x20 ${" ".repeat(meta.column)}${color("~".repeat(meta.length))}`
}

[zaydek]

So far I’ve got JS terminal codes / colors under control here. This solves the ‘how do I nest colors’ problem, so now I just want to make sure I have a clear sense of how esbuild errors should appear outside of esbuild? 🙂

[evanw]

Replicating this exactly will likely be kind of hard because there has been a lot of polish put into them. Some details to think about:

• When computing column counts, keep in mind that JavaScript strings are UTF-16 while the actual files are UTF-8. You may need to operate on UTF-8 buffers instead for sanity. Not sure.
• Lines that are longer than the terminal width are truncated with ... ellipses, potentially on one or both sides (this gets pretty complicated). This helps a lot with errors in minified code but is otherwise probably not necessary.
• Tab stops should be expanded for the underline to line up correctly. This is also relevant when truncating lines since the post-expanded content should be the truncated part.

Another detail that esbuild doesn't yet handle is that some Unicode code points can technically take up zero or two columns instead of just one. This isn't common with the code I work with but may be more of an issue for people working in other languages. Handling this is really gross though and involves large tables of code point values.

[zaydek]

Replicating this exactly will likely be kind of hard because there has been a lot of polish put into them.

I noticed the polish. It really shows! I wish more CLIs felt like esbuild. It has this really nice balance of discoverability / understandability which is ideal for your end users.

Thank you for elaborating on some of the more subtle implementations details. I can mostly understand logger.go now.

Another detail that esbuild doesn't yet handle is that some Unicode code points can technically take up zero or two columns instead of just one. This isn't common with the code I work with but may be more of an issue for people working in other languages. Handling this is really gross though and involves large tables of code point values.

I’ve dealt with a fair share of Unicode / regex / tables and yeah, I know what you mean. I don’t mind starting with a naive implementation for now. Eventually I’d like to respect all of your original considerations.

Here’s my WIP for what it’s worth:

WIP implementation here:

export function formatMessage(message: esbuild.Message): string {
  const loc = message.location!

  let file = ""
  file += path.relative(process.cwd(), loc.file) + ":"
  file += loc.line + ":"
  file += loc.column + 1 // One-based

  const text = message.text

  if (text.endsWith("is not defined")) loc.length = text.slice(0, -" is not defined".length).length

  let code = ""
  code += loc.lineText.slice(0, loc.column)
  code += terminal.green(loc.lineText.slice(loc.column, loc.column + loc.length))
  code += loc.lineText.slice(loc.column + loc.length)

  let gap1 = ""
  gap1 += " ".repeat(3)
  gap1 += loc.line + " "
  gap1 += "│"

  let gap2 = ""
  gap2 += " ".repeat(3)
  gap2 += " ".repeat((loc.line + " ").length)
  gap2 += "│"

  return (
    terminal.bold(` > ${file}: ${terminal.red("error:")} ${text}`) +
    `
${gap1} ${code}
${gap2} ${" ".repeat(loc.column)}${loc.length === 0 ? terminal.green("^") : terminal.green("~".repeat(loc.length))}
`
  )
}

export function formatMessages(messages: esbuild.Message[]): string {
  let str = ""
  str += messages.map(message => formatMessage(message))
  str += "\n"
  str += `${messages.length} error${messages.length === 1 ? "" : "s"}`
  return str
}

I discovered your extractErrorMessageV8 code in lib/common.ts, and used that to help me bootstrap parsing V8 stack traces.

Working implementation here:

import * as esbuild from "esbuild"
import * as fsp from "fs/promises"

// This implementation is heavily based on @evanw’s "extractErrorMessageV8"
// implementation in esbuild.
//
// https://github.com/evanw/esbuild/blob/master/lib/common.ts

export default async function parseV8ErrorStackTrace(error: any): Promise<esbuild.Message> {
  let text = "Internal error"
  let location: esbuild.Location | null = null

  try {
    text = ((error && error.message) || error) + ""
  } catch {}

  // Optionally attempt to extract the file from the stack trace, works in V8/node
  try {
    const stack = error.stack + ""
    const lines = stack.split("\n", 3)
    const at = "    at "

    // Check to see if this looks like a V8 stack trace
    if (!lines[0]!.startsWith(at) && lines[1]!.startsWith(at)) {
      let line = lines[1]!.slice(at.length)
      while (true) {
        // Unwrap a function name
        let match = /^\S+ \((.*)\)$/.exec(line)
        if (match) {
          line = match[1]!
          continue
        }

        // Unwrap an eval wrapper
        match = /^eval at \S+ \((.*)\)(?:, \S+:\d+:\d+)?$/.exec(line)
        if (match) {
          line = match[1]!
          continue
        }

        // Match on the file location
        match = /^(\S+):(\d+):(\d+)$/.exec(line)
        if (match) {
          const contents = await fsp.readFile(match[1]!, "utf8")
          const lineText = contents.split(/\r\n|\r|\n|\u2028|\u2029/)[+match[2]! - 1] || ""
          location = {
            file: match[1]!,
            namespace: "file",
            line: +match[2]!,
            column: +match[3]! - 1,
            length: 0,
            lineText: lineText, // + "\n" + lines.slice(1).join("\n"),
          }
        }
        break
      }
    }
  } catch {}

  const message: esbuild.Message = {
    detail: undefined, // Must be defined
    location,
    notes: [], // Must be defined
    text,
  }
  return message
}

Am I right to assume that when plugins crash, esbuild decorates those errors too?

[evanw]

I noticed the polish. It really shows! I wish more CLIs felt like esbuild. It has this really nice balance of discoverability / understandability which is ideal for your end users.

Thanks! That's great to hear.

I don’t mind starting with a naive implementation for now. Eventually I’d like to respect all of your original considerations.

I wouldn't do the Unicode table thing since esbuild doesn't even do that yet (and may never do it). I think it's actually really hard (or impossible?) to do this right anyway because different terminals have different levels of Unicode support so there is no one code point width table algorithm that works for all terminals. It felt appropriate to bring it up to mention what esbuild doesn't currently handle.

Am I right to assume that when plugins crash, esbuild decorates those errors too?

Yes. And esbuild also indicates where the plugin was registered as well. That looks like this:

Not sure if that's relevant for what you're doing or not.

[zaydek]

I understand Unicode and runes in the abstract but implementation is another story. I previously wrote / forked some code and created this to help me distinguish between ASCII / BMP / astral code points (for a previous project) but using regex for this kind of thing seems crude and offensive: https://github.com/codex-src/codex-wysiwyg/blob/master/src/lib/UTF8/testAlphanum/index.js.

because different terminals have different levels of Unicode support so there is no one code point width table algorithm that works for all terminals

I hadn’t considered that, but yes, terminal encoding is another story. In my own experience I haven’t noticed any hiccups with caret or tilde alignment in terminal output but I’ll be sure to report them if I come across them in the wild.

Yes. And esbuild also indicates where the plugin was registered as well. That looks like this:

Ah. I think I now understand why you are splitting stacks beyond the first stack trace in common.ts:

let lines = (e.stack + '').split('\n', 4)
// ...
location = parseStackLinesV8(streamIn, (e.stack + '').split('\n', 3), '')

It looks like you’re propagating relevant stack traces to the user in this case. Thanks for pointing that out. I wasn’t sure if you were always omitting the stack trace from the user, but now I can see that it depends.

I’m gonna close this since you’ve more than answered my questions. Thank you for taking the time to do that.

I’ll also link this since it’s somewhat relevant to terminal UI / UX: #704 (comment).

[zaydek]

@evanw I just wanted to follow up. I found this fantastic Go package that takes your stderr and converts it to HTML so I can render errors in the browser that are identical in appearance to native esbuild errors. This is great because I can leverage all of your polish in the browser now. How cool is that!

Package: https://pkg.go.dev/github.com/buildkite/terminal-to-html/v3
Demo: https://codepen.io/zaydek/pen/GRrpNVW?editors=1010

I wrapped the output with some small decoration, but other than that it’s the same!