Alternative bootstrap and/or building

[mauricioszabo]

In answer to #420 and other experiments I'm doing - I see multiple problems with the current ./script/bootstrap and ./script/build that Atom uses to start hacking on core editor's functionality and build binaries:

1. Outdated Node.JS version - last version that is 100% working is Node 12, and Node LTS is at version 16 - so it's really old. It also, for some reason, requires some specific env vars to be selected.
2. Build is slow, and you have to build over and over again if you want to change code. There also doesn't seem to be any "recompile only what changed" for example, so hacking on the editor's core is really hard at this point
3. ./script/build is quite unreliable. I tried to build different times the same commit and got different errors
4. 5000 LOC of scripts to maintain - if something breaks, who knows where is the problem?

Now, there are some problems on replacing everything - primarily, Atom is a mix of CommonJS, ESM (for some "core plug-ins"), JSX (of Etch, no React or other lib) some Babel stuff to load CoffeeScript "natively" and CoffeeScript. So, this all gets transpiled into out/app directory before bundling in a binary for Linux, Windows, and Mac. Before bundling, some of this code is required to create a V8 Snapshot that speeds up the load process of the editor (and V8 Snapshots only support JS, no other language is allowed).

Also, currently tests also expect these transpiled files to exist.

So, the question is: do we want to keep these build scripts, or do we want to try another method? If we decide for the second, how can we handle that? I'll try to contribute with some ideas too

[mauricioszabo]

So, for the bootstrap thing, I was actually able to run Atom using yarn and adding some scripts and dev dependencies (even without transpiling anything)

There are some problems though: mainly, packages. Atom expects apm to be on a very specific number of places, and apm is not a "library" that we import on Atom - it's an actual external command that gets called. It also detects "core" packages by parsing Atom's package.json and looking for some metadata that gets added on the ./script/boostrap process. I was able to bypass some of these problems, but not all, on my PR #420. The trouble is that some packages (mainly GitHub) do not work without being transpiled, and also we don't really want to use yarn (but maybe pnpm will work, and that we do want to use).

The advantage of this approach is simple: one can clone the Atom repo, with a recent Node version (LTS), run yarn install and yarn build and then start hacking the editor right away!

The disadvantages:

1. No test will run
2. APM will not work, so listing Atom packages is not working, nor is installing Atom packages under the graphical interface (but both core and use packages do get loaded)
3. I saw different behavior over the editor running under yarn start or with the ./script/build process (which is, honestly, kinda worrying for me)

[mauricioszabo]

For the build, I tried to use electron-builder. That worked well, but also gave some problems:

1. The same problem with the GitHub package happened
2. No transpilation, which means no V8 snapshots, which can mean slower boot time
3. "Core packages" do not appear on the editor's "Package" tabs, which also means that it's impossible to configure

Some of these problems can be worked around too. One solution could be to actually use just the right amount of scripts to transpile all code and generate the snapshots, and then bundle everything with electron-builder. My worry about it is quite simple: if we choose to go thought this route, then we are actually developing Atom with some code, and generating binaries with another. I'm not sure if this is a problem (I, actually, am not comfortable with it but I also am not a Javascript developer full-time) but the thing is, we will still need to keep custom-made scripts for transpilation and V8 snapshots

For now, what I did was to disable github package and do not generate the snapshot at all. I'm also generating the metadata that needs to be generated for apm to capture "core packages" before packing.

The advantage of using electron-build is that we get more targets: flatpack, AppImage, RPM, DEB, DMG, PKG, and some Windows' targets. I also tried to generate a "portable app" for Windows, and while it does work, it's actually quite slow to boot, so I'm unsure if this is the right path for Windows.

Alternative

We could always configure some of these Node.JS transpilers that support watch so we can replace the whole custom-made transpilation scripts to another tool. Again, I'm unsure how much can be replaced by it, but in this case, we keep most of the things working, probably - even the current tests - and instead of loading Atom from the "raw code" that's under src we can load it from the out folder, and bootstrap will not become that hard too... well, at least I hope so...

[aminya]

Here are the steps that the current bootstrap and build scripts take. If the electron-builder does some of these. we can remove the code for that particular step, otherwise, we must keep the current code of that step.

Bootstrapping:

• dependencies installation for scripts/package.json using npm
• dependencies installation for apm/package.json using npm
• dependencies installation for ./package.json using npm

Building:

• cleaning build directory
• copying assets
• custom transpilation
• babel transpilation
• CoffeeScript transpilation
• cson transpilation
• pegjs transpilation
• generating module cache
• less transpilation
• generating metadata
• generating API docs
• dumping symbols for native modules
• packaging the application
• generating V8 snapshots
• signing the executables (different on each platform)
• creating the installers (different on each platform)
• compressing the artifacts
• installing the application

Because I doubt electron-builder does all these steps, we have to integrate it into the current procedure.