Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

doc: document the Melange sources / folder structure in CONTRIBUTING.md #805

Merged
merged 6 commits into from
Oct 22, 2023
Merged

doc: document the Melange sources / folder structure in CONTRIBUTING.md #805

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
46476f7
doc: document the Melange sources / folder structure in CONTRIBUTING.md
anmonteiro Oct 22, 2023
ea57f35
fix: bold, not italic
anmonteiro Oct 22, 2023
624baf6
fix: dist-es6
anmonteiro Oct 22, 2023
0591bed
Update CONTRIBUTING.md
anmonteiro Oct 22, 2023
85bf654
address javi's feedback
anmonteiro Oct 22, 2023
291a3e3
Merge branch 'anmonteiro/compiler-codebase-tour' of github.com:melang…
anmonteiro Oct 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,3 +106,72 @@ To update it, run:
npm install puppeteer
node scripts/build_reserved.js
```

## A whirlwind tour to the compiler codebase

### Folder structure:

- `bin` contains all binaries shipped in a Melange distribution.
- `docs` contains the old [BuckleScript
manual](https://melange.re/melange/Manual.html) which we host to consult
(rarely).
- `jscomp` is where the compiler implementation and some tests live:
- `common` defines the Melange `common` private library, housing code
shared by both the Melange binary and the `melange.ppx` library.
- `core` defines the `core` Melange library, containing the bulk of the
compiler backend implementation.
- `ext` contains the sources for the `ext` private library. This is a
standard library extension with additional functions and data structures
used throughout the Melange code.
- `js_parser` is a vendored copy of the [Flow
parser](https://github.com/facebook/flow/tree/main/src/parser), used in
Melange to classify `%mel.raw` JavaScript code.
- `runtime` is the Melange runtime. It has:
- the `melange.js` library.
- the `Caml_*` modules, implementing the low-level OCaml primitives.
While these are technically part of the `melange.js` library, they:
1. aren't exposed in the `Js.*` module.
2. are only ever accessed from JS: Melange internals don't address
any of these modules directly, and you shouldn't either.
3. are still exposed e.g. as `Js__Caml_array` because of Dune
namespacing implementation details.
- `stdlib` is the Melange stdlib, included in the final distribution.
- Depends on the runtime
- `others` contains the sources for the additional opt-in libraries
distributed with Melange: `melange.belt`, `melange.dom` and
`melange.node`.
- `test` has both:
- the sources for the Melange runtime tests that are executed on every
CI run
- a snapshot of their compilation to JavaScript (in
[`jscomp/test/dist`](https://github.com/melange-re/melange/tree/main/jscomp/test/dist)
and
[`jscomp/test/dist-es6`](https://github.com/melange-re/melange/tree/main/jscomp/test/dist-es6))
- **NOTE**: these snapshots are currently built manually. To do so,
anmonteiro marked this conversation as resolved.
Show resolved Hide resolved
comment the only line in
[`jscomp/dune`](https://github.com/melange-re/melange/blob/main/jscomp/dune)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe worth pointing to the current commit in main now, so that at least if folder layout changes in the future, the docs can point to the state where things were written (applies to all links to code in the document).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking the fact they're on main will let us catch if we ever break them. Meaning the guide should be updated.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking the fact they're on main will let us catch if we ever break them.

I doubt we will know we are breaking them, as there is no mechanism to warn us.

and run `dune build`.
- `ocaml-tree` has a code generation tool that produces files such as
[`core/js_record_fold.ml`](https://github.com/melange-re/melange/blob/main/jscomp/core/js_record_fold.ml),
automating some repetitive traversal tasks of the Melange JavaScript
intermediate representation.
- `ppx` has the code for `melange.ppx`, for example:
- all the `%mel.*` extensions and `@deriving` derivers are declared in
[`ppx/melange_ppx.ml`](https://github.com/melange-re/melange/blob/main/ppx/melange_ppx.ml).
- the Melange FFI (Foreign Function Interface) `external`s and attributes
are interpreted in the PPX.
Comment on lines +161 to +162
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One part that confuses me about the current setup is that the PPX is not self-contained right? In the sense that part of the interpretation of these externals happens in the PPX, but another part happens at lambda compilation time.

E.g. the ppx inlines here some int64 constants

melange/ppx/melange_ppx.ml

Line 655 in 1167ca7

pval_prim = External_ffi_types.inline_int64_primitive s;

that are later picked up by lambda

melange/jscomp/core/lam_convert.ml

Line 544 in 1167ca7

| Ffi_inline_const i -> Lam.const i

I am not sure if it's worth mentioning here that melange.ppx is not a traditional ppx, in that sense, but is designed to work with the rest of the compiler.

- the Melange PPX isn't just a traditional syntax transformation: some
attributes and functions generated by the PPX get interpreted later
during Melange compilation. This [piece of
code](https://github.com/melange-re/melange/blob/1167ca745c7ddc2b950559e53d2ebe43585f3850/jscomp/core/lam_convert.ml#L526-L544)
shows an example of that.

- `rfcs` has the Melange RFC template and the RFCs to Melange that have been
accepted to the project.
- `test` is where the Melange unit tests and the blackbox
[cram](https://dune.readthedocs.io/en/stable/tests.html#cram-tests) tests are
located.
- `vendor` includes the
[`melange-compiler-libs`](https://github.com/melange-re/melange-compiler-libs)
Git submodule.