[Feature]: Introduce a "lazy" haste-map implementation #12231
Labels
Comments
|
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. |
|
Bump to keep this alive. |
|
PR very much welcome adding an alternative crawler! 👍 |
|
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days. |
|
This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one. |
robhogan
added a commit
to robhogan/metro
that referenced
this issue
Apr 27, 2022
Summary: ## Motivation Internally at Meta we need to make various modifications to `jest-haste-map` for use by Metro (eg instrumentation, internal cache support), which up to now we've achieved using a fork, a subclass, the `hasteMapModulePath` option and some monkey-patching. This isn't an ideal situation, and we're also coming up against the limits of what we can do whilst remaining faithful to the `jest-haste-map` API. We don't want to bloat an open source Jest subpackage (which rightly serves Jest first) with Metro-specific accommodations, and we want to be able to make changes, including breaking changes, at Metro's release cadence. ## Approach Therefore, we're publishing our lightly-modified fork of `jest-haste-map@27.3.1` as `metro-file-map` and making it a first-class Metro citizen, with the intention to decouple from Jest and make updates and modifications with fewer constraints. `jest-haste-map` is still heavily in use at Meta by Jest itself, so we'll still look to share/upstream as much as makes sense, and we'll be open to re-partitioning (for direct code sharing) or even re-converging down the line. ## Possibilities Additionally, I'm optimistic that reducing the friction around Metro-oriented file crawling/watching might enable/encourage tackling some longstanding issues, such as [symlink support](facebook#1) and [lazy loading](jestjs/jest#12231). Reviewed By: motiz88 Differential Revision: D35744151 fbshipit-source-id: 0e2b621dccec464b377b0ae0d91436149890b1d7
facebook-github-bot
pushed a commit
to facebook/metro
that referenced
this issue
May 3, 2022
Summary: Pull Request resolved: #812 ## Motivation Internally at Meta we need to make various modifications to `jest-haste-map` for use by Metro (eg instrumentation, internal cache support), which up to now we've achieved using a fork, a subclass, the `hasteMapModulePath` option and some monkey-patching. This isn't an ideal situation, and we're also coming up against the limits of what we can do whilst remaining faithful to the `jest-haste-map` API. We don't want to bloat an open source Jest subpackage (which rightly serves Jest first) with Metro-specific accommodations, and we want to be able to make changes, including breaking changes, at Metro's release cadence. ## Approach Therefore, we're publishing our lightly-modified fork of `jest-haste-map@27.3.1` as `metro-file-map` and making it a first-class Metro citizen, with the intention to decouple from Jest and make updates and modifications with fewer constraints. `jest-haste-map` is still heavily in use at Meta by Jest itself, so we'll still look to share/upstream as much as makes sense, and we'll be open to re-partitioning (for direct code sharing) or even re-converging down the line. ## Possibilities Additionally, I'm optimistic that reducing the friction around Metro-oriented file crawling/watching might enable/encourage tackling some longstanding issues, such as [symlink support](#1) and [lazy loading](jestjs/jest#12231). Reviewed By: motiz88 Differential Revision: D35744151 fbshipit-source-id: c794e9fe1886b6d9599f1ec03349a02845c3189c
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
🚀 Feature Proposal
This proposal is aimed at the react-native developer community. It introduces an experimental crawler to
jest-haste-map, which should dramatically speed upmetrostart times and reduce resource consumption (CPU, memory, disk).The new crawler is named
lazy, and should reside next to thenodeandwatchmancrawlers. Unlikenodeandwatchman,lazydoesn't eagerly collect file metadata. Instead, it reads metadata on demand, as files are requested.This breaks the design of Haste module loading, since there is no longer an in-memory listing of provided modules. In a react-native context, that's ok, as almost no one there uses Haste.
The working theory is that the file graph generated on-demand by
lazy(for bundling) will be much smaller than the entire monorepo file graph, and will therefore use fewer resources and take less time.To validate this, we need a prototype implementation and a set of measurable experiments which look at CPU time, memory usage, disk reads/writes, time to create a bundle, and time to start a bundle server and deliver a bundle to a waiting app.
The metadata found by
lazyshould be cached, just aswatchmanandnodedata is cached.lazymust support file-watching on files it has encountered, to ensure thatmetro(and otherjest-haste-mapclients) remain up to date.Motivation
metrousesjest-haste-mapto crawl the monorepo file-system. It gathers metadata on each file and watches for changes so it can keep the metadata up-to-date. The metadata includes things like a file hash and info about haste annotations within the file (e.g.@providesModule). The haste annotations allow a module to depend on another without using an explicitimportorrequirestatement.metrocrawls the file-system synchronously when it is launched, before it bundles or starts a bundle server. In large monorepos, crawling takes a long time and uses a lot of CPU, memory and disk resources.jest-haste-mapuses caching to alleviate this. Even still, the cached data can take several gigabytes of memory. At this size, the haste map causeswatchmanto become unresponsive or crash at times.Haste, as a module loading system, is not widely used by the react-native developer community. Yet, every developer pays the cost to run it on their machines and in CI loops.
metrorecently landed a PR which lets developers replace the default haste map implementation with their own code. Using this, we can customizejest-haste-mapto use thelazycrawler.Example
No response
Pitch
It is a new crawler and therefore must be in
jest-haste-mapalongside existing crawlers.The text was updated successfully, but these errors were encountered: