Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite IdFactory and IdFactoryWithReuse #8769

Merged
merged 2 commits into from
Jul 17, 2024
Merged

Rewrite IdFactory and IdFactoryWithReuse #8769

merged 2 commits into from
Jul 17, 2024

Conversation

bgw
Copy link
Member

@bgw bgw commented Jul 16, 2024

Description

The real goal here was to extend IdFactory<T> to work with 64-bit ids, which I'll need soon for globally unique (and non-reusable) "execution ids" (#8771) to support the safety requirements of local uncached Vcs.

I got a little carried away and essentially rewrote this:

  • (Debatable if this is an improvement or not) ID generation re-use requires an almost-but-not-entirely-free check of the concurrent queue, so it is now opt-in using the IdFactoryWithReuse.

  • ID generation is always performed with an AtomicU64 (which shouldn't really be any more or less expensive than AtomicU32 on 64 bit architectures).

  • u32 overflow detection is performed by using a TryFrom conversion from a NonZeroU64. Previously we could only detect and panic on the first id generated after overflow. Now we should detect and panic on (basically) all ids generated after overflow.

  • New versions of concurrent-queue make the unbounded constructor const, which allows us to eliminate the use of Lazy.

  • Add a unit test for overflow detection

Testing Instructions

cargo nextest r -p turbo-tasks -p turbo-tasks-memory

Copy link

vercel bot commented Jul 16, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
examples-nonmonorepo ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jul 16, 2024 9:26pm
rust-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jul 16, 2024 9:26pm
8 Skipped Deployments
Name Status Preview Comments Updated (UTC)
examples-basic-web ⬜️ Ignored (Inspect) Visit Preview Jul 16, 2024 9:26pm
examples-designsystem-docs ⬜️ Ignored (Inspect) Visit Preview Jul 16, 2024 9:26pm
examples-gatsby-web ⬜️ Ignored (Inspect) Visit Preview Jul 16, 2024 9:26pm
examples-kitchensink-blog ⬜️ Ignored (Inspect) Visit Preview Jul 16, 2024 9:26pm
examples-native-web ⬜️ Ignored (Inspect) Visit Preview Jul 16, 2024 9:26pm
examples-svelte-web ⬜️ Ignored (Inspect) Visit Preview Jul 16, 2024 9:26pm
examples-tailwind-web ⬜️ Ignored (Inspect) Visit Preview Jul 16, 2024 9:26pm
examples-vite-web ⬜️ Ignored (Inspect) Visit Preview Jul 16, 2024 9:26pm

Copy link
Member Author

bgw commented Jul 16, 2024

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bgw and the rest of your teammates on Graphite Graphite

Copy link
Contributor

github-actions bot commented Jul 16, 2024

🟢 Turbopack Benchmark CI successful 🟢

Thanks

Copy link
Contributor

✅ This change can build next-swc

Copy link
Contributor

github-actions bot commented Jul 16, 2024

⚠️ CI failed ⚠️

The following steps have failed in CI:

  • Turbopack Rust tests (mac/win, non-blocking)

See workflow summary for details

Copy link
Member Author

bgw commented Jul 17, 2024

Merge activity

  • Jul 17, 11:57 AM EDT: @bgw started a stack merge that includes this pull request via Graphite.
  • Jul 17, 11:57 AM EDT: @bgw merged this pull request with Graphite.

@bgw bgw merged commit a80b3cc into main Jul 17, 2024
58 of 60 checks passed
@bgw bgw deleted the bgw/id-factory-rewrite branch July 17, 2024 15:57
bgw pushed a commit to vercel/next.js that referenced this pull request Jul 18, 2024
* vercel/turborepo#8767 <!-- Niklas Mischkulnig -
Update Turbopack members in labeler -->
* vercel/turborepo#8722 <!-- Niklas Mischkulnig -
Use fs and vm for async chunk loading in Node -->
* vercel/turborepo#8758 <!-- Benjamin Woodruff -
Remove unused instant return value from `CaptureFuture` -->
* vercel/turborepo#8760 <!-- Benjamin Woodruff -
Delete dead raw_vc_set module -->
* vercel/turborepo#8761 <!-- Benjamin Woodruff -
Delete dead keyed_cell module -->
* vercel/turborepo#8755 <!-- Donny/강동윤 - feat:
Enable tree shaking in next.js -->
* vercel/turborepo#8769 <!-- Benjamin Woodruff -
Rewrite IdFactory and IdFactoryWithReuse -->
* vercel/turborepo#8771 <!-- Benjamin Woodruff - Add
u64 ExecutionId type -->
* vercel/turborepo#8735 <!-- Tobias Koppers -
turbo-tasks this calls -->
* vercel/turborepo#8775 <!-- Tobias Koppers -
improve failsafe_analyse -->
* vercel/turborepo#8776 <!-- Tobias Koppers - notify
tasks when collectibles change -->
* vercel/turborepo#8778 <!-- Tobias Koppers - fix
collectibles counting -->
* vercel/turborepo#8736 <!-- Tobias Koppers -
Refactor task arguments to be a single one -->
* vercel/turborepo#8737 <!-- Tobias Koppers - remove
ConcreteTaskInput in favor of the original rust datatypes -->
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 25, 2024
### Description

The real goal here was to extend `IdFactory<T>` to work with 64-bit ids, which I'll need soon for globally unique (and non-reusable) "execution ids" (vercel/turborepo#8771) to support the safety requirements of local uncached Vcs.

I got a little carried away and essentially rewrote this:

- (Debatable if this is an improvement or not) ID generation re-use requires an almost-but-not-entirely-free check of the concurrent queue, so it is now opt-in using the `IdFactoryWithReuse`.

- ID generation is always performed with an AtomicU64 (which shouldn't really be any more or less expensive than AtomicU32 on 64 bit architectures).

- u32 overflow detection is performed by using a `TryFrom` conversion from a NonZeroU64. Previously we could only detect and panic on the first id generated after overflow. Now we should detect and panic on (basically) all ids generated after overflow.

- New versions of `concurrent-queue` make the `unbounded` constructor `const`, which allows us to eliminate the use of `Lazy`.

- Add a unit test for overflow detection

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 29, 2024
### Description

The real goal here was to extend `IdFactory<T>` to work with 64-bit ids, which I'll need soon for globally unique (and non-reusable) "execution ids" (vercel/turborepo#8771) to support the safety requirements of local uncached Vcs.

I got a little carried away and essentially rewrote this:

- (Debatable if this is an improvement or not) ID generation re-use requires an almost-but-not-entirely-free check of the concurrent queue, so it is now opt-in using the `IdFactoryWithReuse`.

- ID generation is always performed with an AtomicU64 (which shouldn't really be any more or less expensive than AtomicU32 on 64 bit architectures).

- u32 overflow detection is performed by using a `TryFrom` conversion from a NonZeroU64. Previously we could only detect and panic on the first id generated after overflow. Now we should detect and panic on (basically) all ids generated after overflow.

- New versions of `concurrent-queue` make the `unbounded` constructor `const`, which allows us to eliminate the use of `Lazy`.

- Add a unit test for overflow detection

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 29, 2024
### Description

The real goal here was to extend `IdFactory<T>` to work with 64-bit ids, which I'll need soon for globally unique (and non-reusable) "execution ids" (vercel/turborepo#8771) to support the safety requirements of local uncached Vcs.

I got a little carried away and essentially rewrote this:

- (Debatable if this is an improvement or not) ID generation re-use requires an almost-but-not-entirely-free check of the concurrent queue, so it is now opt-in using the `IdFactoryWithReuse`.

- ID generation is always performed with an AtomicU64 (which shouldn't really be any more or less expensive than AtomicU32 on 64 bit architectures).

- u32 overflow detection is performed by using a `TryFrom` conversion from a NonZeroU64. Previously we could only detect and panic on the first id generated after overflow. Now we should detect and panic on (basically) all ids generated after overflow.

- New versions of `concurrent-queue` make the `unbounded` constructor `const`, which allows us to eliminate the use of `Lazy`.

- Add a unit test for overflow detection

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 1, 2024
### Description

The real goal here was to extend `IdFactory<T>` to work with 64-bit ids, which I'll need soon for globally unique (and non-reusable) "execution ids" (vercel/turborepo#8771) to support the safety requirements of local uncached Vcs.

I got a little carried away and essentially rewrote this:

- (Debatable if this is an improvement or not) ID generation re-use requires an almost-but-not-entirely-free check of the concurrent queue, so it is now opt-in using the `IdFactoryWithReuse`.

- ID generation is always performed with an AtomicU64 (which shouldn't really be any more or less expensive than AtomicU32 on 64 bit architectures).

- u32 overflow detection is performed by using a `TryFrom` conversion from a NonZeroU64. Previously we could only detect and panic on the first id generated after overflow. Now we should detect and panic on (basically) all ids generated after overflow.

- New versions of `concurrent-queue` make the `unbounded` constructor `const`, which allows us to eliminate the use of `Lazy`.

- Add a unit test for overflow detection

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 14, 2024
* vercel/turborepo#8767 <!-- Niklas Mischkulnig -
Update Turbopack members in labeler -->
* vercel/turborepo#8722 <!-- Niklas Mischkulnig -
Use fs and vm for async chunk loading in Node -->
* vercel/turborepo#8758 <!-- Benjamin Woodruff -
Remove unused instant return value from `CaptureFuture` -->
* vercel/turborepo#8760 <!-- Benjamin Woodruff -
Delete dead raw_vc_set module -->
* vercel/turborepo#8761 <!-- Benjamin Woodruff -
Delete dead keyed_cell module -->
* vercel/turborepo#8755 <!-- Donny/강동윤 - feat:
Enable tree shaking in next.js -->
* vercel/turborepo#8769 <!-- Benjamin Woodruff -
Rewrite IdFactory and IdFactoryWithReuse -->
* vercel/turborepo#8771 <!-- Benjamin Woodruff - Add
u64 ExecutionId type -->
* vercel/turborepo#8735 <!-- Tobias Koppers -
turbo-tasks this calls -->
* vercel/turborepo#8775 <!-- Tobias Koppers -
improve failsafe_analyse -->
* vercel/turborepo#8776 <!-- Tobias Koppers - notify
tasks when collectibles change -->
* vercel/turborepo#8778 <!-- Tobias Koppers - fix
collectibles counting -->
* vercel/turborepo#8736 <!-- Tobias Koppers -
Refactor task arguments to be a single one -->
* vercel/turborepo#8737 <!-- Tobias Koppers - remove
ConcreteTaskInput in favor of the original rust datatypes -->
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 15, 2024
* vercel/turborepo#8767 <!-- Niklas Mischkulnig -
Update Turbopack members in labeler -->
* vercel/turborepo#8722 <!-- Niklas Mischkulnig -
Use fs and vm for async chunk loading in Node -->
* vercel/turborepo#8758 <!-- Benjamin Woodruff -
Remove unused instant return value from `CaptureFuture` -->
* vercel/turborepo#8760 <!-- Benjamin Woodruff -
Delete dead raw_vc_set module -->
* vercel/turborepo#8761 <!-- Benjamin Woodruff -
Delete dead keyed_cell module -->
* vercel/turborepo#8755 <!-- Donny/강동윤 - feat:
Enable tree shaking in next.js -->
* vercel/turborepo#8769 <!-- Benjamin Woodruff -
Rewrite IdFactory and IdFactoryWithReuse -->
* vercel/turborepo#8771 <!-- Benjamin Woodruff - Add
u64 ExecutionId type -->
* vercel/turborepo#8735 <!-- Tobias Koppers -
turbo-tasks this calls -->
* vercel/turborepo#8775 <!-- Tobias Koppers -
improve failsafe_analyse -->
* vercel/turborepo#8776 <!-- Tobias Koppers - notify
tasks when collectibles change -->
* vercel/turborepo#8778 <!-- Tobias Koppers - fix
collectibles counting -->
* vercel/turborepo#8736 <!-- Tobias Koppers -
Refactor task arguments to be a single one -->
* vercel/turborepo#8737 <!-- Tobias Koppers - remove
ConcreteTaskInput in favor of the original rust datatypes -->
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 16, 2024
* vercel/turborepo#8767 <!-- Niklas Mischkulnig -
Update Turbopack members in labeler -->
* vercel/turborepo#8722 <!-- Niklas Mischkulnig -
Use fs and vm for async chunk loading in Node -->
* vercel/turborepo#8758 <!-- Benjamin Woodruff -
Remove unused instant return value from `CaptureFuture` -->
* vercel/turborepo#8760 <!-- Benjamin Woodruff -
Delete dead raw_vc_set module -->
* vercel/turborepo#8761 <!-- Benjamin Woodruff -
Delete dead keyed_cell module -->
* vercel/turborepo#8755 <!-- Donny/강동윤 - feat:
Enable tree shaking in next.js -->
* vercel/turborepo#8769 <!-- Benjamin Woodruff -
Rewrite IdFactory and IdFactoryWithReuse -->
* vercel/turborepo#8771 <!-- Benjamin Woodruff - Add
u64 ExecutionId type -->
* vercel/turborepo#8735 <!-- Tobias Koppers -
turbo-tasks this calls -->
* vercel/turborepo#8775 <!-- Tobias Koppers -
improve failsafe_analyse -->
* vercel/turborepo#8776 <!-- Tobias Koppers - notify
tasks when collectibles change -->
* vercel/turborepo#8778 <!-- Tobias Koppers - fix
collectibles counting -->
* vercel/turborepo#8736 <!-- Tobias Koppers -
Refactor task arguments to be a single one -->
* vercel/turborepo#8737 <!-- Tobias Koppers - remove
ConcreteTaskInput in favor of the original rust datatypes -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants