Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add u64 ExecutionId type #8771

Merged
merged 1 commit into from
Jul 17, 2024
Merged

Add u64 ExecutionId type #8771

merged 1 commit into from
Jul 17, 2024

Conversation

bgw
Copy link
Member

@bgw bgw commented Jul 16, 2024

Description

The plan for task-local Vcs is to extend the RawVc type into:

enum RawVc {
    TaskOutput(TaskId),
    TaskCell(TaskId, CellId),
    LocalCell(ExecutionId, LocalCellId),  // new variant!
}

Where ExecutionId is a globally unique value representing the current task execution. That will be used with runtime safety checks to ensure that a given LocalCellId (which is an index into a task-local arena/vec) is valid upon read.

https://www.notion.so/vercel/Resolved-Vcs-Vc-Lifetimes-Local-Vcs-and-Vc-Refcounts-49d666d3f9594017b5b312b87ddc5bff?pvs=4

This PR extends the define_id macro to support arbitrary integer types, and uses that to define ExecutionId.

Testing Instructions

cargo nextest r -p turbo-tasks -p turbo-tasks-memory

Copy link

vercel bot commented Jul 16, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
examples-nonmonorepo ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jul 17, 2024 4:00pm
rust-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jul 17, 2024 4:00pm
8 Skipped Deployments
Name Status Preview Comments Updated (UTC)
examples-basic-web ⬜️ Ignored (Inspect) Visit Preview Jul 17, 2024 4:00pm
examples-designsystem-docs ⬜️ Ignored (Inspect) Visit Preview Jul 17, 2024 4:00pm
examples-gatsby-web ⬜️ Ignored (Inspect) Visit Preview Jul 17, 2024 4:00pm
examples-kitchensink-blog ⬜️ Ignored (Inspect) Visit Preview Jul 17, 2024 4:00pm
examples-native-web ⬜️ Ignored (Inspect) Visit Preview Jul 17, 2024 4:00pm
examples-svelte-web ⬜️ Ignored (Inspect) Visit Preview Jul 17, 2024 4:00pm
examples-tailwind-web ⬜️ Ignored (Inspect) Visit Preview Jul 17, 2024 4:00pm
examples-vite-web ⬜️ Ignored (Inspect) Visit Preview Jul 17, 2024 4:00pm

Copy link
Member Author

bgw commented Jul 16, 2024

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bgw and the rest of your teammates on Graphite Graphite

Copy link
Contributor

github-actions bot commented Jul 16, 2024

🟢 Turbopack Benchmark CI successful 🟢

Thanks

Copy link
Contributor

✅ This change can build next-swc

Copy link
Contributor

github-actions bot commented Jul 16, 2024

⚠️ CI failed ⚠️

The following steps have failed in CI:

  • Turbopack Rust tests (mac/win, non-blocking)

See workflow summary for details

@bgw bgw marked this pull request as ready for review July 16, 2024 22:49
@bgw bgw requested a review from a team as a code owner July 16, 2024 22:49
@bgw bgw requested review from arlyon and sokra July 16, 2024 22:56
Copy link
Member Author

bgw commented Jul 17, 2024

Merge activity

  • Jul 17, 11:57 AM EDT: @bgw started a stack merge that includes this pull request via Graphite.
  • Jul 17, 11:58 AM EDT: Graphite rebased this pull request as part of a merge.
  • Jul 17, 12:05 PM EDT: @bgw merged this pull request with Graphite.

bgw added a commit that referenced this pull request Jul 17, 2024
### Description

The real goal here was to extend `IdFactory<T>` to work with 64-bit ids, which I'll need soon for globally unique (and non-reusable) "execution ids" (#8771) to support the safety requirements of local uncached Vcs.

I got a little carried away and essentially rewrote this:

- (Debatable if this is an improvement or not) ID generation re-use requires an almost-but-not-entirely-free check of the concurrent queue, so it is now opt-in using the `IdFactoryWithReuse`.

- ID generation is always performed with an AtomicU64 (which shouldn't really be any more or less expensive than AtomicU32 on 64 bit architectures).

- u32 overflow detection is performed by using a `TryFrom` conversion from a NonZeroU64. Previously we could only detect and panic on the first id generated after overflow. Now we should detect and panic on (basically) all ids generated after overflow.

- New versions of `concurrent-queue` make the `unbounded` constructor `const`, which allows us to eliminate the use of `Lazy`.

- Add a unit test for overflow detection

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
Base automatically changed from bgw/id-factory-rewrite to main July 17, 2024 15:57
@bgw bgw requested a review from a team as a code owner July 17, 2024 15:57
bgw pushed a commit to vercel/next.js that referenced this pull request Jul 18, 2024
* vercel/turborepo#8767 <!-- Niklas Mischkulnig -
Update Turbopack members in labeler -->
* vercel/turborepo#8722 <!-- Niklas Mischkulnig -
Use fs and vm for async chunk loading in Node -->
* vercel/turborepo#8758 <!-- Benjamin Woodruff -
Remove unused instant return value from `CaptureFuture` -->
* vercel/turborepo#8760 <!-- Benjamin Woodruff -
Delete dead raw_vc_set module -->
* vercel/turborepo#8761 <!-- Benjamin Woodruff -
Delete dead keyed_cell module -->
* vercel/turborepo#8755 <!-- Donny/강동윤 - feat:
Enable tree shaking in next.js -->
* vercel/turborepo#8769 <!-- Benjamin Woodruff -
Rewrite IdFactory and IdFactoryWithReuse -->
* vercel/turborepo#8771 <!-- Benjamin Woodruff - Add
u64 ExecutionId type -->
* vercel/turborepo#8735 <!-- Tobias Koppers -
turbo-tasks this calls -->
* vercel/turborepo#8775 <!-- Tobias Koppers -
improve failsafe_analyse -->
* vercel/turborepo#8776 <!-- Tobias Koppers - notify
tasks when collectibles change -->
* vercel/turborepo#8778 <!-- Tobias Koppers - fix
collectibles counting -->
* vercel/turborepo#8736 <!-- Tobias Koppers -
Refactor task arguments to be a single one -->
* vercel/turborepo#8737 <!-- Tobias Koppers - remove
ConcreteTaskInput in favor of the original rust datatypes -->
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 25, 2024
### Description

The real goal here was to extend `IdFactory<T>` to work with 64-bit ids, which I'll need soon for globally unique (and non-reusable) "execution ids" (vercel/turborepo#8771) to support the safety requirements of local uncached Vcs.

I got a little carried away and essentially rewrote this:

- (Debatable if this is an improvement or not) ID generation re-use requires an almost-but-not-entirely-free check of the concurrent queue, so it is now opt-in using the `IdFactoryWithReuse`.

- ID generation is always performed with an AtomicU64 (which shouldn't really be any more or less expensive than AtomicU32 on 64 bit architectures).

- u32 overflow detection is performed by using a `TryFrom` conversion from a NonZeroU64. Previously we could only detect and panic on the first id generated after overflow. Now we should detect and panic on (basically) all ids generated after overflow.

- New versions of `concurrent-queue` make the `unbounded` constructor `const`, which allows us to eliminate the use of `Lazy`.

- Add a unit test for overflow detection

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 25, 2024
### Description

The plan for task-local Vcs is to extend the `RawVc` type into:

```
enum RawVc {
    TaskOutput(TaskId),
    TaskCell(TaskId, CellId),
    LocalCell(ExecutionId, LocalCellId),  // new variant!
}
```

Where `ExecutionId` is a globally unique value representing the current task execution. That will be used with runtime safety checks to ensure that a given `LocalCellId` (which is an index into a task-local arena/vec) is valid upon read.

https://www.notion.so/vercel/Resolved-Vcs-Vc-Lifetimes-Local-Vcs-and-Vc-Refcounts-49d666d3f9594017b5b312b87ddc5bff?pvs=4

This PR extends the `define_id` macro to support arbitrary integer types, and uses that to define `ExecutionId`.

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 29, 2024
### Description

The real goal here was to extend `IdFactory<T>` to work with 64-bit ids, which I'll need soon for globally unique (and non-reusable) "execution ids" (vercel/turborepo#8771) to support the safety requirements of local uncached Vcs.

I got a little carried away and essentially rewrote this:

- (Debatable if this is an improvement or not) ID generation re-use requires an almost-but-not-entirely-free check of the concurrent queue, so it is now opt-in using the `IdFactoryWithReuse`.

- ID generation is always performed with an AtomicU64 (which shouldn't really be any more or less expensive than AtomicU32 on 64 bit architectures).

- u32 overflow detection is performed by using a `TryFrom` conversion from a NonZeroU64. Previously we could only detect and panic on the first id generated after overflow. Now we should detect and panic on (basically) all ids generated after overflow.

- New versions of `concurrent-queue` make the `unbounded` constructor `const`, which allows us to eliminate the use of `Lazy`.

- Add a unit test for overflow detection

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 29, 2024
### Description

The plan for task-local Vcs is to extend the `RawVc` type into:

```
enum RawVc {
    TaskOutput(TaskId),
    TaskCell(TaskId, CellId),
    LocalCell(ExecutionId, LocalCellId),  // new variant!
}
```

Where `ExecutionId` is a globally unique value representing the current task execution. That will be used with runtime safety checks to ensure that a given `LocalCellId` (which is an index into a task-local arena/vec) is valid upon read.

https://www.notion.so/vercel/Resolved-Vcs-Vc-Lifetimes-Local-Vcs-and-Vc-Refcounts-49d666d3f9594017b5b312b87ddc5bff?pvs=4

This PR extends the `define_id` macro to support arbitrary integer types, and uses that to define `ExecutionId`.

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 29, 2024
### Description

The real goal here was to extend `IdFactory<T>` to work with 64-bit ids, which I'll need soon for globally unique (and non-reusable) "execution ids" (vercel/turborepo#8771) to support the safety requirements of local uncached Vcs.

I got a little carried away and essentially rewrote this:

- (Debatable if this is an improvement or not) ID generation re-use requires an almost-but-not-entirely-free check of the concurrent queue, so it is now opt-in using the `IdFactoryWithReuse`.

- ID generation is always performed with an AtomicU64 (which shouldn't really be any more or less expensive than AtomicU32 on 64 bit architectures).

- u32 overflow detection is performed by using a `TryFrom` conversion from a NonZeroU64. Previously we could only detect and panic on the first id generated after overflow. Now we should detect and panic on (basically) all ids generated after overflow.

- New versions of `concurrent-queue` make the `unbounded` constructor `const`, which allows us to eliminate the use of `Lazy`.

- Add a unit test for overflow detection

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 29, 2024
### Description

The plan for task-local Vcs is to extend the `RawVc` type into:

```
enum RawVc {
    TaskOutput(TaskId),
    TaskCell(TaskId, CellId),
    LocalCell(ExecutionId, LocalCellId),  // new variant!
}
```

Where `ExecutionId` is a globally unique value representing the current task execution. That will be used with runtime safety checks to ensure that a given `LocalCellId` (which is an index into a task-local arena/vec) is valid upon read.

https://www.notion.so/vercel/Resolved-Vcs-Vc-Lifetimes-Local-Vcs-and-Vc-Refcounts-49d666d3f9594017b5b312b87ddc5bff?pvs=4

This PR extends the `define_id` macro to support arbitrary integer types, and uses that to define `ExecutionId`.

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 1, 2024
### Description

The real goal here was to extend `IdFactory<T>` to work with 64-bit ids, which I'll need soon for globally unique (and non-reusable) "execution ids" (vercel/turborepo#8771) to support the safety requirements of local uncached Vcs.

I got a little carried away and essentially rewrote this:

- (Debatable if this is an improvement or not) ID generation re-use requires an almost-but-not-entirely-free check of the concurrent queue, so it is now opt-in using the `IdFactoryWithReuse`.

- ID generation is always performed with an AtomicU64 (which shouldn't really be any more or less expensive than AtomicU32 on 64 bit architectures).

- u32 overflow detection is performed by using a `TryFrom` conversion from a NonZeroU64. Previously we could only detect and panic on the first id generated after overflow. Now we should detect and panic on (basically) all ids generated after overflow.

- New versions of `concurrent-queue` make the `unbounded` constructor `const`, which allows us to eliminate the use of `Lazy`.

- Add a unit test for overflow detection

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 1, 2024
### Description

The plan for task-local Vcs is to extend the `RawVc` type into:

```
enum RawVc {
    TaskOutput(TaskId),
    TaskCell(TaskId, CellId),
    LocalCell(ExecutionId, LocalCellId),  // new variant!
}
```

Where `ExecutionId` is a globally unique value representing the current task execution. That will be used with runtime safety checks to ensure that a given `LocalCellId` (which is an index into a task-local arena/vec) is valid upon read.

https://www.notion.so/vercel/Resolved-Vcs-Vc-Lifetimes-Local-Vcs-and-Vc-Refcounts-49d666d3f9594017b5b312b87ddc5bff?pvs=4

This PR extends the `define_id` macro to support arbitrary integer types, and uses that to define `ExecutionId`.

### Testing Instructions

```
cargo nextest r -p turbo-tasks -p turbo-tasks-memory
```
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 14, 2024
* vercel/turborepo#8767 <!-- Niklas Mischkulnig -
Update Turbopack members in labeler -->
* vercel/turborepo#8722 <!-- Niklas Mischkulnig -
Use fs and vm for async chunk loading in Node -->
* vercel/turborepo#8758 <!-- Benjamin Woodruff -
Remove unused instant return value from `CaptureFuture` -->
* vercel/turborepo#8760 <!-- Benjamin Woodruff -
Delete dead raw_vc_set module -->
* vercel/turborepo#8761 <!-- Benjamin Woodruff -
Delete dead keyed_cell module -->
* vercel/turborepo#8755 <!-- Donny/강동윤 - feat:
Enable tree shaking in next.js -->
* vercel/turborepo#8769 <!-- Benjamin Woodruff -
Rewrite IdFactory and IdFactoryWithReuse -->
* vercel/turborepo#8771 <!-- Benjamin Woodruff - Add
u64 ExecutionId type -->
* vercel/turborepo#8735 <!-- Tobias Koppers -
turbo-tasks this calls -->
* vercel/turborepo#8775 <!-- Tobias Koppers -
improve failsafe_analyse -->
* vercel/turborepo#8776 <!-- Tobias Koppers - notify
tasks when collectibles change -->
* vercel/turborepo#8778 <!-- Tobias Koppers - fix
collectibles counting -->
* vercel/turborepo#8736 <!-- Tobias Koppers -
Refactor task arguments to be a single one -->
* vercel/turborepo#8737 <!-- Tobias Koppers - remove
ConcreteTaskInput in favor of the original rust datatypes -->
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 15, 2024
* vercel/turborepo#8767 <!-- Niklas Mischkulnig -
Update Turbopack members in labeler -->
* vercel/turborepo#8722 <!-- Niklas Mischkulnig -
Use fs and vm for async chunk loading in Node -->
* vercel/turborepo#8758 <!-- Benjamin Woodruff -
Remove unused instant return value from `CaptureFuture` -->
* vercel/turborepo#8760 <!-- Benjamin Woodruff -
Delete dead raw_vc_set module -->
* vercel/turborepo#8761 <!-- Benjamin Woodruff -
Delete dead keyed_cell module -->
* vercel/turborepo#8755 <!-- Donny/강동윤 - feat:
Enable tree shaking in next.js -->
* vercel/turborepo#8769 <!-- Benjamin Woodruff -
Rewrite IdFactory and IdFactoryWithReuse -->
* vercel/turborepo#8771 <!-- Benjamin Woodruff - Add
u64 ExecutionId type -->
* vercel/turborepo#8735 <!-- Tobias Koppers -
turbo-tasks this calls -->
* vercel/turborepo#8775 <!-- Tobias Koppers -
improve failsafe_analyse -->
* vercel/turborepo#8776 <!-- Tobias Koppers - notify
tasks when collectibles change -->
* vercel/turborepo#8778 <!-- Tobias Koppers - fix
collectibles counting -->
* vercel/turborepo#8736 <!-- Tobias Koppers -
Refactor task arguments to be a single one -->
* vercel/turborepo#8737 <!-- Tobias Koppers - remove
ConcreteTaskInput in favor of the original rust datatypes -->
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 16, 2024
* vercel/turborepo#8767 <!-- Niklas Mischkulnig -
Update Turbopack members in labeler -->
* vercel/turborepo#8722 <!-- Niklas Mischkulnig -
Use fs and vm for async chunk loading in Node -->
* vercel/turborepo#8758 <!-- Benjamin Woodruff -
Remove unused instant return value from `CaptureFuture` -->
* vercel/turborepo#8760 <!-- Benjamin Woodruff -
Delete dead raw_vc_set module -->
* vercel/turborepo#8761 <!-- Benjamin Woodruff -
Delete dead keyed_cell module -->
* vercel/turborepo#8755 <!-- Donny/강동윤 - feat:
Enable tree shaking in next.js -->
* vercel/turborepo#8769 <!-- Benjamin Woodruff -
Rewrite IdFactory and IdFactoryWithReuse -->
* vercel/turborepo#8771 <!-- Benjamin Woodruff - Add
u64 ExecutionId type -->
* vercel/turborepo#8735 <!-- Tobias Koppers -
turbo-tasks this calls -->
* vercel/turborepo#8775 <!-- Tobias Koppers -
improve failsafe_analyse -->
* vercel/turborepo#8776 <!-- Tobias Koppers - notify
tasks when collectibles change -->
* vercel/turborepo#8778 <!-- Tobias Koppers - fix
collectibles counting -->
* vercel/turborepo#8736 <!-- Tobias Koppers -
Refactor task arguments to be a single one -->
* vercel/turborepo#8737 <!-- Tobias Koppers - remove
ConcreteTaskInput in favor of the original rust datatypes -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants