macro_rules: Preserve all metavariable spans in a global side table

[petrochenkov]

This PR preserves spans of tt metavariables used to pass tokens to declarative macros.
Such metavariable spans can then be used in span combination operations like Span::to to improve all kinds of diagnostics.

Spans of non-tt metavariables are currently kept in nonterminal tokens, but the long term plan is remove all nonterminal tokens from rustc parser and rely on the proc macro model with invisible delimiters (#114647, #67062).
In particular, NtIdent nonterminal (corresponding to ident metavariables) becomes easy to remove when this PR lands (#119412 does it).

The metavariable spans are kept in a global side table keyed by Spans of original tokens.
The alternative to the side table is keeping them in SpanData instead, but the performance regressions would be large because any spans from tokens passed to declarative macros would stop being inline and would work through span interner instead, and the penalty would be paid even if we never use the metavar span for the given original span.
(But also see the comment on fn maybe_use_metavar_location describing the map collision issues with the side table approach.)

There are also other alternatives - keeping the metavar span in Token or TokenTree, but associating it with Span itsel is the most natural choice because metavar spans are used in span combining operations, and those operations are not necessarily tied to tokens.

[rustbot]

r? @cjgillot

(rustbot has picked a reviewer for you, use r? to override)

[petrochenkov]

@bors try @rust-timer queue
r? @compiler-errors

[bors]

⌛ Trying commit 3212a69 with merge 3dd649f...

[bors]

☀️ Try build successful - checks-actions
Build commit: 3dd649f (3dd649ff6d61bfd47b8ba8ff60e02386e5d5922c)

[rust-timer]

Finished benchmarking commit (3dd649f): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.5%	[0.2%, 3.0%]	54
Regressions ❌ (secondary)	10.3%	[0.2%, 21.7%]	14
Improvements ✅ (primary)	-1.0%	[-1.0%, -1.0%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.5%	[-1.0%, 3.0%]	55

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.9%	[1.9%, 1.9%]	1
Regressions ❌ (secondary)	6.0%	[1.8%, 11.0%]	3
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.0%	[-4.0%, -4.0%]	1
All ❌✅ (primary)	1.9%	[1.9%, 1.9%]	1

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.7%	[1.5%, 2.0%]	2
Regressions ❌ (secondary)	11.7%	[6.7%, 17.8%]	10
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-6.6%	[-6.6%, -6.6%]	1
All ❌✅ (primary)	1.7%	[1.5%, 2.0%]	2

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 668.367s -> 668.851s (0.07%)
Artifact size: 308.49 MiB -> 308.53 MiB (0.01%)

[petrochenkov]

Okay, let's try a light version - the new table insertion logic is disabled for undelimited sequences $($tt)*.
@bors try @rust-timer queue

[bors]

⌛ Trying commit a454c64 with merge b8494cc...

[bors]

☀️ Try build successful - checks-actions
Build commit: b8494cc (b8494cc7858a9797ad6d32db68a9a524e4b1591d)

[rust-timer]

Finished benchmarking commit (b8494cc): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.5%	[0.2%, 2.6%]	40
Regressions ❌ (secondary)	7.4%	[0.2%, 15.5%]	14
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.5%	[0.2%, 2.6%]	40

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.0%	[2.0%, 2.0%]	1
Regressions ❌ (secondary)	3.7%	[1.8%, 5.6%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.9%	[-4.6%, -1.2%]	2
All ❌✅ (primary)	2.0%	[2.0%, 2.0%]	1

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.6%	[1.3%, 1.9%]	2
Regressions ❌ (secondary)	7.2%	[4.2%, 11.2%]	10
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.6%	[1.3%, 1.9%]	2

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 667.404s -> 668.154s (0.11%)
Artifact size: 308.54 MiB -> 308.53 MiB (-0.00%)

[petrochenkov]

While I'm on holidays I'm just going to run rust-perf in the background, without benchmarking anything myself.
@bors try @rust-timer queue

[bors]

⌛ Testing commit 9f8d05f with merge 4c0ba1e...

[bors]

💔 Test failed - checks-actions

[petrochenkov]

@bors r=compiler-errors,cjgillot

[bors]

📌 Commit 12d7bac has been approved by compiler-errors,cjgillot

It is now in the queue for this repository.

[bors]

⌛ Testing commit 12d7bac with merge 2bf78d1...

[bors]

☀️ Test successful - checks-actions
Approved by: compiler-errors,cjgillot
Pushing 2bf78d1 to master...

[rust-timer]

Finished benchmarking commit (2bf78d1): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.4%	[0.1%, 1.3%]	26
Regressions ❌ (secondary)	0.5%	[0.1%, 3.3%]	12
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.4%	[0.1%, 1.3%]	26

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.1%	[-2.1%, -2.1%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-2.1%	[-2.1%, -2.1%]	1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.7%	[0.7%, 0.7%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Bootstrap: 640.58s -> 641.709s (0.18%)
Artifact size: 308.86 MiB -> 308.85 MiB (-0.01%)

[nnethercote]

Some icount regressions here. Was the idea that the perf improvements in #119693 justified the regressions here?

[petrochenkov]

If regressions in the final run are smaller than in the previous run (#119673 (comment)), then #119693 is responsible for that.

In general, with this PR we are doing more work (tracking more spans), so some regressions are expected.
Also this PR enables removal of NtIdent (#119412), which brings improvements.
(I also have a span.ctxt update optimization in the pipeline that should help here, but won't be able to work on it earlier than next week.)

[pnkfelix]

• Instruction count regressions flagged against serde_derive, diesel, serde, html5ever, syn
• Most of these were anticipated in the final perf runs against the try builds.
• Marking as triaged under assumption that the reviewers deemed this change to be worth the predicted cost.

@rustbot label: +perf-regression-triaged