rustc: Tweak custom attribute capabilities #50120

alexcrichton · 2018-04-20T16:05:05Z

This commit starts to lay some groundwork for the stabilization of custom
attribute invocations and general procedural macros. It applies a number of
changes discussed on internals as well as a recent issue, namely:

The path used to specify a custom attribute must be of length one and cannot
be a global path. This'll help future-proof us against any ambiguities and
give us more time to settle the precise syntax. In the meantime though a bare
identifier can be used and imported to invoke a custom attribute macro. A new
feature gate, proc_macro_path_invoc, was added to gate multi-segment paths
and absolute paths.
The set of items which can be annotated by a custom procedural attribute has
been restricted. Statements, expressions, and modules are disallowed behind
two new feature gates: proc_macro_expr and proc_macro_mod.
The input to procedural macro attributes has been restricted and adjusted.
Today an invocation like #[foo(bar)] will receive (bar) as the input token
stream, but after this PR it will only receive bar (the delimiters were
removed). Invocations like #[foo] are still allowed and will be invoked in
the same way as #[foo()]. This is a breaking change for all nightly
users as the syntax coming in to procedural macros will be tweaked slightly.
Procedural macros (foo!() style) can only be expanded to item-like items by
default. A separate feature gate, proc_macro_non_items, is required to
expand to items like expressions, statements, etc.

Closes #50038

alexcrichton · 2018-04-20T16:05:15Z

r? @petrochenkov

cc @nrc, @dtolnay

petrochenkov · 2018-04-20T23:11:55Z

src/libsyntax/ext/expand.rs

@@ -539,6 +541,45 @@ impl<'a, 'b> MacroExpander<'a, 'b> {
        }
    }

+    fn extract_proc_macro_attr_input(&self, tokens: TokenStream, span: Span) -> TokenStream {
+        match tokens.trees().next() {
+            Some(TokenTree::Delimited(_, delim)) => delim.tts.into(),


This still accepts #[attr(delimited token stream) + anything else].
Could you add a test making sure that the example above and also #[attr = "something"] are rejected.

Aha excellent point, tests added and fixed

petrochenkov · 2018-04-20T23:14:45Z

src/libsyntax/ext/expand.rs

+            ExpansionKind::Items => return,
+            ExpansionKind::TraitItems => return,
+            ExpansionKind::ImplItems => return,
+            ExpansionKind::ForeignItems => return,


What happens if a proc macro attribute is applied to something else, like match arm (match 0 { #[attr] _ => {} }), or a generic parameter (fn f<#[attr] T>), or enum variant?

I suspect it already produces some kind of error, but it would be good to have a test.

Interesting! Turns out we don't expand procedural macros in those locations right now so you get an "unknown attribute" error. I've added a test to ensure it's at least an error.

This commit starts to lay some groundwork for the stabilization of custom attribute invocations and general procedural macros. It applies a number of changes discussed on [internals] as well as a [recent issue][issue], namely: * The path used to specify a custom attribute must be of length one and cannot be a global path. This'll help future-proof us against any ambiguities and give us more time to settle the precise syntax. In the meantime though a bare identifier can be used and imported to invoke a custom attribute macro. A new feature gate, `proc_macro_path_invoc`, was added to gate multi-segment paths and absolute paths. * The set of items which can be annotated by a custom procedural attribute has been restricted. Statements, expressions, and modules are disallowed behind two new feature gates: `proc_macro_expr` and `proc_macro_mod`. * The input to procedural macro attributes has been restricted and adjusted. Today an invocation like `#[foo(bar)]` will receive `(bar)` as the input token stream, but after this PR it will only receive `bar` (the delimiters were removed). Invocations like `#[foo]` are still allowed and will be invoked in the same way as `#[foo()]`. This is a **breaking change** for all nightly users as the syntax coming in to procedural macros will be tweaked slightly. * Procedural macros (`foo!()` style) can only be expanded to item-like items by default. A separate feature gate, `proc_macro_non_items`, is required to expand to items like expressions, statements, etc. Closes rust-lang#50038 [internals]: https://internals.rust-lang.org/t/help-stabilize-a-subset-of-macros-2-0/7252 [issue]: rust-lang#50038

alexcrichton · 2018-04-21T02:56:59Z

Updated!

petrochenkov · 2018-04-21T10:14:39Z

@bors r+

bors · 2018-04-21T10:14:40Z

📌 Commit 79630d4 has been approved by petrochenkov

bors · 2018-04-21T13:51:06Z

⌛ Testing commit 79630d4 with merge 222551f...

…nkov rustc: Tweak custom attribute capabilities This commit starts to lay some groundwork for the stabilization of custom attribute invocations and general procedural macros. It applies a number of changes discussed on [internals] as well as a [recent issue][issue], namely: * The path used to specify a custom attribute must be of length one and cannot be a global path. This'll help future-proof us against any ambiguities and give us more time to settle the precise syntax. In the meantime though a bare identifier can be used and imported to invoke a custom attribute macro. A new feature gate, `proc_macro_path_invoc`, was added to gate multi-segment paths and absolute paths. * The set of items which can be annotated by a custom procedural attribute has been restricted. Statements, expressions, and modules are disallowed behind two new feature gates: `proc_macro_expr` and `proc_macro_mod`. * The input to procedural macro attributes has been restricted and adjusted. Today an invocation like `#[foo(bar)]` will receive `(bar)` as the input token stream, but after this PR it will only receive `bar` (the delimiters were removed). Invocations like `#[foo]` are still allowed and will be invoked in the same way as `#[foo()]`. This is a **breaking change** for all nightly users as the syntax coming in to procedural macros will be tweaked slightly. * Procedural macros (`foo!()` style) can only be expanded to item-like items by default. A separate feature gate, `proc_macro_non_items`, is required to expand to items like expressions, statements, etc. Closes #50038 [internals]: https://internals.rust-lang.org/t/help-stabilize-a-subset-of-macros-2-0/7252 [issue]: #50038

bors · 2018-04-21T16:18:17Z

☀️ Test successful - status-appveyor, status-travis
Approved by: petrochenkov
Pushing 222551f to master...

kennytm-githubbot · 2018-04-21T16:19:50Z

📣 Toolstate changed by #50120!

Tested on commit 222551f.
Direct link to PR: #50120

💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk).
💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk).

@Manishearth

Tested on commit rust-lang/rust@222551f. Direct link to PR: <rust-lang/rust#50120> 💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk). 💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk).

nrc · 2018-04-22T20:56:37Z

@alexcrichton

The input to procedural macro attributes has been restricted and adjusted.
Today an invocation like #[foo(bar)] will receive (bar) as the input token
stream, but after this PR it will only receive bar (the delimiters were
removed). Invocations like #[foo] are still allowed and will be invoked in
the same way as #[foo()]. This is a breaking change for all nightly
users as the syntax coming in to procedural macros will be tweaked slightly.

What's the motivation for this change? The previous behaviour was intentional and spec'ed in the RFC. What's the long-term plan for handling delimiters?

alexcrichton · 2018-04-23T14:27:16Z

@nrc most of the discussion happend in #50038 with the tl;dr; (as I understand it) as:

Invocations like foo!() don't have their delimiters in the token stream
Eventually we want foo!() to know what delimiter it has (so macros like vec![] require [])
Path parsing in attributes is somewhat weird, is #[ a :: b ] macro a or macro a::b? @petrochenkov also has some good comments with the meta matcher in the OP of What exactly token streams are passed to procedural macros 1.2 #50038.
In terms of ambiguity, the most conservative route seems to be to only allow #[a ( ... )] which still allows arbitrary syntax but just not next to the name.
Eventually though we'd want to support #[a = b], and the thinking is that just like with foo!() you can learn about the delimiter, and if the delimiter is None then it's #[a = b] rather than #[a ( = b )].

Does that make sense? @petrochenkov may be better able to fill in some holes as well.

SergioBenitez · 2018-04-27T02:27:45Z

The proc_macro_path_invoc feature gate line shouldn't be applied to invocations of decl-macros. Alternatively, the decl_macro feature should imply proc_macro_path_invoc.

jcsoo · 2018-04-27T12:58:30Z

This seems to break crates that are using #![feature(use_extern_macros)] and invoking them like this:

error[E0658]: paths of length greater than one in macro invocations are currently unstable
    --> /Users/jcsoo/bobbin-dev/bobbin-dev/mcu/bobbin-stm32/stm32f74x/src/irq.rs:1413:1
     |
1413 | ::bobbin_mcu::irq!(::dma::Dma2Stream7, IrqDma, Irq70);
     | ^^^^^^^^^^^^^^^^^
     |
     = help: add #![feature(proc_macro_path_invoc)] to the crate attributes to enable

Is that intended?

alexcrichton · 2018-04-27T14:08:23Z

Ah oops yes indeed! I don't think that bang-style macros have the same ambiguity with attribute-related macros, so we should be able to unambiguously allow path-like invocations of bang macros (both decl_macro and #[proc_macro] macros).

@jcsoo or @SergioBenitez want to file a bug to track this?

SergioBenitez · 2018-04-28T04:36:57Z

@alexcrichton Submitted a PR instead: #50295

Don't feature gate bang macros on 'proc_macro_path_invoc'. Fixes oversight from #50120.

After rust-lang/rust#50120 procedural macros do not receive the parentheses as part of their attributes TokenStream. Also the proc_macro_non_items feature is necessary to use async_block.

alexcrichton force-pushed the more-proc-macro-gates branch 3 times, most recently from 21ad04d to de1e1a7 Compare April 20, 2018 16:22

pietroalbini added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 20, 2018

pietroalbini assigned petrochenkov Apr 20, 2018

petrochenkov reviewed Apr 20, 2018

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 20, 2018

alexcrichton force-pushed the more-proc-macro-gates branch from de1e1a7 to 79630d4 Compare April 21, 2018 02:56

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 21, 2018

bors merged commit 79630d4 into rust-lang:master Apr 21, 2018

This was referenced Apr 21, 2018

Add specific never search #49757

Merged

Implement tool_attributes feature (RFC 2103) #50030

Merged

alexcrichton deleted the more-proc-macro-gates branch April 23, 2018 14:24

jcsoo mentioned this pull request Apr 27, 2018

proc_macro_path_invoc feature gate should allow path-like invocation of bang macros #50285

Closed

SergioBenitez mentioned this pull request Apr 28, 2018

Don't feature gate bang macros on 'proc_macro_path_invoc'. #50295

Merged

bors added a commit that referenced this pull request Apr 28, 2018

Auto merge of #50295 - SergioBenitez:master, r=alexcrichton

1e01e22

Don't feature gate bang macros on 'proc_macro_path_invoc'. Fixes oversight from #50120.

ngg mentioned this pull request Apr 29, 2018

Fix async/await support for latest nightly rust-lang/futures-rs#994

Merged

This was referenced May 1, 2018

Fix compilation on nightly PyO3/pyo3#147

Merged

ImportError: dynamic module does not define init function PyO3/pyo3#144

Closed

This was referenced Aug 2, 2018

Remove $:meta matcher in decl macros #49629

Closed

Syntax for multiple attributes rust-lang/rfcs#2516

Closed

Stabilize 'attr_literals' feature. #53044

Merged

jebrosen mentioned this pull request Jan 4, 2019

Tracking issue for procedural macros and "hygiene 2.0" #54727

Open

mystor referenced this pull request in mystor/synstructure Nov 6, 2019

Add documentation and make matchers more flexible

2d9479a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc: Tweak custom attribute capabilities #50120

rustc: Tweak custom attribute capabilities #50120

alexcrichton commented Apr 20, 2018

alexcrichton commented Apr 20, 2018

petrochenkov Apr 20, 2018

alexcrichton Apr 21, 2018

petrochenkov Apr 20, 2018 •

edited

Loading

petrochenkov Apr 20, 2018

alexcrichton Apr 21, 2018

alexcrichton commented Apr 21, 2018

petrochenkov commented Apr 21, 2018

bors commented Apr 21, 2018

bors commented Apr 21, 2018

bors commented Apr 21, 2018

kennytm-githubbot commented Apr 21, 2018

nrc commented Apr 22, 2018

alexcrichton commented Apr 23, 2018

SergioBenitez commented Apr 27, 2018 •

edited

Loading

jcsoo commented Apr 27, 2018

alexcrichton commented Apr 27, 2018

SergioBenitez commented Apr 28, 2018

rustc: Tweak custom attribute capabilities #50120

rustc: Tweak custom attribute capabilities #50120

Conversation

alexcrichton commented Apr 20, 2018

alexcrichton commented Apr 20, 2018

petrochenkov Apr 20, 2018

Choose a reason for hiding this comment

alexcrichton Apr 21, 2018

Choose a reason for hiding this comment

petrochenkov Apr 20, 2018 • edited Loading

Choose a reason for hiding this comment

petrochenkov Apr 20, 2018

Choose a reason for hiding this comment

alexcrichton Apr 21, 2018

Choose a reason for hiding this comment

alexcrichton commented Apr 21, 2018

petrochenkov commented Apr 21, 2018

bors commented Apr 21, 2018

bors commented Apr 21, 2018

bors commented Apr 21, 2018

kennytm-githubbot commented Apr 21, 2018

nrc commented Apr 22, 2018

alexcrichton commented Apr 23, 2018

SergioBenitez commented Apr 27, 2018 • edited Loading

jcsoo commented Apr 27, 2018

alexcrichton commented Apr 27, 2018

SergioBenitez commented Apr 28, 2018

petrochenkov Apr 20, 2018 •

edited

Loading

SergioBenitez commented Apr 27, 2018 •

edited

Loading