Support meaningful spans in the stable version of proc-macro2

[mystor]

This doesn't add support for the diagnostic methods on Span in stable rust. Perhaps we should add support for that, but I'm not sure what it would look like & I don't think we want a diagnostics engine in proc-macro2.

It does add support for things like fetching the line and column numbers of the start and end of a span.

[mystor]

r? @alexcrichton (apparently I can't set you as the reviewer on this repo?)

[alexcrichton]

Looks great to me! I forget what came out of our discussion though, did we figure it was best to hold off on this until there's a "more solid" use case? Or did we decide to go ahead and merge?

[jimblandy]

OOooh, oooh, you definitely decided to go ahead and merge!! ;)

[mystor]

@alexcrichton Blandy apparently wants it, so I figure we might as well merge it.

[alexcrichton]

@jimblandy heh now I'm curious, do you have something that'd benefit from this? If so, could you explain it a bit as well? (to know if this is sufficient)

[jimblandy]

I'd like to implement a crate for Rust source-to-source transformations that preserve formatting and comments, along the lines of the recast library for JavaScript:

The magic of Recast is that it reprints only those parts of the syntax tree that you modify. In other words, the following identity is guaranteed:

recast.print(recast.parse(source)).code === source

Whenever Recast cannot reprint a modified node using the original source code, it falls back to using a generic pretty printer. So the worst that can happen is that your changes trigger some harmless reformatting of your code.

I just learned about the rerast tool for writing Rust; it seems very similar to what I was hoping to write. It simply links with the compiler directly, and requires nightly.

[mystor]

With @Manishearth's work on improving the performance of the find method in rust-lang/rust#46735, it probably won't be useful to have this dependency on memchr anymore, so we might want to drop it.

[Manishearth]

Oh that's why you wanted this.

Sneaky, sneaky 😺

[mystor]

As of rust-lang/rust#46690, the fields have been made public in the proc-macro implementation, so we can probably do a wrap of this type now.

[alexcrichton]

Hm ok so I was about to merge this but it ended up hitting a snag and at this point I think I'd prefer to not merge it.

I originally was going to drop the memchr dep (no need for speed, this crate prioritizes build times) and update LineColumn with a custom type now that support has landed upstream. It turns out, however, that there's more breakage on nightly included with rust-lang/rust#46335.

Overall I feel like these APIs are much more of a moving target on nightly than the initial set of APIs. The original purpose of this crate was to allow authors today to use this crate today on stable and then once procedural macros are stable basically instantly switch over to get better spans. From a stabilization point of view it's hard for me to see how spans and such will make the cut initially as opposed to the apis already here.

In that sense I fear that a feature like this will promote creation of libraries that aren't actually compatible with the first pass of stabilizing procedural macros...

[mystor]

Perhaps we want to have a tool to build syn on top of a separate library which works on stable rust and works for situations where we want to run syn parsing outside of rustc?

There are uses like rust-cpp and @jimblandy's use case which need to have this sort of functionality for reading files from disk, and I think it's important that we find some way to support them with syn. I think that this is one approach to it, but if we want proc-macro2 to be a testing bed for the minimal APIs which will be stabilized on the first proc-macro pass, then I think we need a separate crate for that.

I'm not sure what that would look like in syn? perhaps we could move proc-macro2 behind a default feature flag, and require people to specify --no-default-features --features tokenizer,parsing,printing,full or something like that to run without linking to proc-macro?

We could also make syn or something like that be generic over a particular parsing implementation defined by a trait? I'm worried about that affecting compile times though. I can perhaps look into what that impact would look like, if I added some sort of IntoTokenBuffer trait which produced a SynomBuffer-like object. The big worry is that making everything generic over the span type could kill compile times. I might be able to use some trickery to erase that type inside the parser & fix that problem. Not sure.

[mystor]

@alexcrichton How would you feel about landing this behind an off-by-default feature flag like "experimental-apis" to discourage consumers?

[alexcrichton]

I think one thing that worked well with rayon, for example, is an env var rather than a Cargo feature. That way you have to always explicitly opt into "unstable APIs" which I'd personally be fine doing.

[mystor]

I ran into rust-lang/rust#47077 while updating the APIs, so I had to do some unfortunate hacky stuff to work around it. Hopefully we can remove that in the future.

[mystor]

@alexcrichton Any chance you could look at this & see if we can merge it?

[alexcrichton]

Looks great, thanks!

[mashedcode]

I came across this PR when trying to preserve white-space when converting a TokenStream into a String. It seems to me that no one has worked on this yet so I wrote proc-macro2-whitespace.