Tracking Issue for [*const T|*mut T]::with_metadata_of

[oliver-giersch]

Feature gate: #![feature(set_ptr_value)]

Public API

impl<T: ?Sized> *mut T {
    pub fn with_metadata_of<U: ?Sized>(self, val: *const U) -> *mut U;
}

impl<T: ?Sized> *const T {
    pub fn with_metadata_of<U: ?Sized>(self, val: *const U) -> *const U;
}

Steps / History

• Implementation: adds [*mut|*const] ptr::set_ptr_value #74774
• Change type and add #[must_use]: Requested changes to [*mut T|*const T]::set_ptr_value #75407
• Reverse argument order and rename to with_metadata_of: Refactor set_ptr_value as with_metadata_of #95249
• Const argument for mutable with_metadata_of: Adjust argument type for mutable with_metadata_of (#75091) #103346
• Final commenting period (FCP)
• Stabilization PR

Unresolved Questions

• Correct argument type -- *mut ()? *mut u8? *mut Opaque?
• Is this even needed/useful given that there now also is Tracking Issue for pointer metadata APIs #81513?

[SimonSapin]

As discussed on Zulip this could be replaced with ptr::from_raw_parts(val, ptr::metadata(self)), tracked in #81513

[ChayimFriedman2]

Even if we want to keep these methods I think it's better to write them in terms of the metadata API as it's easy to reason about and less error prone than the current implementation.

[WaffleLapkin]

This needs a better motivational example. From current docs:

This function is primarily useful for allowing byte-wise pointer
arithmetic on potentially fat pointers:

#![feature(set_ptr_value)]
use core::fmt::Debug;
let mut arr: [i32; 3] = [1, 2, 3];
let mut ptr = arr.as_mut_ptr() as *mut dyn Debug;
let thin = ptr as *mut u8;
unsafe {
    ptr = thin.add(8).with_metadata_of(ptr);
    assert_eq!(*(ptr as *mut i32), 3);
    println!("{:?}", &*ptr); // will print "3"
}

This is superceeded by byte_add and co (#96283)

[dtolnay]

dyn-clone perhaps? dtolnay/dyn-clone#8

[HeroicKatora]

flexible-io is the implementation of the solution devised a while ago to the problem and another instance which originally got me invested in this PR. It relies on with_metadata_of. The full metadata API might be an optimization for later, but is way more complex than necessary here.

[bjorn3]

That will use the provenance of from rather than the provenance of onto, which will cause UB if you dereference the resulting pointer, but the provenance of from doesn't allow accessing this address. Miri is entirely correct in rejecting this. With rustc it may work by accident or you may get a miscompilation.

[GnomedDev]

For the "Is this even needed/useful given that there now also is #81513?", I feel like this is super useful even with or without the metadata APIs.

• Without (as stable is now) it allows for working with metadata in a limited fashion without assuming layout (which people are, and will do if not given the correct tools).
• With, this is still useful as a higher level helper function, and doesn't block any of the metadata API stuff as a with_metadata function could be added to attach a Pointee::Metadata to an existing pointer in future.

I'd love to see this stabilized, if that is a good resolution to the last question needed answering.

[Noratrieb]

@rust-lang/libs-api see above

[scottmcm]

Hmm, I think the primary alternative to this would be something like rust-lang/libs-team#246 -- instead of copying the metadata from a full other pointer, just remove the blocker to letting you pass around the metadata directly.

It's not obvious to me that people really ever want this version where you need to pass a full pointer, just to ignore its address. Or is there some situation I'm not thinking of where that's really better than doing a let (p, m) = pointer.parts(); then p.whatever().with_metadata(m)?

[GnomedDev]

As I said above @scottmcm:

• Without pointer metadata: This is a simpler API that can be stable now to allow people to do what they are already doing but soundly (without assuming fat pointer layout).
• With pointer metadata: This is a helper function for the situation where someone does have a full pointer and can avoid splitting it into parts just to add the metadata on.

My motivating example is https://github.com/andylokandy/smallbox which has been assuming fat pointer layout for years and I recently added a nightly feature which uses this API and strict provenance. Since strict provenance is in the process of being stabilized, this being stabilized as well would be amazing for smallbox.

[HeroicKatora]

The ACP is not about stabilizing <T as Pointee>::Metadata but rather only a wrapper. @scottmcm Would it be possible to reflect that in the Unresolved questions of the tracking issue #75091 (comment) that only point to the Pointee::Metadata trait.

That is, a way to represent *const T without an address, for in cases where that address is redundant as here. The value for library authors in having the method proposed here is for storing / dealing with a the unsized pointer whose address is ignored anyways since the provenance is also lost when passed as argument. The proposed struct thus reflects the mechanism quite well. If we had an adjusted type such as that then the signature here might as well be:

• <*const T>::with_metadata_of<U>(PointeeMetadata<U>)

I still think that the with_metadata_of method is more intuitive in usage than any from_raw_parts design, as explained in the PR that changed it originally. However, rust-lang/libs-team#246 's API sketch could be reworked into a similar design. There's no need to do as much with free methods as in the sketch. For instance such a compromise could look like:

pub struct Metadata<T: ?Sized>(<T as Pointee>::Metadata);

impl<T: ?Sized> Metadata<T> {
    /// Not taking up space of `new() where T: Sized`.
    pub fn for_ptr(ptr: *const T) -> Self;
}

// impl for Metadata<T>: Copy, Eq, Ord, Hash

impl<T: ?Sized> *const T {
    pub fn metadata(self) -> Metadata<T>;
    pub fn with_metadata_of<U: ?Sized>(self, other: Metadata<U>) -> *const U;
}

impl<T: ?Sized> *mut T {
    pub fn metadata(self) -> Metadata<T>;
    pub fn with_metadata_of<U: ?Sized>(self, other: Metadata<U>) -> *mut U;
}

---

I've got a bunch of lines of reasoning I pondered going in the other direction but ultimately decided they wouldn't be persuasive:

In terms of simple vs. complex there is prior art: mem::size_of::<T>() vs Layout::new::<T>().size(). It also happens that the semantics of the composite are simpler than the semantics of its parts. This also reduce the number of feature interactions that might complicate any design phases. (Note: size was const-stable before Layout was stabilized and then it took 24 versions to transfer that). Weak because: I don't think time should be too influential in the design of such a smallish feature. And also the alternative is almost as simple.

Answering the question of whether usability, we can look at rustc's usage. I don't think it matters greatly to library authors and usage whether metadata would be stored on its own or in a fat-ptr with null address.

• rc/arc instantly use the metadata pointer after an unsizing cast. That's also addressed by the alternative: "It's not practical to provide unsizing support directly for ::Metadata, but it can be added relatively straightforwardly for a proper Metadata struct". Slightly more verbose but not really more complex:
  • mem.with_metadata_of((ptr as *const ArcInner<T>).metadata()) (in case no unsizing cast and no method is provided)
  • mem.with_metadata_of(ptr.metadata() as Metadata<ArcInner<T>>)
  • mem.with_metadata_of(ptr.metadata().cast<ArcInner<T>>())
• The use in pointer types such as byte_offset is a little odd. They do __modify_address(self).with_metadata_of(self). It shouldn't be problematic to split this into two lines or adding the metadata constructor but it would be a little noisy. Alas it's not entirely clear if they should then be written with from_raw_parts or not. (In particular mask which calls an intrinsic already operating on ()-pointers).