Proposed refactoring to implement core::io

[fluffysquirrels]

Label: T-libs

There have of course been several efforts to refactor std::io to make it usable without std as core::io or similar.

Some links to prior art from a few hours of searching:

• https://github.com/QuiltOS/core-io
• https://crates.io/crates/core_io
• Error is crippled jethrogb/rust-core_io#3
• Move ::std::error to ::core::error rust#33149
• Port libstd to "no operating system" target rust#37133
• https://internals.rust-lang.org/t/refactoring-std-for-ultimate-portability/4301
• Read Write for core BurntSushi/byteorder#51
• https://github.com/rust-lang/rfcs/blob/master/text/0517-io-os-reform.md#coreio
• RFC: io and os reform: initial skeleton #517
• no_std support (via an enabled-by-default "std" feature) rust-lang-deprecated/error-chain#157
• https://internals.rust-lang.org/t/crate-evaluation-for-2017-06-27-error-chain/5362/25
• https://crates.io/crates/genio
• Survey: Crates / libraries / tools you need / want rust-embedded/wg#22

From what I can see all efforts are currently stalled, but I'd like to make progress on using std::io in a no_std environment. My use case is a crate I'm writing that uses std::io::{Read, Write}, and I want it to build with std and when no_std.

As mentioned in https://internals.rust-lang.org/t/refactoring-std-for-ultimate-portability/4301, there are circular dependencies between std::io and the rest of std.

I have an idea that I've not seen mentioned: use conditional compilation in the std::io module to enable use without std. Steps required in more detail:

1. Move rust/src/libstd/io/*.rs to a new directory rust/src/libio_template
2. Add a new rust/src/libstd/io/mod.rs with just an include!("../../libio_template/mod.rs").
3. Add new crate rust/src/libio_nostd, which will also include! libio_template.
4. Add cfg feature "libio_for_std" to both libstd and libio_nostd, with it enabled and disabled by default respectively.
5. Patch libio_template to compile in libstd and libio_nostd.
6. All done. no_std consumers can use libio_nostd, libstd API should be identical, future std::io patches must be correct both in libstd and libio_nostd.

The ideal solution, as proposed in https://internals.rust-lang.org/t/refactoring-std-for-ultimate-portability/4301, is probably a careful refactoring to decouple std, but that seems like a lot of work, which has already stalled once. A templated approach allows quick progress and can be eradicated progressively as more parts of std::io are refactored out of std.

I'm willing to work on a prototype branch or more formal RFC if there is interest from the libs team.

[TimNN]

One problem I see with the include! approach is that std::io::Read and core::io::Read would be different traits, making APIs using either incompatible with each other.

[fluffysquirrels]

Yes, good point! They'd be different traits, but they have to be. std::io::Error may store a heap-allocated Box<Error>, which is a great feature that has been stabilized, but is impossible on a microcontroller with no heap. It would be easy to add wrappers that map between std::io and core::io, losing some data on errors converted to core::io::Error, of course. Then code designed mostly for no_std can simply use core::io and still ergonomically interoperate with std::io code. With this templated approach library authors can easily opt-in to writing for both std and no_std environments by creating their own feature flags to swap between std::io or core::io in their API. That's the use case I'd imagined.

…

On 28 Dec 2017 8:46 a.m., "Tim Neumann" ***@***.***> wrote: One problem I see with the include! approach is that std::io::Read and core::io::Read would be different traits, making APIs using either incompatible with each other. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2262 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAm2kp4rmbmr7xdEPt-NCj-PNjPnrCR0ks5tE1VegaJpZM4RMDvQ> .

[Ericson2314]

I like the blanket impl I propose here https://internals.rust-lang.org/t/refactoring-std-for-ultimate-portability/4301/3 to connect the two traits.

[fluffysquirrels]

@Ericson2314 Yes! I Read that thread with great interest.

I agree the platform abstraction layer sounds like a great approach. I didn't feel up to the task of starting that work and was looking for a quicker fix that could be improved later.

Edit: misread your post.

Yes, it should be possible to rewrite std::io as a concrete layer on top of a generic core::io layer with some wrapper structs to convert between the two, and make it all low-cost. I would be interested in prototyping that approach instead of what I proposed here.

[SimonSapin]

There’s also methods like Read::read_to_vec that mention Vec which is defined in the alloc crate.

[Ericson2314]

@fluffysquirrels I think you closed by mistake?

@SimonSapin Indeed, I imagine we'd want an alloc::io, too.

[fluffysquirrels]

@fluffysquirrels I think you closed by mistake?

I was thinking the linked page may be more appropriate, but now I see your comments were on a forum, not an issue tracker. My mistake.

Regarding Read::read_to_vec and similar, perhaps core::io could have a feature flag to toggle features that require alloc.

[fluffysquirrels]

Relevant GitHub issues raised in the portability working group:

rust-lang-nursery/portability-wg#4
rust-lang-nursery/portability-wg#1
rust-lang-nursery/portability-wg#12

[Ericson2314]

@fluffysquirrels Sure, fine to continue this elsewhere. Thanks again for kicking off the discussion :).

[Ericson2314]

Continued in rust-lang-nursery/portability-wg#12 (comment)

[lygstate]

What's the situation of this?