Switch parser to toml_document or tomllib

[flying-sheep]

This allows us to fix #15

[killercup]

Wow, that looks pretty impressive! Thanks for letting me know.

Do you plan on writing a PR for this? Otherwise I'll probably get around to writing one myself in the few days.

[flying-sheep]

well, it’s not on cargo yet and is probably not feature-complete yet (vosen/toml_document#1) so there’ll probably be some holdups if we don’t want to maintain a fork until everything we need is in there

[killercup]

Right. I was thinking more of a WIP PR, to see how far this would currently work, not one we'd merge immediately. This could potentially also drive the design of toml_document itself.

[flying-sheep]

sure! but without vosen/toml_document#1, we can’t get anywhere 😄

[flying-sheep]

that is done now, we can revisit this!

[killercup]

In #15, https://github.com/joelself/tomllib is discussed. Can someone look into the difference between it and https://github.com/vosen/toml_document/ ?

[flying-sheep]

well, neither have open bugs, but tomllib seems better documented

[killercup]

Yes, but sadly, the documentation (Readme) currently includes:

Some things that you can't do yet, but are planned for the next release are:

• Key/Val insertion

[flying-sheep]

haha well, that’s kinda important

[ordian]

I think we should use toml_document, this crate has everything we need.

[killercup]

Could be. I can't really tell because this crate has basically no documentation and even though there are tests, they are written in quite weird way 😅 Also, it has not been updated for a year, which also makes me hesitant.

[ordian]

So... I've written yet another toml parser for fun - toml_edit (reminds me of this xkcd).

It's not published yet, partly due to the current API being uber ugly and inconsistent. The biggest blocker for using it in cargo-edit is that in toml_edit inline and non-inline tables have different types, which will lead to code duplication, and I don't know what's the best solution here.

Contributions are welcome :)

[killercup]

Nice, @ordian! I saw your gitter link and wrote down a few things while reading some of the code

[bjgill]

This seems like the most significant barrier to more widespread cargo-edit usage at the moment, so it'd be nice to get this moving.

I think we've got three options for the new toml parser:

• toml_document - this looks ideal, with (seemingly) support for adding/removing/editing table entries. Unfortunately, it's completely undocumented and does not have a simple API, which makes integration hard. I had a go at prototyping with toml_document, but failed to get mutable references to table entries. Seemingly no life in the past year.
• tomllib - supports editing table entries, but not (yet?) adding/removing them. Has a nice, simple API, albeit rather stringly typed. Looks as if it would be adequate for cargo upgrade, but would require non-trivial effort to make it suitable for cargo add/rm. Seemingly no life in the past year.
• toml_edit - looks like a better-documented equivalent of toml_document, tailored for use with Cargo manifests (and cargo-edit). Doesn't guarantee that all comments/spaces/relative ordering will be preserved. From the tests, it looks as if integration should be fairly straightforward, albeit with some pain around inline/non-inline tables. Not yet published to crates.io.

Arguably, toml_document is likely to be the best technical solution, given that it claims to provide pretty good guarantees of not altering the toml document unnecessarily. However, given that toml_document is completely undocumented, and seemingly no longer under active development, integration is liable to be excessively painful.

Thus, it looks as if toml_edit may be the best choice. Though this is contingent on toml_edit getting a 0.1 release. @ordian - do you think you'll be able to make the API changes you want to soon? Otherwise, I'm quite tempted to start integrating against the current API.

[ordian]

@bjgill thanks for writing this.

I want to clarify on guarantees provided by toml_edit: it does preserve all comments,
does not preserve spaces only inside headers (e.g. [ ' a '.b ] will be [' a '.b]), and places child tables after parent: e.g.

[dependencies.nom] # comment 1
version = "3"
[dependencies] # comment 2
hello = "1"

will be replaced with

[dependencies] # comment 2
hello = "1"
[dependencies.nom] # comment 1
version = "3"

(Also array of tables is grouped in one place)
Everything else is preserved.

Actually this constraints were only enforced in the latest pr in order to make Table and other structs Cloneable, because cargo-edit's get_sections requires it.

@ordian - do you think you'll be able to make the API changes you want to soon? Otherwise, I'm quite tempted to start integrating against the current API.

Well, I can try to write an abstraction around inline and non-inline tables this weekend-ish.

[LeopoldArkham]

Hey,
I was pinged by @killercup on reddit; I'm currently writing Molten, a TOML parsing library. I actually started it with cargo-edit in mind, among other things, after seeing that tomllib had stalled.

Molten is already fully style, WS, comment, and order preserving in tests, modulo:

• the bugs I'm fixing today
• quoted keys which I'll get to next

After that it needs error management and a proper API.

I'm committed to finishing and maintaining it so it should be a good way to go for cargo-edit.
When will it be done? I've currently got time to work on it a bit every day so hopefully Soon©.

[ordian]

We've switched to toml-edit, so this is no longer relevant.