Additional JsonNode functionality

[steveharter]

Background and motivation

Add missing features that were requested by the community. Note that System.Text.Json.Nodes.JsonNode and derived types are new, added in 6.0 preview 4.

API Proposal

namespace System.Text.Json.Nodes
{
     public abstract class JsonNode
     {
         // In read mode, simply clones the underlying JsonElement without any possible JSON comments.
         // In edit mode this is expensive for values wrapping a non-node value since it need to serialize to JsonNode.
+        public JsonNode DeepClone();

         // In read mode, simply compares against the underlying JsonElement.
         // In edit mode this is expensive for values wrapping a non-node value since it need to serialize to JsonNode.
         // Made `static` for easier `null` comparison
+        public static bool DeepEquals(JsonNode? node1, JsonNode? node2);

         // In read mode, simply uses JsonElement.ValueKind.
         // In edit mode for values wrapping a non-node value that is not a known type (string, int, long, etc) then
         // JsonValueKind.Undefined is returned implying that to return the actual ValueKind one would serialize to a
         // JsonElement\JsonNode along with any possible custom converter.
+        public JsonValueKind GetValueKind()

         // Returns property name from parent object. Parent must be JsonObject otherwise InvalidOperationException.
+        public string GetPropertyName();

         // Returns index from parent array. Parent must be JsonArray otherwise InvalidOperationException.
+        public int GetElementIndex();

         // Replace this instance with a new value. The parent object\array is updated accordingly.
+        void ReplaceWith<T>(T value);
    }

    public class JsonArray
    {
         // For homogeneous arrays, returns enumerator that wraps calls to GetValue<T>().
+        IEnumerable<T> GetValues<T>();
    }
}

API Usage

DeepClone and DeepEquals

JsonNode node = JsonNode.Parse("{\"Prop\":{\"NestedProp\":42}}");
JsonNode other = node.DeepClone();
bool same = JsonNode.DeepEquals(node, other); // true

GetValueKind

JsonNode node = JsonNode.Parse("42");
// In read-mode this is efficient:
JsonValueKind kind = node.ValueKind; // JsonValueKind.Number
// In edit-mode , Undefined is returned
node = new JsonValue(43);
kind = node.ValueKind; // JsonValueKind.Undefined; there could be a custom converter for in32 writing it as a JSON object
// Slower workaround:
kind = node.Deserialize<JsonElement>().ValueKind; // JsonValueKind.Number

node = JsonValue.Create(new Dictionary<string, int>());
kind = node.ValueKind; // JsonValueKind.Undefined 
// Slower workaround; note that dictionaries are written as JSON objects by default
kind = node.Deserialize<JsonElement>().ValueKind; // JsonValueKind.Object

JsonArray.GetValues

JsonArray jsonArray = new JsonArray(1, 2, 3, 2);

// To get all "2" values out today using LINQ requires something like:
IEnumerable<int> values = jsonArray.Select(x => x.GetValue<int>()).Where(i => i == 2);

// With the proposed GetValues this is simpler:
IEnumerable<int> values = jsonArray.GetValues<int>().Where(i => i == 2);

[ghost]

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

Add missing features that were requested by the community. Note that System.Text.Json.Nodes.JsonNode and derived types are new, added in Preview 4.

API Proposal

namespace System.Text.Json.Nodes
{
     public abstract class JsonNode
     {
//       Expensive; does a serialize and deserialize.
+        public JsonNode DeepClone()

//       Expensive; made `static` for easier `null` comparison
+        public static JsonNode DeepEquals(JsonNode? node1, JsonNode? node2);

        // Falls back to JsonElement.ValueKind when possible, otherwise calculates. Expensive in some cases.
+        public JsonValueKind GetValueKind(JsonSerializerOptions options = null)

         // Returns JsonPath-compatible property name ("$", ".MyProperty", "[1]")
         // Based on existing GetPath()
+        public string GetAccessorNameFromPath();

         // Returns property name from parent object. Parent must be JsonObject otherwise InvalidOperationException.
+        public string GetPropertyName();

         // Returns index from parent array. Parent must be JsonArray otherwise InvalidOperationException.
+        public int GetPropertyIndex();
    }

    public class JsonArray
    {
         // For homogeneous arrays, returns enumerator that wraps call GetValue<T>().
+        IEnumerable<T> GetValues<T>();
    }
}

API Usage

DeepClone and DeepEquals

JsonNode node = JsonNode.Parse("{\"Prop\":{\"NestedProp\":42}}");
JsonNode other = node.DeepClone();
bool same = JsonNode.DeepEquals(node, other); // true

GetValueKind

JsonNode node = JsonNode.Parse("42");
// In read-mode this is efficient:
JsonValueKind kind = node.ValueKind; // JsonValueKind.Number

// In edit-mode not sure efficient
node = new JsonValue(43);
kind = node.ValueKind; // JsonValueKind.Number

// And may internally perform serialization if not known
node = new JsonValue(new Dictionary<string, int>());
kind = node.ValueKind; // JsonValueKind.Object // Dictionaries are written as JSON objects

(work in progress; todo)

Risks

No response

Author:	steveharter
Assignees:	steveharter
Labels:	api-suggestion, area-System.Text.Json
Milestone:	6.0.0

[campersau]

GetPropertyIndex() sounds wired to me what about GetArrayIndex()?

[eiriktsarpalis]

GetPropertyIndex() sounds wired to me what about GetArrayIndex()?

I'd probably call this GetElementIndex().

@steveharter not sure I understand how GetAccessorNameFromPath() is different than the existing GetPath() method.

[steveharter]

I'd probably call this GetElementIndex().

Yep thanks.

@steveharter not sure I understand how GetAccessorNameFromPath() is different than the existing GetPath() method.

It returns just the immediate parent's property name \ index. So given "$.Order.Customer.Address", calling that method on Address instance returns ".Customer"

[steveharter]

There are basically two scenarios for the accessor name as I see it:

• Obtaining the string property name ("Address") or integer index value
• Obtaining either the property name or integer value in string JsonPath syntax ("$.Address" or "[1]") for use with JsonPath. Since we don't have any JsonPath functionality (like querying) today other than the Path I don't think this is that relevant.

[steveharter]

These APIs are not blocking as there are workarounds as shown in #55827, so they don't meet the bar for v6 this late in the release.

[PascalSenn]

@steveharter I am currently rewriting JsonNodes to a different structure. It would be great if there was a Replace on JsonArray and JsonObject that does not require to remove the parent first.
Did you consider something like this?

[steveharter]

It would be great if there was a Replace on JsonArray and JsonObject that does not require to remove the parent first.
Did you consider something like this?

Thanks. I added Replace() semantics to JsonNode to the proposal.

The reason why Replace() wasn't added is there is a simple work-around of remove+add. However, that does require knowledge of the parent and corresponding index or property name, so I can see why Replace() could be useful.

[layomia]

Noting another popular request for consideration, to merge two JSON objects, e.g. https://www.newtonsoft.com/json/help/html/MergeJson.htm.

cc @sebastienros who is interested in this.

[eiriktsarpalis]

@layomia FWIW there's an issue for that #31433. Should we update the proposal here and close the other one?

[PascalSenn]

@steveharter saw your reply just now (Github Notification overload). Thanks a lot!

[eiriktsarpalis]

Feedback from #55827

Some feedback after using the new JsonNode apis for some time:

• For F# use JsonNode is missing a convenience method to safely 'dot into' a deep path. F# does not have a null conditional operator to conveniently intersperse between indexer calls. I've had to add something like the following code to make querying workable.

    type JsonNode with
        member node.GetNode(index: int) = Option.ofObj node.[index]
        member node.GetNode(segment: string) = Option.ofObj node.[segment]
        member node.GetNode([<ParamArray>]pathSegments: obj[]) =
            let mutable current, i = node, 0
            while not (isNull current) && i < pathSegments.Length do
                match pathSegments.[i] with
                | :? int as index -> current <- current.[index]
                | :? string as segment -> current <- current.[segment]
                | _ -> failwith "Unknown path segment type, either int or string is supported."
                i <- i + 1
            Option.ofObj current

It could make sense to provide this in the box but mainly I wanted to note this usability issue for F#.

• JsonArray is missing a convenience method to get an IEnumerable or array of * GetValue * results out. I'm assuming the alternative pattern is node.AsArray().Select(x => x.GetValue<int>()). A direct method on JsonArray would be useful for discoverability.
• I'm missing an easy way to access (if available) PropertyName on JsonNode. I can parse it back out via GetPath or track it on the side during folds but it might be helpful to have an easier way for it?
• For dynamic data or when converting JsonNode to another tree representation I'm missing any and all useful information on JsonValue. What kind might it be? Which TryGetValue calls are probably going to succeed? Is the expectation here to just probe TryGetValue against likely types? I get that you want to support arbitrary values to support whatever serialization conversions users may want but as a result handling json data with a conventional shape (primitives/JsonElement) is slightly crazy. Do I serialize JsonValue instances to json and deserialize to a JsonElement? IMO this severly limits the usability to just use it as a mutable version of JsonDocument.
• I really am missing DeepClone, DeepEquals, and Merge. I've written our own versions now but I'm probably not alone. (though again I understand with JsonValue being what it is it would be hard to provide useful and consistent semantics out of the box)

Beyond JsonNode I think a lot of us are still hoping for a 'one-shot' querying method on JsonDocument in the box. I'd be happy with anything given JsonPath was shot down here #31068 (comment). A lot of our code that queries json doesn't need to have a mutable model passed in. Having even the simplest workable querying options on JsonDocument/JsonElement should really come in the box and I'm sad to see any chances for it slip into 7.0.

[eiriktsarpalis]

Feedback from #31433

Using System.Text.Json, is there any way to Merge like Json.Net does?

see Newtonsoft.Json.Linq.JContainer.Merge

public void Merge(
	Object content,
	JsonMergeSettings settings
)

[slang25]

I'm here to upvote the DeepClone functionality, could really do with that 🙂

[terrajobst]

Video

• XDocument uses a copy constructor on XElement instead, but DeepClone seems more discoverable.
• IsRoot doesn't seem that value, as the canonical way would be to check node.Parent is null.
• GetValueKind() should try to return a better value than Undefined such that modifications of primitives doesn't result in unexpected changes from, say, Number to Undefined.
  • Do we need the options? We might need it for settings such as the one that serializes numbers as strings
• In XDocument the Replace method is called ReplaceWith, which makes it a bit more clear that it's in-place semantics
• We need to define what JsonArray.GetValues<object> will do. Return the JsonNode seems the most viable, because boxing the underlying primitives wouldn't be well-defined.

namespace System.Text.Json.Nodes
{
     public partial class JsonNode
     {
        public JsonNode DeepClone();
        public static bool DeepEquals(JsonNode? node1, JsonNode? node2);
        public JsonValueKind GetValueKind(JsonSerializerOptions options = null);
        public string GetPropertyName();
        public int GetElementIndex();
        public void ReplaceWith<T>(T value);
    }
    public partial class JsonArray
    {
        public IEnumerable<T> GetValues<T>();
    }
}

[steveharter]

Per the API review discussion, I looked at JsonValueKind GetValueKind(options) and determined that the options is not needed.

Basically, the algorithm would look like this:

public JsonValueKind GetValueKind()
{
    if (this is JsonElement element) return element.ValueKind;
    else if (TryGetKindFromKnownTypes(this, out kind)) return kind;
    else return JsonValueKind.Unknown;
}

I believe options was there previously because the thought is that there for non-JsonElement types the code would automatically do the expensive serialize(options) in order to obtain the kind. Now, we just return JsonValueKind.Unknown.

[sebastienros]

node = new JsonValue(43);
kind = node.ValueKind; // JsonValueKind.Undefined; there could be a custom converter for in32 writing it as a JSON object

Doesn't seem to match the description:

         // In read mode, simply uses JsonElement.ValueKind.
         // In edit mode for values wrapping a non-node value that is not a known type (string, int, long, etc) then
         // JsonValueKind.Undefined is returned implying that to return the actual ValueKind one would serialize to a
         // JsonElement\JsonNode along with any possible custom converter.
         public JsonValueKind GetValueKind()

Or is Int32 not a "known type" and I misunderstood the api doc?

That would be very unfortunate IMO. I understand that it could be something else when serialized, but at this point it's not serialized, and as a user I would definitely like it to return JsonValueKind.Number.

[eiriktsarpalis]

Doesn't seem to match the description:

Based on @steveharter's latest comment I believe that description is out of date. The current iteration would always return Undefined for all wrapped .NET values.

Per the API review discussion, I looked at JsonValueKind GetValueKind(options) and determined that the options is not needed.

In that case, couldn't it just be a property?

[davidhmays]

Any status on adding json merge functionality?
I keep going in circles seeing where merging JsonDocuments on a key has been requested, merged into other requests, than apparently lost.

Was really hoping to have a simple way to merge on a key, and am frustrated the "upgrade" doesn't offer this. Specifically was hoping to be able to use Linq against the JsonDocument types, especially for left outer join type merges of data from separate API responses that you don't yet need defined types for, and so would prefer merging the JSON directly.

[goldsam]

@eiriktsarpalis You closed #31433 in favor of this issue, but merge functionality doesn't seem to have been addressed. Please reopen #31433 to ensure this critical functionality is not neglected.

I think the community has been very vocal about the need for a simple and intuitive merge/union API. The VERY LAST comment by @davidhmays from back in May was quite eloquent and clear on this matter and his sentiments are echoed throughout this and many other issues. In fact, many of your own Feedback from XXX comments specifically mention this feature request.

Its quite infuriating when after so much time has passed, both this issue and #31433 are now closed without any indication this will be addressed in the future.

[eiriktsarpalis]

@goldsam I've reopened the issue, the particular API seems to have fallen through the cracks.