Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Text.Json serialization order #728

Closed
bugproof opened this issue Dec 10, 2019 · 31 comments
Closed

System.Text.Json serialization order #728

bugproof opened this issue Dec 10, 2019 · 31 comments
Assignees
Labels
area-System.Text.Json enhancement Product code improvement that does NOT require public API changes/additions
Milestone

Comments

@bugproof
Copy link

bugproof commented Dec 10, 2019

AB#1166328
Serializer puts base properties at the end. In JSON net you could create a contract resolver that let you change serialization order.

class Entity
{
    public int Id { get; set; }
}

class Player : Entity
{
    public string Name { get; set; }
}

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(new Player {Id = 1, Name = "noname"}));
    }
}

the output will be:

{"Name":"noname","Id":1}

instead of

{"Id":1,"Name":"noname"} which is more readable

@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added area-System.Text.Json untriaged New issue has not been triaged by the area owner labels Dec 10, 2019
@ahsonkhan
Copy link
Member

ahsonkhan commented Dec 18, 2019

In JSON net you could create a contract resolver that let you change serialization order.

Do you have a scenario in which you need to control the serialization order where the current behavior/order is blocking you? I am asking to understand the help motivate the feature since JSON is generally unordered (i.e. folks shouldn't be relying on the ordering of the properties within the payload).

{"Id":1,"Name":"noname"} which is more readable

Is that the primary benefit? Having the payload written in a specific order so it is easier to read? Adding support for such capabilities would be a bit of work, and this alone doesn't seem worth the trade off (at least when prioritizing features for 5.0).

The contract resolver feature in Newtonsoft.Json has some other capabilities that are currently not supported. How are you controlling the serialization order today (and is that the only capability you need)? Can you please share a code snippet of your resolver.

If ordering is super critical for you, a workaround would be to create and register a JsonConverter<Player>.

public class PlayerConverter : JsonConverter<Player>
{
    public override Player Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        throw new NotImplementedException();
    }

    public override void Write(Utf8JsonWriter writer, Player value, JsonSerializerOptions options)
    {
        writer.WriteStartObject();
        writer.WriteNumber(nameof(value.Id), value.Id);
        writer.WriteString(nameof(value.Name), value.Name);
        writer.WriteEndObject();
    }
}

And here's how you can register and use it:

var options = new JsonSerializerOptions();
options.Converters.Add(new PlayerConverter());

Console.WriteLine(JsonSerializer.Serialize(new Player { Name = "noname", Id = 1 }, options));

@ahsonkhan ahsonkhan added this to the Future milestone Dec 18, 2019
@ahsonkhan ahsonkhan added enhancement Product code improvement that does NOT require public API changes/additions and removed untriaged New issue has not been triaged by the area owner labels Dec 18, 2019
@bugproof
Copy link
Author

bugproof commented Dec 18, 2019

order is very helpful when testing the API with tools like insomnia, postman etc. you instantly see what's the most important at the top, and naturally ID comes first and it's later easier to navigate in code because you know that base properties come first... JSON is meant to be human-readable otherwise I wouldn't use it. The ordering would be only helpful for the developer and many existing APIs put Id as the first JSON property. Why is it hard to do?

code for resolver:

public class BaseFirstContractResolver : DefaultContractResolver
{
    public BaseFirstContractResolver()
    {
        NamingStrategy = new CamelCaseNamingStrategy();
    }

    protected override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization)
    {
        return base.CreateProperties(type, memberSerialization)
            .OrderBy(p => BaseTypesAndSelf(p.DeclaringType).Count()).ToList();

        IEnumerable<Type> BaseTypesAndSelf(Type t)
        {
            while (t != null)
            {
                yield return t;
                t = t.BaseType;
            }
        }
    }
}

no other benefits than readability, I think it would be more natural for base class properties to be serialized in the first place.

@stevendarby
Copy link

stevendarby commented Jan 30, 2020

Have an unusual use case where we have one set of services in .NET for GETs and another set of services for write operations on another platform. We are doing ETag / If-Match checking by hashing JSON responses, so responses have to match across both platforms. It would be nice to be able to control this from the .NET side.

Also, if order really isn't important, could you consider putting the base properties first by default? This just seems to make more logical sense.

@Drawaes
Copy link
Contributor

Drawaes commented Jan 30, 2020

Hashing for cache invalidation/etag is also one I have come across. As it was a service to service scenario we moved to protobuf in the end to get the control over the order and bytes but that is not always possible for everyone

@layomia
Copy link
Contributor

layomia commented Mar 6, 2020

From @Jogge in #1085:

I'm missing an attribute for setting the order of the element in the JSON output.

Example:

I want the @SEGMENT to be the first element in the serialized JSON output of MyClass:

public abstract class MyBase
{
    [JsonPropertyName("@SEGMENT")]
    public virtual string Segment { get; set; } = "1";

    public int ID { get; set; }
}

public class MyClass : MyBase
{
    public string Name { get; set; }
}

The result when serializing MyClass is:

{
    "MyClass": {
    	"Name": "Foo",
    	"@SEGMENT": "1",
    	"ID": 42
    }
}

I would like the serialized JSON output to be:

{
    "MyClass": {
    	"@SEGMENT": "1",
    	"Name": "Foo",
    	"ID": 42
    }
}

or

{
    "MyClass": {
    	"@SEGMENT": "1",
    	"ID": 42,
    	"Name": "Foo"
    }
}

Json.NET has an attribute for this: https://www.newtonsoft.com/json/help/html/JsonPropertyOrder.htm

Related question: .NET core 3: Order of serialization for JsonPropertyName (System.Text.Json.Serialization)

@aktxyz
Copy link

aktxyz commented Mar 9, 2020

+1 for this ... very helpful for DX to be able to have key things at the top

@neelabo
Copy link

neelabo commented Mar 24, 2020

(Reprinted from issue #33854)

Currently, the serialization order is not guaranteed.
I use Json to save application settings, but there is a problem if the order is not guaranteed.

  • Diff cannot be obtained, making it difficult to find differences in settings.
  • Not suitable for version control.
  • The deserialization order is changed. -> The application initialization order is changed.

To fix the order, we propose the following enhancements:

  • Extend so that alphabetical sort flag or sort function (IComparer<PropertyInfo>) can be set in JsonSerializerOptions. This option only affects the output.

@benmccallum
Copy link

benmccallum commented Apr 8, 2020

Adding another use case for this feature. The GraphQL spec recommends here that the errors prop is rendered at the top, and indeed it's bitten me when debugging things to have it after data and not notice the errors as graphql is kind of unique in that it'll return data it resolved successfully and add errors for parts of the tree it couldn't resolve.

Open issue to implement this sorting is here. We'll be able to knock of the NewtonsoftJson serializer, but not the S.T.Json one until this is possible. and I've just realised for S.T.Json we are using a custom converter to write the response, so can just shift the property write ordering in there without need for this feature for S.T.Json. Still, there's another example use case I guess.

@joelwreed
Copy link

Also Algorand's canonical encoding is a MessagePack with sorted json properties. Seems more straightforward to do this with NewtonsoftJson at the moment.

@bjorhn
Copy link

bjorhn commented May 22, 2020

This would be a very useful feature to have. I've been using System.Text.Json for handling "huge" amounts of metadata that get checked into source control for about 8 months now, and until today the order has always been the same. I had assumed it was guaranteed all along.

@gtbuchanan
Copy link

To add another use case for controlling serialization order, Safe Exam Browser requires generating a sorted JSON object for their Config Key algorithm since it involves comparing hashes.

@bugproof
Copy link
Author

bugproof commented Jul 31, 2020

Btw. this is the code responsible for the behaviour:

// We start from the most derived type.
for (Type? currentType = type; currentType != null; currentType = currentType.BaseType)
{
const BindingFlags bindingFlags =
BindingFlags.Instance |
BindingFlags.Public |
BindingFlags.NonPublic |
BindingFlags.DeclaredOnly;
foreach (PropertyInfo propertyInfo in currentType.GetProperties(bindingFlags))
{
// Ignore indexers and virtual properties that have overrides that were [JsonIgnore]d.
if (propertyInfo.GetIndexParameters().Length > 0 || PropertyIsOverridenAndIgnored(propertyInfo, ignoredMembers))
{
continue;
}
// For now we only support public properties (i.e. setter and/or getter is public).
if (propertyInfo.GetMethod?.IsPublic == true ||
propertyInfo.SetMethod?.IsPublic == true)
{
CacheMember(currentType, propertyInfo.PropertyType, propertyInfo, typeNumberHandling, cache, ref ignoredMembers);
}
else
{
if (JsonPropertyInfo.GetAttribute<JsonIncludeAttribute>(propertyInfo) != null)
{
ThrowHelper.ThrowInvalidOperationException_JsonIncludeOnNonPublicInvalid(propertyInfo, currentType);
}
// Non-public properties should not be included for (de)serialization.
}
}

Maybe Stack<T> could be used to push base types and then just pop and get properties in the appropriate order.

var typeStack = new Stack<Type>();
for (Type? currentType = type; currentType != null; currentType = currentType.BaseType)
{
    typeStack.Push(currentType);
}

Whatever will result in the best performance... maybe sorting by MetadataToken ? https://stackoverflow.com/questions/8067493/if-getfields-doesnt-guarantee-order-how-does-layoutkind-sequential-work

Should this be a default behaviour? Should this be configurable? I can confirm that sorting by MetadataToken works fine.

@zcsizmadia
Copy link

zcsizmadia commented Nov 13, 2020

@layomia
Here is a proposed implementation/prototype of JsonProperty ordering

It introduces 2 new attributes: [JsonPropertyOrder] and [JsonPropertyOrderByName]

JsonPropertyOrder
Property or field Attribute to specify the order of the given property (compatible with the Newtonsoft JSON JsonProperty.Order behaviour).

JsonPropertyOrderByName
Class Attribute. If it is specified, the properties will be sorted by name (using an optional StringComparison arg)

Sorting logic
Calling sorting
Sorting function

Examples and expected serialization

        public struct PropertyOrderTestStruct1
        {
            [JsonPropertyOrder(1)]
            public int A { get; set; }
            public int B { get; set; }
            [JsonPropertyOrder(-2)]
            public int C { get; set; }

            public static readonly string e_json = @"{""C"":0,""B"":0,""A"":0}";
        }

        public class PropertyOrderTestClass1
        {
            [JsonPropertyOrder(-1)]
            public int A { get; set; } = 1;
            [JsonPropertyOrder(-1)]
            public int B { get; set; } = 2;
            [JsonPropertyOrder(-1)]
            public int C { get; set; } = 3;

            public static readonly string e_json = @"{""A"":1,""B"":2,""C"":3}";
        }

        public class PropertyOrderTestClass2
        {
            [JsonPropertyOrder(2)]
            public int a { get; set; } = 1;
            [JsonPropertyOrder(1)]
            public int b { get; set; } = 2;
            [JsonPropertyOrder(0)]
            public int c { get; set; } = 3;

            public static readonly string e_json = @"{""c"":3,""b"":2,""a"":1}";
        }

        public class PropertyOrderTestClass3
        {
            public int A { get; set; } = 1;
            public int B { get; set; } = 2;
            [JsonPropertyOrder(-2)]
            public int C { get; set; } = 3;

            public static readonly string e_json = @"{""C"":3,""A"":1,""B"":2}";
        }

        public class PropertyOrderTestClass4
        {
            [JsonPropertyOrder(1)]
            public int A { get; set; } = 1;
            [JsonPropertyOrder(2)]
            public int B { get; set; } = 2;
            [JsonPropertyOrder(-1)]
            public int C { get; set; } = 3;

            public static readonly string e_json = @"{""C"":3,""A"":1,""B"":2}";
        }

        public class PropertyOrderTestClass5
        {
            [JsonPropertyOrder(2)]
            public int A { get; set; } = 1;
            [JsonPropertyOrder(-1)]
            public int B { get; set; } = 2;
            [JsonPropertyOrder(1)]
            public int C { get; set; } = 3;

            public static readonly string e_json = @"{""B"":2,""C"":3,""A"":1}";
        }

        public class PropertyOrderTestClass6
        {
            [JsonPropertyOrder(0)]
            public int A { get; set; } = 1;
            [JsonPropertyOrder(0)]
            public int B { get; set; } = 2;
            [JsonPropertyOrder(0)]
            public int C { get; set; } = 3;

            public static readonly string e_json = @"{""A"":1,""B"":2,""C"":3}";
        }

        public class PropertyOrderTestClass7
        {
            [JsonPropertyOrder(1)]
            public int A { get; set; } = 1;
            public int B { get; set; } = 2;
            [JsonPropertyOrder(-2)]
            public int C { get; set; } = 3;

            public static readonly string e_json = @"{""C"":3,""B"":2,""A"":1}";
        }

        [JsonPropertyOrderByName]
        public class PropertyOrderTestClass8
        {
            public int C { get; set; } = 3;
            public int B { get; set; } = 2;
            public int A { get; set; } = 1;

            public static readonly string e_json = @"{""A"":1,""B"":2,""C"":3}";
        }

        [JsonPropertyOrderByName()]
        public class PropertyOrderTestClass9
        {
            public int cC { get; set; } = 3;
            public int CC { get; set; } = 3;
            public int bB { get; set; } = 2;
            public int BB { get; set; } = 2;
            public int aA { get; set; } = 1;
            public int AA { get; set; } = 1;

            public static readonly string e_json = @"{""AA"":1,""BB"":2,""CC"":3,""aA"":1,""bB"":2,""cC"":3}";
        }

        [JsonPropertyOrderByName(StringComparison.OrdinalIgnoreCase)]
        public class PropertyOrderTestClass10
        {
            public int cC { get; set; } = 3;
            public int CC { get; set; } = 3;
            public int bB { get; set; } = 2;
            public int BB { get; set; } = 2;
            public int aA { get; set; } = 1;
            public int AA { get; set; } = 1;

            public static readonly string e_json = @"{""aA"":1,""AA"":1,""bB"":2,""BB"":2,""cC"":3,""CC"":3}";
        }

        [JsonPropertyOrderByName]
        public class PropertyOrderTestClass11
        {
            [JsonPropertyName("C")]
            public int a { get; set; } = 3;
            [JsonPropertyName("B")]
            public int b { get; set; } = 2;
            [JsonPropertyName("A")]
            public int c { get; set; } = 1;

            public static readonly string e_json = @"{""A"":1,""B"":2,""C"":3}";
        }

        [JsonPropertyOrderByName]
        public class PropertyOrderTestClass12
        {
            [JsonPropertyName("C")]
            [JsonPropertyOrder(-2)]
            public int a { get; set; } = 3;
            [JsonPropertyName("B")]
            public int b { get; set; } = 2;
            [JsonPropertyName("A")]
            public int c { get; set; } = 1;

            public static readonly string e_json = @"{""C"":3,""A"":1,""B"":2}";
        }

@bjorhn
Copy link

bjorhn commented Nov 13, 2020

Looks promising!

Would it make sense to also have a simple JsonSerializerOptions setting that does the same thing as JsonPropertyOrderByName, for the whole structure being serialized, or am I the only one interested in such an option?

@zcsizmadia
Copy link

Would it make sense to also have a simple JsonSerializerOptions setting that does the same thing as JsonPropertyOrderByName, for the whole structure being serialized, or am I the only one interested in such an option?

The implementation should be fairly trivial. It is more of a design/vision question.

@silvairsoares
Copy link

In JSON net you could create a contract resolver that let you change serialization order.

Do you have a scenario in which you need to control the serialization order where the current behavior/order is blocking you? I am asking to understand the help motivate the feature since JSON is generally unordered (i.e. folks shouldn't be relying on the ordering of the properties within the payload).

{"Id":1,"Name":"noname"} which is more readable

Is that the primary benefit? Having the payload written in a specific order so it is easier to read? Adding support for such capabilities would be a bit of work, and this alone doesn't seem worth the trade off (at least when prioritizing features for 5.0).

The contract resolver feature in Newtonsoft.Json has some other capabilities that are currently not supported. How are you controlling the serialization order today (and is that the only capability you need)? Can you please share a code snippet of your resolver.

If ordering is super critical for you, a workaround would be to create and register a JsonConverter<Player>.

public class PlayerConverter : JsonConverter<Player>
{
    public override Player Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        throw new NotImplementedException();
    }

    public override void Write(Utf8JsonWriter writer, Player value, JsonSerializerOptions options)
    {
        writer.WriteStartObject();
        writer.WriteNumber(nameof(value.Id), value.Id);
        writer.WriteString(nameof(value.Name), value.Name);
        writer.WriteEndObject();
    }
}

And here's how you can register and use it:

var options = new JsonSerializerOptions();
options.Converters.Add(new PlayerConverter());

Console.WriteLine(JsonSerializer.Serialize(new Player { Name = "noname", Id = 1 }, options));

@ahsonkhan, I even agree that people don't care about the order of JSON properties. But, what about when you use the JSON received in the request, to convert it into XML, which in turn, must have a specific ordering?
Note: This is a real use case, and it kept me using Newtonsoft ([JsonProperty (Order = xx)])

@alastair-todd
Copy link

alastair-todd commented Nov 18, 2020

Another use-case: iterate through a response array on the client-side using the object's properties to create an html table (and the headings). Useful when a lot of stuff (including the client component) is generic, to have this as a known convention.

@StingyJack
Copy link

I use Json to save application settings, but there is a problem if the order is not guaranteed.

This can be fixed by using XML for config files. Which for most things .net should preserve ordering (preserving formatting may require a little extra work ). Also you get the ability to comment and explain what the settings are for and what eligible values are, which is a win for dev and support teams.

I even agree that people don't care about the order of JSON properties

I care, and we all should care. Not adhering to an order makes supporting the system more difficult and time consuming. There is an order being chosen, and that choice needs to be reliable.

what about when you use the JSON received in the request, to convert it into XML, which in turn, must have a specific ordering?

@silvairsoares - I agree, and this idea extends much farther than just the json->xml. Any machine generated data that may need to be inspected (compared, etc.) or should have a guaranteed and well known ordering. This could be config objects turned into config files, or the result of code generation from an API, .*proj/.sln files, data payloads exchanged between systems, and so on.

@rabidkitten
Copy link

To the commenters asking "why" this feature is necessary, beyond perhaps readability, perhaps what you should be asking is why features that used to work, features that people expected to be there - are no longer there? Sure, one can just deal with the now lack of functionality, but why should anyone have to? Why is it that upon every release code that used to work, tests that used to pass, UI's that used to depend on casing or date formats or custom formatters, all stop because a developer didn't think it was important or didn't feel they had time to do it...

@gtbuchanan
Copy link

@rabidkitten Are you sure System.Text.Json had this functionality in the past? You may be thinking of Newtonsoft.Json, which is a completely different library, not managed by Microsoft, and has been around for about 15 years. As for unmet expectations, that's why this issue exists. AFAIK, Microsoft didn't port System.Text.Json from any other source so your claims of this feature being intentionally excluded are somewhat unfounded. Nonetheless, you can always still use Newtonsoft.Json until this feature is available (see this and this tutorial).

@StingyJack
Copy link

StingyJack commented Dec 15, 2020

@gtbuchanan - the disconnect most of us have is in how System.Text.Json was and is still being presented as the successor and replacement for the newtonsoft library.

This and related blogs/GH posts was the first place many of us heard about the STJ and how fast it was supposed to be, what its potential future was, and how to try it out now.

There is also an official docs page for migration. Even though that has disclaimers at the beginning it uses some fancy words to obfuscate large and unexpected gaps in functionality.

JSON.net has been replaced in all of the recent .NET platform updates by System.Text.Json.

The author of the Newtonsoft library started working for Microsoft a few months before the announcement of the STJ package. I'm pretty sure I read somewhere that he played a large part in the creation of STJ.

So while technically you may be correct, you are also not "right" (no offense intended, I am in this same situation often and its lack of logical sense makes me cray-cray) because all of the presentation and communications from Microsoft give a clear impression that STJ is a continuation or replacement of JSON.net, leading us to expect it to do things that we could do before, but faster or with more features.


EDIT to avoid glomming up the topic (sure would be nice to have threaded conversations here). I'm not frustrated, I'm disappointed that it cant serialize correctly. Sure its fast, but its losing my data so its this makes it unusable.

EDIT2: someone at microsoft should start stressing the importance of reliable ordering of data structures, be it JSON or the elements in a proj file. Everyone who had merge pain with the non-SDK style proj format sticking elements into random but often overlapping positions in the file would have had that pain mostly eliminated had the project fie been ordered in a common and consistent way. AFAICT there is still no ordering for these files, so that problem didnt get fixed, it just got a lowered occurrence rate.

@gtbuchanan
Copy link

@StingyJack I understand your frustration. To be fair, successors do not generally have 100% feature parity right off the bat (hence why they're successors and not just a new version) and the official docs page looks like it makes a decent effort to list the differences. The fact of the matter is Newtonsoft.Json is still under active development and it is possible to swap out System.Text.Json when it does not fit your needs.

leading us to expect it to do things that we could do before, but faster or with more features

I read all the same blog posts and, aside from the speed claim, I saw nobody stating feature parity or more features. Any assumptions anyone makes on that matter are their own. To quote Immo Landwerth:

Furthermore, customers who need more performance can also choose to use the new JSON APIs, at the expense of the rich feature set that Json.NET offers.

I commented to provide an alternative to those who seem to think there isn't one. I have no interest in continuing this conversation about Microsoft's decisions as I'm purely interested in this feature and don't want to see it locked.

@PoP2005
Copy link

PoP2005 commented Jan 10, 2021

I am looking for this as well. In addition to much easier 'human' readability (which is also why the pretty formatting was created I suppose), it would allow for a very basic and efficient equality comparison between two Json serialized output strings (two identical Json objects would give two identical string outputs). I suspect (correct me if I'm wrong) that a simple string comparison would be faster than having to deserialize two objects and then compare all their properties ?

@MPBinSRV
Copy link

One more voice added to the hope of its future inclusion.

@thomas-lm
Copy link

Another use case :
I try to render Object respecting OData V4 Standard. The official documentation (see here) say that @odata.context "MUST be the first property of any JSON response"
There is no possibility to get it in first position when use with Inheritance.

@stevetalkscode
Copy link

Another use case I have is for controlling the ordering properties in models within a Swagger.json. Since Swashbuckle has moved over to STJ, the properties appear in alphabetical order. I would like to be able to control the order of properties using an attribute on the model

@Turnerj
Copy link

Turnerj commented Mar 22, 2021

We're in the same boat with Schema.NET. It affects our tests currently (though that can be overcome) however any users consuming our library may also be dependent on ordering so we wouldn't want to lose that.

I'm trying to work out the stage this issue is in - are we deciding whether this will be implemented, how this will be implemented, or more just waiting on someone to implement it? Happy to help out with implementing the change though it does look like @zcsizmadia basically has done all the work.

@BreyerW
Copy link

BreyerW commented Mar 23, 2021

Another use case is more efficient patching. With guaranteed order patch size is minimal since it involves finding diff in json and with unordered json patch size is undefined instead of smallest

@steveharter
Copy link
Member

Closing; basic property ordering is addressed by a new JsonPropertyOrderAttribute.

@bugproof
Copy link
Author

@steveharter Is there any issue to track "allowing write access to the JsonPropertyInfo.Order property" as explained here #54748 ?

@steveharter
Copy link
Member

Is there any issue to track "allowing write access to the JsonPropertyInfo.Order property" as explained here #54748 ?

For V6, the metadata infrastructure was not opened up, so modifying the Order at runtime is not possible. The issue to expose metadata is tracked by #36785

@ghost ghost locked as resolved and limited conversation to collaborators Aug 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Text.Json enhancement Product code improvement that does NOT require public API changes/additions
Projects
None yet
Development

No branches or pull requests