Possible XmlDataReader?

[electricessence]

Seems like with your existing code base and expertise, this would be easier to do than most.
Is this something of interest? Or no?
I can't find any XML readers out there than will take advantage of modern techniques.

[MarkPflug]

I'm not really interested in pursuing that. The System.Xml implementation is sufficient for everything I've ever needed. When you say "modern techniques" what are you looking for exactly?

[electricessence]

Use of Span, etc. Improved memory use.
No worries tho. Was just curious. :)

[MarkPflug]

System.Xml can be used in a low-allocation way, even though it doesn't support Span. I've used it to create a data reader for excel (.xlsx), that has significantly lower allocation that other .NET excel libraries, including the most popular and closed-source commercial libraries. As an example of the APIs in-use:
https://github.com/MarkPflug/Sylvan.Data.Excel/blob/1d7bd83796f15de96d77fcd6f0381a5f9a8053ac/source/Sylvan.Data.Excel/Xlsx/XlsxReader.cs#L421

[electricessence]

I'm excited to check this out.

[MarkPflug]

@electricessence

To answer your question about the benchmark performance, I think the issue is in the construction of your benchmark against my library.

Taking a look at the link you shared via Twitter:
https://github.com/Open-NET-Libraries/Open.Text.CSV/blob/master/Tests/CsvFileReadTests.cs#L24-L46

The body of the loop is creating a new List<string> and then calling Add repeatedly for each field. Calling Add will cause the list to grow if it isn't large enough. This will cause the List to need to be resized as more elements are added. The default size for a List is 4, and the resize strategy is to double the size of the list each time it grows. Since your test data file has 12 columns, the list will have to grow twice: from 4 to 8 and then 8 to 16, and it will have over-allocated 4 elements (16). If you change line 39 from var row = new List<string>(); to var row = new List<string>(fields); you will avoid the need to resize and overallocation. With this change you should see a more equitable comparison. Your library does this same optimization via the CsvRowBuilder._maxFields field.

Another thing I would point out is that my library also offers optimization/reduced allocation via the strongly typed field accessors. Calling GetString will obviously require a string to be allocated. However, in your test data file 8 of the 12 columns appear to contain integers. Calling GetInt32 instead of GetString will avoid the need to allocate a string for the integer values, as the value can be parsed directly out of the internal buffer without having to produce a string first. This not only reduces the allocation, but is also faster as it avoids a lot of unnecessary work copy allocating/copying character buffers.

The results on my machine with these changes:

Method	Mean	Error	StdDev	Ratio	Gen 0	Gen 1	Gen 2	Allocated
GetAllRowsFromFile	2,023.9 ms	156.55 ms	24.23 ms	1.00	97000.0000	33000.0000	1000.0000	589 MB
GetAllRowsFromFile_Sylvan	1,437.3 ms	96.25 ms	14.89 ms	0.71	97000.0000	34000.0000	1000.0000	590 MB
GetAllRowsFromFile_Sylvan_StrongType	673.0 ms	42.22 ms	6.53 ms	0.33	41000.0000	14000.0000	1000.0000	257 MB

GetAllRowsFromFile_Sylvan is your original benchmark with the List capacity initialized properly.
GetAllRowsFromFile_Sylvan_StrongType is my library binding to a strongly-typed class instead of a List<String>.

I will send a PR with these changes so you can experiment with it.

[electricessence]

Another thing I would point out is that my library also offers optimization/reduced allocation via the strongly typed field accessors. Calling GetString will obviously require a string to be allocated. However, in your test data file 8 of the 12 columns appear to contain integers. Calling GetInt32 instead of GetString will avoid the need to allocate a string for the integer values, as the value can be parsed directly out of the internal buffer without having to produce a string first. This not only reduces the allocation, but is also faster as it avoids a lot of unnecessary work copy allocating/copying character buffers.

Ok I think I'm getting it. The reason why yours is so much faster is because you're making assumptions about the content and can optimize for that.

Thanks for the info. I'm looking forward to sharing and using your library in the future. 😄

[electricessence]

Since your test data file has 12 columns, the list will have to grow twice: from 4 to 8 and then 8 to 16, and it will have over-allocated 4 elements (16).

Each row (List) is created at capacity:
https://github.com/Open-NET-Libraries/Open.Text.CSV/blob/master/Source/CsvRowBuilder.cs#L21

[MarkPflug]

That comment was about your original implementation of the sylvan benchmark, it wasn't specifying an initial capacity and the grow/copy was what was causing my library to appear slow.

[electricessence]

Oh duuhh my bad.

[MarkPflug]

@electricessence I got around to publishing the benchmarks for my Excel library this morning. It does require a bit of effort to avoid allocations when using XmlReader, but it is possible.

[electricessence]

It will definitely be something I look at in the near future. :) I experimented with trying to get it to do the trick but just seemed to daunting at first.