Wire compatibility is broken

[SeanFeldman]

Expected Behavior

To be able to send a byte array from the new client (Microsoft.Azure.ServiceBus) to the old client (WindowsAzure.ServiceBus). Just like the new client can read BrokeredMessages using GetBody<> extension method, it should be possible to send a body with byte array that the old client would be able to read back.

Actual Behavior

The old client throws SerializationException: There was an error deserializing the object of type System.Byte[]. The input source is not correctly formatted.

Stack trace

at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)   at System.Runtime.Serialization.DataContractSerializer.ReadObject(XmlDictionaryReader reader, Boolean verifyObjectName)   at Microsoft.Azure.ServiceBus.InteropExtensions.DataContractBinarySerializer.ReadObject(XmlDictionaryReader reader, Boolean verifyObjectName)   at System.Runtime.Serialization.XmlObjectSerializer.ReadObject(XmlReader reader, Boolean verifyObjectName)   at System.Runtime.Serialization.XmlObjectSerializer.InternalReadObject(XmlReaderDelegator reader, Boolean verifyObjectName)   at System.Runtime.Serialization.XmlObjectSerializer.InternalReadObject(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)   at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)   at System.Runtime.Serialization.XmlObjectSerializer.ReadObject(XmlDictionaryReader reader)   at Microsoft.Azure.ServiceBus.InteropExtensions.DataContractBinarySerializer.ReadObject(Stream stream)   at Microsoft.Azure.ServiceBus.InteropExtensions.MessageInteropExtensions.GetBody[T](Message message, XmlObjectSerializer serializer)   at UserQuery.d__1.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()   at UserQuery.Main()   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)   at System.Threading.ThreadHelper.ThreadStart()

Versions

• OS platform and version:
• .NET Version: 4.5.2
• NuGet packages:

Wicrosoft.Azure.ServiceBus	4.1.3
WindowsAzure.ServiceBus	0.0.7-preview

[SeanFeldman]

I've attached a repro (asb.wire.compat.byte.array.linq.txt) using LinqPad.

[vinaysurya]

@SeanFeldman , can you try received2.GetBody() and convert the stream into bytes using say Stream.ToArray() ?

[vinaysurya]

received2.GetBody<Stream>() i meant*

[SeanFeldman]

@vinaysurya suggested workaround works, but it means implementations using old client require changes to be deployed to work with the messages sent by the new client.

Does that mean that sending byte[] using the old client or the new client both could be received by the old client using msg.GetBody<Stream>() converted later to a byte[]?

[hvetter-de]

Same problem here. I am sending a Message with .NetCore 0.0.7 preview package and I am trying to read the message with the ServiceBus SDK (WebJob). BrokeredMessage.GetBody<string> fails because of XmlSerializer error.

I sent a message with 0.0.7 preview and and the full .Net version for comparison.
To get the string from the preview sent message, I read the Stream, converted it to byte[] and then to Utf8 String.
The result is that the body of both messages are different:

BrokeredMessage

@ 
base64Binary3http://schemas.microsoft.com/2003/10/Serialization/�]{"RdpUserName":null,"RdpPassword":null,"SbConnectionString":null,"Owner":"Owner1","Name":"Name1"�}

Message

{"RdpUserName":"null","RdpPassword":"null","SbConnectionString":"null","Owner":"Owner1","Name":"Name1"}

Name	Version
WindowsAzure.ServiceBus	3.4.5
Microsoft.Azure.ServiceBus	0.0.7-preview

[SeanFeldman]

On the wire compatibility - w/o deploying workaround to all the older endpoints that would consume byte[]-base and Stream-based messages the same way, this would would not work 😞

Do you know @vinaysurya in what version of the old client (Microsoft.Azure.ServiceBus) was the workaround added? In the past that didn't work. Thank you.

[SeanFeldman]

@Azure/azure-service-bus-write would it be possible to trace back in what version of the ASB old client (WindowsAzure.ServiceBus) to see when this workaround became possible? Thank you.

[vinaysurya]

@SeanFeldman, there was no workaround added in the older client to support this scenario. Was there an error when you tried this earlier?

[SeanFeldman]

We had an error in the past @vinaysurya, which led to separation of how body is accessed.
I'll definitely explore this route.

Saying that, it is still not addressing the fact that endpoints using old client cannot receive messages with byte[] body using GetBody<byte[]>() sent from the new client. The only way to achieve that is to re-deploy old endpoints and use the alternative method suggested here. 😞

[vinaysurya]

@SeanFeldman, yes we understand that is going to be an issue for processing with older clients . At this point, we are trying to see what makes sense for interop as a whole, not just for our older dotnetclients (if we did it in a particular way there) but also understand how interop might look like with for ex say ProtonJ/QPID JMS etc and do what will likely be better for all of them. So will update on this thread once we have explored that.

[SeanFeldman]

@vinaysurya should this be labeled as a bug?

[binzywu]

ServiceBus started with SBMP which is a protocol built on top of WCF. At that time, DataContract Serialization became a natural choice to object serialization which is fine. But as it is hidden underneath implicitly. Our BrokeredMessage is actually representing 2 types of messages, Stream (Bytes as we don't really send them as stream) Message and Object Message. When you send stream, you get stream. When you send object, you get object. From this perspective, our .netstd library currently only supports Bytes Message which we sent via AMQP Data section. And you will get Stream from the old client. This also aligns with JMS (Qpid JMS implementation) for it's BytesMessage. This is a sample for sending messages from JMS and receiving from our new client library.

As @vinaysurya mentioned, we will have to support different AMQP sections (Data, Value, Sequence etc.) to have more comprehensive inter-op support. This is not only our old client library but also many other messaging related libraries/standards, particularly JMS.

So 1.0.0 could be the start. How do we extent it for sending and receiving different messages is now up to the collective decision from this community. We would love to hear all suggestions and ideas.

[udidahan]

I'm going to try to reiterate the main issue here which I think is well-stated in the title. Systems in production that currently work with Azure Service Bus should not be broken.

[binzywu]

So there are 2 scenarios, let me explain how things will work with current .netstd client.

1. send via full .net client and receive via .netstd client

type sent via .netfull	receive via .netstd
stream	bytes (which is literally bytes from stream)
object (including string, bytes[]), the object is serialized with DataContract	GetBody<T> (the extension method we provided)

However, in this case, if there is any other customized type of XmlObjectSerializer is passed for object serialization, you will have to get bytes and deserialize with that XmlObjectSerializer.

2. send via .netstd client and receive via full .net clien

type sent via via .netstd	receive via .netfull
bytes	GetBody<Stream> then get all bytes from streams
bytes (the bytes from the XmlObjectSerializer serialization e.g. DataContractBinarySerializer (we will make the class public)	GetBody<T>(XmlObjectSerializer) with default DataContractBinarySerializer or any customized other XmlObjectSerializer

If the systems are running with old client, you will be likely stuck with XmlObjectSerializer (data contract serializer) like our .netfull client. But things will not be broken. Sample code is like below which is exactly what happens underneath in the .netfull client.

var bytes = your bytes;
var serializer = DataContractBinarySerializer<byte[]>.Instance;
using (MemoryStream stream = new MemoryStream())
{
    serializer.WriteObject(stream, bytes);
    var msg = new Message(stream.ToArray());
    var client = new Microsoft.Azure.ServiceBus.QueueClient(ConnectionString, Queue);
    await client.SendAsync(msg);
    await client.CloseAsync();
}

Supporting different body section types for AMQP is another important goal for us. We will later introduce support sending and receiving different AMQP body section types.

[SeanFeldman]

@binzywu I've ended up doing the same thing copying internal DataContractBinarySerializer class to my code. #259 will address this issue. Thank you ☺️

[binzywu]

@SeanFeldman glad it works. also as you also pointed out, expose the internal data contract isn't necessary. So I also close the PR.

[SeanFeldman]

@binzywu this would be a good candidate for a sample.

[clemensv]

This seems resolved, is it? Reopen if it's not.

[SeanFeldman]

@clemensv it's resolved in a way of extension methods that require application to know where messages are coming from (.NET Standard client or Full FW client). It is not resolved automatically. Therefore, I'm not convinced it's considered to be resolved. Unless the team doesn't plan to invest beyond that as the refreshed documentation does mention the following:

The .NET Standard and Java API variants only accept byte arrays, which means that the application must handle object serialization control.

[udidahan]

that require application to know where messages are coming from (.NET Standard client or Full FW client)

It isn't clear to me how a given subscriber could know that, especially in larger team environments where things are evolving in parallel with each other.

[niemyjski]

This needs to be fixed and is unacceptable.

[niemyjski]

I think #138 is related to this issue as well.

[SeanFeldman]

@niemyjski it's not related, it's a subset of this issue.

[Eneuman]

Yeah, this is still big a issue.
As an example: The reference architecture guide is using the new client in the implimentation of events using Azure Service Bus (https://github.com/dotnet-architecture/eShopOnContainers).

But sending a message using Azure Scheduler to one of the buses will crash the subscription implimentation in the reference architecture. I guess Azure Scheduler is using the old client.

Whats the best way to handle this senario?

[SimonCropp]

@clemensv

This seems resolved, is it? Reopen if it's not.

I think it needs to be a project maintainer to reopen an issue. and from the follow up comments it seems it should be re-opened

[NickEricson]

In case it is useful for anyone - here is what I did to be able to send messages from .netstandard and .net462 to a .net462 receiver (roughly):

462 sender

            var busMessage = new ReportingBusMessage(JsonConvert.SerializeObject(message)) { ContentType = message.ContentType };
            return _topic.Value.SendBatchAsync(busMessage);

netstandard 2 sender

            var serializer = DataContractBinarySerializer<string>.Instance;
            using (var buffer = new MemoryStream())
            {
                serializer.WriteObject(buffer, JsonConvert.SerializeObject(message));
                var serializedMessage = buffer.ToArray();
                var busMessage = new ReportingBusMessage(serializedMessage) { ContentType = message.ContentType };
                return _topic.Value.SendAsync(busMessage);
            }

462 receiver

Callback(BrokeredMessage message)
{
     MyObject m = JsonConvert.DeserializeObject<MyObject>(message.GetBody<string>());
}

The message being sent is a json serialized string wrapped in an xml element in 462. Adding the serializer manually in netstandard makes it the same format.