Unable to send "pure" body content

[justdmitry]

Actual Behavior

1. MemoryStream (with simple foo bar text) is converted to byte[] via ToArray() and sent via ServiceBus
2. Real content of message is binary-serialized data like @base64Binar3http://schemas.microsoft.com/2003/10/Serialization/? ?foo bar?

Expected Behavior

1. Send "pure" byte content as message payload, for compatibility with other platforms and old (legacy) clients

Explanation

We use ServiceBus for interaction between applications written on .NET and Java, where message content is XML text, written as "pure" bytes in to Stream of BrokeredMessage. We are planning to introduce new services written in NETCore, which should talk with existing services.

There is no problem to call memoryStream.ToArray() to send byte[] (in new services), but it's unacceptable that this data is being "packed" to some kind of serialization-envelope, because this require additional de-serialization step on receiving.

I wrote sample app with both WindowsAzure.ServiceBus and Microsoft.Azure.ServiceBus, demonstrating the problem, here is output:

How can I send "pure" content in Message? I found #98 with explanation about removing Stream body content, and agree about "need to be in memory" and "sends are taking a long time, when the latency is in fact caused by the reading of the stream", but is it really needed to use serialization-envelope, while byte[] is the only content/payload type?

Versions

• Microsoft.Azure.ServiceBus 0.0.2-preview and Microsoft.Azure.ServiceBus 0.0.3-preview
• WindowsAzure.ServiceBus 4.0.0

[jtaubensee]

@justdmitry - From a quick glance I notice the following:

If I change the SendLegacy method to flush the stream to an array before creating the message, I get the same result.

Here is the change:

public static async Task SendLegacy(MemoryStream stream)
{
	var msg = new BrokeredMessage(stream.ToArray())
	{
		MessageId = Guid.NewGuid().ToString(),
		Label = "WindowsAzure.ServiceBus",
	};

And here is the result:

Sending messages with content 'foo bar':

#d787fdc2-b7fe-441b-89b1-dd1d4dd6bf0c is sent via WindowsAzure.ServiceBus

#2d917d3f-916e-4491-90ba-9669c20b6d80 is sent via Microsoft.Azure.ServiceBus

#d787fdc2-b7fe-441b-89b1-dd1d4dd6bf0c received (from WindowsAzure.ServiceBus):
  Content: '@base64Binar3http://schemas.microsoft.com/2003/10/Serialization/?  ?foo ba?�r'

#2d917d3f-916e-4491-90ba-9669c20b6d80 received (from Microsoft.Azure.ServiceBus):
  Content: '@base64Binar3http://schemas.microsoft.com/2003/10/Serialization/?  ?foo ba?�r'

I believe this has something to do with the serializer in the old client, so we will need to do some further investigation. Usually this isn't the right place for "old client" issues, but this will certainly be relevant to users of the new library as well.

Thanks for the awesome repro!

[justdmitry]

@jtaubensee yes, old clients may be upgraded to send messages, "identical" to new "format". But it's not good "compatibility" point - ask to upgrade all parties to new message transport format...

Also, what about receiving "pure byte stream" messages... I updated my demoapp and found that Microsoft.Azure.ServiceBus can successfully read message, sent in "old way". Cool! But this means that I can receive, but can't send messages in some "format", while this "format" is well-acceptable/compatible by ServiceBus service itself. Why library is limiting my messaging?

Also, if we look at other platforms/languages - are they always be able to deserialize "base64 binary by Microsoft", is this open format?

So I'm back to my original question - its it really needed to perform serialization on "simple" byte sequence?

[NickMoores]

We've been bitten by this.

To me, it's looking as though an AmqpValueMessage is being created for the byte[], and the Amqp library is using it's own encoding (probably ArrayEncoding or BinaryEncoding which is decorating the content.

It has broken our .NET 4.6 clients (including our Azure Functions, which would deserialize to our message types for us!), which we've fixed by doing something like:

byte[] messageContent = brokeredMessage.GetBody<byte[]>(); string messageContentStr = Encoding.UTF8.GetString(messageContent);

...and then deserializing from messageContentStr, which now has the nasties removed.

I understand the desire for the Message class to only use byte[] for holding the data, but if brokeredMessage.GetBody<byte[]>() on the client side doesn't mirror the byte[] that the sender added to the Message, then this feels a bit off to me...

[ghost]

+1 for being able to construct a new Message with a Stream that isn't manipulated via serialization. In other words, it should behave like the full framework client.

[GaryHope]

+1 for being able to construct a new Message with a Stream that isn't manipulated via serialization. In other words, it should behave like the full framework client to support interop. I've been "fixing" previous code to support solutions working with new Microsoft.Azure.ServiceBus code for days now. If we think about interop we need at the very least to provide visible guidance so this doesn't pop-up as a surprise.

[kebeller]

For those that are trying to remove the extra chars from existing messages. Here is the code used in the GetBody method.

public T GetBody<T>(BrokeredMessage brokeredMessage)
{
  var ct = brokeredMessage.ContentType;
  Type bodyType = Type.GetType(ct, true);

  var stream = brokeredMessage.GetBody<Stream>();
  DataContractSerializer serializer = new DataContractSerializer(bodyType);
  XmlDictionaryReader reader = XmlDictionaryReader.CreateBinaryReader(stream, XmlDictionaryReaderQuotas.Max);
  object deserializedBody = serializer.ReadObject(reader);
  T msgBase = (T)deserializedBody;
  return msgBase;
}

[SeanFeldman]

Twitter conversation about old/new client issues by @MisterJames

[SeanFeldman]

Should be labeled as a bug.

[MisterJames]

On my side, I'm not sure that I have a way to help coach Azure Functions into proper deserialization, which happens during parameter binding, before my code is executed. The message won't even convert to string. In previous combinations of the Web Jobs SDK and Service Bus messaging, I would use the BrokeredMessage object, which no longer seems to be part of the library.

I am using Microsoft.Azure.ServiceBus version 0.0.6-preview in my ASP.NET Core project to send the message. Where I'm at right now (putting code together from a few workaround-ish sources) looks like the following:

public async Task Handle(QueueRegistrationCommand message)
{
    // create the message body
    var wrappedMessage = new { QueueRegistrationCommand = message };
    var busMessage = JsonConvert.SerializeObject(wrappedMessage);

    // encode data and create message
    byte[] bytes = Encoding.UTF8.GetBytes(busMessage);            
    var encodedMessage = new Message(bytes) 
    {
        ContentType = "application/json",
        Label = "WindowsAzure.ServiceBus",
        MessageId = Guid.NewGuid().ToString()
    };

    // send message and hold a small vigil 
    await _client.SendAsync(encodedMessage);
}

To note, I've tried wrapping the message in an anon object as the Azure Functions bit was complaining about not finding the QueueRegistrationCommand element. I've also tried simply encoding the message on it's own (without the anon object) and didn't get any further.

The Azure Function itself has a method with the following signature:

public static void Run( [ServiceBusTrigger("my-queue-name", AccessRights.Manage, Connection = "service-bus-connectionstring")]
            //QueueRegistrationCommand registerMessage, 
            string inputMessage,
            TraceWriter log)
    // ...

The Function does not execute (first line of logging is not executed) as binding fails with the message below. You can see above that I've also tried to deserialize to a POCO. Here's the exception that is thrown during binding.

mscorlib: Exception while executing function: Functions.FanOut. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'inputMessage'. Microsoft.Azure.WebJobs.ServiceBus: The BrokeredMessage with ContentType 'application/json' failed to deserialize to a string with the message: 'Expecting element 'string' from namespace 'http://schemas.microsoft.com/2003/10/Serialization/'.. Encountered 'Element'  with name 'base64Binary', namespace 'http://schemas.microsoft.com/2003/10/Serialization/'. '. System.Runtime.Serialization: Expecting element 'string' from namespace 'http://schemas.microsoft.com/2003/10/Serialization/'.. Encountered 'Element'  with name 'base64Binary', namespace 'http://schemas.microsoft.com/2003/10/Serialization/'.