[EventGrid/Core] CloudEvent.from_dict fails to convert a datetime string to datetime object when microsecond exceeds the python limitation

[yunhaoling]

• Package Name: azure-eventgrid
• Package Version: 4.1.1
• Operating System: Windows
• Python Version: 3.7

Describe the bug
CloudEvent.from_dict fails to convert a datetime string to datetime object when the microsecond field exceeds the python limitation.

The time in the input dictionary is "2021-05-10T14:39:23.5811478Z" (This value is set by the event grid service).

Traceback (most recent call last):
  File "d:\projects\azure-sdk-for-python\sdk\core\azure-core\azure\core\_utils.py", line 65, in _convert_to_isoformat
    deserialized = datetime.datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%f")
  File "C:\Users\yuling\Anaconda3\envs\sbenv37\lib\_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "C:\Users\yuling\Anaconda3\envs\sbenv37\lib\_strptime.py", line 362, in _strptime
    data_string[found.end():])
ValueError: unconverted data remains: 8

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:/Projects/azure-sdk-for-python/sdk/eventhub/azure-eventhub/samples/eg.py", line 37, in <module>
    c = CloudEvent.from_dict(json_obj)
  File "d:\projects\azure-sdk-for-python\sdk\core\azure-core\azure\core\messaging.py", line 166, in from_dict
    time=_convert_to_isoformat(event.get("time")),
  File "d:\projects\azure-sdk-for-python\sdk\core\azure-core\azure\core\_utils.py", line 67, in _convert_to_isoformat
    deserialized = datetime.datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S")
  File "C:\Users\yuling\Anaconda3\envs\sbenv37\lib\_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "C:\Users\yuling\Anaconda3\envs\sbenv37\lib\_strptime.py", line 362, in _strptime
    data_string[found.end():])
ValueError: unconverted data remains: .5811478

Process finished with exit code 1

To Reproduce
Steps to reproduce the behavior:

from azure.core.messaging import CloudEvent
raw_data = '{"id":"xxx","source":"xxx","specversion":"1.0","type":"Microsoft.Storage.BlobCreated","dataschema":"#","subject":"xxx","time":"2021-05-10T14:39:23.5811478Z","data":{"api":"PutBlob","clientRequestId":"xxx","requestId":"xxx","eTag":"xxx","contentType":"application/octet-stream","contentLength":104065,"blobType":"BlockBlob","url":"xxx","sequencer":"xxx","storageDiagnostics":{"batchId":"xxx"}}}'
json_obj = json.loads(raw_data)
c = CloudEvent.from_dict(json_obj)

Expected behavior
The sdks handles this instead of raising error or we give a more friendly error.

Additional context
python microseconds docs:

datetime.microsecond
In range(1000000). 

cc: @jongio

[jongio]

Workaround:

import re
datetime_format_regex = re.compile(r'^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{1,}Z$')

def datetime_parser(dct):
    for k, v in dct.items():
        if isinstance(v, str) and datetime_format_regex.match(v):
            dct[k] = dct[k][:-2] + 'Z'
    return dct

receiver = service_bus_client.get_queue_receiver("invoices")
    # Receive message
    with receiver:
        while True:
            msgs = receiver.receive_messages()
            events = [CloudEvent.from_dict(json.loads(str(msg), object_hook=datetime_parser)) for msg in msgs]

[yunhaoling]

more context from @lmazuel :

Datetime is not an azure-core bug, it's a service bug. 3 digits milliseconds is required in order to be parsable from all languages.

[annatisch]

@yunhaoling - ugh .Net datetimes strike again...
What we did for Tables is deserialize to a custom datetime object, where we preserved the original value returned from the service as an attribute in case anyone actually needed the 7 decimal places - but otherwise we trimmed it to be compatible with Python.

Does this need to support round-trip? e.g. if I deserialize and drop the last decimal place, is it going to cause problems if that value no longer matches the one maintained at the service?