You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In mango, we recently created a JsonAttributeStore utility class that allows json and generically nested hashmaps to be flattened into attributes to be used in the EventStore and EntityStore implementations.
The problem is, it has partitioned a lot of assumptions that can be made about the underlying events. For instance, if I could assume that all of my events contain attributes that have been flattened from an arbitrarily nested document, I can do things like querying based on nested attributes in Spark SQL. I can also do things with my UI to determine and display the nested layers. I cannot do such things if my only assumption is that I have an attribute which is a simple key/value object.
I propose we create a new Store that is based on the event store, called EventDocumentStore or JsonEventStore. This store should assume any objects are arbitrarily nested and can be marshalled back into json when queried. This will allow us to manage the schema of the entire document (which items are arrays vs objects) and provide that schema to things like Spark SQL.
Methods in the JsonEventStore interface could look like this:
class JsonEvent {
String type;
String id;
Long timestamp;
Map[String, Object] document;
}
The reason I want to differentiate this object from Mango's Event object is that I want to separate the internal indexing components of the event from the document itself. An Event's ID, for instance, could possibly align with an ID in the json document itself, but it doesn't need to. This gives it flexibility as the Event's fields are just metadata that inform the store how to index while the documents themselves are completely separate entities.
The text was updated successfully, but these errors were encountered:
In mango, we recently created a JsonAttributeStore utility class that allows json and generically nested hashmaps to be flattened into attributes to be used in the EventStore and EntityStore implementations.
The problem is, it has partitioned a lot of assumptions that can be made about the underlying events. For instance, if I could assume that all of my events contain attributes that have been flattened from an arbitrarily nested document, I can do things like querying based on nested attributes in Spark SQL. I can also do things with my UI to determine and display the nested layers. I cannot do such things if my only assumption is that I have an attribute which is a simple key/value object.
I propose we create a new Store that is based on the event store, called EventDocumentStore or JsonEventStore. This store should assume any objects are arbitrarily nested and can be marshalled back into json when queried. This will allow us to manage the schema of the entire document (which items are arrays vs objects) and provide that schema to things like Spark SQL.
Methods in the JsonEventStore interface could look like this:
Where a JsonEvent looks like this:
The reason I want to differentiate this object from Mango's Event object is that I want to separate the internal indexing components of the event from the document itself. An Event's ID, for instance, could possibly align with an ID in the json document itself, but it doesn't need to. This gives it flexibility as the Event's fields are just metadata that inform the store how to index while the documents themselves are completely separate entities.
The text was updated successfully, but these errors were encountered: