Simplifier: GRIP dialect

[bwalsh]

Please write up a use case that describes what this is and when / why to use it.

Use case

As a [role], when [this happens] in order to [describe outcome], the system needs to [feature description]

Implementation details

• What it does, how to invoke it, etc...

[matthewpeterkort]

As a ACED IDP platform developer, in order to import edges and vertices into GRIP, source data files that adhere to the Iceberg schema must be able to be translated into the GRIP format.

For example GRIP vertex has keys {"label": str, "id": uuid , "data": dict} and edge has keys :{"label": str, "to": uuid, "from": uuid}.

The edge/vertex generation command looks for a directory and generates the edges/vertices that are applicable from the ndjson iceberg files located in the directory. It assumes that objects follow the schema provided using the --schema_path option and if they don't errors will occur.

iceberg data simplify tests/fixtures/simplify/synthea newobsss --schema_path https://raw.githubusercontent.com/bmeg/iceberg/main/schemas/graph/graph-fhir.json --dialect GRIP

Edge seems too slow to be a practical solution at scale. Need to think about how to make this faster.

Had to remove observation.ndjson because it was taking too much time to generate edges and DocumentReference.ndjson was causing schema errors.

[bwalsh]

Thanks. Let's talk about it on Monday

…

On Fri, Nov 10, 2023, 11:48 AM matthewpeterkort ***@***.***> wrote: As a ACED IDP platform developer, in order to import edges and vertices into GRIP, source data files that adhere to the Iceberg schema must be able to be generated into the GRIP format. For example vertex has keys {"label": str, "id": uuid , "data": dict} and edge has keys :{"label": str, "to": uuid, "from": uuid}. To invoke these commands run iceberg data simplify tests/fixtures/simplify/synthea newobsss --schema_path https://raw.githubusercontent.com/bmeg/iceberg/main/schemas/graph/graph-fhir.json --dialect GRIP Edge seems too slow to be a practical solution at scale. Need to think about how to make this faster. Had to remove observation.ndjson because it was taking too much time to generate edges and DocumentReference.ndjson was causing schema errors. — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALVQEUSVI57FW2KSBXFLTYDZLFXAVCNFSM6AAAAAA642Y7POVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWGA3TEMRYHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>