Data retention period (and resampling)

[hibke]

Hi,

as there is no dateExpires for entities anymore with NGSI-LD (it is a NGSI-v2 feature), I have been looking how to define a retention duration (and also how to resample) temporal data and I found out that both are natively addressed by TimescaleDB, provided that hypertables are used. It is recommended to use hypertables instead of regular Postgres tables for storing time-series data, in order to take advantage of these features as well as all other Timescale features, including improved insert and query performance. (https://docs.timescale.com/getting-started/latest/tables-hypertables/ )

Therefore, I was wondering why Timescale DB tables are not created by Orion-LD as hypertables from the start, and whether you have a recommendation for managing data retention.

Could you please advise ?

[kzangeli]

ok, not really my table ...
I'll ask for help.
@wistefan ?

[kzangeli]

FYI, we have approved "transient entities" in ETSI ISG CIM and the feature will be standard NGSI-LD in the next release (v1.9.1, planned for September, but I imagine it will get out later - we just released 1.8.1 a few days ago).

We also approved "transient attributes".
The field for this will be a non-reified "expiresAt": "<ISO8601 DateTime>", both on the toplevel (for the entity) and on attribute level.