Data Retention Policy

[scheidtdav]

OSEM needs a data retention policy in order to be sustainable in operation.
Thankfully TimescaleDB comes with a feature for this.

What are reasonable data retention policies and where do we communicate them?
Should we add continuous aggregates?

Here is a first draft as subject for discussion:

• Measurement data
  • Full resolution for 12 months
  • Hourly averages for younger than 18 months
  • Daily averages for data younger than 24
  • Drop data past 24 months

Any opinions?

[zven]

Sounds like a good starting point! I guess this would also be something that could be easily adjusted in case we need to change the retention periods?

How would this affect the data stored in the archive (in case I want to have high resolution data for the last 2 years or 5 years)?

[scheidtdav]

The archive is and will be a place outside of the live data and hosts the raw data without any aggregates at all.
Hence it should stay unaffected by this.

Regarding changing the retention periods: Obviously when extending from 12 to 18 months for example, we don't magically get 6 months added. But it can fill up over time and changing it from 12 to 18 should totally be possible. Same goes for going from 12 to 10.

Should we start with my draft then?

[zven]

Yes! Let's start with your numbers, then. I think as long as "everything" is still available via the archive, we would still enable niche use cases, where a project would need the raw data for multiple years. 👍