Skip to content

Retention

By default, retaining logs forever silently grows your AWS bill. Log Processor manages retention independently at each layer of the pipeline, which reduces the complexity of controlling storage costs and meeting compliance requirements.

Retention by layer

  • CloudWatch Logs — Set via the retentionDays field in your subscription file. Without this, CloudWatch retains logs indefinitely at $0.03/GB/month. Retention is enforced automatically on each subscription run. Specific entries override broader regex patterns.
  • S3 log bucket — The Firehose delivery bucket, controlled by S3 lifecycle rules; old objects are deleted automatically.
  • OpenSearch — Managed by ISM (Index State Management) policies per index type. For example, app indexes delete after 30 days and audit after 365 days. Configurable in your deployment profile.
  • Datalake (Athena) — S3 lifecycle rules per index prefix, typically longer retention than OpenSearch (for example 1–7 years) since S3 storage is cheaper. Queryable via Athena at approximately $5/TB scanned.
  • OpenSearch snapshots — Stored in a separate S3 bucket with its own lifecycle rules, useful for disaster recovery. The bucket can be retained even after stack deletion. See snapshots below.
  • S3 access logs — Access logs for the log bucket and datalake bucket, retained per the tier lifecycle rule and queryable via Athena for auditing. See Compliance Reports and Cross-Account Ingestion.

All retention periods are specified by your deployment tier (Basic, Essential, Advanced, Enterprise). Each index type (app, audit) has retention defaults per tier that you can adjust to your needs, and OpenSearch and the S3 datalake have independent retention policies.

OpenSearch snapshots

AWS-managed automated snapshots (hourly, 14-day retention, free) are sufficient for disaster recovery. The custom S3 snapshot repository is mainly useful for:

  • Long-term archival beyond 14 days.
  • Cross-region restore (copy the S3 bucket).
  • Pre-upgrade backup (snapshot before a major change).

Use the provided snapshot.cmd / snapshot.sh helper script to take a snapshot on demand.