Skip to content

OpenSearch & Athena

Log Processor indexes every ingested log into two query engines from a single pipeline: OpenSearch for full-text search and dashboards, and an Athena datalake for SQL analytics. Subscriptions can target either or both independently.

OpenSearch provides full-text search with pre-built dashboards, index patterns, and ISM (Index State Management) retention policies. The domain ships with pre-built saved objects — index patterns, searches, visualizations, and dashboards — and OpenSearch configuration is applied automatically once the domain becomes healthy after deployment.

Browser access to OpenSearch Dashboards is secured through an ALB with Cognito authentication and an Nginx reverse proxy. See RBAC & SSO.

Why only two indexes (app and audit)

Two indexes keep the cluster simple and cost-effective:

  • app holds application logs with short retention (default 30 days).
  • audit holds compliance-sensitive logs with long retention (default 365 days).

Each subscription stream routes to one of these indexes, so you control retention and access per log type without managing dozens of indexes. Fewer indexes mean a lower shard count, less JVM pressure, and faster cluster recovery. For further separation, attach custom metadata fields to every log event for source filtering, and use ingest pipelines for automatic field processing.

Athena datalake

The datalake queries logs with SQL via partition-projected Glue tables and pre-built queries — no crawlers needed. Datalake writes remain raw, so Athena can parse at query time. Athena is billed by AWS at approximately $5 per TB scanned.

The Basic tier runs datalake-only (no OpenSearch) at roughly $5/month AWS infrastructure cost. Essential and above add OpenSearch.

Routing logs between engines

Because each subscription stream can target Athena and/or OpenSearch, you can route high-volume logs to the datalake only (keeping OpenSearch costs low) while sending critical logs to both for real-time search and visualization.