Frequently Asked Questions

Does my data leave my AWS account?

No. Everything runs inside your VPC with isolated subnets and no internet egress. Logs flow from CloudWatch → Firehose → S3 → Lambda → OpenSearch/Athena, all within your account. AWS services are accessed exclusively through VPC endpoints — no data ever leaves your network boundary. The seller has no access to your account, data, or infrastructure. The only external call is a periodic entitlement check to AWS Marketplace to verify your subscription.

Can I use this without OpenSearch?

Yes. The Basic tier runs datalake-only — query logs with Athena SQL at approximately $5/month AWS infrastructure cost. Essential and above add OpenSearch for full-text search and dashboards. Subscriptions can target Athena and/or OpenSearch independently, so you can route verbose or high-volume logs to the datalake only (keeping OpenSearch costs low) while sending critical application and audit logs to both for real-time search and visualization.

How do I add new log groups?

Use the subscription editor (Essential tier and above) to add log groups, configure stream routing, set retention, and define pattern rules — all from a browser with regex validation and diff preview before saving. On Basic tier, update the subscriptions JSON file directly in the assets S3 bucket. Changes are picked up automatically — no redeployment needed. Version history is maintained in S3, so you can revert to any previous configuration.

What happens if I unsubscribe?

Your data remains in S3 and OpenSearch. Snapshot buckets can be retained for disaster recovery.

Can I customize retention per log type?

Yes. Each index type (app, audit) has retention defaults per tier that you can adjust to your needs. Both OpenSearch and the S3 datalake have independent retention policies. You can also enforce CloudWatch Logs retention per subscription entry using the retentionDays field — specific entries override broader regex patterns. By default, CloudWatch log groups retain logs indefinitely at $0.03/GB/month. Without retention policies, storage costs grow silently and can become a significant expense.

Can I change tiers later?

Yes. Select a tier upgrade or downgrade (Essential | Advanced | Enterprise) and update the stack — node count, instance type, and volume changes are handled in-place via OpenSearch blue/green deployment with zero downtime (typically 15–30 minutes). Feature differences activate immediately based on your Marketplace entitlement dimension. Upgrading from Basic to Essential requires a stack delete and fresh deploy since it adds an OpenSearch domain, VPC, and dashboards infrastructure that cannot be added to an existing Basic stack. Your OpenSearch and S3 datalake remain unaffected during any tier change.

Can I host multiple instances?

Yes. Select a tier and provide a unique stack name. Each stack deploys into its own VPC with isolated resources, so you can run multiple instances side by side without interference — AWS limits apply. This is useful for separate environments (e.g. dev vs prod) or different teams with distinct log management needs. Refer to the Getting Started guide for details.

Can I ingest logs from multiple AWS accounts?

Yes, on the Enterprise tier. Provide the trusted account IDs during deployment. Each source account must create a CloudWatch Logs subscription filter pointing to the cross-account destination. Additionally, external accounts can replicate their S3 access logs to your stack's access log bucket using S3 replication rules — use the provided script(s) to configure partitioned access logging, IAM roles, and cross-account replication in minutes. All logs flow into the same pipeline — searchable in OpenSearch and queryable in Athena alongside your primary account's logs.

How can I provide my system configuration for troubleshooting?

Use the provided support-bundle script contained in the downloadable scripts.zip file. It collects CloudFormation state, Lambda logs and configuration, SQS queue depths, OpenSearch health, Firehose delivery status, CloudWatch alarms, S3 bucket status, VPC endpoints, and the current subscriptions configuration. The output is a timestamped folder — zip it and email it. No sensitive log data is collected, only resource metadata and operational metrics.

Can I use my company's SSO (Okta, Azure AD, etc.)?

Yes. Add your SAML or OIDC identity provider to the Cognito User Pool using the sso helper script. Users will see a "Sign in with [Provider]" button on the login page. After first login, assign groups with the groups script for role-based access. See the helper scripts README for details.

Is this FedRAMP/HIPAA compatible?

The default deployment uses isolated VPC subnets with no internet egress, KMS encryption at rest for all S3 buckets, SQS queues, and the datalake, S3 access logging for request auditing, and TLS-enforced secure transport on all bucket policies. All AWS service communication routes through VPC endpoints — no data traverses the public internet. This architecture meets network isolation requirements for FedRAMP, HIPAA, PCI-DSS, and SOC 2 compliance frameworks out of the box, with no additional configuration required.

How does pattern detection work?

50+ context-aware regex patterns scan log messages inline during processing. Built-in patterns cover identity (SSN, passport, driver's license, DOB), financial (credit card with Visa/Mastercard validation, IBAN), contact (email, phone, IP address), credentials (AWS access keys, secret keys), and SQL (SELECT, INSERT, DROP, injection attempts). Each pattern is categorized by type (PHI, PII, Financial, Secret, SQL) and severity (High, Medium, Low). Configure per stream to redact (mask sensitive values), filter (drop matching events entirely), or tag (annotate for downstream alerting). Add your own custom patterns with category and severity in the subscription editor — no code changes or redeployment needed. Detected patterns are queryable in both OpenSearch and Athena.

Why is regex often preferred over advanced techniques for log pattern detection?

Regex provides speed, precision, and immediate deployment without the need for resource-heavy infrastructure. When you understand your system's log schemas, regex lets you leverage that knowledge for instant, pinpoint accuracy without training complex models. It allows you to hardcode known patterns, filter out predictable noise, and trace with precision. Because you already know what to look for, regex eliminates guesswork and provides immediate, reliable alerting tailored exactly to your environment.

How comprehensive is the pattern detection?

The built-in scanner provides regex-based detection at zero additional cost — no external API calls, no per-scan fees, and no data leaves your VPC. All detections are indexed as structured fields queryable in both OpenSearch and Athena, enabling you to search for specific pattern types across your entire log history. Advanced and Enterprise tiers emit per-pattern CloudWatch metrics with category and severity dimensions for real-time alerting and dashboarding.

What are Enhanced Pattern Metrics?

On Advanced or above tiers, each pattern detection is emitted as a CloudWatch metric with PatternName, Category, and Severity dimensions. This enables multi-dimensional querying — graph individual patterns over time (e.g. "SSN detections per hour" vs "email detections per hour"), aggregate by severity ("all High severity detections in the last 24 hours"), or slice by category ("all PHI detections across all log groups"). Build targeted CloudWatch dashboards, set threshold alarms on specific patterns, and correlate detection spikes with deployment events. Aggregate pattern metrics (total detections and filtered counts) are available on all tiers at no additional cost.

Can I detect anomalies?

Yes. On Advanced or above tiers, create a CloudWatch alarm on any pattern metric dimension — for example, "alert when SSN detections exceed 10 in 5 minutes" or "alert on any High severity detection." Route alarms to the stack's SNS topic for email notifications. Navigate to CloudWatch → Alarms → Create Alarm, select your stack's metric namespace, filter by PatternName, Category, or Severity dimension, and configure your threshold. Integration with the companion AI Monitor product is seamless, eliminating the need for alarms. Custom integration is available as a professional services engagement.

How do ingest pipelines work?

Raw log messages are unstructured text — searching them is like reading a book without an index. Ingest pipelines automatically extract structured fields (IP addresses, status codes, request IDs, latencies, error levels) at index time, turning every log line into a queryable document. Filter by HTTP 500s, sort by response time, aggregate errors by service — without writing a single parser.

Each subscription stream can specify a pipeline name (e.g. "lambda", "nginx", "json"). Parsing runs server-side on the OpenSearch cluster during indexing — no Lambda processing overhead, no additional cost, no code to maintain. The raw message is always preserved alongside extracted fields, so you never lose data. If parsing fails (malformed input, unexpected format), the event is indexed unchanged — no data loss.

14 built-in pipelines cover the most common AWS and application log formats: Lambda, JSON, Nginx, Apache, Syslog, Tomcat, Spring Boot, VPC Flow Logs, ALB, API Gateway, RDS slow queries, EKS/Kubernetes, CloudFront, and RDS PostgreSQL. Create custom pipelines for proprietary formats via the OpenSearch Dev Tools console — they're immediately available to any subscription stream. Pipelines only affect OpenSearch; datalake writes remain raw for Athena query-time parsing.

What if I want to be notified another way?

The stack's SNS topic supports any endpoint type — add your own subscriptions via the AWS Console. For example, add a Slack incoming webhook URL (HTTPS), a PagerDuty integration endpoint, an SQS queue for automation, or a Lambda function for custom routing. Navigate to SNS → Topics → select your stack's notification topic → Create subscription. No changes to the stack or parameters needed.

What are Automated Compliance Reports?

On Advanced and above tiers, the log processor can generate and email an HTML compliance report on a configurable schedule. Reports summarize the past 7 days and include:

• Event Volume — S3 objects processed, log events ingested, processing errors, and active subscription count.
• Pattern Detection Summary — total patterns detected and events filtered across all log groups.
• Detections by Pattern Type — per-pattern breakdown (e.g. SSN: 142 detected, email: 87 detected) from CloudWatch dimensioned metrics.
• Top Log Groups by Volume — the 10 busiest log groups by event count (from OpenSearch).
• Pattern Detections by Log Group — the 10 log groups with the most pattern hits (from OpenSearch).
• System Health — system health diagnostics with traffic-light status indicators across your entire log pipeline.

Reports are saved to the datalake S3 bucket under reports/ and a pre-signed download link is emailed via the alarm SNS topic. Configure which days to auto-generate (e.g. Monday and Friday) in the Subscription Editor, or generate on demand with the Report button.

What does the CloudWatch monitoring dashboard include?

A pre-built dashboard is deployed automatically with your stack. It includes:

• Alarm Status — all alarm states at a glance (OK, ALARM, INSUFFICIENT_DATA).
• Lambda — invocations, errors, throttles, concurrency, duration, and memory.
• SQS Queues — S3 event queue, chunk queue, dead letter queues, and oldest message age.
• Firehose — delivery freshness, incoming records/bytes, and delivery success rate.
• Log Processor Metrics — events processed, errors, retries, pattern detections, and active subscriptions.
• OpenSearch — cluster status, free storage, JVM pressure, CPU, indexing/search rate, and per-index breakdowns.
• Nginx Proxy — CPU, memory, disk usage for the Dashboards reverse proxy.
• Lambda Logs — recent errors and warnings from CloudWatch Logs Insights queries.

Note: The Estimated Costs widgets require Receive Billing Alerts to be enabled in the AWS Billing console (Settings → Billing Preferences). Billing metrics update approximately every 6 hours.

What is the function of the custom OpenSearch-provided snapshot bucket?

The AWS-managed automated snapshots (hourly, 14-day retention, free) are sufficient for disaster recovery. The custom S3 snapshot repo is mainly useful for long-term archival beyond 14 days, cross-region restore (copy the S3 bucket), or pre-upgrade backup (snapshot before a major change). Use the provided .cmd/.sh script, e.g. snapshot.cmd, to take a snapshot on demand.

Why only two OpenSearch indexes (app and audit)?

Two indexes keep the cluster simple and cost-effective. app holds application logs with short retention (default 30 days), while audit holds compliance-sensitive logs with long retention (default 365 days). Each subscription stream routes to one of these indexes, so you control retention and access per log type without managing dozens of indexes. Fewer indexes mean a lower shard count, less JVM pressure, and faster cluster recovery. For further separation, attach custom metadata fields to every log event for source filtering, and leverage OpenSearch pipelines for automatic field processing.

How does retention work across the pipeline?

Retention is managed independently at each layer of the pipeline, reducing the complexity of managing storage costs and compliance requirements:

• CloudWatch Logs — set via retentionDays in your subscription file. Without this, CloudWatch retains logs forever at $0.03/GB/month. Enforced automatically on each subscription run.
• S3 Log Bucket — Firehose delivery bucket, controlled by S3 lifecycle rules. Old objects are automatically deleted.
• OpenSearch — managed by ISM (Index State Management) policies per index type. For example, app indexes delete after 30 days, audit after 365 days. Configurable in your deployment profile.
• Datalake (Athena) — S3 lifecycle rules per index prefix. Typically longer retention than OpenSearch (e.g. 1–7 years) since S3 storage is cheaper. Queryable via Athena at $5/TB scanned.
• OpenSearch Snapshots — stored in a separate S3 bucket with its own lifecycle rules. Useful for disaster recovery. Bucket can be retained even after stack deletion.
• S3 Access Logs — access logs for the log bucket and datalake bucket. Retained per tier lifecycle rule. Queryable via Athena for auditing who accessed your data.

All retention periods are specified in your deployment tier (basic, essential, advanced, enterprise).