Guides & Downloads¶
Reference guides and downloads for Log Processor. These are versioned — the links below point at the current release (26h4c) published on perfware.cloud.
Guides¶
- Getting Started Guide — prerequisites, deployment, post-setup, and first subscription walkthrough.
- Subscription Editor Guide (PDF) — editor UI reference: fields, preview, discovery, maintenance, reports.
- OpenSearch Guide (PDF) — OpenSearch dashboards and queries.
- User Management Guide (PDF) — creating users and groups.
- Single Sign-On Integration Guide (PDF) — SSO (SAML/OIDC) setup.
- Ingest Pipeline Guide (PDF) — automatically extracting structured fields.
- Athena Queries Guide (PDF) — querying the datalake with Athena.
- CloudWatch Dashboard Guide (PDF) — the pre-built monitoring dashboard.
Downloads¶
- Helper scripts (logprocessor-scripts.zip) — includes the
support-bundle,sso,groups, andsnapshothelper scripts.
LogProcessor Helper Scripts¶
26h4c 2026-11-10
Helper scripts for operating a deployed Log Processor stack — user and group management, SSO, domain setup, snapshots, diagnostics, cross-account forwarding, S3 access logging, and cleanup.
Prerequisites¶
- Windows 10/11 or Windows Server 2016+ (for
.cmdscripts) - Linux, macOS, or Git Bash on Windows (for
.shscripts) - AWS CLI v2.13+ (latest recommended)
- AWS credentials configured (
aws configure, SSO, or environment variables) - PowerShell 5.1+ (included with Windows 10/11)
- Python 3.10+ (for
check-quota.pyandcleanup.py)
Run scripts from any directory with AWS CLI access to the target account and
region. Region defaults to us-east-1 if not specified, except
cleanup.py, which requires region explicitly. Stack name is case-sensitive and
must match the CloudFormation stack name.
Most .cmd scripts have an equivalent .sh script with identical
functionality. Exceptions: check-quota / cleanup are Python tools.
Contents¶
- User Management —
users - Group & Role Management —
groups - SSO / Identity Federation —
sso - Domain & Certificate Setup —
setup-domain - OpenSearch Snapshots —
snapshot - Diagnostics & Support —
support-bundle,check-quota - Cross-Account Log Forwarding —
get-destination,setup-cross-account,setup-remote-access,add-account - S3 Access Logging —
add-access-logging,setup-replication - Cleanup —
cleanup.py - Defaults & Notes
- Support
User Management¶
users.cmd / users.sh¶
Manage Cognito users for OpenSearch Dashboards and Subscription Editor access. Each user authenticates via the Cognito-backed ALB and can access Dashboards, the Subscription Editor, and any SSO-protected resources.
users.cmd <stack-name> <action> [args] [region]
| Action | Description |
|---|---|
create <email> [temp-password] |
Create a user (forced password change on first login) |
list |
List all users with username, email, status, and creation date |
delete <email> |
Permanently remove a user |
reset <email> [temp-password] |
Reset password (user sets new one on next login) |
disable <email> |
Block a user from logging in (preserves account) |
enable <email> |
Re-enable a previously disabled user |
Examples
users.cmd LogProcessor create admin@company.com
users.cmd LogProcessor create admin@company.com "MyTemp!Pass99" us-west-2
users.cmd LogProcessor list
users.cmd LogProcessor disable former-employee@company.com
users.cmd LogProcessor reset admin@company.com
Notes
- Default temporary password is auto-generated if not specified.
- Federated (SSO) users appear with their provider prefix in Username.
- Email column may show
Nonefor SSO users (email is in Username).
Requires: AWS CLI, Cognito UserPool deployed (Essential+ tier).
Group & Role Management¶
groups.cmd / groups.sh¶
Manage Cognito user groups for role-based access control (RBAC) in the Subscription Editor. Groups determine what each user can do.
groups.cmd <stack-name> <action> [args] [region]
| Action | Description |
|---|---|
list-groups |
List all groups in the user pool |
create-group <name> [description] |
Create a new group |
delete-group <name> |
Remove a group |
add-to-group <email> <group> |
Assign a user to a group |
remove-from-group <email> <group> |
Remove a user from a group |
list-users [group] |
List all users, or users in a specific group |
Examples
groups.cmd LogProcessor create-group admin "Full editor access"
groups.cmd LogProcessor create-group viewer "Read-only access"
groups.cmd LogProcessor add-to-group admin@company.com admin
groups.cmd LogProcessor add-to-group analyst@company.com viewer
groups.cmd LogProcessor list-users admin
groups.cmd LogProcessor list-groups
Editor RBAC (Advanced+ tier with custos feature)
| Membership | Access |
|---|---|
admin group |
Full access: Save, Sync, Rollback, Import |
viewer group |
Read-only: can view but cannot save (server returns 403) |
| no group | Blocked: Access Denied page |
| no groups configured (custos inactive) | Everyone has full access |
Notes
- Users must log out and back in for group changes to take effect.
- ALB sessions last 1 hour (fixed window, not sliding).
- SSO users must also be assigned to groups after first login.
Requires: AWS CLI, Cognito UserPool deployed (Essential+ tier).
SSO / Identity Federation¶
sso.cmd / sso.sh¶
Manage external SSO (SAML/OIDC) identity providers on the Cognito User Pool. Enables federated login from corporate identity providers (Okta, Azure AD, Google Workspace, ADFS, Ping, etc.) to Dashboards and the Subscription Editor.
sso.cmd <stack-name> <action> [args]
| Action | Description |
|---|---|
setup-saml <name> <metadata-url> |
Add a SAML provider |
setup-oidc <name> <issuer> <client-id> <client-secret> |
Add an OIDC provider |
list |
List configured providers |
remove <name> |
Remove a provider |
info |
Show URLs for IdP configuration |
Examples
sso.cmd LogProcessor info
sso.cmd LogProcessor setup-saml Okta https://dev-123.okta.com/app/xxx/sso/saml/metadata
sso.cmd LogProcessor setup-oidc AzureAD https://login.microsoftonline.com/tenant/v2.0 client-id client-secret
sso.cmd LogProcessor list
sso.cmd LogProcessor remove Okta
Workflow
- Run
infoto get the ACS URL and Entity ID. - Configure your IdP with those values.
- Run
setup-samlorsetup-oidcwith the IdP metadata/config. - Users see "Sign in with
<name>" on the login page. - After first SSO login, assign groups:
groups.cmd <stack> add-to-group <email> admin.
Notes
- Multiple providers can coexist (e.g. Okta + Azure AD).
- Username/password login remains available alongside SSO.
- Federated users appear as
<Provider>_<email>in user listings. - Group assignment is required for editor RBAC (custos feature).
Requires: AWS CLI, Cognito UserPool deployed (Essential+ tier).
Domain & Certificate Setup¶
setup-domain.cmd / setup-domain.sh¶
Configure a custom domain (e.g. dashboards.company.com) for the OpenSearch
Dashboards ALB. Handles ACM certificate creation, DNS validation, and Route 53
alias record creation.
setup-domain.cmd cert-only <domain> <hosted-zone-id> [region]
setup-domain.cmd <stack-name> <domain> <hosted-zone-id> [region]
| Mode | Description |
|---|---|
cert-only |
Run before deployment to request and validate an ACM certificate. Add the certificate ARN to your config before deploying. |
<stack> |
Run after deployment to create the Route 53 CNAME/alias to the ALB. |
Examples
setup-domain.cmd cert-only dashboards.company.com Z0123456789ABC
setup-domain.cmd LogProcessor dashboards.company.com Z0123456789ABC
Workflow
- Run
cert-onlyto get the certificate ARN. - Add
certificateArnanddomainto your config. - Deploy the stack.
- Run with stack name to create the DNS record.
- Access dashboards at
https://dashboards.company.com.
Notes
- Certificate validation uses DNS (Route 53 auto-validation).
- Certificate must be in the same region as the ALB.
- Propagation takes a few minutes after DNS record creation.
Requires: AWS CLI, Route 53 hosted zone for the domain.
OpenSearch Snapshots¶
snapshot.cmd / snapshot.sh¶
Manage OpenSearch manual snapshots for backup and recovery. Snapshots are stored in the S3 snapshot bucket configured during deployment.
snapshot.cmd <stack-name> <action> [name] [region]
| Action | Description |
|---|---|
take <name> |
Create a manual snapshot of all indices |
list |
List all available snapshots |
status <name> |
Check progress/status of a snapshot |
delete <name> |
Remove a snapshot from the repository |
Examples
snapshot.cmd LogProcessor take pre-upgrade-backup
snapshot.cmd LogProcessor list
snapshot.cmd LogProcessor status pre-upgrade-backup
snapshot.cmd LogProcessor delete old-snapshot
Notes
- Snapshots are incremental (only changed data is stored).
- The snapshot repository is automatically registered during deployment.
- Restore via OpenSearch Dev Tools:
POST /_snapshot/repo/name/_restore. - Lifecycle policies auto-expire old snapshots per config.
Requires: AWS CLI, OpenSearch snapshot repository configured (Essential+ tier).
Diagnostics & Support¶
support-bundle.cmd / support-bundle.sh¶
Collect a comprehensive diagnostic bundle for troubleshooting. Gathers configuration, logs, metrics, and health status from all stack components.
support-bundle.cmd <stack-name> [hours] [region]
| Argument | Description |
|---|---|
stack-name |
CloudFormation stack name |
hours |
Hours of log/metric history to collect (default: 24) |
region |
AWS region (default: us-east-1) |
Collects
- CloudFormation stack status and parameters
- Lambda configuration, recent logs, errors, and environment variables
- SQS queue depths and dead letter queue messages
- OpenSearch domain health, cluster status, and index stats
- CloudWatch metrics and alarm states
- SNS topic subscriptions
- S3 bucket status and object counts
- Firehose delivery stream configuration
- Editor Lambda logs
- ALB health and target group status
- Nginx access/error logs
- Current
subscriptions.json - Service quotas and limits
- Cognito configuration
- Tier and build information
Output: support-bundle/<stack-name>-<timestamp>/ folder with JSON/text
files. Zip the folder and email to support@perfware.cloud.
Examples
support-bundle.cmd LogProcessor
support-bundle.cmd LogProcessor 48 us-west-2
Requires: AWS CLI, read access to stack resources.
check-quota.cmd / check-quota.sh / check-quota.py¶
Pre-deployment check of AWS service quotas against Log Processor tier requirements. Identifies potential deployment blockers before you start.
check-quota.cmd <tier> [region]
check-quota.sh <tier> [region]
| Argument | Description |
|---|---|
tier |
Target tier: basic, essential, advanced, enterprise |
region |
AWS region (default: us-east-1) |
Checks
- VPC count and available headroom
- Lambda concurrent executions
- S3 bucket count
- KMS key count
- DynamoDB table count
- Firehose delivery streams
- OpenSearch domain count and instance limits
- ALB count
- Cognito user pool count
- SNS topic count
- VPC interface endpoint count
Output: PASS/WARN/FAIL per quota with current usage, limit, and headroom. Warns at 80% utilization. Fails if deployment would exceed limits.
Examples
check-quota.cmd enterprise
check-quota.cmd basic us-west-2
Requires: Python 3.10+, AWS CLI, service-quotas and resource read permissions.
Cross-Account Log Forwarding¶
get-destination.cmd / get-destination.sh¶
Alternative to using the Editor.
Display the CloudWatch Logs destination ARN for a Log Processor stack. This ARN is needed by external accounts to set up log forwarding.
get-destination.cmd <stack-name> [region]
Example
get-destination.cmd LogProcessor
Output
arn:aws:logs:us-east-1:000000000000:destination:LogProcessor-destination
Notes
- Share this ARN with external account administrators.
- The destination accepts logs from accounts listed in
CrossAccountIdsor any account in the organization specified byOrganizationId.
Requires: AWS CLI, logs:DescribeDestinations permission.
setup-cross-account.cmd / setup-cross-account.sh¶
Alternative to using the Editor.
Manage cross-account log subscriptions. Run in the external (source) account to forward CloudWatch logs to a central Log Processor destination.
setup-cross-account.cmd <action> [args]
| Action | Description |
|---|---|
setup <log-group> <destination-arn> [filter] [region] |
Create subscription filter |
list <stack-name> <log-group> [region] |
List filters for a stack on a log group |
remove <stack-name> <log-group> [region] |
Remove a subscription filter |
Examples
setup-cross-account.cmd setup /aws/lambda/my-fn arn:aws:logs:us-east-1:000000000000:destination:log-dest-LogProcessor-000000000000-us-east-1
setup-cross-account.cmd setup /aws/lambda/my-fn arn:aws:logs:... "ERROR" us-east-1
setup-cross-account.cmd list LogProcessor /aws/lambda/my-fn
setup-cross-account.cmd remove LogProcessor /aws/lambda/my-fn
Workflow
- In the Log Processor account: deploy with
crossAccountIdsincluding this account. - In the Log Processor account: run
get-destination.cmdto get the ARN. - In this (external) account: run
setup-cross-account.cmd setup .... - In the Log Processor account: add a subscription entry with
accountIdmatching.
Prerequisites (in the Log Processor account)
- Enterprise tier deployed with
crossAccountEnabled: true. - This account's ID in the
CrossAccountIdsparameter, or theOrganizationIdparameter set to the shared AWS Organization ID.
Notes
- Filter pattern is optional (empty = all log events).
- Each log group can have max 2 subscription filters (CloudWatch limit).
listandremoverequire the stack name to identify the correct filter (supports multiple Log Processor stacks with different names).- Use
listto verify filters are active.
Requires: AWS CLI, logs:PutSubscriptionFilter,
logs:DeleteSubscriptionFilter, logs:DescribeSubscriptionFilters.
setup-remote-access.cmd / setup-remote-access.sh¶
Set up IAM roles in a remote (source) account to allow the central Log Processor to browse log groups, manage subscription filters, and receive logs via Firehose.
setup-remote-access.cmd setup <central-account-id> [region] [--org <org-id>]
setup-remote-access.cmd remove [region]
setup-remote-access.cmd status [region]
| Action | Description |
|---|---|
setup <central-account-id> [region] [--org <org-id>] |
Create IAM roles trusting the central account |
remove [region] |
Delete both IAM roles |
status [region] |
Show current role configuration |
Examples
setup-remote-access.cmd setup 000000000000
setup-remote-access.cmd setup 000000000000 us-east-1 --org o-abc123
setup-remote-access.cmd status
setup-remote-access.cmd remove
Roles created
| Role | Purpose |
|---|---|
LogProcessor-remote-access |
Assumed by central Lambda to manage subscription filters |
LogProcessor-logs-delivery |
Assumed by CloudWatch Logs to deliver to central Firehose |
Workflow
- In the central account: deploy Enterprise tier with
crossAccountEnabled. - In this (remote) account: run
setup-remote-access.cmd setup <central-account-id>. - The central Log Processor can now browse and manage filters in this account.
- No further configuration needed; the editor auto-discovers this account.
Options (setup only)
--org <org-id>— also trust any account in the specified AWS Organization.
Notes
- Run this in the remote account (not the central Log Processor account).
- The
setupcommand is idempotent (safe to re-run to update trust policies). - Use
statusto verify current role configuration. - Use
removeto cleanly delete both roles and their policies.
Requires: AWS CLI, IAM permissions (iam:CreateRole, iam:PutRolePolicy,
iam:DeleteRole, iam:DeleteRolePolicy), sts:GetCallerIdentity.
add-account.cmd / add-account.sh¶
Alternative to using the Editor.
Register or remove remote account IDs in the central Log Processor so they appear in the Subscription Editor's account dropdown immediately, without waiting for log events to arrive.
add-account.cmd <stack-name> <account-id> [region]
add-account.cmd <stack-name> list [region]
add-account.cmd <stack-name> remove <account-id> [region]
| Action | Description |
|---|---|
<account-id> [region] |
Add a 12-digit account ID |
list [region] |
Show currently registered accounts |
remove <account-id> [region] |
Remove an account from the list |
Examples
add-account.cmd LogProcessor 111222333444
add-account.cmd LogProcessor 111222333444 us-west-2
add-account.cmd LogProcessor list
add-account.cmd LogProcessor remove 111222333444
Workflow
- In the remote account: run
setup-remote-accessto create IAM roles. - In the central account: run
add-accountto register the account ID. - The account immediately appears in the Subscription Editor dropdown.
Notes
- Run this in the central account (where Log Processor is deployed).
- Account IDs must be exactly 12 digits.
- Adding an already-registered account is a no-op (exits cleanly).
- Accounts are stored in
config/accounts.jsonin the assets bucket.
Requires: AWS CLI, SSM read access, S3 read/write access to assets bucket, Python 3.10+.
S3 Access Logging¶
add-access-logging.cmd / add-access-logging.sh¶
Enable S3 server access logging on any bucket, routing logs to the stack's access log bucket. Logs are automatically partitioned and queryable via Athena.
add-access-logging.cmd <stack-name> <action> [args] [region]
| Action | Description |
|---|---|
list |
Show buckets that have delivered logs |
add <source-bucket> |
Enable access logging on a bucket |
remove <source-bucket> |
Disable access logging (existing logs retained) |
Examples
add-access-logging.cmd LogProcessor add my-data-bucket
add-access-logging.cmd LogProcessor add my-other-bucket
add-access-logging.cmd LogProcessor list
add-access-logging.cmd LogProcessor remove my-data-bucket
Query logs in Athena
SELECT * FROM access_logs
WHERE bucket = 'my-data-bucket' AND date = '2026/05/03'
Notes
- Logs may take up to 1 hour to appear after enabling.
- Works with any source bucket encryption (SSE-S3, SSE-KMS, CMK).
- No additional IAM permissions needed for same-account buckets.
- Logs use date-partitioned prefix format for efficient Athena queries.
Requires: AWS CLI, S3 permissions on both source and access log buckets.
setup-replication.cmd / setup-replication.sh¶
Cross-account S3 access log replication. Run in the external account to replicate access logs from your buckets to a central Log Processor stack's access log bucket for unified Athena querying.
setup-replication.cmd <action> [args]
| Action | Description |
|---|---|
setup <target-bucket> [region] [options] |
One-time setup: create log bucket, role, rule |
add <source-bucket> [region] |
Enable logging on a source bucket |
verify <source-bucket> [region] |
Check logging and replication status |
remove [region] |
Remove replication config |
Options (setup only)
--expire-days N— delete objects from local log bucket after N days.--noncurrent-days N— delete noncurrent versions after N days.
Examples
setup-replication.cmd setup logprocessor-access-logs-000000000000-us-east-1
setup-replication.cmd add my-production-bucket
setup-replication.cmd add my-staging-bucket
setup-replication.cmd verify my-production-bucket
setup-replication.cmd remove
Workflow
- In the Log Processor account: deploy with
crossAccountIdsincluding this account, or setOrganizationIdto allow all accounts in the organization. - In this account: run
setup(creates local log bucket + replication to target). - In this account: run
addfor each bucket you want to monitor. - Logs replicate to the central bucket and are queryable via Athena.
Query in Athena (Log Processor account)
SELECT * FROM access_logs
WHERE bucket = 'my-production-bucket' AND date = '2026/05/04'
Prerequisites (in the Log Processor account)
- Enterprise tier deployed.
- This account's ID in the
CrossAccountIdsparameter, or theOrganizationIdparameter set to the shared AWS Organization ID.
Requires: AWS CLI, IAM permissions (iam:CreateRole, iam:PutRolePolicy),
S3 permissions on source and local log buckets.
Cleanup¶
cleanup.py¶
Clean up orphaned resources after stack deletion. CloudFormation may leave behind resources with RETAIN policies or resources created at runtime.
python cleanup.py <stack-name> <region> [-force|-confirm]
| Argument | Description |
|---|---|
stack-name |
CloudFormation stack name (case-sensitive) |
region |
AWS region (required) |
| Mode | Behavior |
|---|---|
| (default) | Dry run — shows what would be deleted without acting |
-confirm |
Interactive — prompts for each resource before deleting |
-force |
Deletes all matched resources without prompting |
Cleans up
- S3 buckets (empties and deletes, including versioned objects)
- Athena workgroup (
<stack-name>-workgroup) - CloudWatch log groups (Lambda, custom resources)
- CloudWatch Logs resource policies
- Lambda VPC ENIs (force-detach from VPC)
- OpenSearch domains
- Orphan Lambda functions
- KMS keys (schedules deletion, 7-day window)
- Orphan VPCs and their resources (endpoints, subnets, SGs, route tables)
Examples
python cleanup.py LogProcessor us-east-1 # dry run
python cleanup.py LogProcessor us-east-1 -confirm # interactive
python cleanup.py LogProcessor us-east-1 -force # delete all
Notes
- Always run dry mode first to review what will be deleted.
- Uses safe matching: regex word-boundary for S3, exact prefix for Lambdas, exact tag match for KMS (prevents accidental cross-stack deletion).
- Prints a Resource Explorer verification hint at the end.
Warning: With
-force, your S3 data (logs, datalake, snapshots) is permanently deleted. To preserve data, copy it first withaws s3 sync.
Requires: Python 3.10+, boto3, appropriate IAM permissions for each resource type.
Defaults & Notes¶
- Region defaults to
us-east-1if not specified, exceptcleanup.py, which requires region as an explicit argument. - Stack name is case-sensitive and must match the CloudFormation stack name.
- All scripts exit with code 0 on success, non-zero on error.
.shscripts requirechmod +x <script>.shbefore first use.- Scripts auto-detect resources via CloudFormation stack outputs and tags.