Skip to content

Guides & Downloads

Reference guides and downloads for AI Monitor. These are versioned — the links below point at the current release (26h4c) published on perfware.cloud.

Guides

Downloads


AI Monitor — Operational Scripts

26h4c 2026-11-10

Each script ships in two forms that take the same arguments and do the same thing:

Form Platform
.cmd Windows (cmd.exe)
.sh macOS / Linux (bash)

Both forms are referenced throughout: each Usage block lists the .cmd and .sh invocation. Examples are shown in the .cmd form; on macOS/Linux use the matching ./<name>.sh ... with the same arguments.

Prerequisites

  • Windows 10/11 / Windows Server 2016+, or macOS / Linux with bash
  • AWS CLI v2.13+ (latest recommended)
  • AWS credentials configured (aws configure, SSO, or environment variables)
  • PowerShell 5.1+ (included with Windows 10/11; used by support-bundle.cmd)
  • Python 3.10+ (used by check-quota and cleanup)

Script index

Script Purpose
users Manage Cognito editor users
groups Manage Cognito groups for RBAC
sso Manage SAML/OIDC identity providers
setup-domain Custom domain + ACM cert for the editor
support-bundle Collect diagnostics for support
check-quota Validate AWS quotas before deploy
setup-cross-account-metrics Remote-account metric delivery
cleanup.py Tear down leftover stack resources

users

Manage Cognito users for the AI Monitor editor.

users.cmd <stack-name> <action> [email] [temp-password] [region]
./users.sh <stack-name> <action> [email] [temp-password] [region]
Action Description
create <email> [temp-password] Create a new user (prompted to reset on first login)
list List all users
delete <email> Delete a user
reset <email> [temp-password] Reset password (temporary, must change on login)
disable <email> Disable a user
enable <email> Re-enable a user

Notes

  • region defaults to us-east-1 and is auto-detected if passed in the password slot.
  • create defaults a temporary password if none is given.

Examples

users.cmd AIMonitor create admin@example.com "Temp*88=888Pass123!@#"
users.cmd AIMonitor list
users.cmd AIMonitor disable user@example.com

groups

Manage Cognito groups for RBAC. Roles apply on the advanced tier and above. On lower tiers RBAC is off: every authenticated user has full (admin) access and these groups are ignored.

When RBAC is on, users in the admin group have full access; everyone else (the viewer group, any other group, or no group) is read-only.

groups.cmd <stack-name> <command> [args] [region]
./groups.sh <stack-name> <command> [args] [region]
Command Description
list-groups List all groups
create-group <name> [description] Create a new group
delete-group <name> Delete a group
add-to-group <user> <group> Add a user to a group
remove-from-group <user> <group> Remove a user from a group
list-users [group] List users (optionally in a specific group)

Examples

groups.cmd AIMonitor create-group admin "Full access editors"
groups.cmd AIMonitor create-group viewer "Read-only access"
groups.cmd AIMonitor add-to-group admin@example.com admin
groups.cmd AIMonitor list-users admin

sso

Manage external SSO (SAML/OIDC) identity providers on the Cognito User Pool for federated login to the editor. Requires the advanced or enterprise tier.

sso.cmd <stack-name> <command> [args]
./sso.sh <stack-name> <command> [args]
Command Description
info Show Cognito pool details (ACS URL, Entity ID, Redirect URI)
setup-saml <name> <metadata-url> Add a SAML provider (Okta, Azure AD, ADFS)
setup-oidc <name> <issuer> <client-id> <secret> Add an OIDC provider (Google, Auth0)
list List configured identity providers
remove <name> Remove an identity provider

Workflow

  1. Run sso.cmd <stack> info to get your ACS URL and Entity ID.
  2. Configure your IdP (Okta, Azure AD, etc.) with those values.
  3. Run sso.cmd <stack> setup-saml <name> <metadata-url> to register it.
  4. Users see a "Sign in with <name>" button on the Cognito login page.
  5. After first login, assign groups: groups.cmd <stack> add-to-group <federated-username> admin

Notes

  • Federated usernames have the format <ProviderName>_<email> (e.g. Okta_user@domain.com).
  • Group assignments must use the federated username, not the email alone.
  • JIT provisioning is automatic (no pre-creation needed in Cognito).
  • Removing a provider immediately prevents SSO login via that provider.
  • Native Cognito users are unaffected by IdP changes.

Examples

sso.cmd AIMonitor info
sso.cmd AIMonitor setup-saml Okta https://dev-xxx.okta.com/app/yyy/sso/saml/metadata
sso.cmd AIMonitor setup-oidc Google https://accounts.google.com client-id client-secret
sso.cmd AIMonitor list
sso.cmd AIMonitor remove Okta

Requires: AWS CLI, Cognito User Pool deployed, advanced or enterprise tier. See SSOGuide.pdf for detailed IdP setup instructions.


setup-domain

Configure a custom domain (e.g. monitor.company.com) for the editor ALB. Handles ACM certificate creation, DNS validation via Route 53, and the ALB alias record.

setup-domain.cmd  cert-only <domain> <hosted-zone-id> [region]
setup-domain.cmd  <stack-name> <domain> <hosted-zone-id> [region]
./setup-domain.sh cert-only <domain> <hosted-zone-id> [region]
./setup-domain.sh <stack-name> <domain> <hosted-zone-id> [region]
Mode Description
cert-only Request and validate an ACM certificate without modifying the stack. Run before deployment if you want HTTPS on a custom domain.
<stack-name> Full setup: create the Route 53 alias record pointing your domain to the ALB. Run after deployment. Creates a certificate if none exists.

Workflow

  1. (Optional) Pre-create cert: setup-domain.cmd cert-only monitor.company.com Z0123456789
  2. Deploy the stack with the CertificateArn parameter.
  3. Create DNS alias: setup-domain.cmd AIMonitor monitor.company.com Z0123456789
  4. Access the editor at https://monitor.company.com/editor.

Notes

  • Certificate must be in us-east-1 (same region as the ALB).
  • DNS validation typically takes 1–5 minutes; the script waits for it.
  • Existing certificates for the same domain are reused.

Examples

setup-domain.cmd cert-only monitor.perfware.cloud Z04ABCDEF123
setup-domain.cmd AIMonitor monitor.perfware.cloud Z04ABCDEF123
setup-domain.cmd AIMonitor monitor.perfware.cloud Z04ABCDEF123 us-east-1

Requires: AWS CLI, Route 53 hosted zone for your domain.


support-bundle

Collects diagnostic information for troubleshooting into a timestamped bundle directory: Lambda configs, recent logs, DynamoDB state, CloudWatch metrics, alarm status, discovery cache, AI circuit breaker state, and more. No sensitive metric data is collected — only resource metadata and status.

support-bundle.cmd <stack-name> [hours]
./support-bundle.sh <stack-name> [hours]
  • hours — log/metric lookback period (default: 24).
  • Region is auto-detected from your AWS CLI configuration.

Output

A <stack-name>-<timestamp>/ directory of JSON/text files. Run analyze-bundle.py on it for a summary report, then zip and send the folder to support.

Examples

support-bundle.cmd AIMonitor
support-bundle.cmd AIMonitor 48
python analyze-bundle.py AIMonitor-20250115-143022/

check-quota

Validates AWS service quotas and infrastructure readiness for a given tier before deployment. Checks VPC endpoints, Lambda concurrency, DynamoDB capacity, S3, SQS, Firehose, CloudWatch Metric Streams, OpenSearch, and Bedrock availability.

check-quota.cmd <tier> [region]
./check-quota.sh <tier> [region]
  • tier — basic, essential, advanced, or enterprise.
  • region — AWS region (default: us-east-1).

Bedrock checks

  • Verifies Bedrock Runtime is available in the region.
  • Confirms the Anthropic Claude model is listed.
  • Tests InvokeModel (catches the use-case form requirement).
  • Reports the daily token quota and warns if insufficient.

Examples

check-quota.cmd advanced
check-quota.cmd enterprise us-east-1

setup-cross-account-metrics

Run in the remote account to set up cross-account metric delivery into the central AI Monitor stack's S3 bucket.

Remote:  Metric Stream -> Firehose (+ transform Lambda) -> Central S3
Central: S3 event -> SQS -> Collector Lambda -> OpenSearch

CloudWatch Metric Streams require Firehose in the same account; Firehose supports cross-account S3 destinations natively.

Prerequisites

  • Central AI Monitor stack deployed with CrossAccountIds including the remote account, or OrganizationId set to allow any account in the AWS Organization.
  • AWS CLI configured for the remote account.
setup-cross-account-metrics.cmd  <central-account-id> <central-stack-name> <central-region> [options]
./setup-cross-account-metrics.sh <central-account-id> <central-stack-name> <central-region> [options]
Option Description
-filter=all Stream ALL metrics (default)
-filter=discover Auto-discover active namespaces in this account
-filter=file://path.txt Read namespaces from a file (one per line)
--org=<org-id> AWS Organizations mode (no explicit account IDs needed)
--status Verify existing setup and show resource health
--remove Remove all cross-account resources for this stack

Examples

setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 -filter=discover
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 -filter=file://namespaces.txt
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 --org=o-abc123xyz
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 --status
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 --remove

The discover option calls aws cloudwatch list-metrics to find all namespaces with active metrics in the remote account, configures the Metric Stream to include only those namespaces, and saves the list to namespaces.txt for reuse.

To stream billing data from a payer/management account (where AWS/Billing EstimatedCharges lives), run this script in that account with the bundled billing-ns.txt filter:

setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 -filter=file://billing-ns.txt

billing-ns.txt contains the single AWS/Billing namespace. Billing metrics only exist in us-east-1, so run with that central region. Allow ~2 days of history before cost monitoring has enough datapoints to baseline.

The --org option is informational — it adjusts the completion message to note that the central stack uses OrganizationId for S3 bucket-policy access rather than explicit CrossAccountIds. The resources created are the same.

The --status option checks an existing setup: Metric Stream state and last update, Firehose status, transform Lambda, IAM roles (firehose, stream, transform, metrics-reader), and an S3 bucket connectivity test.

The --remove option tears down everything the script created: the Metric Stream, Firehose delivery stream, transform Lambda, and the four IAM roles.

Example namespaces.txt:

AWS/Lambda
AWS/EC2
AWS/RDS
AWS/DynamoDB

Creates in the remote account:

  1. Transform Lambda (splits Metric Stream concatenated JSON)
  2. Firehose delivery stream with Lambda processor + cross-account S3 destination
  3. CloudWatch Metric Stream → local Firehose
  4. Metrics Reader role (lets the central account list metrics in this account)

cleanup.py

Tears down an AI Monitor stack's resources that CloudFormation can't remove on its own — S3 buckets (including versioned objects), DynamoDB tables, CloudWatch log groups, Metric Streams, Firehose delivery streams, Lambda VPC ENIs, Cognito user pools, and orphan resources. Run this before deleting the stack.

Runs in dry-run mode by default (lists what would be deleted; changes nothing).

python cleanup.py <stack-name> <region> [-force|-confirm]
Flag Behavior
(none) Dry run — list what would be deleted
-confirm Prompt before each deletion
-force Delete everything without prompting

Examples

python cleanup.py AIMonitor us-east-1
python cleanup.py AIMonitor us-east-1 -confirm
python cleanup.py AIMonitor us-east-1 -force

See GettingStarted.pdf (Teardown) for the full teardown procedure and ordering.


AWS Organizations support (advanced/enterprise)

Instead of listing individual account IDs in CrossAccountIds, set the OrganizationId parameter (e.g. o-abc123xyz) on the central stack. Any account in the organization can then deliver metrics without updating the stack each time a new account is added.

Both methods can coexist:

  • OrganizationId — any org member account gets automatic access.
  • CrossAccountIds — explicit access for accounts outside the org.

When using OrganizationId, remote accounts still run setup-cross-account-metrics to create their Firehose and Metric Stream. The only difference is the central stack's S3 bucket policy uses aws:PrincipalOrgID instead of explicit account principals.

Enabling AWS/Billing metrics in remote accounts

AWS/Billing metrics (EstimatedCharges) are not enabled by default. Each account that needs billing monitoring must enable this separately:

  1. Sign in to the AWS Console as root or an IAM identity with billing permissions.
  2. Open the AWS Billing and Cost Management console.
  3. In the left nav, choose Billing preferences.
  4. Under Alert preferences, choose Edit.
  5. Select Receive CloudWatch Billing Alerts.
  6. Choose Save preferences (or Update).

Note: Billing metrics are only published in us-east-1 regardless of where your resources run. The remote-account Metric Stream must be in us-east-1, or you must create a separate us-east-1 stream for AWS/Billing if your primary stream is in another region.

After enabling, AWS/Billing EstimatedCharges metrics begin publishing within a few hours and can be streamed cross-account like any other CloudWatch metric.