Guides & Downloads¶
Reference guides and downloads for AI Monitor. These are versioned — the links below point at the current release (26h4c) published on perfware.cloud.
Guides¶
- Getting Started Guide — prerequisites, deployment, post-setup, and first subscription walkthrough.
- Subscription Editor Guide (PDF) — editor UI reference: fields, preview, discovery, maintenance, reports.
- OpenSearch Guide (PDF) — OpenSearch dashboards, queries, and troubleshooting.
- User Management Guide (PDF) — creating users and groups.
- Single Sign-On Integration Guide (PDF) — SSO (SAML/OIDC) setup.
- Monitoring Guide (PDF) — monitoring features, configuration, and operations.
Downloads¶
- Helper scripts (aimonitor-scripts.zip) — includes the subscription, SSO, user-management, and diagnostics helper scripts.
AI Monitor — Operational Scripts¶
26h4c 2026-11-10
Each script ships in two forms that take the same arguments and do the same thing:
| Form | Platform |
|---|---|
.cmd |
Windows (cmd.exe) |
.sh |
macOS / Linux (bash) |
Both forms are referenced throughout: each Usage block lists the .cmd and
.sh invocation. Examples are shown in the .cmd form; on macOS/Linux use the
matching ./<name>.sh ... with the same arguments.
Prerequisites¶
- Windows 10/11 / Windows Server 2016+, or macOS / Linux with bash
- AWS CLI v2.13+ (latest recommended)
- AWS credentials configured (
aws configure, SSO, or environment variables) - PowerShell 5.1+ (included with Windows 10/11; used by
support-bundle.cmd) - Python 3.10+ (used by
check-quotaandcleanup)
Script index¶
| Script | Purpose |
|---|---|
users |
Manage Cognito editor users |
groups |
Manage Cognito groups for RBAC |
sso |
Manage SAML/OIDC identity providers |
setup-domain |
Custom domain + ACM cert for the editor |
support-bundle |
Collect diagnostics for support |
check-quota |
Validate AWS quotas before deploy |
setup-cross-account-metrics |
Remote-account metric delivery |
cleanup.py |
Tear down leftover stack resources |
users¶
Manage Cognito users for the AI Monitor editor.
users.cmd <stack-name> <action> [email] [temp-password] [region]
./users.sh <stack-name> <action> [email] [temp-password] [region]
| Action | Description |
|---|---|
create <email> [temp-password] |
Create a new user (prompted to reset on first login) |
list |
List all users |
delete <email> |
Delete a user |
reset <email> [temp-password] |
Reset password (temporary, must change on login) |
disable <email> |
Disable a user |
enable <email> |
Re-enable a user |
Notes
regiondefaults tous-east-1and is auto-detected if passed in the password slot.createdefaults a temporary password if none is given.
Examples
users.cmd AIMonitor create admin@example.com "Temp*88=888Pass123!@#"
users.cmd AIMonitor list
users.cmd AIMonitor disable user@example.com
groups¶
Manage Cognito groups for RBAC. Roles apply on the advanced tier and above. On lower tiers RBAC is off: every authenticated user has full (admin) access and these groups are ignored.
When RBAC is on, users in the admin group have full access; everyone else (the
viewer group, any other group, or no group) is read-only.
groups.cmd <stack-name> <command> [args] [region]
./groups.sh <stack-name> <command> [args] [region]
| Command | Description |
|---|---|
list-groups |
List all groups |
create-group <name> [description] |
Create a new group |
delete-group <name> |
Delete a group |
add-to-group <user> <group> |
Add a user to a group |
remove-from-group <user> <group> |
Remove a user from a group |
list-users [group] |
List users (optionally in a specific group) |
Examples
groups.cmd AIMonitor create-group admin "Full access editors"
groups.cmd AIMonitor create-group viewer "Read-only access"
groups.cmd AIMonitor add-to-group admin@example.com admin
groups.cmd AIMonitor list-users admin
sso¶
Manage external SSO (SAML/OIDC) identity providers on the Cognito User Pool for federated login to the editor. Requires the advanced or enterprise tier.
sso.cmd <stack-name> <command> [args]
./sso.sh <stack-name> <command> [args]
| Command | Description |
|---|---|
info |
Show Cognito pool details (ACS URL, Entity ID, Redirect URI) |
setup-saml <name> <metadata-url> |
Add a SAML provider (Okta, Azure AD, ADFS) |
setup-oidc <name> <issuer> <client-id> <secret> |
Add an OIDC provider (Google, Auth0) |
list |
List configured identity providers |
remove <name> |
Remove an identity provider |
Workflow
- Run
sso.cmd <stack> infoto get your ACS URL and Entity ID. - Configure your IdP (Okta, Azure AD, etc.) with those values.
- Run
sso.cmd <stack> setup-saml <name> <metadata-url>to register it. - Users see a "Sign in with
<name>" button on the Cognito login page. - After first login, assign groups:
groups.cmd <stack> add-to-group <federated-username> admin
Notes
- Federated usernames have the format
<ProviderName>_<email>(e.g.Okta_user@domain.com). - Group assignments must use the federated username, not the email alone.
- JIT provisioning is automatic (no pre-creation needed in Cognito).
- Removing a provider immediately prevents SSO login via that provider.
- Native Cognito users are unaffected by IdP changes.
Examples
sso.cmd AIMonitor info
sso.cmd AIMonitor setup-saml Okta https://dev-xxx.okta.com/app/yyy/sso/saml/metadata
sso.cmd AIMonitor setup-oidc Google https://accounts.google.com client-id client-secret
sso.cmd AIMonitor list
sso.cmd AIMonitor remove Okta
Requires: AWS CLI, Cognito User Pool deployed, advanced or enterprise tier. See
SSOGuide.pdf for detailed IdP setup instructions.
setup-domain¶
Configure a custom domain (e.g. monitor.company.com) for the editor ALB.
Handles ACM certificate creation, DNS validation via Route 53, and the ALB alias record.
setup-domain.cmd cert-only <domain> <hosted-zone-id> [region]
setup-domain.cmd <stack-name> <domain> <hosted-zone-id> [region]
./setup-domain.sh cert-only <domain> <hosted-zone-id> [region]
./setup-domain.sh <stack-name> <domain> <hosted-zone-id> [region]
| Mode | Description |
|---|---|
cert-only |
Request and validate an ACM certificate without modifying the stack. Run before deployment if you want HTTPS on a custom domain. |
<stack-name> |
Full setup: create the Route 53 alias record pointing your domain to the ALB. Run after deployment. Creates a certificate if none exists. |
Workflow
- (Optional) Pre-create cert:
setup-domain.cmd cert-only monitor.company.com Z0123456789 - Deploy the stack with the
CertificateArnparameter. - Create DNS alias:
setup-domain.cmd AIMonitor monitor.company.com Z0123456789 - Access the editor at
https://monitor.company.com/editor.
Notes
- Certificate must be in
us-east-1(same region as the ALB). - DNS validation typically takes 1–5 minutes; the script waits for it.
- Existing certificates for the same domain are reused.
Examples
setup-domain.cmd cert-only monitor.perfware.cloud Z04ABCDEF123
setup-domain.cmd AIMonitor monitor.perfware.cloud Z04ABCDEF123
setup-domain.cmd AIMonitor monitor.perfware.cloud Z04ABCDEF123 us-east-1
Requires: AWS CLI, Route 53 hosted zone for your domain.
support-bundle¶
Collects diagnostic information for troubleshooting into a timestamped bundle directory: Lambda configs, recent logs, DynamoDB state, CloudWatch metrics, alarm status, discovery cache, AI circuit breaker state, and more. No sensitive metric data is collected — only resource metadata and status.
support-bundle.cmd <stack-name> [hours]
./support-bundle.sh <stack-name> [hours]
hours— log/metric lookback period (default:24).- Region is auto-detected from your AWS CLI configuration.
Output
A <stack-name>-<timestamp>/ directory of JSON/text files. Run
analyze-bundle.py on it for a summary report, then zip and send the folder to
support.
Examples
support-bundle.cmd AIMonitor
support-bundle.cmd AIMonitor 48
python analyze-bundle.py AIMonitor-20250115-143022/
check-quota¶
Validates AWS service quotas and infrastructure readiness for a given tier before deployment. Checks VPC endpoints, Lambda concurrency, DynamoDB capacity, S3, SQS, Firehose, CloudWatch Metric Streams, OpenSearch, and Bedrock availability.
check-quota.cmd <tier> [region]
./check-quota.sh <tier> [region]
tier—basic,essential,advanced, orenterprise.region— AWS region (default:us-east-1).
Bedrock checks
- Verifies Bedrock Runtime is available in the region.
- Confirms the Anthropic Claude model is listed.
- Tests
InvokeModel(catches the use-case form requirement). - Reports the daily token quota and warns if insufficient.
Examples
check-quota.cmd advanced
check-quota.cmd enterprise us-east-1
setup-cross-account-metrics¶
Run in the remote account to set up cross-account metric delivery into the central AI Monitor stack's S3 bucket.
Remote: Metric Stream -> Firehose (+ transform Lambda) -> Central S3
Central: S3 event -> SQS -> Collector Lambda -> OpenSearch
CloudWatch Metric Streams require Firehose in the same account; Firehose supports cross-account S3 destinations natively.
Prerequisites
- Central AI Monitor stack deployed with
CrossAccountIdsincluding the remote account, orOrganizationIdset to allow any account in the AWS Organization. - AWS CLI configured for the remote account.
setup-cross-account-metrics.cmd <central-account-id> <central-stack-name> <central-region> [options]
./setup-cross-account-metrics.sh <central-account-id> <central-stack-name> <central-region> [options]
| Option | Description |
|---|---|
-filter=all |
Stream ALL metrics (default) |
-filter=discover |
Auto-discover active namespaces in this account |
-filter=file://path.txt |
Read namespaces from a file (one per line) |
--org=<org-id> |
AWS Organizations mode (no explicit account IDs needed) |
--status |
Verify existing setup and show resource health |
--remove |
Remove all cross-account resources for this stack |
Examples
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 -filter=discover
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 -filter=file://namespaces.txt
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 --org=o-abc123xyz
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 --status
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 --remove
The discover option calls aws cloudwatch list-metrics to find all namespaces
with active metrics in the remote account, configures the Metric Stream to include
only those namespaces, and saves the list to namespaces.txt for reuse.
To stream billing data from a payer/management account (where
AWS/Billing EstimatedCharges lives), run this script in that account with the
bundled billing-ns.txt filter:
setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 -filter=file://billing-ns.txt
billing-ns.txt contains the single AWS/Billing namespace. Billing metrics
only exist in us-east-1, so run with that central region. Allow ~2 days of
history before cost monitoring has enough datapoints to baseline.
The --org option is informational — it adjusts the completion message to note
that the central stack uses OrganizationId for S3 bucket-policy access rather than
explicit CrossAccountIds. The resources created are the same.
The --status option checks an existing setup: Metric Stream state and last
update, Firehose status, transform Lambda, IAM roles (firehose, stream, transform,
metrics-reader), and an S3 bucket connectivity test.
The --remove option tears down everything the script created: the Metric
Stream, Firehose delivery stream, transform Lambda, and the four IAM roles.
Example namespaces.txt:
AWS/Lambda
AWS/EC2
AWS/RDS
AWS/DynamoDB
Creates in the remote account:
- Transform Lambda (splits Metric Stream concatenated JSON)
- Firehose delivery stream with Lambda processor + cross-account S3 destination
- CloudWatch Metric Stream → local Firehose
- Metrics Reader role (lets the central account list metrics in this account)
cleanup.py¶
Tears down an AI Monitor stack's resources that CloudFormation can't remove on its own — S3 buckets (including versioned objects), DynamoDB tables, CloudWatch log groups, Metric Streams, Firehose delivery streams, Lambda VPC ENIs, Cognito user pools, and orphan resources. Run this before deleting the stack.
Runs in dry-run mode by default (lists what would be deleted; changes nothing).
python cleanup.py <stack-name> <region> [-force|-confirm]
| Flag | Behavior |
|---|---|
| (none) | Dry run — list what would be deleted |
-confirm |
Prompt before each deletion |
-force |
Delete everything without prompting |
Examples
python cleanup.py AIMonitor us-east-1
python cleanup.py AIMonitor us-east-1 -confirm
python cleanup.py AIMonitor us-east-1 -force
See GettingStarted.pdf (Teardown) for the full teardown procedure and ordering.
AWS Organizations support (advanced/enterprise)¶
Instead of listing individual account IDs in CrossAccountIds, set the
OrganizationId parameter (e.g. o-abc123xyz) on the central stack. Any account
in the organization can then deliver metrics without updating the stack each time
a new account is added.
Both methods can coexist:
OrganizationId— any org member account gets automatic access.CrossAccountIds— explicit access for accounts outside the org.
When using OrganizationId, remote accounts still run
setup-cross-account-metrics to create their Firehose and Metric Stream. The only
difference is the central stack's S3 bucket policy uses aws:PrincipalOrgID
instead of explicit account principals.
Enabling AWS/Billing metrics in remote accounts¶
AWS/Billing metrics (EstimatedCharges) are not enabled by default. Each account
that needs billing monitoring must enable this separately:
- Sign in to the AWS Console as root or an IAM identity with billing permissions.
- Open the AWS Billing and Cost Management console.
- In the left nav, choose Billing preferences.
- Under Alert preferences, choose Edit.
- Select Receive CloudWatch Billing Alerts.
- Choose Save preferences (or Update).
Note: Billing metrics are only published in
us-east-1regardless of where your resources run. The remote-account Metric Stream must be inus-east-1, or you must create a separateus-east-1stream for AWS/Billing if your primary stream is in another region.
After enabling, AWS/Billing EstimatedCharges metrics begin publishing within a
few hours and can be streamed cross-account like any other CloudWatch metric.