Inspecting AWS Resources

Quick reference for finding what's deployed and checking logs, metrics, and errors.

All commands use the polo AWS CLI profile. Set it up with aws configure --profile polo.

Lambda Functions

List deployed collectors

aws lambda list-functions --profile polo \
  --query "Functions[?starts_with(FunctionName, 'polo-')].{Name:FunctionName, Runtime:Runtime, LastModified:LastModified}" \
  --output table

Check recent invocations

# Recent errors for a specific collector
aws logs filter-log-events --profile polo \
  --log-group-name /aws/lambda/polo-config-ec2 \
  --filter-pattern "ERROR" \
  --start-time $(date -v-1H +%s000)

# Tail logs live
aws logs tail /aws/lambda/polo-config-ec2 --profile polo --follow

Check Lambda metrics (last hour)

# Errors
aws cloudwatch get-metric-statistics --profile polo \
  --namespace AWS/Lambda \
  --metric-name Errors \
  --dimensions Name=FunctionName,Value=polo-config-ec2 \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Sum

# Duration
aws cloudwatch get-metric-statistics --profile polo \
  --namespace AWS/Lambda \
  --metric-name Duration \
  --dimensions Name=FunctionName,Value=polo-config-ec2 \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Average,Maximum

List all collector log groups

aws logs describe-log-groups --profile polo \
  --log-group-name-prefix /aws/lambda/polo- \
  --query "logGroups[].logGroupName" --output table

EventBridge Schedules

# List rules that trigger collectors
aws events list-rules --profile polo \
  --name-prefix polo- \
  --query "Rules[].{Name:Name, Schedule:ScheduleExpression, State:State}" \
  --output table

# Check targets for a rule
aws events list-targets-by-rule --profile polo --rule polo-config-ec2

ClickHouse

Connect

If using ClickHouse Cloud, the host and credentials are stored as Wrangler secrets (see deployment.md). For local dev:

# Local (Docker)
curl 'http://localhost:8123/?query=SELECT+1'

# Or via clickhouse-client
docker exec -it polo-clickhouse clickhouse-client

Health checks

-- Table sizes and row counts
SELECT table, formatReadableSize(total_bytes) AS size, total_rows
FROM system.tables WHERE database = 'polo' ORDER BY total_bytes DESC;

-- Recent inserts (last hour)
SELECT table, count() AS queries, sum(written_rows) AS rows_written
FROM system.query_log
WHERE event_time > now() - INTERVAL 1 HOUR AND type = 'QueryFinish' AND query_kind = 'Insert'
GROUP BY table ORDER BY rows_written DESC;

-- Slow queries (last 24h)
SELECT query_duration_ms, query
FROM system.query_log
WHERE event_time > now() - INTERVAL 1 DAY AND query_duration_ms > 1000
ORDER BY query_duration_ms DESC LIMIT 10;

-- Disk usage
SELECT formatReadableSize(total_space), formatReadableSize(free_space)
FROM system.disks;

Cloudflare Worker

The Worker (polo-api) serves the API and SPA. It's managed via Wrangler, not AWS.

# Check deployment status
cd components/api && npx wrangler deployments list

# Tail live logs
cd components/api && npx wrangler tail

# Check configured secrets
cd components/api && npx wrangler secret list

IAM Roles

Role	Purpose	Scope
{env}-PoloReadRole	Collector Lambdas assume this in target accounts	Read-only EC2, EBS, VPC, Cost Explorer, Organizations, Tagging
polo-action-role	Action Lambda assumes this (planned)	Scoped write access for remediation

# Check if role exists in a target account
aws iam get-role --role-name staging-PoloReadRole --profile polo

# Check what the Lambdas' execution role can do
aws lambda get-function-configuration --profile polo \
  --function-name polo-config-ec2 \
  --query "Role"

CDK Stacks

# List deployed stacks
aws cloudformation list-stacks --profile polo \
  --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE \
  --query "StackSummaries[?starts_with(StackName, 'polo') || starts_with(StackName, 'Polo')].{Name:StackName, Status:StackStatus, Updated:LastUpdatedTime}" \
  --output table

# Check specific stack resources
aws cloudformation list-stack-resources --profile polo \
  --stack-name PoloStack \
  --query "StackResourceSummaries[].{Type:ResourceType, LogicalId:LogicalResourceId, Status:ResourceStatus}" \
  --output table

Quick Triage Checklist

When investigating an issue:

1. Is the collector running? Check EventBridge rule state and recent invocations
2. Is it erroring? Check CloudWatch Logs for ERROR/exceptions
3. Is data landing? Query resource_events for recent rows from that collector
4. Is ClickHouse healthy? Check disk, slow queries, failed inserts
5. Is the API serving? Hit the Worker health endpoint or tail Wrangler logs