Infrastructure Diagram

Status: IMPLEMENTED — Added in PR #179. The /api/v1/diagram/graph endpoint, the React Flow-based diagram page, and the ELK.js layout engine are live.

Polo provides an interactive infrastructure diagram that visualizes the full resource hierarchy — customers, clusters, indexes, and individual AWS resources — in a zoomable, filterable canvas. It replaces the legacy GoJS-based diagram from Polo v1.

Purpose

The diagram answers "what does our infrastructure look like right now?" by rendering:

• Hierarchy groups nested inside each other (customer > cluster > index), derived from resource metadata fields
• Resource nodes (EC2 instances, EBS volumes, NAT gateways, load balancers, etc.) positioned within their hierarchy group
• Relationship edges between resources (volume attached to instance, instance backed by load balancer, etc.)

Unlike the tabular views (Resources, Hierarchy), the diagram shows spatial relationships and containment at a glance.

Data Sources

The diagram draws from two ClickHouse tables:

Table	Role in Diagram
resource_snapshots	Leaf nodes (individual AWS resources with cost, state, type) and group derivation (via marqo_customer, marqo_cluster, marqo_index fields)
resource_relationships	Edges between resources (attached_to, in_subnet, in_vpc, backed_by, etc.)

Group containers (customer, cluster, index) are derived client-side from the marqo_customer, marqo_cluster, and marqo_index fields on each resource — no hierarchy tables are queried.

API

GET /api/v1/diagram/graph

Single endpoint returning all data needed to render the diagram.

Parameters:

Param	Type	Default	Description
aws_account_id	string	—	Filter to a single AWS account (required in UI — account selector defaults to Staging)
limit	number	1000	Max resources to return (capped at 2000)

Response:

{
  "nodes": [
    { "resource_arn": "arn:aws:ec2:...", "resource_type": "ec2:instance", "resource_name": "marqo-serving-1", "aws_account_id": "468036072962", "aws_region": "us-east-1", "state": "running", "marqo_customer": "acme", "marqo_cluster": "acme-prod-1", "marqo_index": "products", "marqo_env": "prod", "cost_daily_usd": 12.50 }
  ],
  "edges": [
    { "source_arn": "arn:aws:ebs:...", "target_arn": "arn:aws:ec2:...", "relationship": "attached_to", "is_cost_parent": 1 }
  ]
}

Nodes are ordered by cost_daily_usd DESC and limited to limit. Edge subqueries use the same ordering and limit to ensure they match the returned nodes.

Visualization

Technology

• React Flow (@xyflow/react) — React-native interactive diagram canvas with pan, zoom, minimap, and grouped node support
• ELK.js (elkjs) — Automatic hierarchical layout engine supporting compound graphs (groups containing groups)

Node Types

Group nodes (hierarchy containers, derived client-side):

Type	Color	Contains
customer	Blue border	Clusters
cluster	Purple border	Indexes
index	Green border	Resources

Group nodes display: name, type badge, resource count, and aggregate cost.

Resource nodes (leaf nodes):

Type	Icon	Key Properties
ec2:instance	Server	Name, instance type, state, cost
ebs:volume	HardDrive	Name, size, state, cost
vpc:nat_gateway	Router	Cost
elbv2:load_balancer	Network	Type, cost
ebs:snapshot	Archive	Age, cost
ec2:eip	Globe	Association, cost
Other types	Box	Name, cost

Resource nodes display: icon, name, type badge, daily cost, state indicator.

Edge Types

Relationship	Style	Meaning
attached_to	Solid	EBS volume attached to EC2 instance
backed_by	Solid	Load balancer routing to instance
in_subnet	Dashed	Resource located in a subnet
in_vpc	Dashed	Subnet located in a VPC
snapshot_of	Dotted	EBS snapshot of a volume
associated_with	Solid	EIP associated with instance
eni_of	Light	ENI belonging to instance

Layout

ELK.js with algorithm selection per group:

• Groups with edges: layered algorithm
• Groups without edges: rectpacking algorithm
• Root level: box algorithm
• Layout computed asynchronously (skeleton shown during computation)

Filtering

All filtering is client-side on the already-fetched dataset (no re-fetching needed).

Dimension Filters

Dropdown selectors matching the Explore page pattern:

• Resource Type
• Customer
• Environment
• Region
• State

Selecting a value hides all resources that don't match that dimension.

Text Search

An input field where the user types space-separated terms. A node is visible only if all terms match some property of the node (name, ARN, type, account, customer, cluster, state, etc.). This enables queries like prod acme instance to find production EC2 instances belonging to Acme.

Type Toggles

Checkboxes for each resource type and edge type present in the data:

• Resource types: ec2:instance, ebs:volume, vpc:nat_gateway, etc. — toggling off hides all nodes of that type
• Edge types: attached_to, in_subnet, backed_by, etc. — toggling off hides edges of that type

Edge Visibility Rule

An edge is hidden when both of its connected nodes are hidden. If at least one endpoint is visible, the edge remains visible (pointing off-screen if needed, or hidden if both are filtered out).

Group Pruning

Groups with all children hidden are automatically hidden to avoid empty containers cluttering the diagram.

Account Selection

The diagram UI defaults to the Staging AWS account. The user selects an account from a dropdown (using the hardcoded account names from lib/accounts.ts), which triggers a data refetch and re-layout.

Performance

• Server: only active resources returned (running, in-use, available); bounded by limit parameter; edges use subqueries with matching ORDER BY to ensure overlap with returned nodes
• Client: filtering via useMemo on cached data; ELK layout runs async; React Flow virtualizes rendering (only visible nodes in DOM)
• Expected scale: ~200-1000 nodes, well within React Flow's capabilities

Comparison with Legacy (Polo v1)

Aspect	Legacy (v1)	New (v2)
Library	GoJS (commercial)	React Flow (MIT)
Layout	Custom PoolLayout + swimlanes	ELK.js hierarchical
Data source	DynamoDB via Python	ClickHouse via CF Worker
Filtering	Category checkboxes + fold-to levels	Dimension dropdowns + text search + type toggles
Grouping	AWS Account > Account > Cluster > VPC > Subnet > Index	Customer > Cluster > Index (derived from resource fields)
Edges	ELB→instance, instance→EBS only	All resource_relationships
Cost overlay	HSL heat map toggle	Cost displayed on each node