ECS Observability with Prometheus, Grafana, and Crash Testing

CloudWatch logs tell you what happened after something goes wrong. They are reactive. What I wanted was something that shows me what is happening right now. CPU, memory, request rates, the shape of the system while it is running. That is observability, and CloudWatch alone is not enough for it.

So I added Prometheus and Grafana, ran a load test, and then deliberately crashed the app to watch ECS recover on its own.

The Stack

Node.js app on ECS Fargate  → exposes /metrics endpoint
        ↓
Prometheus on EC2            → scrapes /metrics every 15 seconds
        ↓
Grafana on EC2               → visualizes the data from Prometheus
        ↓
Uptime Robot                 → monitors public URL availability

Prometheus and Grafana run together on a t3.micro EC2 instance using Docker Compose. Free tier eligible. No managed services needed.

Step 1: Expose Metrics from the Node App

Prometheus works by scraping an HTTP endpoint that exposes metrics in a specific format. The app needs to provide that endpoint.

Install the Prometheus client library:

npm install prom-client

Add the metrics endpoint to index.js:

const express = require('express')
const client = require('prom-client')

const app = express()
const port = 3000

const collectDefaultMetrics = client.collectDefaultMetrics
collectDefaultMetrics()

app.get('/', (req, res) => {
  res.json({ message: 'Welcome to Three Tier App' })
})

app.get('/health', (req, res) => {
  res.json({ status: 'ok' })
})

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType)
  res.end(await client.register.metrics())
})

app.listen(port, () => {
  console.log(`App listening on port ${port}`)
})

collectDefaultMetrics() automatically collects Node.js runtime metrics, heap memory, CPU time, event loop lag, garbage collection. No manual instrumentation needed for the basics.

Hitting /metrics now returns something like:

# HELP nodejs_heap_size_used_bytes Process heap size used from Node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 4567890

Push this change through the GitHub Actions pipeline and the running ECS task will expose metrics at /metrics through the ALB.

Step 2: Run Prometheus and Grafana on EC2

I added a monitoring EC2 instance to my existing compute Terraform module. The user_data script installs Docker, installs Docker Compose, creates the configuration files, and starts both containers on boot.

Prometheus config:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'three-tier-app'
    static_configs:
      - targets: ['your-alb-dns-name']
    metrics_path: '/metrics'

Prometheus scrapes the ALB URL at /metrics every 15 seconds. The ALB forwards requests to the healthy ECS task.

Docker Compose:

services:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  prometheus-data:
  grafana-data:

The named volumes mean Prometheus data and Grafana dashboards survive container restarts. If the containers restart the historical data is still there.

Security group for the monitoring instance:

Port 3000 → open to 0.0.0.0/0  (Grafana, public)
Port 9090 → not open            (Prometheus, internal only)
Port 22   → open to my IP only  (SSH management)

Prometheus is not exposed publicly. It contains raw infrastructure metrics you do not want the public reading. Grafana talks to Prometheus internally using the Docker Compose service name, not localhost.

The Docker Networking Lesson

When I first tried to connect Grafana to Prometheus I used http://localhost:9090 as the URL. It failed immediately.

The reason is that inside a Docker container localhost refers to that container itself, not the host machine and not other containers. Prometheus is a separate container. To reach it from Grafana you use the service name defined in Docker Compose:

<http://prometheus:9090>

Docker Compose creates an internal network where containers can reference each other by service name. This is fundamental Docker networking and easy to miss the first time.

Step 3: Connect Grafana to Prometheus

Once both containers are running:

Open Grafana at http://your-ec2-ip:3000
Go to Connections → Data Sources → Add data source → Prometheus
URL: http://prometheus:9090
Save and Test

The response should say the Prometheus API was queried successfully.

Then create a dashboard. The most useful metrics to start with:

nodejs_heap_size_used_bytes   → memory usage over time
process_cpu_seconds_total     → CPU consumption

Step 4: Load Testing

With the dashboard open I ran a load test using hey:

hey -n 10000 -c 50 <http://your-alb-url/>

This sends 10,000 requests with 50 concurrent connections. Watching the Grafana dashboard during the test showed memory climbing and CPU spiking in real time. The metrics endpoint was doing its job.

Step 5: Crash Testing

The most interesting part of observability is proving your system recovers from failure. I added a crash endpoint to the app:

app.get('/crash', (req, res) => {
  res.json({ message: 'Crashing...' })
  process.exit(1)
})

Deployed it through the pipeline, then triggered it:

curl <http://your-alb-url/crash>

What happened next:

Container process exited
        ↓
ECS detected unhealthy task
        ↓
Task deregistered from ALB target group
        ↓
ECS started a new task automatically
        ↓
New task passed health checks
        ↓
Task registered with ALB
        ↓
App responding again

Recovery took about 1 to 2 minutes. During that window the ALB returned 503s. Uptime Robot sent a down alert within 5 minutes, then a recovery alert once the new task was healthy.

The Grafana dashboard showed a gap in metrics during the crash window, then metrics resuming when the new task came up.

What This Proves

Nobody manually intervened. The container crashed, ECS detected it, a new task started, and traffic resumed. That is self-healing infrastructure working as designed.

The combination of Prometheus metrics, Grafana dashboards, and Uptime Robot gives three different views of the system:

Uptime Robot  → is it up or down right now
Grafana       → what is the system doing over time
CloudWatch    → what did the application log

Each answers a different question. Together they give you enough visibility to understand your system under normal load and under failure.

Observability for ECS: Prometheus, Grafana, and Crash Testing