Collecting Container Output on EC2 With Cloud Logging Pipelines

This post is a full record of the issues and implementation details I hit while configuring container output collection on EC2 in a real environment.

Why central remote logging is worth it

Before this setup, checking container output always meant SSH into EC2 and running docker logs. Once environments grow (staging + prod), that becomes hard to manage. A centralized stream lets you inspect output in AWS Console or CLI directly, without SSH on every check.

End-to-end topology

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

Docker containers on EC2
  ├── nginx
  ├── api
  └── postgres
        │
        │  each container's stdout/stderr
        │  sent via Docker's awslogs driver
        │  directly to AWS CloudWatch Logs
        ▼
CloudWatch Logs
  ├── /side-project/dev/nginx
  ├── /side-project/dev/api
  └── /side-project/dev/postgres

To make this work, you need three things:

1. CloudWatch has log groups ready to receive data (created by Terraform)
2. EC2 has permission to write output streams (role + instance profile)
3. Containers know the destination (awslogs driver configured in docker-compose)

How to inspect output quickly

1
2
3
4
5
6
7

# CLI (requires AWS profile)
aws logs tail /side-project/dev/api --follow --profile side-project
aws logs tail /side-project/dev/postgres --follow --profile side-project
aws logs tail /side-project/dev/nginx --follow --profile side-project

# Console:
# CloudWatch > Log groups > /side-project/dev/*

Local development does not use remote log shipping; use docker compose logs directly.

Credential and Trust Model Concepts

Access card object model

A role is like an access card. It defines:

• who can wear this card (Trust Policy / Assume Role Policy)
• which rooms this card can open (Permission Policy)

1
2
3

aws_iam_role "ec2"
├── Trust Policy:      only EC2 service can assume this role
└── Permission Policy: allows writing to CloudWatch Logs

Badge holder object model

EC2 cannot attach a role directly. It must attach through an Instance Profile as an intermediate container. This is an AWS platform constraint:

1

IAM Role (access card) ──▶ Instance Profile (badge holder) ──▶ EC2 Instance (person)

Terraform example:

1
2
3
4
5
6
7
8
9

resource "aws_iam_role" "ec2" { ... }

resource "aws_iam_instance_profile" "ec2" {
  role = aws_iam_role.ec2.name
}

resource "aws_instance" "api" {
  iam_instance_profile = aws_iam_instance_profile.ec2.name
}

Note: adding iam_instance_profile to an existing EC2 instance triggers recreation (AWS limitation).

Delegation right for handing over the card

GitHub Actions uses Terraform to attach a role to EC2. AWS requires the executor to have permission to hand over that card: iam:PassRole.

Without this limit, anyone with broad IAM rights could attach an admin role to any EC2 instance.

1
2
3
4
5
6
7
8
9

{
  Action   = ["iam:PassRole"]
  Resource = "arn:aws:iam::*:role/side-project-ec2-*"
  Condition = {
    StringEquals = {
      "iam:PassedToService" = "ec2.amazonaws.com"
    }
  }
}

Best practice is to isolate PassRole in its own statement, separate from other IAM operations (CreateRole and so on).

Permission Statement Anatomy

Each IAM statement has three parts:

1
2
3
4
5

{
  Action   = "some operation"
  Resource = "which resources"
  Condition = { ... }
}

Why Some Operations Need Wildcard Scope

In this category, the pattern is Resource = "*", and operations like logs:DeleteLogGroup target specific log groups, so you can scope resources:

1
2
3
4

{
  Action   = "logs:DeleteLogGroup"
  Resource = "arn:aws:logs:us-east-1:*:log-group:/side-project/*"
}

But logs:DescribeLogGroups is a list operation. AWS does not evaluate your resource pattern for this call, so it must use *. Since this is read-only, the risk is acceptable.

Legacy and Current Endpoint Names

AWS sometimes renames APIs. These two pairs are functionally equivalent:

Terraform AWS provider now uses the newer names, but for compatibility I keep both sets.

Foundational Stack and Delivery Stack

Flow:

1
2
3
4
5
6

1. run bootstrap terraform apply locally
   → GitHub Actions gets new permissions

2. push to main
   → infra.yml: terraform apply (creates IAM role, log groups, EC2)
   → deploy.yml: SSH into EC2, deploy containers

Container Output Shipping Settings

1
2
3
4
5
6
7
8
9

# docker-compose.prod.yml
services:
  api:
    logging:
      driver: awslogs
      options:
        awslogs-group: /side-project/${APP_ENV}/api
        awslogs-region: ${AWS_REGION}
        tag: api

• driver: awslogs: Docker does not keep output locally, it sends to CloudWatch
• ${APP_ENV} / ${AWS_REGION}: loaded from .env
• tag: identifies the log stream in CloudWatch

Final Takeaways

After centralizing container output into CloudWatch, troubleshooting speed improved a lot, and multi-environment operations became cleaner. This pattern is not just for side projects; it can be reused for any EC2 + Docker deployment.

Reference links

• Docker awslogs logging driver
• IAM Roles for Amazon EC2 - Instance profiles
• Grant permissions to pass IAM roles to AWS services (EC2)
• IAM condition keys - iam:PassedToService
• Amazon CloudWatch Logs - Service authorization reference
• AWS CLI - logs tail