~/ nmadiraju.com Amazon EKS · Observability
arrow_back Back to nmadiraju
Visual Walkthrough · Operations

How Observability Works on Amazon EKS

Follow a 2 a.m. checkout incident from kubectl to logs, metrics, traces and the audit log, then check yourself with 8 questions.

1Visual Walkthrough

Observability on an EKS cluster: log, metric and trace agents on each node, AWS services that store them, and control-plane logs The EKS control plane holds ConfigMap checkout-config and the kube-apiserver with audit logging on. Your VPC holds node-1 and node-2, each with a checkout Pod, its container log files, a fluent-bit Pod and a cloudwatch-agent Pod; node-2 also runs metrics-server. On the right, CloudWatch Logs, Container Insights, Amazon Managed Service for Prometheus, Amazon Managed Grafana and AWS X-Ray. The walkthrough follows an on-call engineer from kubectl to logs, metrics, traces, a high-cardinality label and the audit log that shows who changed the ConfigMap. You, on call paged at 02:07 CloudTrail AWS API calls: CreateCluster, CreateNodegroup… EKS CONTROL PLANE · AWS-MANAGED · CLUSTER prod ConfigMap checkout-config namespace shop DB_POOL_SIZE: "5" kube-apiserver every kubectl request control-plane log types: all off by default · audit on YOUR VPC · NODE GROUP ng-general node-1 m6i.large node-2 m6i.large checkout-5f8b-k4m REPLACED REPLACED /var/log/containers/ stdout + stderr of each container REPLACED /var/log/containers/ stdout + stderr of each container REPLACED fluent-bit DaemonSet · tails log files REPLACED fluent-bit DaemonSet · tails log files REPLACED cloudwatch-agent DaemonSet · metrics, traces REPLACED cloudwatch-agent DaemonSet · metrics, traces REPLACED metrics-server current CPU and memory only REPLACED AWS SERVICES · REGION us-west-2 CloudWatch Logs /aws/containerinsights/prod/application /aws/eks/prod/cluster (control plane) Container Insights cluster · node · Pod · container metrics Managed Service for Prometheus workspace prod · managed scraper, 30 s Managed Grafana dashboards · PromQL queries AWS X-Ray traces: one request, every span

    2Quiz to check understanding

    Check yourself