Collecting metrics with the CloudWatch agent
In this lab we'll be storing metrics in an Amazon Managed Service for Prometheus workspace which is already created for you. You should be able to see it in the console:
Open APS console
To view the workspace, click on the All Workspaces tab on the left control panel. Select the workspace that starts with eks-workshop and you can view several tabs under the workspace such as rules management, alert manager etc.
To gather metrics from the Amazon EKS cluster we'll use the Amazon CloudWatch Observability EKS add-on, which deploys the CloudWatch agent. The agent runs an embedded OpenTelemetry collector, and because that collector includes the Prometheus receiver and the Prometheus Remote Write exporter, we can point the add-on at Amazon Managed Service for Prometheus instead of CloudWatch.
There is nothing special about the CloudWatch agent here. Any OpenTelemetry-compatible collector configured with a Prometheus receiver, the sigv4auth extension and the Prometheus Remote Write exporter can send metrics to Amazon Managed Service for Prometheus. We use the add-on because it is a managed, supported way to run that collector without deploying and maintaining a collector or operator yourself.
Grant the CloudWatch agent permissions
The CloudWatch agent needs IAM permissions to remote-write metrics to your Amazon Managed Service for Prometheus workspace. We'll grant them using Amazon EKS Pod Identity, which lets a Kubernetes service account assume an IAM role without any static credentials. The EKS Pod Identity Agent add-on, a prerequisite for Pod Identity, has already been installed on the cluster for you by prepare-environment.
Create an IAM role that the CloudWatch agent can assume. The trust policy allows the EKS Pod Identity service principal, and we attach the AWS managed AmazonPrometheusRemoteWriteAccess policy for AMP ingestion along with CloudWatchAgentServerPolicy for the agent's baseline operation:
Associate the role with the cloudwatch-agent service account that the add-on will create in the amazon-cloudwatch namespace:
Configure the add-on to write to AMP
We provide the add-on with an OpenTelemetry pipeline that scrapes Prometheus metrics from the cluster and remote-writes them to your workspace:
# Configuration values for the amazon-cloudwatch-observability add-on.
#
# The add-on deploys the CloudWatch agent, which is built on the OpenTelemetry
# collector. Here we attach an OpenTelemetry pipeline that scrapes the Prometheus
# metrics exposed by the application pods and remote-writes them to Amazon Managed
# Service for Prometheus (AMP) using the prometheusremotewrite exporter and the
# sigv4auth extension.
#
# The pipeline is attached to the "cloudwatch-agent-cluster-scraper" agent only
# (a single-replica Deployment) rather than the global agent config. This is
# deliberate: the global config is inherited by the node-level DaemonSet too, so
# every node would scrape and remote-write the same series, and AMP rejects the
# resulting duplicate samples. Scoping the pipeline to the single cluster-scraper
# means each series is written exactly once.
#
# ${AMP_ENDPOINT} and ${AWS_REGION} are substituted by envsubst before this file
# is handed to the add-on.
otelContainerInsights:
enabled: true
logs:
enabled: false
agents:
- name: cloudwatch-agent
- name: cloudwatch-agent-cluster-scraper
mode: deployment
otelConfig:
extensions:
sigv4auth:
region: ${AWS_REGION}
service: aps
receivers:
prometheus/amp:
config:
global:
scrape_interval: 30s
scrape_timeout: 15s
external_labels:
cluster: eks-workshop
scrape_configs:
# Kubelet and cAdvisor are scraped directly on each node (role: node
# sets __address__ to <node>:10250) rather than through the API server
# proxy. The proxy path would require a "$1" capture-group replacement,
# and the CloudWatch agent expands "$1" as an environment variable — so a
# direct scrape with a static metrics_path avoids the escaping problem and
# feeds the Kubernetes cluster dashboard in Grafana.
- job_name: kubelet
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- job_name: cadvisor
scheme: https
metrics_path: /metrics/cadvisor
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- job_name: kubernetes-pods
honor_labels: true
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels:
[__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels:
[
__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow,
]
action: drop
regex: true
- source_labels:
[__meta_kubernetes_pod_annotation_prometheus_io_scheme]
action: replace
regex: (https?)
target_label: __scheme__
- source_labels:
[__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_pod_phase]
regex: Pending|Succeeded|Failed|Completed
action: drop
processors:
batch/amp:
timeout: 30s
exporters:
prometheusremotewrite/amp:
endpoint: "${AMP_ENDPOINT}api/v1/remote_write"
auth:
authenticator: sigv4auth
service:
extensions: [sigv4auth]
pipelines:
metrics/amp:
receivers: [prometheus/amp]
processors: [batch/amp]
exporters: [prometheusremotewrite/amp]
A few parts of this configuration matter. The prometheusremotewrite/amp exporter sends metrics to the AMP workspace's remote-write endpoint, and the sigv4auth extension signs those requests with the credentials the agent obtains through Pod Identity. The Prometheus scrape_configs collect application metrics from annotated pods (kubernetes-pods) and cluster metrics from the kubelet and cadvisor endpoints on each node.
The pipeline is attached to the cloudwatch-agent-cluster-scraper agent, a single-replica Deployment, rather than the global agent configuration. The add-on also runs the CloudWatch agent as a DaemonSet on every node. If we attached the remote-write pipeline globally, every node would scrape and write the same series, and AMP would reject the duplicate samples. Scoping it to the single cluster-scraper means each series is written exactly once.
Install the amazon-cloudwatch-observability add-on with this configuration. We use envsubst to substitute your workspace endpoint and Region into the file before handing it to the add-on:
The add-on deploys the CloudWatch agent into the amazon-cloudwatch namespace. Confirm the pods are running:
NAME READY STATUS RESTARTS AGE
cloudwatch-agent-8xk2p 1/1 Running 0 72s
cloudwatch-agent-df9wz 1/1 Running 0 72s
cloudwatch-agent-cluster-scraper-6dfdc8f88-458kl 1/1 Running 0 72s
fluent-bit-2s7zn 1/1 Running 0 72s
fluent-bit-krl6t 1/1 Running 0 72s
Notice the mix of components the add-on manages for you: the cloudwatch-agent DaemonSet, the single cloudwatch-agent-cluster-scraper Deployment that runs our AMP pipeline, and the fluent-bit DaemonSet, all open source projects packaged and supported as a managed add-on.