AWS Cloud S3 monitoring

Amazon Simple Storage Service (Amazon S3) is an object storage service. SkyWalking leverages AWS Kinesis Data Firehose receiver to transfer the CloudWatch metrics of s3 to OpenTelemetry receiver and into the Meter System.

Data flow

1. AWS CloudWatch collect metrics for S3, refer to S3 monitoring with CloudWatch
2. CloudWatch metric streams stream CloudWatch metrics of S3 to AWS Kinesis Data Firehose
3. AWS Kinesis Data Firehose delivery metrics to AWS Kinesis Data Firehose receiver through the HTTP endpoint

Set up

1. Create CloudWatch metrics configuration for S3, refer to S3 metrics configuration
2. Create an Amazon Kinesis Data Firehose Delivery Stream, and set AWS Kinesis Data Firehose receiver's address as HTTP(s) Destination, refer to Create Delivery Stream
3. Create CloudWatch metric stream, and select the Firehose Delivery Stream which has been created above, set Select namespaces to AWS/S3, Select output format to OpenTelemetry 0.7. refer to CloudWatch Metric Streams

Read Monitoring AWS EKS and S3 with SkyWalking for more details

S3 Monitoring

SkyWalking observes CloudWatch metrics of the S3 bucket, which is cataloged as a LAYER: AWS_S3 Service in the OAP.

Supported Metrics

Monitoring Panel	Unit	Metric Name	Catalog	Description	Data Source
4xx Errors	count	aws_s3_4xx	Service	The number of HTTP 4xx client error status code requests made to the S3 bucket	S3 monitoring with CloudWatch
5xx Errors	count	aws_s3_5xx	Service	The number of HTTP 5xx client error status code requests made to the S3 bucket	S3 monitoring with CloudWatch
Downloaded	bytes	aws_s3_downloaded_bytes	Service	The number of bytes downloaded for requests made to an Amazon S3 bucket	S3 monitoring with CloudWatch
Uploaded	bytes	aws_s3_uploaded_bytes	Service	The number of bytes uploaded for requests made to an Amazon S3 bucket	S3 monitoring with CloudWatch
Request Average Latency	bytes	aws_s3_request_latency	Service	The average of elapsed per-request time from the first byte received to the last byte sent to an Amazon S3 bucket	S3 monitoring with CloudWatch
First Byte Average Latency	bytes	aws_s3_request_latency	Service	The average of per-request time from the complete request being received by an Amazon S3 bucket to when the response starts to be returned	S3 monitoring with CloudWatch
All Requests	bytes	aws_s3_delete_requests	Service	The number of HTTP All requests made for objects in an Amazon S3 bucket	S3 monitoring with CloudWatch
Get Requests	bytes	aws_s3_delete_requests	Service	The number of HTTP Get requests made for objects in an Amazon S3 bucket	S3 monitoring with CloudWatch
Put Requests	bytes	aws_s3_delete_requests	Service	The number of HTTP PUT requests made for objects in an Amazon S3 bucket	S3 monitoring with CloudWatch
Delete Requests	bytes	aws_s3_delete_requests	Service	The number of HTTP Delete requests made for objects in an Amazon S3 bucket	S3 monitoring with CloudWatch

Customizations

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in /config/otel-rules/aws-s3/. The AWS Cloud EKS dashboard panel configurations are found in /config/ui-initialized-templates/aws_s3.