Price monitoring for Amazon EMR on Amazon EKS

Resolution overview

A job in EMR on EKS incur prices primarily on two dimensions: compute sources and a marginal uplift cost for EMR on EKS utilization. To trace the price related to every of the scale, we use knowledge from three sources:

AWS CUR – We use this to get the EMR on EKS value uplift per job and for Kubecost to reconcile the compute value with any saving plans or reserved occasion used. The supporting infrastructure for CUR is deployed as outlined in Establishing Athena utilizing AWS CloudFormation templates.
Kubecost – We use this to get the compute value incurred by the executor and driver pods.

The associated fee allocation course of contains the next elements:

The compute value is supplied by Kubecost. Nevertheless, with the intention to do an in-depth evaluation, we outline an hourly Kubernetes CronJob on it that begins a pod to retrieve knowledge from Kubecost and shops it in Amazon Easy Storage Service (Amazon S3).
CUR recordsdata are saved in an S3 bucket.
We use Amazon Athena to create a view and supply a consolidated view of the entire value to run an EMR on EKS job.
Lastly, you possibly can join your most popular enterprise intelligence instruments utilizing the JDBC or ODBC connections to Athena. On this put up, we use Amazon QuickSight native integration for visualization functions.

The next diagram reveals the general structure in addition to how the completely different elements work together with one another.

emr-eks-cost-tracking-architecture

We offer a shell script to deploy our the monitoring answer. The shell script configures the infrastructure utilizing an AWS CloudFormation template, the AWS Command Line Interface (AWS CLI), and eksctl and kubectl instructions. This script runs the next actions:

Begin the CloudFormation deployment.
Create and configure an AWS Price and Utilization Report.
Configure and deploy Kubecost backed by Amazon Managed Service for Prometheus.
Deploy a Kubernetes CronJob.

Stipulations

You want the next stipulations:

This put up assumes you have already got an EKS cluster and run EMR on EKS jobs. Should you don’t have an EKS cluster prepared to check the answer, we recommend beginning with a normal EMR on EKS blueprint that configures a cluster to submit EMR on EKS jobs.

Arrange the answer

To run the shell script, full the next steps:

Clone the next GitHub repository.
Go to the folder cost-tracking with the next command:

cd cost-tracking

Run the script with following command :

sh deploy-emr-eks-cost-tracking.sh REGION KUBECOST-VERSION EKS-CLUSTER-NAME ACCOUNT-ID

After you run the script, you’re prepared to make use of Kubecost and the CUR knowledge to know the price related along with your EMR on EKS jobs.

Monitoring value

On this part, we present you the best way to analyze the compute value that’s retrieved from Kubecost, the best way to question EMR on EKS uplift knowledge, and the best way to mix them to have a single consolidated view for the price.

Compute value

Kubecost affords numerous methods to trace value per Kubernetes object. For instance, you possibly can monitor value by pod, controller, job, label, or deployment. It additionally permits you to perceive the price of idle sources, like Amazon Elastic Compute Cloud (Amazon EC2) cases that aren’t absolutely utilized by pods. On this put up, we assume that no nodes are provisioned if no EMR on EKS job is working, and we use the Karpenter Cluster Autoscaler to provision nodes when jobs are submitted. Karpenter additionally does bin packing, which optimizes the EC2 useful resource utilization and in flip reduces the price of idle sources.

To trace compute value related to EMR on EKS pods, we question the Kubecost allocation API by passing pod and labels within the combination parameter. We use the emr-containers.amazonaws.com/job.id and emr-containers.amazonaws.com/virtual-cluster-id labels which are all the time current in executor and driver pods. The labels are used to filter Kubecost knowledge to get solely the price related to EMR on EKS pods. You’ll be able to assessment numerous ranges of granularity on the pod, job, and digital cluster degree to know the price of a driver vs. executor, or of utilizing Spot Situations in jobs. You can even use the digital cluster value to know the general value of a EMR on EMR when it’s utilized in a namespace that’s utilized by purposes apart from EMR on EKS.

We additionally present the instance_id, occasion measurement, and capability sort (On-Demand or Spot) that was used to run the pod. That is retrieved via querying the Kubecost belongings API. This knowledge may be helpful to know the way you run your jobs and which capability you employ extra usually.

The information about the price of working the pods in addition to the belongings is retrieved with a Kubernetes CronJob that submits the request to the Kubecost API, joins the 2 knowledge sources (allocation and belongings knowledge) on the instance_id, cleans the information, and shops it in Amazon S3 in CSV format.

The compute value knowledge has a number of fields which are of curiosity, together with cpucost, ramcost (value of reminiscence), pvcost (value of Amazon EBS storage), effectivity of use of CPU and RAM, in addition to whole value, which represents the mixture value of all of the sources used, both at pod, job, or digital cluster degree.

To view this knowledge, full the next steps:

On the Athena console, navigate to the question editor.
Select athenacurcfn_c_u_r for the database and cost_data for the desk.
Run the next question:

SELECT job_id,
vc_id,
sum(totalcost) as value
FROM "athenacurcfn_c_u_r"."compute_cost"
GROUP BY job_id, vc_id

The next screenshot reveals the question outcomes.

To question the information about info on the pod degree, you possibly can run the next SQL assertion:

SELECT
split_part(title, '/', 1) as pod_name,
job_id,
vc_id,
totalcost,
instance_id,
"properties.labels.node_kubernetes_io_instance_type",
capacity_type
FROM "athenacurcfn_c_u_r"."compute_cost";

EMR on EKS uplift

The associated fee related to EMR on EKS uplift is out there via AWS CUT and is saved in an S3 bucket. The script you ran within the setup step created an Athena desk related to the information within the S3 bucket. The next steps take you thru how one can question the information:

On the Athena console, navigate to the question editor.
Select athenacurcfn_c_u_r for the database and cur_data for the desk.
Run the next question:

SELECT
split_part(line_item_resource_id, '/', 5) as job_id,
split_part(line_item_resource_id, '/', 3) as vc_id,
sum(line_item_blended_cost) as value
FROM athenacurcfn_c_u_r.automated
WHERE product_product_family='EMR Containers'
GROUP BY line_item_resource_id

This question gives you with the price per job. The next screenshot reveals the outcomes.

You’ll have to wait as much as 24 hours for the CUR knowledge to be accessible. As such, you need to solely run the previous question after the CUR knowledge is out there and you’ve got run the EMR on EKS jobs.

Total value

To view the general value and carry out evaluation on it, create a view in Athena as follows:

CREATE VIEW emr_eks_cost AS
SELECT
split_part(line_item_resource_id, '/', 5) as job_id,
split_part(line_item_resource_id, '/', 3) as vc_id,
sum(line_item_blended_cost) as value,
'emr-uplift' as class
FROM athenacurcfn_c_u_r.cur_data
WHERE product_product_family='EMR Containers'
GROUP BY line_item_resource_id
UNION
SELECT
job_id,
vc_id,
sum(totalCost) as value,
'compute' as class
FROM "athenacurcfn_c_u_r"."compute_cost"
group by job_id, vc_id

Now that the view is created, you possibly can question and analyze the price of working your EMR on EKS jobs:

SELECT sum(value) as total_cost, job_id, vc_id
FROM "athenacurcfn_c_u_r"."emr_eks_cost"
GROUP BY job_id, vc_id;

The next screenshot reveals an instance output of the question on the created view.

Lastly, you should use QuickSight for a graphical high-level view in your EMR on EKS spend. The next screenshot reveals an instance dashboard.

emr-eks-compute-cost-quicksight-dashboard

Now you can adapt this answer to your particular wants and construct your customized evaluation.

Clear up

All through this put up, you deployed and configured the required infrastructure elements to trace value on your EMR on EKS workloads. To keep away from incurring further costs for this answer, delete all of the sources you created:

Empty the S3 buckets cost-data-REGION-ACCOUNT_ID and aws-athena-query-results-cur-REGION-ACCOUNT_ID.
Delete the Athena workgroup kubecost-cur-workgroup.
Empty and delete the ECR repository emreks-compute-cost-exporter.
Run the script destroy-emr-eks-cost-tracking.sh, which can delete the AWS CloudFormation deployment, uninstall Kubecost, delete the CronJob, and delete the Price and Utilization Experiences.

Conclusion

On this put up, we confirmed how you should use Kubecost capabilities alongside Price and Utilization Experiences to intently monitor the prices for Amazon EMR on EKS per digital cluster or per job. This answer permits you to obtain extra granular prices for chargebacks utilizing Athena, Amazon Managed Service for Prometheus, and QuickSight.

The answer offered steps to arrange Price and Utilization Experiences and Kubecost, and configure a CronJob on an hourly foundation to get the price of working pods spun by EMR on EKS. You’ll be able to modify the offered answer to run at longer intervals or to gather knowledge on completely different EKS clusters. You can even modify the Python script run by the CronJob to additional clear knowledge or cut back the quantity of knowledge saved by eliminating fields you don’t want. You should utilize the insights supplied to drive value optimization efforts over time, detect any enhance of prices, and measure the affect of latest deployments or explicit occasions on useful resource utilization and price efficiency. For extra details about integrating EMR on EKS in your present Amazon EKS deployment, seek advice from Design issues for Amazon EMR on EKS in a multi-tenant Amazon EKS setting

In regards to the Authors

Lotfi Mouhib is a Senior Options Architect working for the Public Sector group with Amazon Internet Companies. He helps public sector clients throughout EMEA notice their concepts, construct new providers, and innovate for residents. In his spare time, Lotfi enjoys biking and working.

Hamza Mimi Principal Options Architect within the French Public sector group at Amazon Internet Companies (AWS). With an extended expertise within the telecommunications {industry}. He’s at present working as a buyer advisor on matters starting from digital transformation to architectural steerage.

[ad_2]