Run Hamilton Functions as an AWS SageMaker Processing Job

AWS SageMaker is a comprehensive platform that facilitates the creation, training, and deployment of machine learning (ML) models. This guide demonstrates deploying a “hello-world” processing job using Hamilton functions on SageMaker.

Prerequisites

• AWS CLI Setup: Ensure that the AWS CLI is configured on your machine. Follow the Quick Start guide for setup instructions.

Step-by-Step Guide

1. Build the Docker Image

Navigate to the container directory and build the Docker image:

cd container/ \
    && docker build --platform linux/amd64 -t aws-sagemaker-hamilton . \
    && cd ..

2. Create AWS ECR repository.

Ensure the AWS account number (111122223333) is correctly replaced with yours:

• Authenticate Docker to Amazon ECR:

  Retrieve an authentication token to authenticate your Docker client to your Amazon Elastic Container Registry (ECR):

  aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com
  

• Create the ECR Repository:

  aws ecr create-repository \
      --repository-name aws-sagemaker-hamilton \
      --region us-east-1 \
      --image-scanning-configuration scanOnPush=true \
      --image-tag-mutability MUTABLE
  

3. Deploy the Image to AWS ECR

Ensure the AWS account number (111122223333) is correctly replaced with yours:

docker tag aws-sagemaker-hamilton 111122223333.dkr.ecr.us-east-1.amazonaws.com/aws-sagemaker-hamilton:latest
docker push 111122223333.dkr.ecr.us-east-1.amazonaws.com/aws-sagemaker-hamilton:latest

4. Create simple role for AWS SageMaker ScriptProcessor.

• Create the Role:

  Example of creating an AWS Role with full permissions for ECR and S3.

  aws iam create-role \
      --role-name SageMakerScriptProcessorRole \
      --assume-role-policy-document '{"Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "sagemaker.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
  

• Attach Policies to the Role:

  Here we grant full access to ECR, S3 and SageMaker as an example. For production environments it's important to restrict access appropriately.

  aws iam attach-role-policy \
      --role-name SageMakerScriptProcessorRole \
      --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
  aws iam attach-role-policy \
      --role-name SageMakerScriptProcessorRole \
      --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess
  aws iam attach-role-policy \
      --role-name SageMakerScriptProcessorRole \
      --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
  

5. Install additional requirements

pip install -r requirements.txt

6. Execute the Processing Job

Find the detailed example in notebook.ipynb to run the processing job.