AWS S3 Registry Setup

This guide provides a comprehensive walkthrough on how to leverage an AWS S3 bucket as a centralized registry for storing your machine learning models and facilitating their deployments. By following the outlined steps, you will:

• Create an S3 Bucket: Establish the foundational storage location for your models.
• Generate a Read-Only Policy: Ensure secure, read-only access to your models, safeguarding them from unauthorized modifications.
• Create an IAM User: Set up a dedicated identity for managing and accessing the S3 bucket.
• Issue Access and Secret Keys: Obtain credentials for the IAM user, enabling authenticated interactions between your platform and the S3 bucket.

This guide does not cover the process of adding the S3 bucket as a registry within the HPE Machine Learning Inferencing Software platform. For instructions on how to add a registry, refer to the Add Registry guide.

Before You Start #

• Ensure that you have an AWS account and the AWS CLI installed
• Ensure that you have the necessary permissions to create an S3 bucket, policies, and IAM users

How to Set Up an AWS S3 Registry #

1. Create an S3 Bucket #

1. Sign in to the AWS CLI or Console.
2. Create an S3 Bucket.

   aws s3 mb s3://<BUCKET_NAME> --region <REGION>

   Copy

2. Create a Read-Only Policy #

1. Define the policy details as a JSON file.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:ListBucket",
                   "s3:GetBucketLocation"
               ],
               "Resource": [
                   "arn:aws:s3:::<BUCKET_NAME>",
                   "arn:aws:s3:::<BUCKET_NAME>/*"
               ]
           }
       ]
   }

   Copy
2. Create the policy.

   aws iam create-policy --policy-name <POLICY_NAME> --policy-document file://<POLICY_FILE>.json

   Copy

3. Create an IAM User & Attach the Policy #

1. Create a new IAM user.

   aws iam create-user --user-name <BUCKET_NAME>-read-only 

   Copy
2. Attach the policy to the user.

   aws iam attach-user-policy --user-name <BUCKET_NAME>-read-only --policy-arn <POLICY_ARN>

   Copy

4. Create an Access & Secret Key #

1. Create an access key for the user.

   aws iam create-access-key --user-name <BUCKET_NAME>-read-only

   Copy

   Creating access key for user: model-registry-24f62156-read-only
   {
       "AccessKey": {
           "UserName": "model-registry-24f62156-read-only",
           "AccessKeyId": "AKIAYLZZO5KKY7PPOME7",
           "Status": "Active",
           "SecretAccessKey": "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",
           "CreateDate": "2024-03-21T16:18:33+00:00"
       }
   }

   Copy
2. Save the accessKeyID and secretAccessKey values. You will need these credentials to add the S3 bucket as a registry within the HPE Machine Learning Inferencing Software platform.

You’re now ready to add an S3 registry to HPE Machine Learning Inferencing Software.

Script for Registry Setup #

You can use the following script to automate the process of setting up an AWS S3 bucket as a registry. Replace the placeholder values with your actual bucket name and region.

#!/bin/bash

RANDOM_IDENTIFIER=$(openssl rand -hex 4)

# Define variables
REGION=""
BUCKET_NAME="model-registry-${REGION:-us-east-2}-${RANDOM_IDENTIFIER}"
POLICY_NAME="${BUCKET_NAME}-read-only-policy"
USER_NAME="${BUCKET_NAME}-read-only"
POLICY_FILE="${BUCKET_NAME}_policy.json"

# Create the policy JSON file
cat <<EOF >${POLICY_FILE}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET_NAME}",
                "arn:aws:s3:::${BUCKET_NAME}/*"
            ]
        }
    ]
}
EOF

# Step 1: Create an S3 Bucket
echo "Creating S3 bucket: ${BUCKET_NAME}"
aws s3 mb s3://${BUCKET_NAME} --region ${REGION}

# Step 2: Create a Read-Only Policy
echo "Creating IAM policy: ${POLICY_NAME}"
POLICY_ARN=$(aws iam create-policy --policy-name ${POLICY_NAME} --policy-document file://${POLICY_FILE} --query 'Policy.Arn' --output text)

# Step 3: Create an IAM User & Attach the Policy
echo "Creating IAM user: ${USER_NAME}"
aws iam create-user --user-name ${USER_NAME}

echo "Attaching policy to user"
aws iam attach-user-policy --user-name ${USER_NAME} --policy-arn ${POLICY_ARN}

# Step 4: Create an Access & Secret Key
echo "Creating access key for user: ${USER_NAME}"
aws iam create-access-key --user-name ${USER_NAME}

echo "Setup completed successfully. Remember to securely store the generated access and secret keys."

# Cleanup the policy file
rm ${POLICY_FILE}
echo "Policy file ${POLICY_FILE} removed."