Configuring HTTPS/TLS Certificate Trust for External Repositories

Some external repositories may use certificates that are either self-signed or issued by non-public Certificate Authorities (CAs), making them untrusted by default. This guide provides instructions on configuring MLIS to recognize and trust these certificates. Note these steps are not required for external repositories such as huggingface.co or s3.amazonaws.com, which use valid SSL/TLS certificates issued by trusted certificate authorities (CAs) for their website.

Before You Start #

• Obtain the self-signed certificate or the certificate issued by the non-public CA from the external repository you want to trust.

How to Configure Custom Certificates for External Repository Access #

To add custom certificates to the MLIS controller and deployment pods, you have two options:

1. Manually create ConfigMaps: Add certificates by creating ConfigMaps that contain your certificates directly in the namespace where the MLIS controller is installed, as well as in the namespaces where deployments will be made.

2. Use Trust Manager for automated ConfigMap creation: Configure Trust Manager to automatically create and distribute a ConfigMap containing the necessary certificates across the desired namespaces.

For configuring Trust Manager, refer to the Trust Manager documentation to control which namespaces will automatically receive the ConfigMap.

Once the ConfigMap containing your certificates is ready, either manually or via Trust Manager, specify its name during the Helm installation process as shown below:

helm install mlis \
   --set trustedCAsConfigMap=<CONGIFGMAP_NAME> \
   --set 'global.imagePullSecrets[0].name=regcred' \
   --set 'global.imagePullSecrets[1].name=hpe-mlis-registry' \
   --set imageRegistry=hub.myenterpriselicense.hpe.com/hpe-mlis/<LOWERCASE_SKU> \
   --set defaultPassword=<CREATE_ADMIN_PASSWORD> \
   --values values.yaml \ # Required if you have custom configurations
   <SKU>_aioli-helm-chart_1.2.0.tgz

Once deployed, the certificates in the bundle will be automatically mounted into:

• The controller
• All newly created deployment pods

Example: Configuring MLIS to Trust a Self-Signed MinIO Certificate Using Trust Manager #

This example demonstrates how to configure MLIS to trust a self-signed certificate named minio-self-signed.crt for a MinIO server. It assumes you have already installed Trust Manager in your cluster.

1. Create a Secret for the Certificate

   Create a secret named minio-cert-secret in the cert-manager namespace that contains the self-signed certificate.

   kubectl create secret generic minio-cert-secret --from-file=./minio-self-signed.crt -n cert-manager

   Copy

   Example output:

   secret/minio-cert-secret created

   Copy

2. Verify the Secret’s Contents

   Use the following command to examine the contents of the minio-cert-secret:

   kubectl describe secret minio-cert-secret -n cert-manager

   Copy

   Example output:

   Name:         minio-cert-secret
   Namespace:    cert-manager
   Labels:       <none>
   Annotations:  <none>
   
   Type:  Opaque
   
   Data
   ====
   minio-self-signed.crt:  1302 bytes

   Copy

3. Create a Trust Manager Bundle Configuration

   Create a file named mlis-trust-manager-bundle.yaml with the following contents. The spec.target.configMap.key specifies the filename that will be created under the /etc/ssl/certs/ directory in the container. Use the useDefaultCAs: true option to specify that the bundle should include the system’s default trusted Certificate Authorities (CAs) along with any additional certificates you specify.

   apiVersion: trust.cert-manager.io/v1alpha1
   kind: Bundle
   metadata:
     name: mlis-trust-manager-bundle
   spec:
     sources:
     - useDefaultCAs: true
     - secret:
         name: minio-cert-secret
         key: minio-self-signed.crt
     target:
       configMap:
         key: "ca-certificates.crt"

   Copy

4. Apply the Trust Manager Bundle

   Apply the configuration to create the ConfigMap in the namespaces.

   kubectl apply -f mlis-trust-manager-bundle.yaml

   Copy

   Example output:

   bundle.trust.cert-manager.io/mlis-trust-manager-bundle created

   Copy

5. Configure Helm to Use the ConfigMap

   Specify the ConfigMap during the Helm installation:

   --set trustedCAsConfigMap=mlis-trust-manager-bundle

   Copy

Example: Configuring MLIS to Trust a Self-Signed MinIO Certificate Using a Manually Created ConfigMap #

1. Create the ConfigMap in the MLIS controller’s namespace, as well as in the namespaces where deployments will be made.

   kubectl create configmap my-cert-configmap --from-file=ca-certificates.crt=minio-self-signed.crt -n <MLIS_NAMESPACE>

   Copy

   kubectl create configmap my-cert-configmap --from-file=ca-certificates.crt=minio-self-signed.crt -n <DEPLOYMENT_NAMESPACE_1>
   kubectl create configmap my-cert-configmap --from-file=ca-certificates.crt=minio-self-signed.crt -n <DEPLOYMENT_NAMESPACE_2>
   ...
   kubectl create configmap my-cert-configmap --from-file=ca-certificates.crt=minio-self-signed.crt -n <DEPLOYMENT_NAMESPACE_N>

   Copy

   Note: To include public certificates along with your MinIO certificate, you can combine them into a single file before creating the ConfigMap:

   cat /etc/ssl/certs/ca-certificates.crt minio-self-signed.crt > combined-certs.crt

   Copy

   Then, create the ConfigMap with the combined certificates:

   kubectl create configmap my-cert-configmap --from-file=ca-certificates.crt=combined-certs.crt -n <YOUR_NAMESPACE>

   Copy

2. Configure Helm to Use the ConfigMap

   Specify the ConfigMap during the Helm installation:

   --set trustedCAsConfigMap=my-cert-configmap

   Copy

Verifying the Certificate in the MLIS Controller #

To ensure the certificate was mounted and is being recognized by the MLIS controller, perform the following verification steps:

1. Extract the first line of your certificate, excluding the BEGIN CERTIFICATE line:

   awk '/BEGIN CERTIFICATE/{found=1; next} found {print; exit}' minio-self-signed.crt

   Copy

   Example output:

   MIIDlDCCAnygAwIBAgIUFvnJdhuBfM6V94FG4KPEt3t68P0wDQYJKoZIhvcNAQEL

   Copy

2. Check that the certificate is present in the MLIS controller’s /etc/ssl/certs/ca-certificates.crt file:

   kubectl exec -it svc/aioli-master-service-aioli -- grep "MIIDlDCCAnygAwIBAgIUFvnJdhuBfM6V94FG4KPEt3t68P0wDQYJKoZIhvcNAQEL" /etc/ssl/certs/ca-certificates.crt

   Copy

   Example output:

   MIIDlDCCAnygAwIBAgIUFvnJdhuBfM6V94FG4KPEt3t68P0wDQYJKoZIhvcNAQEL

   Copy

Environment Variables #

Depending on the tool or application used to download models from external repositories that use self-signed or non-public CA certificates, you may need to set environment variables that specify the location of trusted certificates.

For example, if you’re using the kserve:storage-initializer container to download models from S3 storage, set the environment variable AWS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt in your packaged model or deployment’s environment settings. This example assumes ca-certificates.crt is the key used in your ConfigMap, though the actual value should match the key specified in your custom ConfigMap.

Note

For version kserve/storage-initializer:v0.11.2, only the AWS_CA_BUNDLE environment variable is required. For newer versions, you also need to set the CA_BUNDLE_CONFIGMAP_NAME variable to any non-empty value. This additional variable is necessary due to kserve’s custom CA bundle configuration requirements. For more information, refer to the Kserve storage-initializer configuration instructions.

Other common environment variables for specifying CA bundle locations include REQUESTS_CA_BUNDLE and SSL_CERT_FILE, depending on the application being used for model downloads.