Blob/Object Storage

You can enable blog/object storage for HPE Machine Learning Data Management by updating the pachd.storage section in your Helm chart. The necessary configuration options depend on your chosen blob storage provider.

Before You Start #

• You should be familiar with the storageURL options in goCDK related to your blob storage provider.
  • AWS S3
  • Azure Blob
  • GCS Blob
• You should have the necessary credentials and permissions to create a bucket or container in your blob storage provider.

How to Set Up Blob Storage #

1. Navigate to your values.yaml file or obtain your current Helm values.yaml overrides:

   helm get values pachyderm  > values.yaml

   Copy
2. Add the following pachd.storage fields to your values.yaml file:

   pachd:
     storage:
       gocdkEnabled: true 
       storageURL: "s3://my-bucket" or "gs://my-bucket" or "azure://my-container"

   Copy
3. Update the storageURL to include provider-specific configuration options as needed; for options, see the related goCDK packages.
   • AWS S3
   • Azure Blob
   • GCS Blob
4. Save your changes and upgrade your cluster:

   helm upgrade pachyderm pachyderm/pachyderm -f values.yaml

   Copy

Limitations #

Some configuration settings such as verifySSL may not be passable via the storageURL as query parameters. In such cases, you can use the pachd.storage section to set these options.