Release Notes Highlights for MLDM

2.10.0

January 1, 0001

  • Feature: For enterprise customers using MLDM (Pachyderm) & MLDE (Determined) in a combined cluster environment, the MLDE Notebooks now include support for the Jupyter Pachyderm extension by default.
  • Feature: You can now set up and maintain metadata on your HPE Machine Learning Data Management artifacts. This includes clusters, projects, repos, branches, and commits.
  • Enhancement: The blog storage configuration attribute GOCDK_ENABLED is now set to true by default in your Helm chart values; in 2.11.0 the option to disable it will be removed.
  • Enhancement: The Console UI has undergone several improvements, including:
    • Improved file browsing experience
    • Improved DAG visualizations
      • Interactive DAG edge highlighting
      • Distinguishing colors and/or patterns based on pipeline types
      • Enhancing the ability to understand more about connections by their edges (like joins)
      • Visual indications of parallelism for pipelines when spec calls for it and when running
    • Pipeline and repo table paging

2.10.1

January 1, 0001

  • Enhancement: Editing Commit and Repo Metadata is now available in Console.

2.10.4

January 1, 0001

  • Enhancement: Editing Project and Cluster Metadata is now available in Console.
  • Enhancement: pachctl is now available in Determined notebooks.
  • Fix: Resolve an unexpected file loading error when opening multiple files in the Jupyter extension explore view.
  • Fix: Pachyderm Clients are now correctly loaded into Determined notebooks.

2.10.6

January 1, 0001

  • Feature: You can now create pipelines in Console using jsonnet-compatible pipeline templates.
  • Enhancement: Console now displays more information in the side panel for repos and pipelines, including the pipeline’s active workers and parallelism spec.
  • Enhancement: Several web-responsive improvements have been made to Console: the side-panel now supports responsive sizing, and the DAG view side panel tab headers become scrollable on small screen sizes.
  • Fix: Action buttons in the sidebar now become dropdown ellipsis menus when resizing the sidebar to a narrower size.
  • Fix: When trying to download a single file, the file browser will download the file instead of opening the file in a new tab
  • Fix: Correctly configured custom TLS certificates in Console.
  • Fix: Improved our consistent hashing algorithm’s use of locking.

2.11.0

January 1, 0001

  • Feature: Users can now manage metadata (as key:value pairs) in Console for projects and repositories from the User Metadata tab of the details side panel.
  • Enhancement: Projects, pipelines, branches, and commits now include the following dervied metadata by default: created_at, created_by, updated_at, and updated_by.
  • Feature: Pre-built Jsonnet templates are now available in Console when creating a pipeline:
    • Snowflake Integration: Creates a cron pipeline that can execute a query against a Snowflake database and return the results in a single output file.
    • Hugging Face Downloader: Creates a cron pipeline to download datasets or models from huggingface on demand.
  • Enhancement: Several enhancements have been made to improve the integration between HPE Machine Learning Data Management and Machine Learning Development Environment.
  • Feature: The Pachyderm SDK now has an extras package cdr that you can install (pip install pachyderm_sdk[cdr]) to make use of Common Data Refs (CDRs) in your user code. CDRs improve performance and speed by downloading version-controlled data directly from HPE Machine Learning Data Management’s underlying Object Storage bucket and caching that data locally on your machine, allowing datasets to be assembled entirely locally and incrementally updated.
  • Security: The HPE Machine Learning Data Management repository is now available at Iron Bank, a hardened container image repository owned and maintained by the U.S. Department of Defense (DoD) that supports the end-to-end lifecycle for modern software development. If you plan to download and install from Iron Bank, please reach out to ai-support@hpe.com or your Customer Success Engineer for assistance.
  • Notice: The gocdk_enabled attribute has been removed from the Helm Chart Values as it is now the default object storage driver.

2.5.5

January 1, 0001

  • Fix: Resolved an issue with “Get File URL” when authentication is enabled by modifying its functionality to operate on file sets instead of commits, leveraging the capability-based authentication of the file set API.
  • Fix: Resolved an issue where input files from an s3 were being downloaded by the worker’s storage container, causing worker pods to be evicted due to disk pressure. Now, the input files from s3 are not downloaded.
  • Enhancement: Previously, if a user turned off authentication before upgrading, the auth_tokens table would not get migrated. Now, if a user turns off authentication before upgrading, the auth_tokens table will still get migrated.

2.5.6

January 1, 0001

  • Fix: Corrected “Get File URL” functionality by removing leading slashes from output path and restoring support for output path prefixes that was available prior to 2.5.0

2.5.7

January 1, 0001

  • Enhancement: We now wrap database calls in retries to catch connection flakiness.

2.5.8

January 1, 0001

  • Security: Upgraded Go to 1.20.6 from 1.20.5
  • Security: Added security enhancements to prevent HTTP/2 Stream Cancellation Attacks
  • Enhancement: Added logs to 2.5.0 migration

2.6.0

January 1, 0001

  • Feature: Datum Batching is now available. Datum Batching is a performance optimization process that enables processing multiple datums sequentially.
  • Feature: The JupyterLab Pipeline Extension (PPS Extension) is now available, allowing users to push notebook code directly into a pipeline to create and run it. This feature is in Alpha, so we encourage you to share your feedback with us as you use it.
  • Enhancement: New RBAC roles have been added to Projects: ProjectViewerRole, ProjectWriterRole, ProjectOwnerRole, and ProjectCreatorRole. You can read about the roles here.
  • Enhancement: The Console UI has undergone some substantial improvements, including a revamped file browser and more detailed information about pipeline and job performance.
  • Enhancement: The Documentation site has undergone a substantial information architecture overhaul, making it easier to find the information you need. Content is now stored in top-level folders that follow the natural progression of learning about and using HPE Machine Learning Data Management.