A big part of building and maintaining a DAG of pipelines is understanding how to explore the resources, input/output data, and logs associated with your pipelines. Doing so will enable you to trouble-shoot issues, validate resource creation, and get a better understanding of how your pipelines are processing data.
Congratulations! You’ve successfully created a DAG of pipelines that process video files into a collage. However, we’ve only just scratched the surface of what you can do with HPE Machine Learning Data Management. Now that you have a working pipeline, try out some of these commands to explore all of the details associated with the DAG.
For a comprehensive list of operations, check out the Build DAGs section of the documentation or browse the Command Library.
How would you update the glob pattern in the video converter pipeline spec (video_mp4_converter.yaml) to only process video files in the raw_videos_and_images repo? That would enable you to reduce the complexity of the user code in def process_video_files and make the pipeline more efficient.
Congratulations!
You've successfully completed the Explore Resources, Data, & Logs.