DVC
Open-source version control system for Data Science and Machine Learning projects. Git-like experience to organize your data, models, and experiments.
We’re thrilled to welcome the DVC Community to the lakeFS family. Keep updated on blog posts with our RSS Feed! We use cookies to improve your experience and understand how our site is used. Learn more in our Privacy Policy We provide short articles on common data science scenarios where DVC can help. Our example scenarios are not written to be run end-to-end like tutorials. For more hands-on experience with DVC, see Get Started. Even with all the success we've seen in machine learning, especially with deep learning and its applications in business, data scientists still lack best practices for organizing their projects and collaborating effectively. This is a critical challenge: while ML algorithms and methods are no longer tribal knowledge, they are still difficult to develop, reuse, and manage. If you store and process data files or datasets to produce other data or machine learning models, and you want to Choose a page from the navigation sidebar to the left. ✅ Check out our GitHub repositories: DVC give us a ⭐ if you like the project! We use cookies to improve your experience and understand how our site is used. Learn more in our Privacy Policy
Metaflow
Build and manage real-life ML, AI, and data science projects with Metaflow.
Open-source Metaflow makes it quick and easy to build and manage real-life ML, AI, and data science projects. Explore with notebooks, develop with Metaflow, and test and debug locally. Results are stored and tracked automatically for easy analysis. Break out from the confines of a laptop or a single notebook. Scale out easily to the cloud, utilizing GPUs, multiple cores, and multiple instances in parallel. Metaflow organizes the work for easy collaboration on the way. Deploy experiments to production with a single click without changing anything in the code. Make flows react to updating data and other events automatically. Get started easily on a laptop. When you are ready to scale, deploy the Metaflow stack on your cloud account or on-premise Kubernetes cluster. Metaflow integrates seamlessly with your existing infrastructure, security, and data governance policies. To get a taste of Metaflow in the cloud, try Metaflow Sandbox in the browser. Deploy on EKS and S3, or AWS Batch & AWS Step Functions. Deploy on AKS and Azure Blob Storage. Deploy on GKE and Google Cloud Storage. For maximum flexibility, deploy on a custom Kubernetes cluster. Metaflow was originally developed at Netflix to address the needs of developers and data scientists who work on demanding real-life ML, AI, and data projects. Netflix open-sourced Metaflow in 2019. Today, Metaflow is used by hundreds of companies across industries, powering diverse projects from state-of-the-art GenAI and compute vision to business-oriented data science, statistics, and operations research. Create flows incrementally step-by-step with the new spin command Build agentic systems with the new recursive and conditional steps Compose flows with reusable custom decorators Use uv to manage dependencies, from dev to cloud Setup the full Metaflow stack on your laptop with one click Checkpoint long-running model training and other tasks with the new @checkpoint decorator Configure flows freely with the new Config object New APIs allow you to run and deploy Metaflow in notebooks and scripts Learn about various patterns of scalable compute with Metaflow. Train and fine-tune large language models and other generative AI models on AWS Trainium. Build observable ML/AI systems with cards that update in real-time. Install dependencies from PyPI as well as Conda in your Metaflow steps. Connect to external services securely using the new @secrets decorator. Metaflow 2.9 allows you to trigger workflows based on real-time events. Apache Arrow and Metaflow.S3 make it easy to process data fast. Learn how to use Metaflow for demanding GPU tasks. Develop with Metaflow, deploy on your existing Apache Airflow servers. Deploy and operate Metaflow on GCP and all other m
DVC
Metaflow
DVC
Metaflow
Metaflow (1)
Only in DVC (4)
DVC
Metaflow
DVC
Metaflow
