KubeCon + CloudNativeCon Europe 2021 Virtual

This talk presents the challenges that corporations have when they evaluate technologies for data pipelines automation or for MLOps workflows, this includes processes like ETL, ML model generation and ML model deployments. At the same time, this talk shows how open source technologies can help to reduce implementation time and infrastructure costs for MLOps systems. The talk also presents the advantages of using k3s as a lightweight Kubernetes for POCs, the lessons learned at work when trying to implement MLOps using Scikit Learn for ML Models and Apache Spark for ETL processes. As a complement for k3s the talk describes how Argo can replace or complement pipelines designed with Apache Airflow and how these types of technologies prepare companies for Edge Computing. At the end of the session the speaker presents a short demo showing a basic MLOps workflow implementation using k3s and Argo and showing how k3s and Argo interact with each other to deploy ML models ready to be used.