09-19, 11:20–11:55 (Europe/Amsterdam), Van Gogh
The talk will cover the core ML Monitoring Flow necessary to maintain and maximize the business impact of models deployed to production. We will focus on the three main steps of the Flow: Performance monitoring, Root Cause Analysis, and Issue resolution. In the performance monitoring part, we will cover the two core algorithms that allow us to estimate the predictive performance without ground truth: Confidence-Based Performance Estimation (CBPE) and Direct Loss Estimation (DLE). The Root Cause Analysis part will go over various Drift Detection algorithms, focusing especially on multivariate drift detection and linking the drop in performance to drift signals. In the issue resolution part, we will briefly cover the typical steps to fix ML failure and their applicability and limitations.
The talk will focus on the three main steps of the Monitoring Flow: Performance monitoring, Root Cause Analysis, and Issue resolution. In the performance monitoring part, we will cover the two core algorithms that allow us to estimate the predictive performance without ground truth: Confidence-Based Performance Estimation (CBPE) and Direct Loss Estimation (DLE). The Root Cause Analysis part will go over various Drift Detection algorithms, focusing especially on multivariate drift detection and linking the drop in performance to drift signals. In the issue resolution part, we will briefly cover the typical steps to fix ML failure and their applicability and limitations.
Wojtek Kuberski is an AI professional and entrepreneur with a master's in AI from KU Leuven. He co-founded NannyML, an OSS in Python for ML monitoring and Post-Deployment Data Science. At NannyML, he leads the research and product teams as the CTO, contributing to novel algorithms in model monitoring.