STEP 5: DEPLOYMENT AND ITERATION IN CONTEXT
What is the monitoring and evaluation process and who is responsible for it?
Setting up a project’s monitoring and evaluation practices is key in ensuring appropriate thresholds and targets. Understanding what the threshold is for 'sufficient' accuracy is and establishing Key Performance indicators (KPIs) is crucial in the process. MLOps are also important in the deployment and maintenance of ML models. Stakeholders must determine whose responsibility it is for monitoring and evaluating the model against the defined thresholds and, once determined, they must establish what the mechanism for feedback and making adaptations is. Learning and feedback loops from users and buyers need to be actively incorporated into the design and iteration of the model while embedding contextual factors.
It is also important to acknowledge the difficulty of monitoring interventions at the outcome level as there is no ‘one-size-fits-all’ M&E practice. When designing the model, all KPIs and MLOps must be included and discussed with all stakeholders to decide who reports to whom. When testing, it is advisable to decide KPIs and DevOps principles for ML systems (MLOps) based on the feedback received by the users. It is also important to decide how users will participate in the testing (e.g. individual discussion, online, group sessions, feedback collection). Overall, KPIs must also be updated periodically, with flexibility as products might change and technologies might evolve.
See Step 4, Decision 1
Please find below a legend of what can be found within the framework:
📚Resources - e.g. reports, articles, and case studies
🛠Tools - e.g. guidelines, frameworks and scorecards
🔗Links - e.g. online platforms, videos, hubs and databases
❌Gap analysis - tools or resources are currently missing
👥 List of stakeholders which should be included in the specific decision point
-
📚MLOps tools and principles - Set of resources providing guidance on machine learning development. The website also contains a set of principles to establish best practices and tools to test, deploy, manage, and monitor ML models in real-world production
📚End-to-End Framework for Internal Algorithmic Auditing - Paper introducing a framework for algorithmic auditing that supports end-to-end AI system development. The proposed auditing framework is intended to contribute to closing the accountability gap in the development and deployment of large-scale AI systems by embedding a robust process to ensure audit integrity
-
📚 Continuous delivery and automation pipelines in machine learning - A resource on techniques for implementing and automating continuous integration (CI), continuous delivery (CD), and continuous training (CT) for ML systems. This document is for data scientists and ML engineers who want to apply MLOps
-
👥MEAL experts, program officers, in-country managers, software developers, technical experts, end users
❌M&E guidance on AI/ML policies and programs