Machine Learning in production- An overview

So this article started with a Google search a while back about ML and it was the percentage of ML models that made it to production

This actually got me interested. I mean, I do this exact thing for work. I work as a software developer in the data science team of an Ecomerce company.

So this write an article about the 10% of the models that make it to production and share some of the design principles.

So what we will do here is this:

• Look at the introduction into the problem and build the context of the problem and the idea of an AI product.
• Look at a particular use case for the credit risk assessment ML model and apply all that we learned to that use case in the next article(credit scoring use case).

How does ML Look like in Production?

So let’s start with what everyone thinks when they think about machine learning and artificial intelligence. I am sure it is below.

For anyone who is new to the industry, machine learning is the below:

• We define the problem statement
• We already have data
• We choose the right algorithm for our ml problem.
• Tune the model and we are done.

This is far from the truth. The question is does ML have value by itself or is the value generated from a business using the end product built?.

So let’s look at ways the ML model used in production:

1. ML model added to an existing product as a feature

An e-commerce site adding recommendations or improving recommendations. There is a product existing already.

2. An App Built to use the ML Model.

A bank wants to do a credit risk assessment of loan applicants. So the bank wanted an ML model build to solve this use case which can be used by their loan officers.

ML In Production: Challenges

We have a few engineering challenges and some non-engineering ones when it comes to ML in production. Let’s look at them in brief:

Leadership and focus:

So a company wants to work on machine learning but they might not be sure of what exactly they want to achieve.

Data:

• The core of machine learning is data, to get data required in a clean and consumable way.
• We know we need data but where do we get it from and how do we get it from even if the systems are internal.
• Everyone system has its own scheme, different types of storage, etc.

AI Development needed but disorganized teams

Data scientists build models, engineers build a software product, DevOps does deployment.

• You do not have data scientists thinking of scale and reliability while building models.
• Engineering doesn’t think of enough of machine learning while developing their products around ML code.
• DevOps just thinks of CI/CD.

So now that we know of some challenges, let’s look at a way we can use this knowledge to build AI products that make it to production.

ML Model vs AI Products

The fact is that the ML model is the engine that is the core that can be used independently.

Or you could build AI products(car).

To build products, you need to know things like the millage of the car, the top speed, 0–60, how many people it can hold(model metrics, performance, input data required, etc). But you also require the below:

• You need a firm chassis on which you can have other parts build(backend)
• You need a seat a steering wheel to drive the car(usage and UI)
• You need doors, windows, bonnet, etc.(UI)
• Fuel indicator, speedometer, and accessories(metrics and monitoring)

AI Product Design Metrics

AI product development needs cross-functional knowledge. Distributed systems, Machine learning, and DevOps. Let’s so let’s look at the core design concepts that are cross-functional that we can apply to AI products.

• Reliability — Have we considered how to handle software errors, hardcore faults, and human errors?
• Scalability — Do you know the load your system has to address? Number of users each day or month, number of searches, what do you do if there is more load to your system. What if you need support batch and real-time operations.
• Maintainability/Usability — Do we have enough application logs and metrics. This might be to and the model performance and handles application failures. Is the application built in a configurable way? This so that the operations team can take care of simple maintenance tasks.
• Model Monitoring and versioning- Can we monitor how the ML model is performing . ? Is it as expected and what is the degradation of the model over time? Are we considering the fact that we might need to have several versions of the model to be there in the production? Should have the ability to be able to switch versions?
• CI/CD — So consider this we have application code, we have data pipelines, and ml models and tools. Which ones must CI, which ones don’t?

AI Product Tech Template

We took a look at the car and some design principles, next step is to define the components in our car. Let’s do that.

We will look at individual components and apply them to our use case in the next article.

Summary:

We looked at ML in production, AI products, defined some design principles to consider, and different components to our AI product. We will look at an actual example i had at work in the next article(credit scoring use case)