ML Engineer vs AI Engineer: Who Should You Hire for Your AI Project?

It’s a classic startup story these days: hiring requests get framed around a solution (‘we need AI’) instead of the actual problem. That’s why so many companies end up stuck trying to figure out whether they actually need integration work or model training — and whether the right hire is an ML engineer vs AI engineer.

“The situation nowadays is similar to the early 2000s: back then, a business without a web presence risked falling behind. Now it’s the same — a company that doesn’t integrate AI risks being left behind,” says Nick Kliestov, CTO at ITExpert. The market has been swept by a wave of job postings with ambiguous titles, and companies are massively confused about who to hire.

This piece is a practical breakdown based on experience filling these roles, covering:

• the difference between ML and AI engineer
• what does an ML engineer do
• what does an AI engineer do
• which companies each role fits
• a checklist of criteria for choosing an ML engineer vs AI engineer
• how to assess candidates in an interview.

ML Engineer vs. AI Engineer: What’s the Difference?

Before the ChatGPT boom, the distinction was pretty simple:

• If a company needed to build something from scratch — actual R&D work — they hired a Data Scientist.

• If they needed to implement existing data science tools into a product, they hired an ML Engineer.

That distinction still holds up for classic machine learning, but once modern large language models (LLMs) came onto the scene, it stopped covering most business use cases.

LLMs from OpenAI, Anthropic, and Google are standardized, with clear, well-documented ways to integrate them. Think of it like a public API — similar to Google Maps, which you can just drop into a website and start using. You don’t need to understand how the model works under the hood any more than you’d need to understand cartography to embed a map. That’s exactly why, for most of these tasks, companies don’t need an ML Engineer or a Data Scientist — they need something new: an AI Engineer. Someone who can quickly and skillfully “plug in the map” instead of drawing one from scratch.

Meet the Classic ML Engineer: Frameworks, Libraries, Training from Scratch

The classic ML Engineer profile grew up around a specific toolkit: frameworks like TensorFlow and Keras, computer vision libraries like OpenCV, and text-processing libraries like BERT.

Anything that wasn’t covered by an out-of-the-box library had to be configured by hand — and that is exactly where the real expertise came in. It’s the kind of know-how you build up over years of working hands-on with these tools, and you can spot that kind of specialist a mile away just by scanning their CV for that list of technologies.

Important caveat: none of this means the classic ML Engineer is obsolete. Tasks that require a model trained specifically on a client’s data — scoring, forecasting, recommendation systems, custom computer vision — haven’t gone anywhere, and that classic expertise is still absolutely critical there.

However, with the rise of powerful, ready-to-use models, a different — and much bigger — category of work has emerged: clients don’t want a model built from scratch anymore. They want a working product built on top of an existing LLM: a chatbot, an assistant, document workflow automation, an agent.

Welcome the New Profile: The AI Integrator (What’s Now Called an “AI Engineer”)

Most of the time, this specialist’s job isn’t about training a model at all. It comes down to integrating public APIs from LLM providers into the company’s existing IT systems.

The best candidate for this role is an experienced Python developer (since most ready-made tools are built in Python) who’s spent the last few years hands-on with AI integrations specifically.

To hire the right AI Engineer — and separate that candidate from someone who’s just “heard something about AI” — ITExpert looks for three specific markers on a resume:

• Vector databases — where a company’s texts and documents get stored so an AI tool can search them. This ties directly into RAG (Retrieval Augmented Generation).

• MCP (Model Context Protocol) — if the vector database is the agent’s “memory,” MCP is its “hands.” It’s what lets you plug in functions from the company’s CRM, ERP, or ecommerce platform, so the AI agent can actually do things — add a product to a cart, pick a color and quantity, complete a purchase — on its own.

• The LangChain, LangGraph, and LangSmith family — the go-to frameworks for building fast, straightforward LLM integrations into a product.

*The term “AI engineer” sometimes means a broader role — for example, in the definition used by researcher Chip Huyen (author of the book AI Engineering), it covers not just integrating ready-made APIs, but also fully building apps on top of foundation models through prompts and guardrails (a set of control mechanisms that limit a model’s behavior — what it can and can’t answer, do, or generate), quality evaluation, and agent orchestration as separate areas of work.

Which Companies Need an ML Engineer, and Which Need an AI Engineer?

Whether you need an ML engineer or an AI engineer doesn’t come down to company size or how trendy the technology is. Instead, it comes down to the task at hand. Are you building your model from scratch on your own data, or integrating ready-made AI tools into a product? Here’s a breakdown of which companies and tasks lean toward each role.

A Startup with a 6-Month Idea → AI Integrator

If a startup isn’t positioning itself as building fundamentally new technology — say, its own LLM — it doesn’t need someone with a deep understanding of how LLMs work behind the scenes. That’s the route China’s DeepSeek took, for instance, when it released its model. But most startups go a different route: they bolt AI onto an existing idea that had nothing to do with AI in the first place.

What you need is an AI integrator — someone who works with frameworks like LangChain and can spin up a prototype or MVP in a few weeks. These frameworks are built for exactly that: quick prototypes and demos that show off what the technology can do. Whether a candidate knows their way around them is a solid signal of whether they’re the right fit for a startup.

A Company Wants Its Own Language Model → A Classic ML Engineer

If the goal is building your small language model (SLM), you need a classic ML specialist. These specialists usually cut their teeth on word2vec-style libraries that turn text into vectors, so they’ve got a solid handle on word vector spaces and how language models actually get built. Look for terms like “model distillation” and “fine-tuning on local data” in their experience.

One important thing: you can’t “fine-tune” a closed model like ChatGPT. That requires open-source models — Llama, for example. If a candidate has experience with models like that, it’s a good sign they didn’t just plug in ready-made APIs — they actually adapted models for specific use cases.

Enterprise with Large Amounts of Data → It Depends on Volume, Confidentiality, and the Nature of the Data

You’d think the more data a company has, the more obvious it is that they need an ML engineer to train a model on it. Let’s see if that actually holds up.

Most companies just don’t have the data volume to fully train an LLM. If a company has confidential data that can’t be sent to external APIs — a bank, say — the go-to solution is usually a RAG approach: pairing local search over a vector database with an existing LLM. This works well even for businesses handling sensitive data, since the company can run a local open-source LLM and “blend in” knowledge from local storage.

Whether that requires an ML Engineer really depends on scale. A small online store with a handful of product descriptions, or a support center with a modest ticket history, can build RAG without a dedicated ML specialist. But at national scale — think a large bank with terabytes of data — building a solid RAG system at that level absolutely requires an ML Engineer.

There’s one more wrinkle here: if a business can’t use public APIs for security reasons and has to run an LLM locally, that means bringing in a separate infrastructure role — a data center admin for language models. Why? Because these systems run on GPUs, not standard servers. So at enterprise scale, you’re often not looking at two roles but three: an AI Integrator, an ML Engineer, and an infrastructure specialist for GPU clusters.

At the end of the day, data volume alone doesn’t tell you much — what really matters is the scale of the RAG system and how strict the confidentiality requirements are.

Criteria for Choosing an ML Engineer vs an AI Engineer: A Quick Checklist

Before you write a job posting, it’s worth answering three questions:

#1. Are you building something fundamentally new — a proprietary model, custom training — or are you integrating a ready-made model (think GPT, Claude, or Gemini) into your product?

#2. Does your data — in terms of volume or confidentiality — require local deployment and a complex RAG system, rather than just calling a public API directly?

#3. What’s the actual scale of the task — a quick prototype you need in a few weeks, or enterprise-level infrastructure handling terabytes of data?

If your product already involves your predictive models or building something from the ground up, it’s worth digging into how to hire a Data Scientist (Machine Learning), and how that role stacks up against the two engineering profiles above.

And if you’re having trouble pinning down the right profile on your own — that’s totally normal. This market is so young that even seasoned CEOs and recruiting teams mix these roles up all the time. In cases like that, it’s worth reaching out to a company that specializes in recruiting AI/ML talent. A industry-specific recruiter can usually figure out in a 15–20 minute conversation which profile — ML or AI Engineer — actually solves your business problem.

How to Assess a Candidate During an Interview

There are already some clear markers that tell these profiles apart:

• An AI integrator will bring up RAG and MCP.
• A classic ML engineer will bring up fine-tuning and model distillation*.

*This is a model-compression technique where a large “teacher” model trains a smaller “student” model to mimic the teacher’s behavior, instead of training that smaller model from scratch on raw data.

From there, candidate evaluation gets more nuanced. So we’ve broken it down into three levels: a filter question (weeds out the wrong fit in two minutes), a depth question (shows you how deep the candidate’s knowledge actually goes), and a practical task (takes more time, but it’s the most reliable method by far).

Assessing an AI Integrator (AI Engineer)

Level 1 — filter question

“Describe how you built RAG for a specific project: where did the data come from, which vector database did you choose, and why that one?”

This question immediately splits candidates into two groups. The first group are those who genuinely built RAG in production: they name a specific database (Pinecone, Weaviate, Chroma, pgvector, Qdrant) and explain the choice through constraints — budget, the need for self-hosting due to data confidentiality, or search speed at a specific document volume.

The second group are those who followed a tutorial. Their answer sounds like “we plugged in a vector database, it stores embeddings” — correct, but without a single detail you couldn’t read in the first paragraph of the documentation.

🚩 Red flag: the candidate can’t name which embedding model they used (OpenAI text-embedding-3, sentence-transformers, Cohere) — this means RAG was a black box for them, at the level of “copied the code from an example.”

Level 2 — depth question

“Imagine RAG is returning irrelevant chunks of text, even though the correct information is definitely in the knowledge base. What causes would you check first?”

This distinguishes someone who really debugged RAG in production from someone who only ran it on test data. A good answer covers several levels:

• chunk size and chunking strategy (maybe the text chunks are too large or split mid-sentence)

• embedding quality for the specific language or domain 

• the similarity metric (cosine vs. dot product)

• whether the candidate even checked relevance through reranking, or just relied on the top-k results from vector search without additional filtering.

“What happens if the agent gets an error, a timeout, or incomplete data from an MCP tool? How do you handle that at the architecture level?”

An experienced candidate will bring up retry logic with exponential backoff, fallback scenarios (for example, degrading to an answer without the tool, with a warning to the user), and, most importantly, validating the model’s response before passing the result on: whether the model is hallucinating a call to a nonexistent tool, and whether it’s correctly parsing the JSON schema.

“Compare LangChain/LangGraph with calling the model’s API directly without a framework — when would you choose each approach, and why?”

A good answer acknowledges the trade-off in both directions: a framework speeds up MVP development and provides ready-made patterns for complex agentic flows (LangGraph is especially useful for multi-step state machines), but it adds abstraction, makes debugging harder, and makes the project dependent on someone else’s API updates. An experienced integrator will say that for a simple chatbot, they’d write directly against the Anthropic or OpenAI SDK, and would only reach for a framework when the orchestration complexity justifies the overhead.

Level 3 — practical task (15–20 minutes)

“We have a database of 5,000 support documents. We need an AI agent to answer customers and, if needed, create a ticket in an external system. Sketch the architecture in Miro in 10 minutes.”

What to look for:

• Does the candidate separate the vector database for document search from MCP or function calling for creating the ticket? This reflects a basic understanding of RAG vs. the agentic approach.

• Do they mention guardrails — what happens if the agent tries to create a ticket with incomplete data, and whether there’s human-in-the-loop for critical actions.

• Do they build in monitoring and logging — experienced integrators always think about how to debug the system after launch, not just how it works in the ideal scenario.

🚩 An empty diagram with a single “LLM API” block and no surrounding detail is a sign that the person has never designed such a system themselves, and has only ever followed ready-made instructions.

Assessing a Classic ML Engineer

Level 1 — filter question

“Tell me about a time you fine-tuned an open-source model. What data did you use, and how did you evaluate the result?”

What matters here isn’t the mere mention of fine-tuning, but the details: the size and source of the dataset, whether the data was cleaned, and which evaluation metrics were used (perplexity, BLEU/ROUGE for text generation, F1 for classification, or human evaluation via A/B testing).

🚩 Red flag: the candidate says “we fine-tuned the model on our data, and it got better” and can’t specify what “better” actually means.

Level 2 — depth question

“What’s the difference between model distillation and quantization, and when have you used each approach?”

This is a question without a template answer, which makes it hard to prepare for in advance. A technically sound answer: distillation is training a smaller student model to imitate the behavior of a larger teacher model (reduces size, but requires retraining), while quantization is reducing the precision of the model’s weights (for example, from FP32 to INT8) without changing the architecture, which is faster and cheaper but risks losing quality on complex tasks.

A candidate who has actually done this will describe a specific trade-off from their project — for example, “we chose quantization because we didn’t have the resources for retraining, and we needed to reduce latency for a mobile app.”

“What metrics did you use to prove to the business that a custom model was better than just calling GPT with a good prompt?”

This checks not just technical competence but also an understanding of business context. A classic ML engineer needs to understand that a custom solution only justifies its higher development and maintenance cost if there’s a measurable gain — in accuracy, latency, inference cost at scale, or the ability to work offline with confidential data.

If a candidate can’t give a concrete comparison, their entire fine-tuning experience was probably limited to training exercises without production deployment, where such numbers are always calculated.

“What kinds of data leakage have you caught while training models, and how did you detect them?”

Data leakage is a specific problem that only people who’ve actually gone through the full model-validation cycle will recognize. A good answer includes a concrete example — for instance, a time-based feature ended up in both the training and test sets, or duplicate records were split between the train/test split. The absence of any example from real work is a sign that the candidate never brought a model to production quality, where such mistakes always surface during validation.

Level 3 — practical task (20–30 minutes)

If the team has the technical resources, give the candidate a small dataset (for example, a few thousand lines of text) and ask them to spend 30 minutes showing how they’d approach fine-tuning a small open-source model for a specific classification or generation task. Don’t expect working code in that time — watch the process: does the candidate first analyze the data distribution, or jump straight into writing a training loop; do they mention a baseline model for comparison; do they plan for cross-validation or at least a train/val/test split instead of training on all the data at once.

Before posting a job listing, it’s worth honestly asking yourself whether you’re building something new or integrating something that already exists, and what scale of data and confidentiality you’re working with. Then hiring stops being a lottery, and the job posting won’t be asking for “5+ years of experience” with a technology that isn’t even three years old yet.