Does no one care about Data Drift in LLMOps ?

(This article is a transcript in English of one of my french LinkedIn posts)

By the way, there is indeed a term to describe the life cycle management of language models: LLMOps (In the Ops family, following Dev and ML, meet the newest member, LLMOps).

But back to our question: I had this thought: why doesn't Data Drift seem to concern many when using LLMs? By Data Drift, I mean the evolution of input data (the prompts) over time, which could impact the model's performance. I searched scientific publications, blogs, content specialized in LLMOps... Not much to find.

It simply doesn't seem to be a topic.

Is it because this emerging discipline still lacks maturity and tools to tackle this issue? Admittedly, "classic" MLOps hasn't reached its full maturity either.

Or is it because LLMs are trained on such massive amounts of data that the risk of drift between production and training data is minimal? Therefore, it wouldn't affect the model's performance?

Or, is it because the input data (the prompts) are shaped and reshaped by the user until the desired result is obtained? So the drift of the prompts wouldn't be an issue.

However, even if we assume that drift doesn't affect LLMs much (which is debatable), monitoring it seems crucial in many cases...

Suppose users find a flaw in your production LLM, spread the word, and overnight, hundreds of them have fun generating malicious code with your service.

Without data drift monitoring, how can you detect that something is happening?

But then again, how to measure it effectively?

I guess one solution would be to regularly collect a representative sample of embeddings associated with user prompts, and then devise a clustering and divergence measurement solution based on that.

Assuming this works and is scalable, there are still data privacy concerns.

If you store the embeddings of the prompts, it means that you could, using certain techniques, roughly reconstruct the original prompts (though it's not that straightforward)…

I have so many questions!

In short, data drift with LLMs seems to be a blind spot even though its monitoring seems crucial to me. Between the lack of maturity of tools, the massive amount of training data, and user intervention on input data, several hypotheses might explain this "lack of interest."

I am very curious to hear your thoughts on this!

I'm also looking for resources on the topic