Unmasking Confidential Data in ML Models with Membership Inference Attacks

(This article is a transcript in English of one of my french LinkedIn posts)

MIA, or Membership Inference Attack, is a technique that lets you determine whether specific data was used to train a ML model. It can be used for malicious purposes or, on the contrary, to audit cybersecurity.

In particular, when auditing a model, MIA can be used to check if confidential data was unknowingly used by the model.

The major challenge is that when you test an online AI service, you don't have access to the model's internal architecture or how it was trained. You're faced with a "black box": you feed it data and get back predictions, without knowing what's happening inside.

But how do we proceed?

ML models learn from a training dataset. Meanwhile, Data Scientists must ensure that these models don't overfit and can generalize to new data. A strong assumption used by MIA is that, despite precautions, these models still tend to be more confident about data they've seen during training.

For instance, for an image classification into 3 categories, if prediction probabilities are [99%, 0.5%, 0.5%], one could argue the model is "unusually" certain. Hence, it's likely that the image was part of the training data.

However, at this stage, we generally don't have access to these probabilities. So, we need to find a technique to indirectly estimate them.

To do this, we'll create our own models. Either we manage to gather information on the target architecture, or we "guess" it or at least create models with similar behavior. These exact or approximate clones are called "shadow models."

Now, we need data to train these shadow models. This is probably the most complex part since you either need access to data similar to the target model's or you must generate a synthetic dataset by iterating on the target model using specific techniques.

Lastly, we need another model known as the "attack" model. This takes the probabilities from the shadow model and will determine if the data was part of the target model's training set or not.

There are some reservations:
It's a relatively complex method with limited applicability. It's primarily effective on modest-sized models, supervised learning, or GANs. Moreover, recent criticisms of MIA suggest their efficiency largely depends on models' overfitting.

It's safe to say that they can't really be applied to LLMs right now.

But in the near future, who knows?"