The Fuzzy Notion of Overfitting with GPT4 - What Does it Mean to Overfit on All Knowledge?

What could it possibly mean to overfit on 'all available knowledge'? The notion of overfitting for GPT4 is fuzzy. Does it still make sense?

With the emergence of huge language models like GPT4, the classical notion of overfitting becomes abstract, at least for me.

In traditional machine learning, we try to prevent the model from overfitting the training data. We look for a good balance between overfitting and underfitting to maintain the model's ability to generalize, beyond the training set. In short, so it can adapt to new data.

With the advent of enormous language models like GPT4, what could it possibly mean to overfit on 'all available knowledge'?

In this case, isn't overfitting what we're looking for?

Assuming we have an infinite training budget, one could consider that we want GPT-4 to "overfit" as much as possible the extent of human knowledge. If GPT4 overfits, wouldn't that be a sign that it captures a large part of the subtleties and nuances of human language?

The term overfitting itself is probably not very suitable for very large LMs, because even if it sees "the same information" several times, it will be with different nuances.

On closer reflection, we probably don't want to overfit on all this knowledge because there is a good chance that the information present is not reliable in its entirety.

Of course, my thinking remains wishful, but I find that it illustrates well how very large language models push the boundaries of traditional concepts in machine learning.