The Non-Determinism of OpenAI and Anthropic Models

OpenAI and Anthropic's models are not deterministic, even at temperature zero and with a fixed seed, and this can be a real problem.

How is it that some models are not deterministic under these conditions? This is a question I've been asking myself for some time now.

In manipulating models, I observed that with a fixed prompt and temperature at zero, the output was not always the same. At the time, I thought it was a bug, a rounding issue, or that there was deliberately some residual randomness for some unknown reason. I didn't really investigate the issue thoroughly.

But in some cases, it's quite essential to guarantee reproducible results, for example in a testing/experimentation tracking phase.

Temperature is the parameter that controls a dose of randomness and therefore the model's dose of creativity.

This "T" comes into play in the model's final layer, the softmax, to transform the output values into a probability distribution. When T tends towards 0, it acts as an amplifier and the highest probability tends towards 1, all others towards 0. In other words, this amounts to always selecting the most probable word (like an argmax).

Conversely, when T tends towards 1, it reduces differences between probabilities and the distribution becomes more uniform, thus giving the model the opportunity to vary its choice of words.

OpenAI and Anthropic's models, even with a temperature of 0, are not deterministic. What? A result that changes 1 out of 5 times can't be a rounding problem. It also can't be a bug shared by 2 different services.

Anthropic and OpenAI only provide vague answers on this subject.

Some have hypothesized that since GPT-4 would be a Mixture of Experts, it would have to choose an expert for each token, and this choice would depend on its availability. In other words, if it's already busy processing other requests, another expert takes over, thus causing a difference.

GPT-3.5 Turbo also isn't deterministic, so according to the preceding hypothesis it too would be a Mixture of Experts.

What is also interesting is that Mixtral models seem to be deterministic, even though they too are Mixtures of Experts...

So, MoE? Floating point operations? Designers' choice?