Most embedding models have linguistic bias

Most embedding models have linguistic bias. Whether for semantic search or LLM-powered QA (which is itself just semantic search with an LLM at the end of the chain), these techniques rely on creating embeddings - high-dimensional vectors capturing semantics. These embeddings, which can be seen as points in a space, are designed to be close together if they come from semantically similar sentences.

However, these models are often linguistically biased. Most models I've tested, whether via Hugging Face's sentence transformers or OpenAI's API, tend to bring texts in the same language closer together, even when more relevant texts exist in other languages.

Concretely, this linguistic bias means that a semantic search engine will tend to prioritize results in the same language as the query formulated by the user. This can be problematic in some cases but overall, it's not necessarily a bad thing. Suppose a user queries the engine with a legal question - since the law depends on the country, it would be relevant for them to get results in the language of their query, which probably corresponds to their country.

Where it gets tricky is when for example the user queries in French to ask for details on English law. Among the relevant documents, the French texts will likely be favored. This is where having an LLM at the end of the chain can compensate for this bias and strengthen the relevance of the response based on the documents but also on its own knowledge.