The Role of Language in Reasoning and Conceptualization

Yann LeCun's recent statements about language not being necessary for reasoning are as fascinating as they are divisive. Among other things, he emphasizes that animals or children acquire basic knowledge through their senses. He adds that sight allows for collecting a wealth of information much faster than one could by reading a book. Language would ultimately be the cherry on top, but one would first need to build the cake.

However, even if we read fewer words per second than we see images, language seems to be very efficient in compressing concepts into relatively few symbols. Something that struck me when I started getting interested in Stable Diffusion and Imagen, the famous models that generate images in response to a prompt, is that language plays a predominant role.

Indeed, these models are composed of two sub-models: the first encodes the text of the prompt into embeddings, which are then passed to the second diffusion model (UNet) to generate an image. In Google's publication on Imagen, one can read that the researchers found, quite surprisingly, that increasing the number of parameters in the text encoder contributed much more to improving the quality of the generated image than increasing the number of parameters in the UNet.

From this empirical observation, one is tempted to say that even if language is not a priori necessary for reasoning, it still seems to simplify many things in the process of creation and conceptualization.

The boundaries between AI, philosophy, and neuroscience are sometimes thin, and these fields intersect and influence each other in fascinating ways.