Cosine similarity: an Intuitive and simple explanation

5 minute read

Cosine similarity is one of the most important metric in Natural Language Processing.
It is most often used to quantify a distance between two embeddings (or 2 vectors).
As embeddings are supposed to gather semantic information, the cosine similarity between two embeddings becomes somewhat a semantic distance.

If you are not familiar with this metric in ML and you just heard of the cosine from high school, you may wonder what the hell the adjacent side over the hypotenuse on a right triangle has to do with vectors and similarities :D . You'll see in this article that the cosine serves different purposes.

As always in Machine Learning and especially in Linear Algebra, it's always good to use simple examples in order to understand concepts, generally, you'll be able to generalize the reasoning afterwards. In Linear algebra, whenever it's possible, I recommand you to build your inuition in 2 dimensions first, simply because you can plot 2D vectors on a graph so you can visualize what's going on.

The question we are going to answer is "what is the cosine of two vectors on 2D ?", but before let met introduce you the idea behind the cosine measure. In fact, it's a very simple concept, you just want to know if 2 vectors point towards the same direction.
Indeed, if it's the case well you can say that the vectors are very similar, otherwise if they are perpendicular, they are very different.

Before considering the cosine itself, an intuitive way to determine if 2 vectors points to the same direction is to measure the angle between them.

first_examples

Let's examine these cases:

  • in the first example, the vectors u and v point towards the same direction, the angle between them is 0°.
    the vectors can't be more similar, in this case, the similarity is equal to 1.
  • in the second example, the vectors u and v are perpendicular, they form an angle of 90°, they can't be more different than that. In this case the similarity is equal to 0
  • in the last example. the angle between u and v is a bit less than 45°. In this case the similarity lays between 0 and 1 (the result is quite close to 1 though.)

By looking at the first example, we should realize that if we are only interested in directions then the magnitude of the vectors don't matter. So why don't we just get rid of the magnitudes and see if it doesn't simplify the problem ? To do so, it's very simple, we just have to divide the vectors by their respective norms so they become unit vectors.

example_2

  • In the first example, Now u and v have the same magnitude and point towards the same direction, they are identical. Note that if we were to project v on u the magnitude of the projected vector would be the same as the magnitude of v which is 1.

  • In the second example, if we were to project u on v, the projected vector would have a magnitude of 0.

  • In the third example, the projection of u on v would have a magnitude of +/- 0.89.

Wait a minute, instead of calculating an angle between two vectors, can't we just calculate a projection? The answer is yes :) and this is exactly what is a cosine but don't tell anybody.

cosine_sine

So here is the key takeway: measuring the angle between the vectors is closely related to measuring the magnitude of the projected vector (if u and v are unit vectors). You can visualize in the animation bellow that the magnitude of the projection decreases as the angle between the vectors increases.

Note: as long as you are working with unit vectors, it doesn't matter if you project u on v or v on u, the result is the same.

quadrant

So in fact the cosine similarity is all about calculating the projection of a vector on another when these have a magnitude of 1. Luckily it's very easy to calculate it, it's just a dot product.
Please read my blog post about projections if you don't feel confortable with this assertion.

Note: $P_v(u)$ stands for projection of u on v.

You can always make a vector have a magnitude of 1 by dividing it by its norm.
so the formula that always work to calculate a cosine it the following one.

cosine_formula

In fact, you can see a cosine as a normalized version of a dot product.

Let's get more concrete with a simple example.

Up until now, theses vectors remained somewhat abstract. Now suppose that you have two documents u and v, for each one you want to know if it deals with cats or dogs. So for each document you'll count the number of repetitions of the words "cat" and "dog".

example_with_words

  • In the first example, on document u, the frequency of cat is 2 while the frequency of dog is 0, thus we can say that the document is all about cats. On document v it's also all about cats with a frequency of 4.
    In that regards both documents are similar in the sense that the topic is the same.In this case, the similarity is equal to 1. The word cat could have a frequency of 900, it won't change the fact that it's all about cats, this is why the magnitude of the vector doesn't matter, we only need the normalized version of it or in other words, we only need the direction of the vectors.

  • In the second example, u is all about dogs while v is all about cats, the documents can't be any different thus the similarity is 0, this is the case where the vectors are perpendicular.

  • In the third example, u is mostly about cats and a bit about dogs while v is all about cats. The documents are still a lot similar, the value of the similarity if you were to compute the cosine is 0.89.

By now, I hope you can get a more pragmatic sense of what having the same direction for two vectors really means. In fact it means that the vectors (or the documents if you want to be more concrete) have similar ratios or distribution across the features.

That's it for this small article folks :) , I hope it helped you getting a sense of what the cosine is and why we use it as a measure of similarity in NLP.


You May Also Enjoy