Understanding projections - They are everywhere in Machine Learning !
These days I felt like writing an article about projections. Simply because I noticed that projections are one of these mathematical tools that people daily use without even realizing it.
I'll try to make things clear by going step by step but I assume you have some remembrance of high school level maths and some basic notions of linear algebra. Hopefully this article won't be to difficult to read.
The problem is that you are not exactly sure where to put the mattress. That's where projections come into action.
In a sense, yo can see it as measuring the shadow of the leaning pole, caused by the sunlight that casts rays perpendicular to the ground. That's what we call an orthogonal projection.
Before jumping into the problem, I would like to be clear about some basic notations.
let a be a data point that has two coordinates x=3 and y=4.
then, knowing these coordinates, the data point is fully defined and you can put a(3,4) on the graph.
But the same coordinates are also useful to define not the data point anymore but the vector.
You can differentiate them with the little arrow above the a.
These two guys are defined the same way but they are not exactly the same mathematical object.
Firstly, this is interesting to note that a(3,4) is sufficient to define the object a as a vector because we are working with position vectors and such vectors always starts at the origin (0,0). So when we write a(3,4) we implicitly say that this vector starts from (0,0) and goes to the point (3,4).
There are several ways you can intepret vectors but for now let's just stick with the basic way to see it:
It's an object that has a magnitude and points towards a direction.
So we can use this vector a(3,4) to model our leaning pole.
Also note that i is the unit vector that points towards the point (1,0).
It is called a unit vector because its norm or magnitude is 1.
So here is to problem, knowing that the leaning pole's height is 5m and that it points towards the point (3,4), what is the distance d ?
the answer is quite immediate with a little bit of high school math.
the answer is $d = h\cos \theta$
- also h is just the magnitude of the vector a which can be written |a|
- $\cos \theta$ can be written using the dot product between a and i.
generally, you can see a dot product being written <a, i> or $a^T \cdot i$ or even just a.i.
I'will use the linear algebra notation which is $a^T \cdot i$
So the distance d can now simply be written:
And now you can see why it's interesting to take a unit vector, because the final expression becomes so simple ! The projection of a on i is just the dot product of a and i
And as you might have guessed (because the example is actually trivial) the result is 3
Note that the dot product of two vector is always a scalar. Interestingly in french the dot product is called a "produit scalaire" because it is the product of two vectors that returns a scalar.
So we just have to put the mattress on the x axis (on the ground) and put it 3m away from the leaning pole so this gentleman can safely jump.
Vector projection of a vector
Now, sometimes we don't really care about just having a scalar as a final output.
Instead we want the result as a vector which means having the coordinates of the projected point.
To do so, we can simply associate the scalar d with the direction of i.
This is simply done with the product d.i
now you get the projection as a vector (you can check on the graph that the answer is correct).
Basically what we have done is that we transformed a magnitude into a vector by giving this magnitude a direction.
This operation is called an outer product, like a dot product, an outer product involves two vectors.
In our case d can be seen as a 1x1 vector.
Now things are becoming interesting !
What if we didn't have only one vector to project onto u but 2 and we wanted to do all the projection with a single operation ? This is what matrices are made for !
Let's get back to the original graph with an extra vector b
We can then stack the vectors into a matrix called A
This way, we can perform the dot product with u for each line of the matrix (for each vector).
We end up with the projection of a and b onto u (i becomes u because it's just more handy to have a unit vector called u). These are wrapped up in the output vector. As seen previously these scalars 3 and 4 are the magnitudes of the projections.
Now let's give those magnitudes a direction with the outer product so they can become vectors.
As a conclusion, if u is a unit vector these simple formulas can get vectors projected !
- for a vector a onto u : $a \cdot u \cdot u^T$
- for a matrix A onto u : $A \cdot u \cdot u^T$
Note if the vector you are projecting on is not normalized, you can normalize it by dividing it with its norm !
Now let's take an example where the vector you are projecting on is less trivial and doesn't have a magnitude of 1.
First, let's normalize this bad boy v.
We'll now use u which points towards the same direction than v but has a magnitude of 1.
The projection of a on u is written $p_u(a)$
We can see on the graph that the projected vector is indeed (2.5, 2.5)
Projections are all over the place in ML, especially in the linear models like PCA or Linear Regression. Previously what I've explained in that a matrix A can be projected on a vector u just with a matrix multiplication A.u.
Does it ring a bell ?
If it doesn't let's take a closer look at how we can interpret A.u.
Remember that we can interpret a matrix as a stacking of data points/vectors (rows) or a stacking of features (columns).
A.u can be written like so
This can now be read as a linear combination of the columns. Everytime we multiply a matrix with a vector, we are calculating a weighted sum of the columns, the weights being the coefficients in the vector.
Now this should explain a lot of things as linear combinations are ubiquitous in ML.
Indeed, think about PCA where we are looking for the linear combination of the features/columns that maximizes variance. In fact this linear combination X.u is just the projection of X on u. If we rephrase it, we are looking for a vector u so that the projection of X on u has the highest possible variance.
In a sense, we could say that linear combinations and projections are "cousins".
That's all for today folks !
