From Classic ML to LLMs: Similar Development Trajectories?

It's interesting to see how certain patterns in Machine Learning seem to recur, including with LLMs:

Classic ML:

• Step 1: The nature of the model matters.
• Step 2: The quantity of data matters.
• Step 3: Feature engineering matters, you have to find the magic feature.
• Step 4: The quality of the data matters.
• Step 5: No point trying to get an exceptional model, it's more interesting to have an ensemble of average models.

LLMs:

• Step 1: The number of parameters matters.
• Step 2: The quantity of data matters.
• Step 3: The length of the context window matters.
• Step 4: The quality of the data matters.
• Step 5: No point trying to get an exceptional model, it's more interesting to have an ensemble of specialized models (Mixture of Experts).

Are we rediscovering the same best practices viewed from a different angle every time? Or is this rather a natural evolution of priorities as ML models develop? That is, first emphasize the model itself, then the quality of the data, and finally ensemble approaches.

In any case, two principles seem universal:

The first: "Garbage doesn't turn into gold, no matter how much math you throw at it" (I believe that's a quote from Cassie Kozyrkov).

The second: united we stand. With tabular data, the star algorithms come from ensemble methods: XGBoost, LightGBM, Random Forest, etc. LLMs seem to be following the same trajectory.