From Data Scientist to Data Engineer… ultimately to AI Engineer.

(This article is a transcript in English of one of my french LinkedIn posts)

Lately, a clear trend of many Data Scientists moving towards Data Engineer roles has been observed.

This phenomenon can be attributed to several factors:

First and foremost, the job market is undergoing significant changes, and Data Engineers capable of building robust and scalable data pipelines are in high demand. Consequently, it is currently easier to find positions as a Data Engineer than a Data Scientist. Notably, Data Scientists (and ML Engineers for that matter) often possess knowledge in cloud, Python, and tools like Docker, GitLab, or Terraform, which facilitates a transition to the Data Engineer profession (requiring a varying degree of effort).

No doubt, some Data Scientists are frustrated by spending considerable time on "data wrangling" (data cleaning and preparation). Moreover, certain Data Scientists might feel weariness after dedicating much time to develop POCs (proof of concept) that don't yield tangible outcomes. On the contrary, some might feel stagnant working on the same use-cases for extended periods. By transitioning to a Data Engineer role, they can rejuvenate their experience by focusing on building sturdy data infrastructures and gaining solid technical skills.

Other Data Scientists, who were torn between software engineering and the mathematical/statistical aspect, find that they are more inclined towards the software side.

To top it all, the DS to DE transition can often take place within the same company.

More recently, a new trend is emerging with the rise of LLMs (Large Language Models). Indeed, these models become accessible through simple API calls, greatly reducing the entry barrier to the world of AI and ML. Subsequently, one can develop comprehensive applications focusing mainly on the software development aspect. Hence, an increasing number of software engineers are getting involved in building AI solutions.

It's fascinating to observe the overlaps emerging between the various data professions, and this might just be the beginning…

With the current enthusiasm and the "sexy" allure of generative models, combined with the need for ever-more sophisticated architectures to deploy open-source LLMs like Llama-2 in the cloud, might we see a reverse trend? Perhaps, witnessing Data Engineers transitioning to AI Engineer roles?