Is prompt engineering a scalable long-term approach ?

The Scalability Concerns of Relying on Prompt Engineering.

I'm wondering if prompt engineering is a scalable long-term approach. I feel like today a large part of solutions based on LLMs rely on OpenAI/Azure API calls and use more or less complex prompt engineering (chains of prompts, chains of thoughts, routing, etc.).

All these prompts are very sensitive - sometimes a comma changes the output behavior. Prompt templates, like any app feature, need to evolve over time. Some use cases even require periodically adding new ones.

Evolving these prompts inevitably means breaking a precarious balance and going back to a few hours of manual testing. There are still no mature processes for versioning and monitoring prompts and hyperparameters. Simply because there are no very sophisticated methods yet to automatically evaluate text generated by LLMs. This step is still very manual.

Having an open source model with finetuning pipelines seems like a more scalable approach to me (but much more complex and expensive for now). I agree that in terms of cost, relying on APIs is much cheaper for the vast majority of use cases. However, I can't help but think open source is the future.

Curious to hear your thoughts.