Open source LLMs are sometimes cheaper than APIs

There are cases where using an open source LLM is cheaper than going through API calls. The main obstacle to using open source models is the inference cost associated with serving, but you'll see that there are exceptions. I'm extending here Pierre-Carl Langlais' line of thinking. In one of his Linkedin post he describes a text annotation use case via LLMs that is cheaper and faster for him than using APIs.

More generally, I think we need to distinguish between two situations: streaming, where we need to process data in real time, and batch processing, where we can afford to wait.

The chatbot is a good illustration of the streaming case: the user sends their request and expects an immediate response. In this case, going through APIs like OpenAI, Bedrock, Azure or VertexAI is generally more cost effective, because you only pay for what you consume, without having to rent remote machines 24/7 to host the model.

On the other hand, for batch processing, we can afford to wait to build up a large enough data set. We can then rent a machine for the duration of processing and do everything in one go. For example, an A10G GPU (64GB VRAM) on SageMaker only costs $2/hour. We then repeat this routine periodically as needed.

Use cases that lend themselves to batch processing include: text annotation, complex labeling, generating descriptions from summary information, text correction, exam grading, text summarization, annotated dataset generation, document translation, named entity extraction, customer service QA...

And I haven't even talked about all the other advantages, besides cost, that using open source models offers.