Is There a Future for On-Device Open Source LLMs?

One obstacle to using open source LLMs is the cost of hosting them on GPU instances in the cloud. But perhaps there is a future for on-device open source?

From what I've seen, the overwhelming majority of AI products using LLMs are based on API calls to services like OpenAI, Bedrock or Vertex AI. And that makes sense - it's quick to set up, cheaper to run, and there's no heavy infrastructure to manage.

In a previous post I made a distinction between use cases requiring an instant response (streaming) like chatbots, and those that can be handled in batches like annotation, proofreading, etc. In the latter case, you just rent a GPU instance in the cloud for the duration of the task, not 24/7. It's an inexpensive way to benefit from open source LLMs.

I imagine that in the future, a new trend may emerge for open source models - on-device. For example, for a corporate chatbot, instead of designing cloud infrastructure and a web UI for employees, the work would consist of directly (or partially) downloadable applications that run from laptops (also bringing technical and practical challenges).

When you think about it, LLMs are increasingly optimized and the chips embedded in computers more and more powerful. There are also massive investments happening in the space right now.

Obviously with all this, Apple MacBooks with their M-series chips are ahead and seem like natural candidates to run LLMs locally, even if AMD and Intel are working hard playing catch up.

I wonder if for some time now Apple has been secretly conceiving its own on-device LLM that will interact with the entire macOS ecosystem.