January 2025 state of local LLMs

I just wanted to put the current state of local open source language models on the record, because a few years ago I thought these would be a huge deal by now. They aren't.

Local models would be incredibly useful

I think the usecases for running LLMs locally on your phone really obvious. They can give you instructions for emergencies (e.g. changing a flat tyre). You can use them as a sounding board for thinking out load, with full confidence that the conversation will remain private. You can look something up without having access to the internet (e.g. a long flight). This isn't even considering the countless integrations that could be really useful (e.g. it could read your notes app).

Apps for running small models have gotten better

A year ago the best app for running local models on Android was MLC Chat, which was crap. Now there's Pocket Pal LLM which is way better. LLM Studio is also excellent on desktop compared to older more barebones approaches like llama.cpp

Models

Open source models that can run on powerful desktops (in my case, my MacBook) have gotten significantly better, but 999 times out of 1000 if I'm using my laptop then I have an internet connection.

So models that can run on smartphones (where I'm regularly without a good internet connection) are potentially far more useful and interesting, but the issue is that the models are just bad right now. llama-3.2-7B runs slow on my S21 Ultra (granted, kind old but very high end phone) and it's responses are just not useful whatsoever.

Stuff I need to look more into

I have a little bit more looking into this problem to do. I'm particularly interested in the Gemma model that's built into Chrome now, and I'm sure that there is better models I can run with Pocket Pal on my phone, I just need to experiment a bit more. I'm not an expert in this stuff so absolutely sure there is better ways to run better models on my phone, but I think this is a rough lay of the land in early 2025.