RUN YOUR OWN AI CHATBOT ON A RASPBERRY PI

Stop sharing your data with cloud AI services.

- 

There are plenty of cloud-based AI chat services, but they all have downsides: you have no direct control over your privacy, the experience can abruptly change from day to day, and to get the best results you must often pay a subscription fee. If you’d rather avoid all that, why not run your own AI chatbot?

Doing so can be a lot cheaper and easier than you might think. General-purpose AI platforms are known for consuming vast quantities of energy and computing power, but it’s quite possible to run your own large-language model (LLM) on a humble Raspberry Pi. And to help you do so, Raspberry Pi has just released the AI HAT+ 2 for a very reasonable £125 inc VAT.

The board adds a Hailo-10H AI accelerator chip to the platform, putting 40 TOPS of INT4 inference performance at your disposal. That’s enough to run a smart and useful chatbot at an acceptable speed, and roughly equivalent to the NPUs found in Ryzen AI 300 and Core Ultra 200V chips from AMD and Intel. It’s a big improvement on the original AI HAT+, launched in 2024, which came in two variants offering 13 and 26 TOPS.

The AI HAT+ 2 board also includes 8GB of dedicated RAM, which allows it to support both text- and vision-based AI applications. You could, for example, use it to count cars driving past your home, or to monitor items passing along a production line. What we’re interested in, though, is running a large-language model on our own network – and here’s how to do it.

Hardware requirements

As well as the HAT, you'll need a Raspberry Pi 5; any variant should be usable, but a 4GB or 8GB model is preferable to make sure the various software elements we'll be installing will run together smoothly. You'll want a 27W power supply, as AI eats up power, and you should ideally fit some form of cooling for the Raspberry Pi itself. The official Active Cooler is fine, as the AI HAT + 2 ships with standoffs that lift it clear of the fan.