T ThinkEdge Learn

Local LLM: Your Own Model, Your Own Machine

Running a language model on your own machine costs you nothing per token and keeps your data in the room. It also means working inside the memory you actually have.

local llm run llm locally local large language model llm on raspberry pi
AI and Electronics · The memory arithmetic, the real speeds, and the Arduino's job

What running it locally actually buys you

Three things. Your data stays on the machine, there is no bill per token, and it keeps working when the internet does. The price is that the model has to be small enough for the memory you have, small models are noticeably weaker, and you have to do the setup yourself instead of pasting an API key into a web page.

Privacy is the one that matters in a school. Anything you type into a cloud tool may be stored and may be used to improve the service. A model running on a Raspberry Pi in your classroom sends nothing anywhere, so a student can experiment with real data without a permission form, and you can find out exactly what the software does because you can read it.

The RAM arithmetic, and where the Arduino fits

A 4-bit quantised model needs roughly 0.7 GB of RAM per billion parameters, plus room for the context cache. So a 3B model is about 2.0 GB resident and an 8B model about 5.0 GB. On a Raspberry Pi 5 with 8 GB you can hold a 3B model comfortably and an 8B model only just, once the operating system has taken its share. Speed follows memory bandwidth, not cleverness: a 3B model generates roughly 5 to 7 tokens per second, a 1B model about 14, and an 8B model about 1.4, which is painful.

The microcontroller is not the model, and pretending otherwise is the usual mistake. The model does one job: turn a sentence into a number. Send that number over serial at 9600 baud, roughly 10 ms for a short command, and the Arduino acts on it within a few milliseconds while the model is still thinking about the next question. It is the same division of labour as a car: the model decides, the microcontroller drives.

Step by step

  1. Do the RAM sum before you download

    Parameters times 0.7 GB, plus about 1 GB for the operating system and the context cache. If it does not fit in your RAM, it will not run at a useful speed, however good the runtime is.

  2. Start on a desktop

    A laptop with 16 GB and a graphics card runs the same model many times faster. Get the prompt and the output format right there before you move to a slower board.

  3. Move to the Raspberry Pi 5

    Use llama.cpp or Ollama, pull a 1B to 3B model at 4-bit, and measure the tokens per second yourself rather than trusting someone else's benchmark.

  4. Make the output a number

    Ask for a single integer or a short keyword. A model that answers 15 cm is easy to parse, while a model that writes a paragraph needs a parser you have to write yourself.

  5. Send it to the Arduino over serial

    At 9600 baud the link carries about 960 bytes per second. Print the number, read it with a non-blocking parser, and act on it without blocking the sensor loop.

  6. Add a timeout and a default

    If nothing arrives within 500 ms, fall back to the last known good value. A model that hangs must never be able to freeze the robot.

What you'll need

ItemQtyWhy
Raspberry Pi 5 with 8 GB1Quad-core Cortex-A76 at 2.4 GHz and 8 GB of RAM, which is the working ceiling for a 1B to 3B model at 4-bit. Around $130.
microSD card 32 GB1Holds the operating system, the runtime and the model files. A single 4-bit model download is a couple of gigabytes.
Arduino Uno R31The real-time half: it reads the sensors and acts within milliseconds, whatever the model is doing.
HC-SR04 ultrasonic sensor1The observation the model reasons about and the board reports.
LED bar graph module1Shows what the model decided, so you can tell a bad decision from a bad parse.
USB cable and jumper wires1 setThe serial link between the Pi and the board, plus the sensor wiring.

Practise it before you build it

Every step above has a matching circuit in the ThinkEdge Learn simulator. The simulator runs a real Arduino sketch against a modified-nodal-analysis solver, so digitalWrite actually drives the LED on screen and the readings change when you change the wiring — no hardware, no burnt components.

Common questions

How fast is a local LLM on a Raspberry Pi?

On a Pi 5 with 8 GB, a 1B model at 4-bit generates about 14 tokens per second, a 3B model about 5 to 7, and an 8B model about 1.4, which is slow enough to be frustrating. The bottleneck is memory bandwidth, so a cleverer model does not automatically run faster. Treat 5 tokens per second as the planning number for a classroom project.

Do I need a GPU to run a local LLM?

Not to start. A Pi 5 runs a small model on its CPU, and the Raspberry Pi AI HAT with an NPU helps but changes the speed by a factor rather than unlocking a much bigger model. A desktop with 16 GB of memory and a graphics card is where comfortable token rates come from, and that card is the single biggest cost in the whole build.

Is a local model better than a cloud one?

For privacy and offline work, yes. For quality, no, and the gap is wide. A 3B model running on a Pi will not write a better essay than a large cloud model, and it will confidently get simple factual questions wrong. Use it where the answer is a number you can check, such as a distance, a temperature or a motor speed.

Related guides

Edge AI: Decisions Made on the DeviceEdge AI runs the decision on the device instead of in the cloud. Here is what that buys you in latency and privacy, and what a 32 KB microcontroller can really do. What an AI Agent Actually IsAn AI agent is a loop of model calls, tools and memory. Learn how that loop maps onto the sense, decide, act cycle of a robot you can build this term. Raspberry Pi 5: What Changed Inside and How to Teach With ItThe Raspberry Pi 5 has a faster CPU, a new RP1 I/O chip and a 5 V power supply. What changed, what it costs, and how to use it in a classroom. ESP32 IoT Projects: From a Single Sensor to a Live DashboardFive ESP32 IoT projects that teach sensors, MQTT publishing, deep sleep and reconnection, each staged so the logic runs in a browser simulator first. Artificial Intelligence for StudentsArtificial intelligence for students: what a model really is, the three ways to learn it that are not hype, and the projects that prove each one landed.

Try the full simulator with your class

103 guided circuit projects, 21 lessons and an in-browser Arduino simulator. The first 15 projects are free, no account needed. NCERT Science mapping on 47 projects, with Arabic and English copy in the regional builds.