# From Old Drives to Pocket AI: Building a Portable Local LLM Router ![](https://cdn.hashnode.com/uploads/covers/6133a0b7489de51a6a7ae0d3/c2feae98-85b3-4d05-9ac2-53ad1013c050.png align="center") *How I repurposed an old pendrive and external hard disk to store local AI models, offload storage from my MacBook, and build a decision-model-powered router that sends prompts to specialized LLMs.* ## The idea: My AI models were eating up my storage Running large language models locally is exciting. You get to experiment with different models, explore their capabilities, and build AI applications without relying on a hosted API for every request. But there is a practical problem: **model weights take up a lot of storage.** As I started experimenting with local models, my Ollama model directory grew to around 75 GB. Keeping all those weights on my MacBook's internal SSD didn't feel like the best use of space, especially when I wanted to experiment with more models. I already had something that could help: an old pendrive and a WD Purple 1 TB hard disk. Instead of buying more storage immediately, I decided to repurpose what I already owned. The idea was simple: move model storage off my MacBook, make the models accessible through Ollama, and turn the setup into something closer to a portable AI workstation. But I didn't want to stop at storing and running models. I also wanted to solve another problem with the usual local chatbot experience. ## Part 1: Turning old storage into a portable AI setup The first part of the project was about making better use of existing hardware. I wanted to keep my MacBook's internal storage free for development tools, source code, applications, and everyday work, while keeping large model weights on external storage. The pendrive offered portability. The external hard disk offered substantially more room for model files and experimentation. The resulting setup separates the main components: - **MacBook:** Runs the application and provides the computing resources for inference. - **External storage:** Holds the downloaded model weights. - **Ollama:** Manages and serves the locally available models. - **Local router:** Decides which model should handle an incoming prompt. - **Chat interface:** Lets me interact with the system and inspect its routing and inference behavior. The important distinction is that external storage solves the *storage-capacity* problem, not the entire inference-performance problem. Models still need to be loaded into system memory or accelerator memory as required by the runtime. A slower external drive can also increase loading times. The goal wasn't to make an old pendrive magically faster than an SSD. It was to reuse available hardware and build a more flexible local AI environment. There are practical details to consider when doing this, including filesystem limitations, drive reliability, macOS compatibility, and configuring Ollama to use the intended model location. Rather than duplicate every installation command and configuration detail here, I've documented the project in the repository. 🗂️ **Project repository:** [Raghul-M/local-llm-router](https://github.com/Raghul-M/local-llm-router) ## Part 2: Why use one model when you can route between several? Once the storage problem was addressed, I started thinking about the application itself. A typical chatbot sends every prompt to the same model. That can work, but different tasks benefit from different models and capabilities. **Consider these prompts:** - `"Explain how Python decorators work."` - `"Research the trade-offs between two model-serving architectures."` - `"Help me understand a general concept."` They may all look like ordinary chat messages, but their requirements differ. A model suited to coding may not be my first choice for research, and a general-purpose model may not be the best choice for every request. I wanted a small model to make that initial decision before invoking a larger, specialized model. That led to the idea of a **local LLM router powered by a decision model**. Instead of treating the application as one chatbot backed by one model, I designed it as a small orchestration layer that sits between the user and a collection of local models. ## Part 3: A small decision model as the router The interesting part is that the router doesn't need to generate the final answer itself.Its job is to understand the intent of the incoming prompt and select an appropriate destination. I use a small decision model, approximately 0.8 billion parameters, to classify prompts into categories such as: - **Coding:** Programming questions and software-development tasks. - **Research:** Investigations, comparisons, and deeper technical questions. - **General:** Everyday questions and requests that don't fit the other categories. The classifier evaluates the prompt against category-specific questions and produces scores. The application uses those scores to choose a route, and the selected model generates the actual response. This separates two responsibilities: deciding *which model should answer* and deciding *what the answer should be*. The decision model is deliberately smaller than the models used for generation. Its purpose is to provide a lightweight routing step rather than consume the resources needed for the main task. A small classifier is not automatically accurate, though. Prompt classification is itself a language-understanding problem. For example, asking for a blog post about Python decorators is a writing request about a programming topic; it isn't necessarily a request to write code. The router needs to distinguish the subject of a prompt from the type of work being requested. That distinction is one of the more interesting engineering challenges in this project. ## Part 4: The architecture The high-level workflow looks like this: ```text User prompt | v Small decision model (~0.8B) | v Prompt classification | +-----------+-----------+ | | | v v v Coding Research General | | | v v v Specialized Specialized General-purpose model model model | | | +-----------+-----------+ | v Generated answer | v Chat interface ``` The diagram represents the logical flow rather than a guarantee that every category maps permanently to a particular model. The available models and routing configuration determine the actual destinations. The main components are: **1\. Prompt classification** The decision model evaluates the user's request and determines which category best matches its intent. **2\. Model selection** The router selects a model based on the classification scores and the configured mapping between categories and models. **3\. Local inference** Ollama serves the selected model. The router doesn't need to generate the final answer itself; it delegates that work to the selected model. **4\. Conversation history** The application can include a bounded amount of previous conversation in subsequent requests. This helps preserve context without sending the entire conversation indefinitely. **5\. Observability** The application exposes useful information about routing and inference, including the selected model, classification latency, scores, recalled messages, token counts, and generation timings. This makes the router more than a model-selection demo: it also provides visibility into how each request moves through the system. ## Part 5: From a routing experiment to a usable application #### Defining a route : Each route is a `noul` question with instructions and explicit true/false criteria: ```python CLASSIFIER_QUESTIONS = { "coding": { "type": "noul", "instructions": ( "Determine whether the user's PRIMARY goal is to implement, modify, " "debug, review, or understand software or code. Classify based on " "the main deliverable, not merely the presence of technical terms." ), "criteria": { "true": "The main deliverable is working code, a bug fix, a script, " "a code review, or a technical implementation.", "false": "The main deliverable is an article, report, comparison, or " "explanation. Code is only illustrative or incidental.", }, }, # ... research, general } ``` All three questions get scored. Highest score wins: ```python scores = {name: answer.noul for name, answer in response.answers.items()} route = max(scores, key=scores.get) ``` Then the prompt goes to whichever model owns that route: ```python ROUTE_DEFAULTS = { "coding": {"model": "nemotron-3.5-bolt:e4b", "prompt_suffix": ""}, "research": {"model": "qwen3.8:latest", "prompt_suffix": " Please provide a detailed research plan..."}, "general": {"model": "gpt-oss-safeguard:latest", "prompt_suffix": ""}, } ``` That's the whole router. About forty lines. I wanted to interact with the router as a real application rather than test everything through isolated terminal commands. The project includes a Streamlit-based chat interface as well as a terminal-based workflow. The interface makes it possible to submit prompts, receive responses, and inspect the decisions behind them. Some of the information I track includes: - Classification latency: how long the routing decision takes. - Routing scores: how strongly the classifier scores the available categories. - Selected model: which model receives the request. - Conversation context: how many previous messages are recalled. - Prompt and output tokens. - Model loading and prompt-evaluation durations. - Generation time and tokens per second. These measurements help separate different sources of latency. A slow response might be caused by loading a model, evaluating a long prompt, or generating tokens—not necessarily by the routing step. That distinction becomes especially useful when experimenting with models of different sizes and when using external storage. ## Part 6: The trade-offs This project brings together two ideas: making local model storage more flexible and making local inference more task-aware. Both introduce trade-offs. ### External storage versus internal SSD External storage lets me reclaim internal disk space and reuse hardware I already own. The downside is that loading model files can be slower, depending on the device, interface, filesystem, and model size. The external drive is primarily a place to keep model weights; it doesn't eliminate the RAM or compute requirements of inference. ### A small classifier versus direct model selection A decision model adds another inference step before the actual response. That step introduces latency, and an incorrect classification can send a prompt to a less suitable model. The benefit is that routing decisions become explicit and inspectable instead of relying on one model for every task. ### Specialized models versus one general-purpose model Specialization gives me the flexibility to choose models for different kinds of work. However, it also means managing several model downloads, configurations, and resource requirements. More models do not automatically mean better answers. The routing strategy and the quality of each selected model matter just as much. ### Conversation context versus resource usage Keeping relevant conversation history helps the application respond coherently to follow-up questions. But longer histories consume additional context-window capacity and can increase prompt-processing time. The application therefore uses bounded conversation history rather than continually sending every message from the session. ## Part 7: Try the project yourself If you want to explore the implementation, start with the repository: 🗂️ [**GitHub: local-llm-router**](https://github.com/Raghul-M/local-llm-router) The repository is the source of truth for the current code, dependencies, model configuration, and setup instructions. Follow its README rather than relying on commands copied into an article that may become outdated. Before running the application, make sure that: - Ollama is installed and working. - The configured model storage location is accessible. - The models referenced by the application are available locally. - Your Mac has enough memory and compute capacity for the model you select. - The external storage device is connected and mounted whenever those model files are needed. Once the required model weights are downloaded, local inference can run without sending prompts to a hosted model API, provided the application and runtime are configured to remain local. Initial downloads, software installation, and updates may still require an internet connection. ## What's next? The current design is a starting point, not a claim that routing is a solved problem. There are several improvements I'd like to explore: - **Confidence thresholds:** Avoid blindly accepting a weak classification when two categories have similar scores. - **Fallback routing:** Use a sensible default or ask for another decision when the classifier is uncertain. - **Better routing evaluation:** Build a representative set of prompts and measure classification accuracy instead of relying only on a few manual examples. - **Smarter context management:** Retain useful conversation context without unnecessarily increasing prompt length. - **Model-aware selection:** Consider memory requirements, model availability, loading time, and observed inference performance alongside task category. - **More reliable storage workflows:** Make it easier to switch between external drives and recover gracefully when a device is unavailable. These improvements would move the project from a straightforward category-based router toward a more robust local model orchestration system. ## Final thoughts ![](https://cdn.hashnode.com/uploads/covers/6133a0b7489de51a6a7ae0d3/cd6da34b-2cd4-4e08-94e0-4fe5423bd9bb.gif align="center") What started as an attempt to free up storage on my MacBook turned into a more interesting experiment: combining repurposed hardware, local model serving, and lightweight AI orchestration. The bigger lesson is that building a useful local AI system isn't only about finding the largest model that can run on your machine. It's also about making thoughtful decisions about storage, resource usage, model selection, and the application architecture around the models. An old pendrive and hard disk won't replace powerful inference hardware. But with the right setup, they can become useful pieces of a portable local AI environment and a good excuse to experiment with building your own model router. **The code is here:** [github.com/Raghul-M/local-llm-router](https://github.com/Raghul-M/local-llm-router) If you build something similar, I'd be interested in how you approach model routing, external model storage, and keeping local inference practical on everyday hardware. Connect with me on Linkedin: [**Raghul M**](https://www.linkedin.com/in/m-raghul/)