KnightOS
Personal · single-user · localhostpersonal ai assistant
A self-hosted, voice-capable assistant for macOS that pairs a chat interface with an agentic layer able to drive the machine and pull live data.
- lines, solo
- ~3,900lines, solo
- REST / SSE endpoints
- 17REST / SSE endpoints
- routed OS actions
- 34routed OS actions
- intent patterns
- 60+intent patterns
screenshot slot · knightosdrop a file in /public/shots and set `shot` in case-studies.ts
Cloud assistants can hold a conversation but cannot touch your machine, and the ones that can are usually thin shells around a terminal. I wanted one process that could do both — talk properly, and actually operate macOS — while staying usable when the network or the cloud provider fails. Everything runs on my own hardware.
- 01
Deterministic intent routing instead of LLM tool-calling
key callSystem commands never reach the model. A routing engine resolves natural language against 60+ prioritised regex patterns to 34 concrete actions first; only genuinely open-ended input falls through to an LLM. Asking a model whether to open an app costs a network round-trip and can pick the wrong action from a plausible-looking set — neither is acceptable for something you say out loud and expect to happen. Routing is O(patterns), runs locally, and is testable without a model in the loop.
- 02
Hybrid cloud/local model routing with automatic fallback
Gemini 2.0 Flash handles the primary conversational path; a failed cloud call fails over to local Ollama rather than surfacing an error. Model selection is task-specialised — a coder model for code generation, a vision model for local image understanding — so quality does not get traded away for a single general endpoint. The assistant degrades rather than stops.
- 03
SSE for token streaming, not WebSockets
Once a prompt is submitted the traffic is one-directional, server to client. SSE is a plain HTTP response with built-in reconnection, so there is no second protocol to run, proxy, or keep alive. Tokens stream as they are produced.
- 04
Two-tier persistent user modelling
Hard facts are pulled synchronously by regex the moment they appear — cheap and instant. A second asynchronous LLM pass runs off the critical path to mine personality, preferences and mood. Both feed a system prompt composed fresh per conversation, so the profile keeps deepening without adding latency to any single reply.
- 05
AppleScript as the OS control surface
App launching, volume and brightness, media transport, Spotlight file search, Calendar, Mail and notifications all go through AppleScript — the sanctioned automation path on macOS, which keeps the permission model honest and the action layer readable.
- 06
Multimodal ingestion and a generation studio alongside the chat
PDF, DOCX, XLSX, CSV and images are ingested into the conversation. A generation studio covers text-to-image, text-to-video and neural TTS. A separate news-intelligence dashboard aggregates five RSS feeds and plots geo-tagged articles on an interactive world map.
- Node.js
- Express 5
- React 19
- SSE streaming
- Gemini API
- Ollama
- AppleScript
- Web Speech API
Runs as a daily driver on one machine. It is deliberately a single-user localhost tool — there is no multi-tenancy, no hosted deployment and no auth layer, because none of those were the point. What it demonstrates is the routing, fallback and user-modelling architecture holding up under real daily use.