Edge & On-Device AI

Current State

Running AI locally — on phones, laptops, and edge devices — has gone from hobbyist to mainstream. Apple Intelligence runs on-device for iPhone/Mac. Samsung and Google embed AI into their devices. NVIDIA DGX Spark ($4,699) makes local inference of 200B-parameter models possible on a desktop device.

The software stack for local AI is maturing: Ollama provides simple local model serving, llama.cpp enables efficient CPU/GPU inference, MLX optimizes for Apple Silicon, and ONNX Runtime covers cross-platform. Quantization (GGUF, GPTQ, AWQ) makes large models fit in limited memory.

Privacy is a primary driver: enterprises and individuals want AI without sending data to the cloud. The trade-off between local (private, fast, offline) and cloud (most capable, always updated) is narrowing as local models improve.

Key Players

Player	Product	Notable
NVIDIA	DGX Spark, RTX AI	Desktop/workstation local AI
Apple	Apple Intelligence, MLX	On-device for consumer devices
Ollama	Ollama	Simplest local model serving
llama.cpp	llama.cpp / ggml	Foundation of local inference
Google	Gemini Nano, MediaPipe	Android on-device AI
Qualcomm	Snapdragon X Elite, NPU	Mobile/laptop AI chips
Intel	OpenVINO, Core Ultra NPU	PC AI inference
Jan.ai	Jan	Local AI desktop app

Recent Signals

Date	Signal	Significance	Source
2026-04-27	Luce DFlash: 2× speculative decoding speedup for Qwen3.6-27B on single RTX 3090 — MIT, standalone C++/CUDA (no Python/vLLM/llama.cpp). Mean 1.98× speedup: 2.24× HumanEval, 1.99× Math500, 1.71× GSM8K. KV cache compressed to TQ3_0 (3.5 bpv), fits 256K context in 24GB VRAM. Bit-identical to AR output in verify mode. → 2× inference speedup on consumer GPU, no retraining, embeddable in native apps.	notable	Reddit LocalLLaMA
2026-04-27	Multi-GPU VRAM pooling via Vulkan: 30B-dense models on 16GB + 6GB secondary GPU — RTX 5070Ti + RTX 2060: 186 t/s prompt processing, 19 t/s generation (vs 4 t/s single card). Model fits in combined 22GB VRAM. Config: `dev=Vulkan1,Vulkan2`. → Practical ~5× throughput multiplier using a secondary low-end GPU; no hardware upgrade required for 30B model class access.	notable	Reddit LocalLLaMA
2026-04-11	DFlash speculative decoding — 85 tok/s on M5 Max, 3.3x speedup Qwen3.5-9B via MLX	notable	reddit.com
2026-04-08	LM Studio acquires Locally AI for cross-device local inference — LM Studio (a popular desktop app for running LLMs locally) acquired Locally AI, a platform focused on cross-device local inference (running models across multiple local devices in concert). → Consolidation in the local AI tooling space; cross-device inference could enable users to pool compute from multiple personal devices (laptop + desktop + phone) for running larger models locally.	notable	lmstudio.ai
2026-03-26	Apple opens Siri to rival AI in iOS 27 — Extensions will let users pick Claude, Gemini as AI handler. Multi-model orchestrator strategy.	significant	Bloomberg
2026-03-24	Apple testing standalone Siri app for iOS 27 — Bloomberg reports chatbot app with text/voice, chat history, Dynamic Island, "Ask Siri" button. WWDC June 8 target. → Apple's competitive response to Claude and ChatGPT.	notable	bloomberg.com
2026-03-23	NVIDIA NVFP4/NVFP8 quantization formats ship — new floating-point formats (FP4 uses 4 bits per weight, FP8 uses 8 bits, compared to the standard 16 or 32 bits) that reduce model memory footprint by ~60% and deliver 3x inference speed. Lower bit-width means less data to move through memory bandwidth, which is the primary bottleneck for on-device inference. → Makes running larger models on consumer GPUs and edge devices practical without the quality loss of aggressive integer quantization	notable	edge-ai-vision.com
2026-03-20	Qwen3-Coder-Next: 3B active parameters on 8GB VRAM — frontier-quality coding model that fits in consumer laptop memory using MoE (Mixture of Experts — the model has more total parameters but only activates 3B per query, keeping memory and compute low). 8GB VRAM is the baseline for most modern laptops. → Demonstrates that specialized coding AI no longer requires cloud APIs or expensive hardware	notable	localaimaster.com
2026-03-16	Apple Siri struggles delaying hardware launches — Apple's difficulty getting on-device AI to production quality is reportedly causing delays to hardware product timelines. On-device AI requires fitting capable models into tight memory/power budgets while maintaining response latency under ~200ms. → Highlights that the engineering challenge of on-device AI remains a bottleneck even for the largest consumer hardware company	notable	macdailynews.com
2026-03-14	Tether BitNet LoRA — 1-bit fine-tuning on phones/laptops, 77.8% less VRAM	notable	tether.io
2026-03-12	DGX Spark shipping with Ollama pre-installed, March firmware update	notable	developer.nvidia.com
2026-03-10	Microsoft BitNet renewed surge +6,457 stars/wk	significant	github.com
2026-03-03	Qwen 3.5 Small (0.8-9B) optimized for edge devices	significant	venturebeat.com
2026-03-01	Apple Siri rebuilt on Gemini shipping to 2.2B devices via iOS 26.4	breakthrough	webpronews.com

30-Day Trend

Accelerating. Apple Siri rebuilt on Gemini reaching 2.2B devices is a breakthrough moment for on-device AI at consumer scale. 1-bit fine-tuning (BitNet LoRA) slashing VRAM 77.8% makes local model customization viable on phones and laptops. DGX Spark now ships with Ollama pre-installed, and Qwen 3.5 Small targets edge explicitly. The BitNet ecosystem is surging (+6,457 GitHub stars/wk). On-device AI is shifting from niche to default.

What to Watch For

Local models matching cloud API quality for specific tasks
New quantization techniques that preserve more quality
NPU utilization improvements in consumer hardware
Privacy-first AI products gaining market share
Enterprise adoption of local AI for sensitive data
Offline-first AI applications
DGX Spark / similar products creating a new product category

Builder's Notes

(To be filled by daily scan — Phase 5)