AI-augmented cockpit: local generative AI, in a buck and on the road
A year of unknowns turned into shipped systems: a self-hosted generative AI stack of LLM agents, speech, music, sound effects, images, and haptics, built as modular services and orchestrated multimodally across a seating buck and a real vehicle, first milestone in 3.5 months.
- Client
- zigzag, for Volkswagen Group
- Year
- 2025 / Berlin, with on-site integration at the client's proving ground
- Elapsed
- ~1 year / first milestone shipped in 3.5 months
- Role
- Team of five; personally end to end on the AI server and runtime, heavy technical build, R&D and integration, plus PM, ops, and client management
- Stack
- Local LLM agents (Ollama) / open-weight TTS / latent-diffusion music and SFX / ComfyUI image pipelines / LLM-driven haptics / dual-GPU dockerized server with monitoring / REST, SSE and WebSocket middlewares
- Unlock
- Internal Innovation Award / several solutions advanced to patent applications / business brought in personally
The ask
A vehicle interior as an immersive environment: lighting, sound, haptics, and UI enriched by generative AI, generated locally for privacy and latency, on two platforms at once, a static seating buck and a drivable vehicle. Nobody knew yet what local generative AI could carry; the project was structured to find out and ship at the same time.
Designed for AI, not just with AI
UX concepts and AI systems evolved together. Designers and technologists sat side by side evaluating outputs collectively. I built the AI services not as invisible backend processes but as visible, modular tools with their own interfaces, so every system could be understood, discussed, and reused. My job was to build these fast enough for designers to work with, while simultaneously hardening them for integration, and to bring designers and stakeholders up to speed on what the systems could and could not do.
Generative when it mattered, curated when it counted
Not every moment required real-time generation. The system switches between pre-curated content and live generative output per experience, with both sharing the same structural logic, and a designed upgrade path from lookup-table to fully generative in a later phase. I proposed, demonstrated, and sold this hybrid model to the client as the way to balance creative unpredictability with narrative coherence.
Beyond the screen
Experiences orchestrated across content, sound, music, lighting, and vibration as one coherent whole, with system logic responding to user preference, driving data, and environment: generated music and sound effects, spoken narration, and haptic patterns tuned per platform, down to different signal libraries for the buck's and the vehicle's exciters. I guided colleagues and senior designers step by step on designing for multimodal, generative contexts, and carried QA and stakeholder communication.
Engineered for complexity
Shared business logic across AI services, client software, and middleware, each with clear responsibilities. Everything containerized and individually testable; unified control layers (moderator panels, logging, container management, GPU dashboards) when the experience demanded the whole. CI/CD moved builds between development rigs and final platforms as hardware evolved, through to on-site integration at the proving ground. A bespoke Immersion Player orchestrated all major off-HMI media through a single coordinated system (a colleague-led build; I co-developed the architecture).
Result
The first milestone shipped in 3.5 months for the client's future-mobility showcase; the full system matured over a year. A later internal showcase earned an internal Innovation Award, several solutions advanced toward patents, and the engagement itself was business I brought in, planned, led, built, and delivered.
