Serendipity: AR Quests on Smart Glasses

Overview

A context-aware AR platform for MentraOS smart glasses that generates personalized quests from where you are and what the world is doing.

Smart glasses are the first platform in years where the interface constraints are genuinely hostile: no keyboard, unreliable network, tiny compute budget. Designing for that changes how you think about every API call.

The problem

MentraOS runs on hardware with strict compute and connectivity limits. Building anything real-time there means solving data sync and degraded-network behavior before you write a single feature.

What I built

Context-aware quest generation combining Claude with GPS position and live weather APIs.
Fully voice-controlled interface using Vosk and Whisper so the platform never assumes a keyboard.
A data sync layer engineered specifically for low-connectivity operation.
Google Gemini integration for dynamic quest management as conditions change.

Notes

Things that surprised me, or that I got wrong first.

Designing for a network that drops

The sync layer assumes connectivity is the exception, not the rule. Quests resolve locally and reconcile later rather than blocking the wearer on a round trip that may never return.

Voice as the only input

With no keyboard, intent recognition is not a convenience feature. It is the entire input surface. Vosk handles the fast local path, Whisper the accurate one.

Say hi

If you want to talk about Serendipity, or anything on this page, I'd love to hear from you. Email me or use the contact page.

At a glance