VoiceMem

Most memory layers built for LLM agents were designed around text chat, then adapted for voice as an afterthought, which shows up as latency and a flat, facts-only notion of what is worth remembering. VoiceMem, from researcher xzf-thu, starts from voice instead. Its own framing is direct: it aims to give voice agents the final missing component, something like a soul, so they come to understand a user better over time. The project backs that ambition with a technical report on arXiv and a genuinely streaming architecture rather than a batch-processed memory layer bolted onto an existing voice pipeline.

The core idea is a dual-brain split. A 'left brain' organizes factual information through schemas and entities for precise retrieval, while a 'right brain' independently tracks emotions, personality traits, and relationships through their own and cross-entity nodes. A streaming pipeline handles audio segmentation, transcription, and memory extraction while the conversation is still happening, rather than after the call ends, using FunASR or sherpa-onnx for speech recognition, 3D-Speaker for speaker verification, and multilingual-e5 embeddings. At query time, the system routes, ranks, and injects only the top-scoring memories into the model's context, keeping token usage down.

The natural audience is teams building real-time voice agents, assistants, companions, or support bots, who need the agent to recall both facts (a user's preferences, past requests) and emotional context (how a relationship has evolved) across sessions, without breaking the low-latency requirement that voice interaction demands. The repository ships a full evaluation harness against established benchmarks, LoCoMo and PersonaMem, plus a web demo, which makes it usable as a research starting point rather than only a paper's supplementary code.

The caveats reflect a project still very early in its life. The first release, v0.0.1, and the current v0.0.2 both shipped on the same day, September 1, 2026, so there is essentially no track record of stability across releases yet. The headline comparisons against Mem0, a 91.2 percent LoCoMo score versus 61.68 percent, a 134 millisecond response time versus 1,440 milliseconds, and roughly 430 memory tokens versus 6,956, are the project's own benchmark results and have not been independently reproduced. Open issues also flag that emotion labels and personality traits default to Chinese even for English-speaking users, a concrete localization gap.

The verdict: VoiceMem is a serious, well-documented research project addressing a real gap, memory that understands both facts and emotional context for voice agents, backed by a paper and a reproducible evaluation suite rather than marketing alone. Adopt it if you are building or researching voice agents and want a head start on emotion-aware, low-latency memory, accepting that this is week-one software. It is too early for production voice products, or for anything needing verified non-Chinese-default behavior and independently confirmed benchmark numbers.