Perceptor

Context

Perceptor started as the real-time sensory organ for a multimodal bot: a bot that watches and listens to a live stream (Twitch or YouTube) and decides when to speak in chat. The core problem: turn unstructured inputs — text chat, audio, video — into a single stream of structured, timestamped events.

The output is a JSONL file where each line is an event with capture timestamp:

{"ts":1783070000.123,"source":"chat","user":"marc3lly99K","text":"che figata questa live"}
{"ts":1783070005.991,"source":"speech","speaker_id":"spk_01","text":"questo è un test di trascrizione"}
{"ts":1783070010.441,"source":"video","caption":"The scene shows a streamer looking at the monitor."}

Architecture

The system uses multiprocessing (not threads) and Redis as message broker. Every component runs in its own process and only communicates through queues.

flowchart TB
  subgraph CHAT["Chat pipeline"]
      C1["Twitch IRC / YouTube chat"] --> C2["Chat Worker"] --> AGG
  end
  subgraph AUDIO["Audio pipeline"]
      A1["ffmpeg (PCM 16kHz)"] --> A2["VAD<br/>WebRTC"]
      A2 --> A3["Speaker ID<br/>sherpa-onnx"]
      A3 --> A4["ASR<br/>faster-whisper"]
      A4 --> AGG
  end
  subgraph VIDEO["Video pipeline"]
      V1["ffmpeg (frames)"] --> V2["pHash filter"]
      V2 --> V3["VLM captioning<br/>(Qwen3-VL)"]
      V3 --> AGG
  end
  AGG["Aggregator<br/>(single writer)"] --> OUT["data/perceptions.jsonl"]

Why multiprocessing

Python has the GIL (Global Interpreter Lock): with threads, a heavy task blocks the others. Perceptor uses separate processes — chat, audio VAD, speaker, ASR, video, aggregator, producers — so chat, audio and video keep running in parallel without interference.

Why Redis

Redis acts as an in-RAM message broker: processes never talk directly, only through queues (queue:chat_raw, queue:audio_raw, queue:speech_segments, …). Benefits: producer/worker decoupling, resilience when a worker is slower, queue monitoring, selective worker scaling.

Main components

• Producers: Twitch over IRC (OAuth, JOIN, PRIVMSG parsing, PING/PONG); YouTube via Data API REST; audio and video via ffmpeg (normalised to PCM 16kHz mono for audio; JPEG frames for video); platform metadata (title, viewer count, category) straight to queue:perceptions.
• Audio pipeline: VAD (WebRTC, CPU) → speech segments → speaker identification (sherpa-onnx or mock, per-channel persisted centroids) → ASR with faster-whisper (CPU/CUDA, int8/float16).
• Video pipeline: frame extraction → pHash filter (drops visually identical frames, avoiding useless API calls) → VLM captioning (OpenRouter, e.g. Qwen3-VL) with an LRU for already-seen scenes.
• Aggregator: the only component writing to data/perceptions.jsonl, with at-least-once consumption, processing/dead-letter queues and deduplication.

Reactor (MVP)

On top of Perceptor sits the Reactor: a module that follows the JSONL and decides when the bot should speak in chat. It uses deterministic triggers (mentions, questions, repeated topics), a FocusSelector with per-source TTL, a PromptBuilder with short memory/facts, a JSON LLM decision, anti-stale guardrails and dry-run.

Design choices

• Processes instead of threads: the GIL makes threads unsuitable for real-time multi-modal work; the memory cost of processes is acceptable compared with guaranteed latency.
• Redis as a buffer: if Whisper is slow, the queue grows but chat and video don’t stall. Debugging is done by measuring queue lengths (documented in the README).
• pHash before the VLM: VLM calls cost money; the visual novelty filter cuts them drastically without losing semantic events.
• .env + profiles config: operational defaults live in profiles/*.toml, credentials and tuning in .env; the launcher python -m perceptor.launch twitch <channel> derives sources from the channel.

Results

A working end-to-end system that captures real live streams (Twitch/YouTube/local files), produces a synchronised JSONL stream and feeds a bot that intervenes in chat in a controlled way. The code is private, but the full architecture is documented in the README and I’m happy to present it in detail.

Notes

Roadmap noted in the project: migrate queues to Redis Streams/consumer groups, Prometheus metrics on latency and drops, offline diarisation, anonymous session replay.