Drift — Emotion-Driven Ambient Music

Context

Drift was born during the Generative Music AI workshop organised by the Music Technology Group at Universitat Pompeu Fabra, Barcelona — Europe’s leading experts in AI music generation. There were 30 participants: 20 technologists and 10 musicians, split into international teams.

The project is inspired by Michael Viega’s concept of “listening in the ambient mode”: instead of a structured song, Drift generates a soundscape that is, in Brian Eno’s words, “as ignorable as it is interesting”.

The system creates a bio-creative loop: your voice generates music that reflects your emotional state, and this soundscape in turn invites reflection and introspection.

What I did

I was team leader of an international team (with Jing Yu and Lianne Sánchez-Rodríguez, mentored by Fernando García), working with people from different fields — the classic tech+music hybrid teamwork. I contributed to the pipeline architecture and to the emotion analysis / MIDI generation side.

How it works

The system processes audio through a multi-stage pipeline:

flowchart LR
  V["User's voice"] --> A1["Prosodic analysis<br/>Wav2Vec2<br/>(intonation, rhythm, timbre)"]
  V --> A2["Semantic analysis<br/>Whisper + NRC-VAD<br/>(transcription + lexicon)"]
  A1 --> F["Weighted fusion<br/>(valence, arousal)"]
  A2 --> F
  F --> M["MIDI-Emotion model<br/>(music tokens)"]
  M --> O["Real-time OSC"]
  O --> X["Max/MSP<br/>granular ambient synthesis"]
  X --> S["Evolving soundscape"]

1. Dual-path voice analysis (Python).

   • Prosodic: a pre-trained Wav2Vec2 model analyses intonation, rhythm and timbre to extract valence and arousal.
   • Semantic: Whisper transcribes the speech; the NRC-VAD lexicon assigns valence/arousal scores to the words used.
   • Both paths are merged with a weighted average for a more robust emotional reading.

2. Emotion→MIDI generation (Python). The (valence, arousal) pair feeds the midi-emotion generative model, producing a continuous stream of symbolic music events (channel, pitch, velocity) that represent the emotion.

3. OSC transmission. MIDI events are not saved to a file: they are streamed in real time to Max/MSP over OSC (Open Sound Control) for a fluid, low-latency experience.

4. Ambient synthesis (Max/MSP). The patch is not a plain MIDI player but a custom synth:

   • Granular synthesis (GRANULAR, CLOUD): MIDI notes trigger engines that break sound into grains, creating evolving pads and ethereal clouds.
   • Spatialisation: deep reverbs (GIGAVERB) for an immersive soundscape.
   • Slow envelopes: long attacks and releases (ADSR) avoid harsh transients — everything fluid, everything “drifty”.

Design choices

• Dual-path emotion: prosody alone is not enough (the same sentence said differently), semantics alone is not enough (words without intonation). Weighted fusion gives a more robust reading.
• Symbolic MIDI instead of direct audio: the model operates on music tokens (the same transformer family used at the workshop), leaving final synthesis to Max/MSP — the right tool for ambient sound design.
• OSC streaming: essential for interactivity: no intermediate files, minimal latency.

Connection to my thesis

This project is the practical counterpart of my MSc thesis — “Conditioning Sound Generation with Emotions: An Investigation on Generative AI for Controllable Audio Synthesis” (supervisor Prof. Antonio Rodà, co-supervisor Dott. Matteo Spanio) — which investigates exactly how to condition sound generation on emotional content.

Results

The project was presented at the end of the workshop as a working interactive installation. The experience gave me: international multidisciplinary teamwork, leadership, and direct exposure to frontier research in generative music.

• Thesis: PDF
• Repo: Drift-Music-AI on GitHub