mediapipe-jweb-audio-classifier

Real-time audio classification in Max/MSP using Google's YAMNet model via MediaPipe Tasks Audio. No build step, no local server — just open the patch and go.

Inspired by and follows the same pattern as robtherich/jweb-mediapipe, which brings MediaPipe vision tasks into Max's jweb object. This extends the approach to audio.

---

What it does

The patch embeds a self-contained HTML page inside Max's jweb object. The page:

• Loads the YAMNet TFLite model (521 AudioSet sound classes) directly from Google's CDN
• Captures your microphone at 16 kHz using the Web Audio API
• Runs classification every N milliseconds (configurable hop size)
• Sends results back to Max via window.max.outlet()

Everything runs client-side in the Chromium engine built into Max — no Python, no Node, no server.

---

Files

File	Description
audio-classifier.html	Self-contained web page — the classifier UI and all logic
audio-classifier.maxpat	Max patch — loads the HTML in a jweb object and routes the outlet messages

---

Requirements

• Max 8 or later (tested on Max 8.x with the built-in jweb / Chromium engine)
• An internet connection on first launch (to download the YAMNet model and MediaPipe WASM from CDN)
• Microphone permission granted to Max when prompted by macOS/Windows

---

Getting started

1. Clone or download this repository into your Max file search path (e.g. ~/Documents/Max 8/Projects/).
2. Open audio-classifier.maxpat in Max.
3. The jweb object will load audio-classifier.html automatically. Wait for the status to read model ready.
4. Click start (or send a start message to the patch inlet) and allow microphone access when prompted.
5. Classification results stream out of the patch as Max messages — see the section below.

Offline use: after the first successful load, the YAMNet model is cached by the browser engine. Subsequent launches work without internet access.

---

Max messages

Out of the patch (from jweb → Max)

Message	Arguments	Description
top	<label> <score>	Highest-scoring class for the current frame
class	<rank> <label> <score>	One message per result (rank starts at 1)
inference_ms	<ms>	How long classification took in milliseconds
status	<text>	Human-readable status string
running	0 or 1	Capture state changed
done	—	End-of-frame marker (useful for bangs / triggers)
error	<where> <msg>	Error description
device	<index> <id> <label>	Sent for each audio input device during enumeration
ready	1	Model finished loading

Into the patch (Max → jweb)

Send these as messages to the patch inlet:

Message	Arguments	Description
start	—	Begin capturing and classifying
stop	—	Stop capture
threshold	<0.0–1.0>	Minimum confidence score to include a result
maxResults	<int>	Maximum number of classes to return per frame
hop	<ms>	Classification interval in milliseconds (100–975)
device	<deviceId>	Switch to a specific audio input device ID
listDevices	—	Re-enumerate audio input devices

---

Parameters

Parameter	Default	Description
max results	5	How many top classes to return each frame
threshold	0.10	Minimum score — lower values return more (noisier) classes
hop ms	480	How often classification fires. 480 ms ≈ every half-second. Lower = more CPU.

YAMNet always analyses a ~975 ms sliding window of audio regardless of hop size. Reducing the hop makes classification fire more often but does not reduce the analysis window.

---

Output format example

When someone whistles, Max might receive:

status  listening @ 16000 Hz
top     Whistling 0.8723
class   1 Whistling 0.8723
class   2 Music 0.0412
class   3 Human voice 0.0310
inference_ms  18
done

Connect the route object (already in the patch) to separate these streams and drive whatever Max logic you need — samplers, synths, parameter automation, etc.

---

How it works

The HTML page uses the MediaPipe Tasks Audio JavaScript library loaded from CDN:

import { AudioClassifier, FilesetResolver } from
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-audio/+esm";

Audio is captured via getUserMedia, resampled to 16 kHz by the browser's AudioContext, and fed into a ring buffer. A ScriptProcessor fires every 4096 samples; when enough samples have accumulated (based on the hop setting) the ring buffer snapshot is passed to classifier.classify(). Results are immediately sent to Max via window.max.outlet().

The Max bridge is a thin shim — window.max is only defined when the page runs inside a jweb object, so the HTML file also works standalone in a regular browser for testing.

---

Related projects

• robtherich/jweb-mediapipe — vision tasks (hands, face, pose, objects) in Max jweb; same pattern, great starting point
• google-ai-edge/mediapipe-samples-web — official MediaPipe web demos including the audio classifier reference implementation

---

License

MIT — do whatever you want, credit appreciated.