Talk into the microphone and the mouth follows. Visemes are detected in the browser from loudness, pitch, formants and hiss, then blended across the 17 mouth shapes with idle breathing, blinks and emotions on top.
Hold any shape above to drive the mouth by hand. That is the same call a WebSocket handler would make.
Drive it
Hold a long "ee", "ah" and "oo": the dot should land in I, A and U. If vowels read one region off, nudge Voice range. Raise the gate if room noise moves the mouth.
Preview only: letters are mapped to shapes by rule and re-synced on each spoken word. Your real lip sync replaces this.
With a cue file, the mouth follows audio.currentTime exactly, including seeks. Without one, it falls back to loudness.
Emotion
Actions
Wire in your lip sync
// Sprite layers cut from the reference sheet
const sprites = AvatarSprites.create(AVATAR_SPRITES.manifest, n => AVATAR_SPRITES.images[n]);
const rig = new AvatarRig(canvas, { draw: sprites.draw });
// Microphone → visemes → mouth, all in the browser
const mic = new MicVisemes({ onViseme: (v, w) => rig.setViseme(v, w) });
await mic.startMic(); // https or localhost only
// mic.attachElement(audioEl); // same detector on TTS playback
// A. Cues with timestamps (Rhubarb / Azure / your own)
rig.play(AvatarRig.toCues(json), () => audio.currentTime);
// B. Streaming TTS: start a clock, add chunks as they land
const t0 = ctx.currentTime;
rig.play([], () => ctx.currentTime - t0);
ws.onmessage = e => rig.append(AvatarRig.toCues(JSON.parse(e.data)));
// C. One viseme at a time, live
ws.onmessage = e => rig.setViseme(e.data); // 'O', 'M', 'REST'…
// Character state
rig.setEmotion('happy');
rig.trigger('wave');
// Your own layered art
new AvatarRig(canvas, { draw(ctx, f) { /* f.viseme, f.mouth, f.emo, f.blink, f.pose */ } });