Creating sound with gestural harmony: AethericGeometry

A build log on Aetheric Geometry, a musical instrument you play with your bare hands in front of a webcam. The first half is how hand geometry becomes harmony. The second half is the seven things that were wrong with it while it sounded perfectly good. Point a webcam at yourself. MediaPipe finds 21 landmarks per hand, and the geometry of your hands becomes the sound. Pinch both hands to start a thread between your index fingers. Bring them together, or make a pyramid, and the thread becomes a polygon whose vertices are your fingertips. That polygon is the instrument. Webcam frame → flip → MediaPipe → assign_hands() → waveform selector (5-frame debounce) → gesture detection (pinch, kiss, pyramid, cross, prayer, open palm) → state machine (IDLE / THREAD / POLYGON) → compute_poly_sound() → SynthEngine.set_params() → OpenCV render Pure Python. No JUCE and no external synth: dsp.py implements PolyBLEP oscillators, a TPT filter and a Schroeder reverb from scratch, and sounddevice plays them. It ships as a PyInstaller executable and has a pytest suite that needs no camera, no microphone and no audio device. The whole mapping is one function, compute_poly_sound(), and it is deliberately small: Geometry Musical parameter Why that axis centroid X pitch of the root left to right is the obvious pitch axis centroid Y filter cutoff raising your hands opens the sound number of fingertips number of voices in the chord you hold up the chord you want polygon area reverb wet, from 6 vertices a bigger shape is a bigger room aspect ratio tremolo depth, from 8 vertices stretching wide makes it pulse extended finger count waveform with a 5-frame debounce The third row is the one the instrument is built around. Every extended fingertip is both a polygon vertex and a chord voice, so the shape you make and the chord you hear are the same object counted two ways. Open one more finger and the polygon gains a corner at the same instant the chord gains a note. Two of the mappings are deliberately locked behind complexity. Reverb needs 6 vertices, tremolo needs 8. You cannot reach the full parameter space with a lazy gesture, which turns "more fingers" into a progression rather than a switch. Fraction, not float The chord is built on exact just-intonation ratios above a movable root: CHORD_RATIOS[8] = (UNISON, MAJOR_SECOND, MAJOR_THIRD, PERFECT_FOURTH, PERFECT_FIFTH, MAJOR_SIXTH, HARMONIC_SEVENTH, OCTAVE) and they are fractions.Fraction, not decimals. An earlier version stored 1.333 for the perfect fourth and 1.667 for the major sixth, which looks harmless. It is not, and the reason is the whole point of just intonation. Against the true 4/3, storing 1.333 is an error of 1200 · log2(1.333 / (4/3)) ≈ -0.43 cents which is inaudible as pitch. Nobody hears 0.43 cents. But just intonation does not exist to make intervals sound in tune, it exists to make partials of different voices land on exactly the same frequency so the beating between them disappears. And beating is a difference of frequencies, which is a far more sensitive measurement than pitch. Work it out. A fourth above 220 Hz is 293.333... Hz exact and 293.26 Hz rounded. The third partial of the root sits at 660 Hz. The second partial of the fourth sits at 586.67 Hz exact and 586.52 Hz rounded. Against the root's harmonic series those two disagree by roughly 0.15 Hz, which is a slow wow with a period of about seven seconds across a sustained chord. That is not a tuning error you hear as pitch. It is a tuning error you hear as movement, and it is precisely the artefact just intonation was chosen to remove. Storing exact fractions costs nothing, and the only rounding left in the system is one float conversion per partial at the final multiply. Here is what those exact ratios are, measured against the equal-tempered intervals they replace: The fifth is 1.96 cents sharp of tempered and the fourth 1.96 flat, which is why those two barely matter on a piano. The major third is 13.69 cents flat, the sixth 15.64 flat, and the harmonic seventh is 31.17 cents flat, nearly a third of a semitone. That last one is not a badly tuned minor seventh, it is a different interval: the one that appears naturally as the 7th partial, and that 12-TET has no room for. The voicings are ordered on purpose too. Voices enter as fifth, then major third, then octave, then ninth, then harmonic seventh, then fourth, then sixth: simplest ratios first, because the simpler the ratio, the sooner its partials coincide with the root's and the more strongly the two voices fuse into one sound rather than two. The instrument is justly tuned within a chord and equal-tempered between chords: the ratios are exact, but the root they sit on is quantised to a pentatonic subset of 12-TET. That is a real compromise and it is worth saying out loud rather than hiding. A fully just system has no single answer for what happens when the root moves, because stacking exact fifths never closes the octave. Twelve of them overshoot by the Pythagorean comma: (3/2)¹² / 2⁷ ≈ 1.0136 about 23.5 cents Somebody has to absorb those 23.5 cents. Choosing a fixed ratio set over a tempered root sidesteps comma drift entirely, at the cost of transposition purity. For an instrument you wave your hands at, that is the right trade. Two more details in the pitch path. The pitch map is exponential, not linear. f = f_lo · (f_hi/f_lo)^t over 110 Hz to 880 Hz, so a centimetre of hand travel is the same number of cents everywhere in the range. A linear map would make the bottom of the gesture range musically cramped and the top sparse. The root is quantised to A minor pentatonic before the ratios are applied. Five notes per octave means that with your hand anywhere in the frame, the root is one of a set with no bad intervals in it. The instrument cannot be played out of key, which for a gesture instrument with the precision of a human arm is not a limitation, it is what makes it playable. Here is the turn. dsp.py, tuning.py and knob.py are free of camera, audio and MediaPipe imports. The entire numeric core is importable with nothing plugged in. That sounds like ordinary hygiene. In practice it is what let me write analysis/measure_dsp.py, a script that renders buffers from the real DSP primitives and FFTs them. Every number below comes out of it, and I can regenerate all of it with one command and no hardware. Without that separation, checking "is my sawtooth aliasing" means opening the app, playing a note and listening. With it, it means a number. Here is what the numbers found. My favourite, because it is so easy to write and so hard to hear. The oscillator renders a block, then writes the final phase back for the next block. I wrote back the phase at the last sample, not the phase after the last sample. One sample short, every block, forever. The left panel should be a single spike at 440 Hz. Instead the block period stamps a whole harmonic series onto it, because a periodic discontinuity at the block rate is a modulator. The right panel shows where they land: sidebands at exactly fs/N = ±86.1 Hz around the carrier, at −40 dB. And the tuning error it causes: 3.385 cents flat. Not enough to sound out of tune on its own. Enough to be wrong against anything else, and note the irony: I had just spent all that care getting the ratios exact to fractions of a cent, on top of an oscillator that was 3.4 cents off. phase = (phase + n_samples * inc) % 1.0 # not (n_samples - 1) One character of arithmetic. A naive digital sawtooth (phase * 2 - 1) has a discontinuity per cycle, and a discontinuity has infinite bandwidth. Everything above Nyquist folds back. At 2 kHz the naive saw sits at about −12 dB alias-to-signal. A twelfth of the output is inharmonic garbage that moves the wrong way as you play up, which on an instrument built around exact ratios is an unusually expensive thing to get wrong. PolyBLEP corrects the samples immediately around each discontinuity with a polynomial approximation of a band-limited step, and the triangle is generated by integrating the BLEP square. Across the range that buys roughly 16 dB of alias rejection. It does not eliminate aliasing; nothing cheap does. It moves it from "obviously wrong" to "below what you will notice", for a handful of operations per sample. The reverb is a classic Schroeder: four damped comb filters in parallel into two allpass diffusers. Combs need different delay lengths; that is the point, it decorrelates them. I gave all four the same feedback gain g, which seems reasonable until you write down what T60 depends on: T60 = -3 · D / (fs · log10(g)) D is in there. The same g across four different delay lengths means four different decay times: measured, 1.12 s to 1.28 s, a 13.7 % spread. The tail does not decay, it stages. The short comb dies first, then the next, and the character of the reverb changes as it fades. The fix inverts the relationship: take one T60 target and solve each comb's own gain from its own delay length. Right panel: all four combs now cross −60 dB together, at the T60 you asked for, damped or not. Left panel is what I had. No allpass diffusion at all. Four combs with no diffusers are not a reverb, they are four echo trains, and on a percussive attack you hear them as four distinct repeats rather than a wash. Two allpass stages after the combs smear the echo density without touching the magnitude response. It is the difference between a delay bank and a room. Every envelope time changed with the sample rate. The envelope stored raw one-pole coefficients, and 0.999 means one thing at 44.1 kHz and something else at 48 kHz, so attack and release changed depending on which audio device you plugged in. Store the time constant in seconds and derive the coefficient at the current rate: coeff = math.exp(-1.0 / (tau_seconds * fs)) Every MIDI note was an octave flat. MIDI note 0 is C−1, not C0, so 12·log2(f/C0) is not a MIDI note number. Everything was consistently an octave off, which is why it sounded internally correct and only broke when the MIDI drove something else. The triangle overshot by 85 % on every note. The triangle comes from integrating a BLEP square, integrators need an initial condition, and mine started from zero. That is only right if the note begins at exactly the phase where the triangle is zero. Start anywhere else and the integrator spends its first moments walking to where it should have started: measured, 85 % overshoot for about 16 ms on every note onset. A click on every note, which I had mentally filed as "attack character". Defect Measured effect Phase written back one sample short −3.385 cents, sidebands at ±86.1 Hz Naive saw and square −12 dB alias-to-signal at 2 kHz One feedback gain shared by four combs T60 spread 1.12 s to 1.28 s No allpass diffusion four bare echo trains Envelope stored as raw coefficients every envelope time varied with fs 12·log2(f/C0) used as a MIDI note every note an octave flat Triangle integrator started from zero 85 % overshoot for ~16 ms per note Every single one of these sounded fine. One more, since I fixed the filter at the same time: the old lowpass used exp(-2π·fc/fs), which is 2 dB out by 20 kHz. The replacement is a two-stage TPT with a prewarped coefficient g = tan(π·fc/fs), which is exactly −3 dB at the requested cutoff. The instrument also takes voice commands, offline, via Vosk, in Spanish. "jarvis, ponme una onda cuadrada." This turned into its own measurement problem, for a reason I did not anticipate. It is keyword spotting, not language understanding: a result is scanned word by word and anything in the vocabulary fires. Which means ordinary conversation in the room plays the instrument. Four gates, each added because a measurement demanded it: Gate Question it answers Measured without it Final results only is the utterance finished? partials fired 11 commands in 25 s of silence min_level (peak RMS) did anyone actually speak? room tone fired one per 45 s at high confidence Length triage was it meant for us? 11 commands in 40 s of ordinary room sound min_confidence which word was it? wrong-word substitutions on noisy input The third is the interesting one. Requiring a wake word for everything would make the instrument unplayable, because you have to be able to bark "eco" mid-performance. Requiring nothing lets the room conduct. So length decides: Utterance Wake word? Result congela no fires, short commands need no ceremony sube el eco no fires, still short pues mira sube un poco mas no dropped whole, despite two command words in it eter pillame el eco y baja treinta por ciento yes parsed fully, selects reverb, lowers it 30 % A long utterance without a wake word is discarded entirely, not scanned for the commands it happens to contain. That one rule is what made voice control usable in a room with other people in it. A Vosk grammar can only contain words the model already knows. Anything else is silently dropped with a warning on native stderr, so your command simply never fires: no error, no clue. The Spanish lexicon is unaccented. mas, triangulo, atras exist. más, triángulo, atrás do not. There is now a test enforcing ASCII-only Spanish keys. The obvious technical words are missing. Neither sinusoidal nor reverb is in the Spanish model, and tremolo is in neither model. The plain words (seno, eco, temblor) are what work. English sine is /saɪn/, not "SEE-neh". A Spanish speaker reading the word aloud produces something the English model cannot map, hence round, smooth, pure. aether is not in the Spanish lexicon. The wake word is eter, which is. Filler words earn their place. ponme, una, onda map to nothing, but without them in the grammar the other words of a sentence smear onto commands. Every voice command has a keyboard equivalent, so the instrument never depends on the microphone or on Vosk being installed at all. A different flavour of the same lesson. Text goes through Pillow with real TrueType faces rather than cv2.putText, because the Hershey fonts OpenCV ships are single-stroke vector outlines with no hinting or kerning, and they were the main reason the interface looked unfinished. Panels rasterise into RGBA tiles at 2× and box-filter down, which antialiases Pillow's arcs and rounded corners for free. Doing all that from scratch every frame cost 39 ms. Four changes: Change Effect Cache rasterised tiles keyed by content most frames became a blit Premultiply and convert RGB to BGR once at rasterise time blit 5.9 ms to 1.4 ms Composite with OpenCV instead of numpy expressions SIMD, no temporaries Split the hold panel into chrome / dial / rows tiles turning a knob 21 ms to ~6 ms Steady state: 5.7 ms, and 11.6 ms while a knob is turning. One trap worth the price of admission. The effect-meter loop in draw_status_bar used key as its loop variable, which shadowed the cache key computed just above it. Tiles were stored under the wrong key and the cache never hit once. The panel looked completely correct and simply ran at full cost, forever. Only the profiler knew. Make the gesture and the harmony the same object. Fingertips are polygon vertices and chord voices at once, so there is nothing to learn twice. Exact ratios matter for beating, not for pitch. 0.43 cents is inaudible as tuning and audible as a seven-second wow across a sustained chord. Name your compromise. Just within the chord, tempered between chords, because the Pythagorean comma has to go somewhere. Split your numeric core from your I/O. Not for testing purity, so you can measure it. Every number here exists because dsp.py imports nothing that needs hardware. "It sounds fine" is not evidence. All seven bugs sounded fine, and two of them I had unconsciously reinterpreted as character. Write the formula down. T60 = -3D/(fs·log10 g) has D in it. Reading that once would have saved the shared-gain bug. Profile before you optimise, and read your loop variables. A 39 ms frame caused by a shadowed variable is not something you reason your way to. Aetheric Geometry is on GitHub, PyInstaller-packaged, with analysis/measure_dsp.py reproducing every measured figure above. It also became the engine underneath AirStems, which won the LALAL.AI Special Prize at the Musixmatch Musicathon 2026: same hands, but what they play is the separated stems of a real song. I'm an audio DSP student at UPC. I build VST plugins and instruments you play with your hands.