Gesture Synth: What Real User Experiments Reveal
Gesture Synth is a good fit for people who want to explore chords and musical expression with hand movements; it is a weaker fit for players who need precise note-by-note control or fully verified song charts. The main friction reported by users is learning the gestures and checking community arrangements for missing sections or mismatched chords.
Gesture Synth at gesturesynth.com is a browser-based hand-tracking instrument. It has two modes: Gesture mode uses the left hand for chords and the right hand to shape their sound; Theremin mode maps the left hand to volume and the right hand to pitch. (project documentation)
In Gesture mode, raising one to five fingers on the left hand selects chord degrees I through V. The index-and-pinky combination selects VI, and adding the thumb selects VII. Wrist tilt changes between major and minor. On the right hand, one to four raised fingers select chord voicing; hand height controls volume, wrist tilt controls filter tone, and an extended thumb lowers the octave. (project documentation)
The published requirements call for a modern browser, with Chrome or Edge recommended, and a webcam. The project documentation says microphone permission is not required; camera access requires HTTPS or localhost. Those requirements describe the browser instrument in this repository. (project documentation)
The name can be confusing because other projects also use “Gesture Synth.” Eric Wei’s project page describes a separate browser instrument designed to let someone play chords with their hands while singing the melody. The reports below concern the instrument connected to Ekmand’s repository and community, with related Reddit experiments identified as separate projects. (Eric Wei’s project page, Ekmand’s project documentation)
What first-time use can feel like
One Reddit tester tried gesture.live, a separate project whose maker described it as an exploration inspired by Gesture Synth. The tester initially said they had trouble getting the melody to work when they wanted it to. In an update, they reconsidered and said they were probably performing some gestures incorrectly. (Reddit discussion)
For a new player, a wrong-sounding melody may be a gesture-learning problem rather than a tracking failure. The Reddit correction changes the diagnosis: the tester first blamed melody control, then suspected their own hand movements. That account is from a related project, so it does not establish how often this happens in gesturesynth.com. It does show why a new player should learn and check the gesture vocabulary before treating every missed note as a system fault.
A different Reddit post discussed a maker’s custom hand-tracked music project, also inspired by a gesture-synth website. The maker said tracking delay sometimes made it hard to control exact pitch with the other hand. That delay report belongs to the maker’s own build; it is not a measured flaw in gesturesynth.com. (Reddit discussion)
A hand gesture can feel expressive for broad chord movement and still frustrate note-by-note timing. This is a design distinction suggested by the two Reddit accounts, not a benchmark of the official instrument. One report is about a tester correcting their understanding of gestures; the other is about delay in a separate implementation. Keeping them separate avoids turning adjacent user experience into a product-wide claim.
What community arrangements reveal
The Gesture Synth community lets people publish arrangements for others to play and practice. The entries include user-written guidance, such as instructions to repeat a chord sequence or notes about which section of a song an arrangement covers. (Gesture Synth community feed)
One first-time contributor said they did not know how to notate suspended chords in the editor. They wrote that their arrangement covered only verse one and the chorus because verse two used those chords. The missing verse is a concrete content gap the contributor connected to an editor limitation, not just a performance choice. (Seasons arrangement in the community)
The same contributor also wrote that a minor section was supposed to be followed by Bb and C, but that the selected key meant the arrangement would not teach that part; they said they planned to fix it. When a key setting leaves out an intended chord, a practice arrangement can teach the wrong harmonic path even if its notes are playable. (Seasons arrangement in the community)
Another contributor added, “idk if I made this right tbh,” to an arrangement. That uncertainty is a cue for learners to check an arrangement, not proof that all community charts are inaccurate. The community provides playable material, but these posts also show that some arrangements come with partial coverage or unresolved author questions. (needy arrangement in the community)
The community arrangements are not only songs to play; they also reveal what the editor lets contributors describe. Users supply both musical material and some of the instructions needed to work through it. A learner may therefore need to check an arrangement’s key, chord sequence, and section coverage before using it as a complete guide. These are practical checks drawn from the examples above, not a claim that every arrangement has a problem.
Recording and sharing
A contributor on the project’s GitHub issue page described teenagers recording themselves singing over Gesture Synth backing tracks with separate screen-recording tools or by filming the screen with a phone. The contributor proposed adding recording inside the instrument and shared a test that captured webcam video and microphone audio, with settings to change the selected devices. (recording feature request)
In the proposed test, stopping a recording downloaded the video, and the contributor said it was not saved on the server. The test also included options for monitoring the microphone and recording without visual overlays. The project maintainer replied that they would review the code. The linked pull request page says its branch has not been deployed. (recording feature request, pull request status)
This request came from a reported sharing workaround: separate capture tools or a phone camera. Until the proposed change is deployed, users should not assume the project has an in-app recording control. The issue documents one contributor’s experience and proposal; it does not establish how many users need the feature.
A sensible first test is to check that the camera sees the hand, try a single chord change, and then play an arrangement while checking its key and section coverage. Gesture Synth is best approached as a hands-on way to explore chord movement, with precise control and arrangement accuracy checked separately.