We were given the incredible opportunity to have a free trip to Japan where the World Expo 2025 was taking place. We first worked on music and AI challenges in a Buddhist temple called 'Gyokuzoin Temple' on Mount Shigi in Nara Prefecture, and built a solution using Logitech's VR-pen, the MX Ink.
Group Photo - All Hackers at the Temple: ↗
We designed the pen to function as an equalizer. You map frequency bands to one of the x, y, or z axes and then simply move the pen around to hear the music change in real-time. We called our project 'Slight Adjustments with AI', or 'Slai'. We brought our prototype to a working state and let fellow participants interact with the system at a demo station we set up.
The VR-pen features multiple infrared emitters, enabling precise 3D tracking through devices like the Meta Quest 3. We couldn't rely on GPS or simple computer vision tracking for this project, as these methods lack the precision required for applications like orchestral conducting or professional DJing.
However, the pen's tracking depends on remaining within the Quest headset's field of vision, which we considered acceptable for our proof of concept.
Interface Overview - Trajectory Plot and Equalizer: ↗
On the left side of our interface, we displayed a real-time plot showing the pen's trajectory in 3D space for a user-defined time period (e.g., last 2 seconds). On the right side, you see the equalizer with frequency bands that can be mapped to the x, y, or z axes. The rightmost section features a 'click me' button for selecting specific tracks (such as a drumbeat, or perhaps Chicken Attack by Takeo Ischi), allowing users to control mid-tones with the y-dimension and piano with the z-dimension while adjusting drumbeat volume through the equalizer (where values <0 disable the drumbeat).
Demo Video - Proof of Concept: ↗
In our demo video, we mapped the x-dimension to a drumbeat and the z and y dimensions to some frequency bands for a Mozart piece. Note that I'm not a musician, so please judge the solution independently of my musical control skills! I intentionally used extreme settings to make the functionality clearly visible to the audience.
This video demonstrates the working system, though I was sleep-deprived during recording. I held the pen too high, making some movements less visible on the left plot (trust me, it normally displays beautifully), and as someone without musical training, I focused on demonstrating extreme volume variations across frequency bands rather than subtle adjustments. I also forgot to scroll down to show the complete mapping interface. Ideally, I'd re-record this demo with better preparation, but given the hackathon's nature (<24 hours of working time with limited internet access in a temple in Japan, where we also wanted to explore and travel), what did you expect—a perfect pitch?
We experimented with using pen velocity (direct, 10ms, 100ms averages) and even acceleration (direct, 10ms, 100ms averages) instead of position, but concluded this felt unnatural for conductors. When a conductor raises their baton and holds it there, they expect the orchestra to maintain that dynamic level. Position matters: pen up, orchestra up. Of course, this creates the challenge of establishing the reference point (0,0,0), which we address through VR pen synchronization. However, we didn't advance our PoC enough to enable seamless reference point recalibration, which limited our live demo's effectiveness. We had ambitious plans—imagine playing Mozart, then suddenly stepping right and switching to Chicken Attack to surprise the audience! It would've been magnificent.
The name 'Slight Adjustments with AI' reflects our vision: pen movements should create subtle musical changes, which we still believe represents the ideal final product. However, for our demo, we showcased extreme variations instead of subtle adjustments because, as non-musicians, it would've been difficult to convince the audience the system was working otherwise.
Our project is open source (check out our GitHub repository) and we invite others to draw inspiration and explore the pen's capabilities through different approaches. We implemented two 'backends' and one 'frontend'—if these terms still mean anything today—essentially ensuring location data flowed correctly to the source application where musical parameters could be configured.
Meeting Maywa Denki, Founder of Otamatone: ↗
We had the honor of meeting and listening to Maywa Denki, founder of the Otamatone, who, like us, was invited as a VIP of Austria to the expo. What a talented musician!
Thank you to everyone for this incredible event! Special thanks to Florian Bauer and Nobuko Okazaki for the outstanding organization!
Final note: Osaka is extremely humid. You aren't prepared. You cannot prepare. Just embrace it.