Massive Technologies releases major update to AR Pianist with new MIDI and Audio features
Massive Technologies (MT) newest AR Pianist update shows the unique power of combining MIDI Data with AI and VR technologies and is an incredibly engaging combination of new technologies.
They gave The MIDI Association the inside scoop on their new update to AR Pianist.
One of the major new features is the ability to import MIDI files to create virtual performances.
We’re excited to announce that a major update of AR Pianist is to be released on May 25th. We’ve been working on this update tirelessly for the past two years.
The update brings our AI technology to users’ hands, and gives them the ability to learn any song by letting the app listen to it once through the device microphone.
Using AI, the app will listen to the audio, extract notes being played, and then show you a virtual pianist playing that song for you with step by step instructions.
The app also uses machine learning and augmented reality to project the virtual avatar onto your real piano, letting you experience the performance interactively and from every angle.
Users can also record their piano performance using the microphone (or MIDI), and then watch their performance turn into a 3D / AR virtual concert. Users can share it as a video now, and to VR / AR headsets later this year.
The update also features songs and content by “The Piano Guys”, along with a licensed Yamaha “Neo” designer piano.
by Massive Technologies
A.I. Generates 3D Virtual Concerts from Sound:
“To train the AI, we brought professionally trained pianists to our labs in Helsinki, where they were asked to simply play the piano for hours. The AI observed their playing through special hardware and sensors, and throughout the process the pianist and we would check the AI’s results and give it feedback or corrections. We would then take that feedback and use it as the curriculum for the AI for our next session with the pianist. We repeated that process until the AI results closely matched the human playing technique and style.”
by Massive Technologies
Massive Technologies used MIDI Association Member Google’s Tensor Flow to train their AI model.
The technology’s main potential is music education, for piano teachers to create interactive virtual lessons for remote teaching—or for virtual piano concerts, and film or games creators who want to incorporate a super-realistic pianist in their scenes.
The key to it all is MIDI
If you look at the work being done by Google, Yamaha, Massive Technologies, The Pianos Guys and others in the AI space, MIDI is central to all of those efforts.
Why? Because MIDI is the Musical Instrument Digital Interface so to connect with AI and Machine Learning algorithms you usually have to convert Music into MIDI.
How Does AR Pianist work and what can you do with it?
AR Pianist combines a number of proprietary Massive Technologies together.
- Multi pitch recognition
Massive Technologies’ in house ML models can estimate pitch and extract chords from audio streams, on the fly, in realtime.
This allows you to convert audio files of solo piano recordings into MIDI data that the AI engine can analyze. Of course you can also directly import MIDI data.
- Object pose estimation
Their proprietary models can estimate the 3d position and orientation of real instruments from a single photograph.
This allows you to point your mobile device’s camera at your 88 note keyboard. The app can then map your keyboard into 3D space for use with Augmented Reality.
- Motion synthesis and 3D Animation Pipeline
MT developed new machine learning algorithms that can synthesize novel and kinematically accurate 3d musical performance from raw audio files, for the use in education and AR / VR. Their tools can perform advanced full body and hand inverse kinematics to fit the same 3d musical performance to different avatars.
This is the part that almost seems like magic.
The app can take a MIDI or Audio performance (the Audio performance should be piano only), analyze it and generate musically correct avatar performances with the correct fingerings and hand positions including complex hand crossovers like those often used in classical or pop music (think the piano part from Bohemian Rhapsody).
- Music notation rendering, in 3D
Massive Technologies has built a notation rendering engine, that can be used to display music scores in 3D and inside virtual environments, including AR / VR.
This allows you to see the notation for the performances. Because the data is essentially MIDI like data you can slow the tempo down, set the app to wait for you to play the right note before moving forward and other practice techniques that are widely used in MIDI applications.
A.I. Plays Rachmaninoff Playing Himself (First Person View):
An audio piano roll recording of Rachmaninoff himself playing his famous Prelude, from 1919, reconstructed into 3d animation by Massive Technologies AI.
A virtual camera was attached to the virtual avatar’s head, where its movement is being driven by the AI, simulating eye gaze and anticipation.
Massive Technologies is Fayez Salka, M.D Medical Doctor, Musician, Software Developer and 3D Artist and
Anas Wattar, BCom Graduate from McGill University, Software Developer and 3D Artist.
AR Pianist is available for on the Apple App Store and Google Play store
The app is free to download and offers in app purchases for libraries of songs. You can check out Jon Schimdt of the Piano Guys virtual demoing the AR Pianist at any Apple retail store.
Check out the press release with details on the original app introduction
