Categories: Software Prototypes/Non-Commercial Products

Submitted by:Max Graf and Mathieu Barthet

24 May 2023

Elevator Pitch


Discover Netz, an immersive mixed reality musical instrument and controller featuring a keyboard-like interface with expressive MPE MIDI that learns your gestures using AI.

Product Description


Netz is a novel AI-powered keyboard-like mixed reality (MR) musical instrument developed by Max Graf and Mathieu Barthet at Queen Mary University of London. It is inspired by Euler’s Tonnetz which provides a method to organise pitches according to the musical intervals between them. The Netz interface appears as a network where nodes represent notes. Each note is directly connected to six consonant notes, and simple or complex chords can be obtained by playing groups of connected notes. The Netz interfaces is a so-called isomorphic keyboard where the same sequence or combination of intervals is obtained with the same pattern in different keys. Netz extends the ideas of the Tonnetz to enable expressive musical performance in mixed reality…
Contrary to virtual reality (VR) which disconnects users from their environment, mixed reality blends virtual and real worlds by displaying 3D virtual objects onto the real environment. The Netz MR instrument can be performed by keeping awareness of the physical world, and by being able to interact with studio equipment or other musicians and the audience when playing live. The Netz interface can be mapped to a tangible surface for tactile feedback or be positioned in the air.
Netz is a self-contained instrument as it features an embedded sound engine producing sounds directly from the head-mounted display (e.g. Meta Quest 2). However, Netz can also be used as an MPE MIDI controller to interface with external virtual instruments over Wi-Fi or USB. Notes in Netz have MPE capabilities enabling performers to individually control their dynamics, pitch bend, timbre (filters), as well as other assignable attributes. Polyphonic glissandos and portamentos are also supported.
The AI technology integrated into Netz further enhances its interactive capabilities. Using machine learning algorithms, Netz tracks the performer's hand poses and gestures in real time. This not only enables a more natural interaction in MR but also opens new possibilities for controlling instrument parameters. Netz can translate subtle hand movements to expressive musical controls. Gestures such as the opening and closing of fingers, or intricate movements like wrist rotation, are recognised and interpreted by the system. Each of these can be assigned to specific instrument parameters and MIDI controls. This feature allows for a high degree of expressivity and personalisation, as performers can design their unique gestural vocabulary to control the instrument.

How It's Innovative

Netz enables expressive, interactive performances by integrating virtual and real environments, allowing musicians to manipulate dynamics, pitch, and timbre through gestures including hand poses. The embedded sound engine enables flexible instrument positioning in the physical space.
The use of artificial neural network (ANN) models for hand pose recognition in Netz allows for highly accurate, real-time classification of complex hand poses and gestures. This enables musicians to interact with the instrument in an intuitive manner. Performers can train a model quickly from within the immersive application. The learned gestures can be mapped onto different control parameters in the instrument. By employing ANN models, Netz aims to push the boundaries of traditional musical interfaces, offering an engaging experience that fosters creative exploration.

See MIDI Innovation In Action

Most Inspiring Use Cases

Netz enables embodied musical interaction with a virtual and extendable instrument that does not disconnect the performer from her or his physical environment. From home music production to live ensemble performance, Netz can be used in a hybrid way. Performers can switch between physical instruments and Netz seamlessly, or even play both simultaneously.

Expansion Plans

We envision the following expansion plans:
1. Product finalisation: by continuing our research and development and user evaluations, we aim to refine the user interface and user experience of Netz. We will integrate our novel multimodal deep learning model for hand pose estimation, which combines wearable (surface electromyography armband) and HMD signals. The model provides a low latency solution to keep track of hand gestures in self-occlusion situations, improving performance over hand-tracking based on the HMD alone.
2. Diversify the Netz product line: we envision to build variants of Netz catering to different skill levels, musical genres, preferences, and budgets (e.g. alternative layouts, sound libraries, and features). Users will be able to adapt the instrument to their own liking, and the instrument ecosystem can be made modular so that users can build their own MRMIs.
3. Increasing exposure: we aim to work with artist endorsers and promoters to showcase Netz in live venues and music festivals.
4. Cross-platform compatibility: we will make Netz compatible with various MR headsets and platforms to maximise its accessibility and market reach.
5. Collaboration with software developers: we plan to partner with developers to create custom applications, plugins, and extensions for popular digital audio workstations and music production software, further integrating Netz into the digital music ecosystem.
6. Develop a user community: we will encourage collaboration, knowledge sharing, and networking among Netz users by creating an online platform with forums, tutorials, and user-generated content.
7. Build other MRMIs: we also plan to develop other MRMIs, covering a wider spectrum of MR musical interactions.
8. Exploring non-musical applications: we will investigate the potential of Netz's technology in other creative fields, such as visual arts, gaming, or therapy, to find new avenues for growth and diversification.


Our route to commercialisation includes the following steps:
1. Spin-out company formation: we will investigate options to create a spin-out and incubator schemes.
2. Investment/licensing: we aim to secure funding through investment and/or licensing by partnering with music industry actors and investors.
3. Building partnerships: we aim to collaborate with music technology companies, musicians and artists to showcase Netz's capabilities, gather valuable feedback, and bring mixed reality musical instruments to the attention of individuals and the industry.
4. App development: the product will be packaged as an app for XR product platforms, such as the Meta Quest Store.
5. Marketing and promotion: a comprehensive marketing strategy will be set out including social media, online advertising, product demonstrations, and endorsements from musicians/artists. Video performances, tutorials, and testimonials will be used to showcase the instrument's features and applications.
6. Widening distribution channels: we aim to establish partnerships with MR and music retailers (e.g., Meta, HTC, Apple) to distribute Netz worldwide.
7. Education and workshops: we plan to organise workshops and online courses to educate musicians and producers about the benefits and capabilities of Netz and MRMIs, fostering a community of users and enthusiasts.
8. Customer support and continuous improvement: we will provide excellent customer support, including troubleshooting, software updates, and user guides.