Netz

Netz is a novel AI-powered keyboard-like mixed reality (MR) musical instrument developed by Max Graf and Mathieu Barthet at Queen Mary University of London. It is inspired by Euler’s Tonnetz which provides a method to organise pitches according to the musical intervals between them. The Netz interface appears as a network where nodes represent notes. Each note is directly connected to six consonant notes, and simple or complex chords can be obtained by playing groups of connected notes. The Netz interfaces is a so-called isomorphic keyboard where the same sequence or combination of intervals is obtained with the same pattern in different keys. Netz extends the ideas of the Tonnetz to enable expressive musical performance in mixed reality…
Contrary to virtual reality (VR) which disconnects users from their environment, mixed reality blends virtual and real worlds by displaying 3D virtual objects onto the real environment. The Netz MR instrument can be performed by keeping awareness of the physical world, and by being able to interact with studio equipment or other musicians and the audience when playing live. The Netz interface can be mapped to a tangible surface for tactile feedback or be positioned in the air.
Netz is a self-contained instrument as it features an embedded sound engine producing sounds directly from the head-mounted display (e.g. Meta Quest 2). However, Netz can also be used as an MPE MIDI controller to interface with external virtual instruments over Wi-Fi or USB. Notes in Netz have MPE capabilities enabling performers to individually control their dynamics, pitch bend, timbre (filters), as well as other assignable attributes. Polyphonic glissandos and portamentos are also supported.
The AI technology integrated into Netz further enhances its interactive capabilities. Using machine learning algorithms, Netz tracks the performer's hand poses and gestures in real time. This not only enables a more natural interaction in MR but also opens new possibilities for controlling instrument parameters. Netz can translate subtle hand movements to expressive musical controls. Gestures such as the opening and closing of fingers, or intricate movements like wrist rotation, are recognised and interpreted by the system. Each of these can be assigned to specific instrument parameters and MIDI controls. This feature allows for a high degree of expressivity and personalisation, as performers can design their unique gestural vocabulary to control the instrument.

Netz enables expressive, interactive performances by integrating virtual and real environments, allowing musicians to manipulate dynamics, pitch, and timbre through gestures including hand poses. The embedded sound engine enables flexible instrument positioning in the physical space.
The use of artificial neural network (ANN) models for hand pose recognition in Netz allows for highly accurate, real-time classification of complex hand poses and gestures. This enables musicians to interact with the instrument in an intuitive manner. Performers can train a model quickly from within the immersive application. The learned gestures can be mapped onto different control parameters in the instrument. By employing ANN models, Netz aims to push the boundaries of traditional musical interfaces, offering an engaging experience that fosters creative exploration.

Netz enables embodied musical interaction with a virtual and extendable instrument that does not disconnect the performer from her or his physical environment. From home music production to live ensemble performance, Netz can be used in a hybrid way. Performers can switch between physical instruments and Netz seamlessly, or even play both simultaneously.