PhotoSynth > InterFACE

An extended-reality musical synthesizer – completely controlled by the face! Sometimes referred to as the Smile Powered Synthesizer =D

By piggybacking onto a person’s own natural movements using motion-tracking in real-time, we reduce the complexity required to learn a new instrument to mere moments whilst still retaining full expression and precise control.

Simply open your mouth to begin – it is a lot like singing. The head becomes the primary interface and the mouth the main control – we can convert smiles into musical notes and winks into audio controls. By observing the features on the face over time, we know at any point the angle and orientation of the face as well as basic facial emotions and moods, and we can use these in order to trigger and modify sounds.

This virtual interface has no moving parts, requires no extra hardware, and can run on most modern computers and mobile phones. We can capitalise on a person’s own expressions and movements to create unique personalised synthesized music but thanks to MIDI we can also control and manipulate almost any electronic musical instrument too.

Simply connect your MIDI instrument to instantly be able to play it with your head! Without any effort, the face is now an accessible monophonic MIDI controller, whilst with a friend in duet mode you can have polyphony too!

This is a zero-touch interface that piggy backs onto your natural movements to imbue them with new properties and skills. By recognising universally familiar movements, such as smiling and nodding, along with reading simple human emotions, we can form the basis of a new type of smart interface that can reduce the complexity of operation whilst still retaining full expression and control. Without any extra equipment or gear, your computer and studio can now be completely and simply operated by your face. The brief was simple but the end product requires much complexity in order to offer that simplicity :

● Requires no touch to be played
● Be suitable for all ages
● Have little to no learning curve
● Be accessible, inclusive and collaborative
● Be engaging, fun and a little bit magical

The scope of this project’s ability falls far beyond a musical instrument. This is as much a research project into perfecting natural-instinct interfaces for the future of human-machine interaction. Current state of the art motion capture devices are complicated, expensive and bespoke pieces of equipment – we intend to remove the need for any specialised equipment entirely and remove the cost so as to completely democratise accessible interface technology.

This is a collaborative project with a number of disabled artists and charities including accessible technology specialists, Drake Music Charity, who have been providing feedback and essential testing.

Ultimately it has dreams of becoming the most inclusive musical instrument and set a precedent for how interactions with other devices could be simplified in the future. Without any training or instructions this technology empowers anybody with a musical ability. We can use a person’s own expressions and movements to control and manipulate hardware, software and data. Music is a great form of expression but also one of precision and depth – if this software is successful in it’s endeavor it will be simple to then refine the technology to control practically anything. It will feature as an installation in the Accessible Musical Instrument Museum which is the world’s first accessible technology site dedicated to representing and demonstrating musical equipment that anybody can play, but thanks to the internet this software can also be in every home and school and as it works on mobile devices, this can be used by anyone anywhere too. This technology has the capacity to provide people new capabilities and opportunities. It aims to bring more equality to skills and leverages technology to empower and include the most diverse and varied audience – giving confidence to the most underrepresented and unprivileged.