Creating a brain controlled musical instrument

Wednesday, 24 September 2008

I first became interested in creating a brain controlled musical instrument after meeting Scott Eggert in late 2007. Scott has extensive experience in this area in the past -- we both imagined that a brain interface like this would be fun and extremely useful as a controller for audio platforms.  We then started a plan to create a system that accepts can input information from Pure data to Ableton Live. Our first plan of action was to choose a controller. Scott had worked on a device many years ago that has evolved into a product now available on the market, the NIA. The NIA focuses receiving on generalised rather than than specific brainwave activity. It also has a following in the gaming community and makes use of muscle and eye glance movement.  This device can output both MIDI and keystrokes which are actuated from various brain waves. To use the controller as an out of the box musical instrument, we first had to learn how to interact with the controller. It was not easy at first, but with practice, it became much easier to invoke the various signals.  Initially it was difficult to get waves in the higher alphas and betas to work properly.  Over time we discovered that relaxation is the key to the successful utilisation of these frequencies.    As i had never used the device before I made a number of observations about the way I found it to respond.  The easiest part to control is definitely the muscle and glance parameters. The glance particularly would give a very accurate reading of left to right. Up and down was easy when play the pong game that comes with the device, however, once in Ableton Live, getting the volume to read downwards was much more difficult.  This may be caused by the software, or may be a more indicative of a more difficult feat to accomplish with the brain. We have done a number of tests and have so far been unable to connect the device as is into Pure Data or Max/MSP but Scott is working on a variety of methods, including the creation of a TCP socket that will send the data to these programs. In the meantime we have been using the controller to input computer keyboard data into Ableton Live, and, coupled with the software that came with the device, have been able to trigger samples, make melodic instruments, and, with practice, move volume sliders and effects modulation parameters . The new prototype we are currently working on focuses on the brain controller as a MIDI device, able to be connected to a diverse range of hardware or software, and allow artists to sonify, or visualise the output of their brain and head muscles.   Our prototype will be hardware agnostic, so we can test it on a variety of hardware solutions in the marketplace.  We are also integrating a variety of pattern matching systems into the tool, so more precise muscle sensing and better emotional receptivity can be detected.   Our interests lie in the musical and sonic potential of the interface, how it can be used in performance, as a biofeedback device, emotional gauge, a trigger device or control interface.  Its uses to artists and disabled persons is unlimited.    We are currently building a team of researchers and programmers, and are looking for research funding opportunities to assist in continuing in this pioneering field.

 

Next >