Welcome: what am I doing?

Welcome to my diary of the hunt for a way to walk inside a piece of music!

I’ve been composing impossible music for two decades, using the digital studio to simulate reality.  The challenges to true realism in simulated performance are still enormous and I look forward to my forthcoming research fellowship with Taiwan’s brilliant SCREAM-Lab working on performance modelling.

In his visionary 1936 essay “The Liberation of Sound”, post-futurist composer Edgard Varèse wrote of ‘Zones of Intensities’ where listeners could differentiate the timbres and lines of a composition:

“Today with the technical means that exist and are easily adaptable, the differentiation of the various masses and different planes as these beams of sound could be made discernible to the listener by means of certain acoustical arrangements. Moreover, such an acoustical arrangement would permit the delimitation of what I call Zones of Intensities.”

What those acoustical arrangements might have been is unclear but we will be proposing a configuration of binaural audio simulation controlled by motion sensors, allowing the listener to differentiate the masses, planes and beams, finally delineating the zones after all. Prototype demo early in 2013.

Posted in Locative Music | Tagged , , , , , , , , , , , , | Leave a comment

1st prize at #SXSC3 Dragon’s Den Business Plan Competition

It’s a little hard to take in winning this incredible prize!

It will go a long way towards getting 3DBARE to proof-of-concept and beyond. 

Many, many thanks to the panel and the conference organisers of #SXSC3 !

For more about 3DBARE, our engine for Music You Can Walk Inside, and to get involved, please get in touch via 

Twitter (#benjaminmawson),  Soundcloud (benjamin_mawson) or Linkedin.

Some key things about 3DBARE: 

'Threadbare'

3D Binaural Audio Rendering Engine

  • 3DBARE packages site- and event-specific audio content, creating a connection between digital and real-world experience through being situated at a physical location.software to give listeners virtual experience of walking inside music.
  • translates motion-tracking into multi- channel binaural audio rendering.

It will be used at a new kind of event that combines elements of concert, exhibition, silent disco, funhouse, public garden, dreamscape.

3DBARE allows listeners to move and make a virtual exploration of a soundscape: permits annotation of a space with audio that enhances both listening experience and individuals’ connection to a place. 

Why we think 3DBARE is essential:

• Digital studios permit ‘impossible’ music and sonic textures of greater complexity than can be fully heard through standard (loudspeaker) relay.

• Loudspeaker listening same as a CD or a gramophone record – it is passive and fixed, identical on each audition.

• Loudspeaker spatialisation is unconvincing, complex and expensive.

• Wireless headsets offer freedom to investigate sound as though it were a physical structure.

• This permits recorded digital sound to be explored from continually changing perspectives.

• 3DBARE achieves a significant step towards “digital liveness”.

To find out more and to get involved – send us your email address by comment or social platforms, above. 

 
Posted in Locative Music | Leave a comment

Technical implementation of 3DBARE

I’ve been working with Iyad Assaf on implementation of 3DBARE for some months now. He’s the coding guru on this team.  My initial concept for 3DBARE was inspired by the visionary avant-gardist Edgard Varese who, writing in the 1930s was able to envision technical solutions to musical problems that would not materially exist until decades later. There remains a significant gap between the ‘impossible’ music it is now possible to make in the digital studio, and the still very limited, fixed means of listening which we use. 3DBARE is the search for what Varese refers to as ‘zones of intensities’, the ‘differentiation of masses of sound, hitherto unobtainable’.

Back in 2011, I approached and eventually persuaded  ISVR (Institute of Sound & Vibration Research) to get on board and build an initial 3D audio engine. I remember the naïve optimism of thinking that this would be the bulk of the work done.

They built two interesting engines that were in some ways impressive, with different benefits and drawbacks, I’ll write more elsewhere about these, but they had a common problem. They guzzled data at such a rate that they’d only work with a single user listening to very few sounds at once. This was to make both tools essentially irrelevant to our intended purpose: for a sizable audience to walk inside an apparently live performance and hear it according to the way they explored the physical space.

Iyad heard about some of the work I was doing with  GPS-based tracking for music listening, using noTours software and offered to help out: a partnership was born. Over the past months, he has been working on a number of possible solutions for the real crux of the engine, the tracking controllers.

After trying for some time to obtain a workable multi-user solution to ISVR’s engine, we went back to the drawing board.

Iyad’s initial, apparently simple task of creating a user interface to control the second of ISVR’s audio engines was to turn into a complex ongoing development project which is at last yielding some very interesting fruit.

The ISVR engine created a 3D virtual sound world by convolving head related transfer files (HRTF), made by recording sounds from different orientations around a dummy head (a mannequin with microphones where the ears should be). It worked reasonably well with very few audio sources but became completely inefficient with more than a couple of sounds. It was built using Max MSP, which is intrinsically data-heavy and difficult to implement in other platforms due to lack of readily accessible text-based code (the user works in a graphical layout).

The new interface was created using Processing, a java-based creative coding platform used widely by artists, and controlled a much simpler binaural plugin, again using MAX. The UI allows mapping where audio should be placed in a virtual space in relation to the listener. At it’s current state, the user can export the source data to be used on an independent device.

processing_jhg

This is what it looks like (the listener is the black dot at the hub of radiating red lines, individual sounds are marked as blue circles. Only the ones connected to the listener by red lines are audible, for a good reason I’ll explain later).

The creator of the second engine at ISVR was doing a Masters in Acoustics and about to head straight off into a job in sound proofing. He was therefore pretty bemused by our excitable urgency regarding imaginary musical instruments dotted around an otherwise empty room. We couldn’t always work out what he’d done or why, in the programming and communication  difficulties finally led to starting from first principles on a completely new audio engine.

Iyad started in MAX MSP, using an external plug-in called ‘binaural+’, which did all of the directional processing, removing the need to load individual HRTF files. The engine was designed to be controlled by the user interface (above), creating an instance of the binaural panning external for each audio source, scaling the loudness depending on user distance from a source and mixing it with all other sources to create the impression of multiple sources emanating from points in the physical space.

We were still limited to a single plane of rotation – a horizontal ring in which to rotate sounds’ positions – and this is of course a major block to realism.

Acknowledging the complexity of adding the other two planes (or a total of “6 degrees of freedom”) we also remained convinced of the need to develop in such a way that would allow these to be added later without complete reconstruction. At first though, without much thought for how we would track the user, the concept was to have a base computer performing extensive audio processing and sending the audio to the audience members via wireless headsets.

We remembered how much data was used by the first audio engine, even with a single user turning an onscreen rotary slider to create the binaural panning and listening to just two audio sources. (This engine too, worked not in 3D but, in single planar rotation only. Rather than sounds everywhere, as you would hear in the real world, the effect was similar to 1970s and 80s ‘ambisonics’ with rings of speakers around a seated audience.)

So we looked at how, instead of streaming all the data not only about a listener’s movement but the audio itself – as we intended to do for lots of listeners at once! – we could use independent mobile devices.

This had several advantages, perhaps the biggest of which is that all of the audio processing is onboard. These devices use wireless technologies like Wi-Fi and Bluetooth to track the user, the application can be downloaded and used by many independent listeners at the same time and it can be implemented at low cost.

Mobile devices and Max/MSP don’t work together so if that was our framework, the code  would need complete rewriting for compatibility with mobile operating systems.

We researched hardware and coding solutions that innovators around the web have been developing at incredible speed over the past year or two. Whether to start with  iOS, Android or even the raspberry pi?

The current way forward is to take a version of the processing interface online, using openAL  (Open Audio Library), a cross platform, open source audio library that can place audio in a 3D environment. In this way we can demonstrate the audio rendering remotely. Here the tracking is user-operated as an avatar seen from above.

The library responds to relative vector-based positions of listener and source to control perceived distance and direction of sounds.

Similar libraries such as FMOD are used in many first person video games to give a greater sense of presence and immersion into the gaming environment where for example you might hear a vehicle approaching from behind, where the audio library ‘places’ the sound behind the user with realistic spatialised effect.

Developing for iOS and testing on the almost identical OSX platform, substantial experimental work has now been completed into the audio playback system. One influential finding was that openAL is not optimized for lengthy synchronized audio files, which the project requires, so the audio has to be played through the “audio toolbox framework”, and then streamed into openAL.

Ultimate development for Android is also anticipated although this is going to require another complete reengineering of the code to work with this very different platform.

Our task at the moment is to have a working prototype for iOS, written in a language called Objective-C, (Android works on Java).  Both platforms do however support the openAL framework.

The main aspect to the project at the moment is developing really accurate tracking and using this to control the audio rendering as the listener moves, to determine their proximity and orientation in relation to each virtual source.

In terms of user technology, we have looked at two equally plausible systems, infra-red camera tracking and wireless signal strength tracking. Infra-red involves several cameras at different points around a space and software to recognize where in the room a participant is. This is highly dependent on camera positioning, lighting and it can be tricky to eliminate blind-spots.

Wireless signal tracking works by measuring Bluetooth or Wi-Fi signal strength of several anchor points – if the receiving device (smartphone) knows where the anchor points are, it can roughly determine it’s own location. An issue with this is that the signal strength can fluctuate significantly in different environmental conditions.

Iyad is now working with wireless signal strength tracking via Bluetooth; the processing can be done on the device and the lowest cost option for implementation over a wide physical space. Signal strength fluctuations can be minimized using smoothing algorithms based upon static anchor point locations. Both Bluetooth and Wi-Fi are of this, but since Apple’s removal of Wi-Fi support Bluetooth is the chosen method. In the last two generations of iOS devices, Apple has moved to the ‘Bluetooth Low Energy’ (Bluetooth LE) or ‘Bluetooth 4.0’ specification. In contrast to previous iterations that send continual data to other devices, Bluetooth LE can be used to send small bits of information without rapidly draining battery power.

This works well for this project as the only information the smartphone requires is the signal strength value, so very little data needs to be transmitted. For this purpose, Iyad has been experimenting with four Bluetooth LE stickers that run on watch batteries for up to a year.

Sold as ‘The StickNFind’, they are intended to allow the user to locate their lost belongings by way of signal strength – exactly what we need for this project. An early concern that since the technology uses very little energy, the update rate would be insufficient to create a fluid tracking model, but it is in fact frequent enough to remove the requirement for complex interpolation algorithms.

The 3DBARE product will be intended for the following applications:

-3DBARE Composer: A user interface where the composer can place and set attributes to their sources in different positions around a given space.

-3DBARE Setup: A means to set up the wireless tracking system within a space, the user measures the area and the computer returns vector-position values for where the trackers must be positioned in order to track users effectively.

-3DBARE Client: The 3DBARE audio application as used by listeners, which will be available on iOS and Android.

Another future possibility of the project involves streaming live musicians through the system in order to have live, but virtual performance. This would involve one or more microphones per instrument to be sent into a computer, and then streamed over a local network to the mobile devices. This means that the handset devices always have to be within range of the Bluetooth device.

A further possibility is for the development of external control to change the perceptual positions of active audios sources via automation, in a kind of ultra-acousmatic setting where listeners are motionless in a motion-filled sonic field.

The current stage involves synchronous multi-channel audio playback through Audio Toolbox and openAL and control of virtual spatialisation via Bluetooth tracking.

We are also conducting some really exciting research into using the new HTML5’s audio rendering capability which will be multi-browser, multi-device and here you will shortly be able to try out our first online demo with an original composition from the digital studio, another experiment in advanced virtual performance simulation.

Subscribe or follow on Twitter; #benjaminmawson for updates and to be the first to hear it!

Posted in Locative Music | Leave a comment

Virtual Sounds of the Mesolithic

pech-merle_small

Bradshaw Foundation

Working with the MIRA team at Hull University, led by Professor Joy Porter, to contribute immersive simulated sound to their planned virtual prehistoric caves – “Why?”, you reasonably ask: four reasons.

Prehistoric caves are even more fragile than excavated ruins. So much so that public visits have been all but completely ruled out for most. Certainly the famous French and Spanish cave art discoveries of the past few decades can’t be seen in situ byany but a few specialists.

1st,then: otherwise impossible access for many, to things of wonder.

The virtual space by contrast can stand deconstructing, warping, before being identically reconstituted: it is possible experimentally to reconfigure a virtual site without changing the underlying ‘objects’, what is called non-destructive editing. So we can simulate an environment then try out different arrangements of it. There are somany unknowns about these caves, their images and artists: virtuality provides the best opportunity yet to test our theories and imaginative extensions of the very few known facts.

2nd: the unreality of the virtual is rich in imaginative possibility, permitting, encouraging otherwise unfindable links between the visual, physical evidence and the possible spiritual and social bases for these complex images.

Connections in the virtual world are starting to be formed between the visual – arguably the most noticed, conscious of our senses – and other, often overlooked ones: smell, touch, taste and most important to me at least, hearing. In the virtual cave we are planning a multi-sensory evocation of a series of environments which would be impossible to rebuild physically. Here, specialists not only in the virtual sensory experience will collaborate on a concerted outcome that embraces all of these and more importantly, does it synchronously. Listener action triggers environmental responses to all the senses in tandem, giving a uniquely sensuous, even visceral experience of these mysterious, foreign spaces.

3rd: a unique opportunity to collaborate with technologists and theorists on the construction of an ambitious holistic virtual experience.

The last reason is a little more personal and goes a bit like this:

My connection with caves started with a visit to Chislehurst aged 6. Our guide told the story of a challenge that used to exist for anyone who wanted, to spend the night alone in the caves. A man had placed a chair in the centre of a pool under limestone overhangs and drawn on the walls by torchlight, drinking whisky to warm and comfort himself. After some hours, a ghastly female figure had risen from the water and started to walk towards him. Blind with terror, he ran through the darkness towards what he thought was the exit, striking his face and falling unconscious. When he was found in the morning he was alive and able to recount the event. Shortly afterwards he slipped into a dreamlike, unreachable state from which he never recovered, apparently never speaking thereafter, his gaze fixed in frozen awe at some unseeable horror in the middle distance.

Ten years ago, somewhat recovered from the childhood nightmares of ghouls rising from subterranean water, I half-accidentally bought a cave in the Loire valley, surrounded by fields and vineyards. The most magical tranquility surrounds the place by day, even at its lightless, utterly silent interior. Closing the doors and lighting a fire in the large chimney, opening a little drop of something and sitting in the still calm is a peace unlike any other. As darkness outside falls, the calm of the remote fields and woods all around is imperceptibly turned into a deeper, more impenetrable silence. After some hours, a scratching at the heavy door begins. You feel your own eyelashes without moving a hand. The air in your chest stops moving and your fingernails seem to throb, as you wait, hearing yourself listen and believing this moment to be your last. Of course the moment passes, the curious badger or fox moves off and absolute silence descends again.

Once this has happened a few times you actually start to get accustomed to the environment. You can tune into or out of your heartbeat, breathing, the creaks of invisible objects expanding with the heat from the fire. It is at last possible to sit in the absolute silence and consider a lovely protrusion on the limestone rock. Finding a camping knife and a bottle top I bring a better light and sit on the ledge in front of the uneven surface, feeling its curves and starting to see a long, impervious face. A few marks define the outline, some deeper cuts begin to show where the features will rise from. I have no skill at carving or drawing. I realise that three hours have passed, in a contented absorption like nothing available to us in our highly-connected, urban drama. It was a glimpse of both the serenity and mystical terror that may have characterisedmuch of life for cave dwellers tens of thousands of years ago that both drew me to my cave and to this incredible project.

I can’t wait!

cwa21p56

 

Posted in Locative Music | Leave a comment

Fluid Narratives of Virtual Music

When I started building an engine to allow the listener to walk inside a piece of music, I thought it was a tool to help me carry on doing what I was doing – just a better way of listening.

But allowing a listener to determine the temporal content of music by their position and route through space means that all or many of the former controls held by the composer / performer / producer are removed. The listener is in charge: they are attracted or repelled by sounds and their combinations, and they negotiate a path ‘blindly’, feeling without signposting for a way through the experience.

If the ‘work’ presented is to tell a story or – now more likely – allow a series of associations between the materials experienced and personal memory, visceral responses to these, thereby giving the listener the tools to construct their own narrative, how can we determine the outcome, some aspects of the whole?

The narrative of a ‘classical work’ is necessarily abandoned. Its message resides as firmly in form as in tonal colour and harmonic content. The formula of embarking on a sometimes choppy but ultimately protected journey before returning to the comfortable shores of the original key (and ‘tune’) have, for all but a few music-makers, become necessarily historical.

The scope which we now enjoy for example of the exploration of textures was simply impossible in the pre-electronic age. Digital transformation of the familiar and unfamiliar into and through each other lets us explore weird new identities. The skewed reality of dreams becomes communicable, merging and overlaying in impossible but plausible juxtapositions allows music to reflect the complexities of our sensory and cognitive experience.

Music is not something we present people, like a cake or a pair of shoes. Music is the ordering into communicative structures of sound. You cannot touch, see or smell sound. You cannot write anything about it that approaches the reality of hearing those sounds.

So why have we spent a couple of centuries telling ourselves that the musical composition was an object like a cake, like shoes, like a painting? I refer you to the million-word discussions of others on this thorny matter. My business is struggling to make the er, not-stuff, that music is.

So if we want a narrative in our music, let’s put one there. I witness stories on the top deck of London buses, as I drive through landscape, sit in a city square. Amalgamations of snapshots, overheard snippets, accents, phrases, references, calling up an un-knowable back story from every voice, each noise that flashes past and evaporates.

Composers cannot hope to control the tale they tell: there is no more agreement about the import of a Mozart string quartet than those of Morton Feldman. We do though, have access to an unprecedented level of complexity in the material we present to our audience and the combinations in which these may, endlessly, be sensed. The big issue for me has been how to deliver all this magical, vertiginous potential: no one can play it, read it, hum it. Sounds that cannot be reproduced. Combinations that cannot be heard if sounding all at once.

Here’s what we do: let the listener combine the materials as they proceed, like Amelie collecting photo booth snaps, or Cage with his same-different-same seas of traffic. Why don’t we present the listener with a shoebox full of letters (maybe some distractions thrown in, certain pages strategically removed) – and ask them to tell us the life story of the unknown protagonist?

Then we can proceed beyond the need for a narrative altogether: removing the imposition of structure, particularly the temporal, is not an act of abandonment, of irresponsibility – it is the most generous gift you can make to an audience. To present them with a collection of the most closely, finely wrought pieces of work possible, in placements and combinations that have been tested, over and again, until the swirling whole, this whirlpool of memory and desire, amusement, terror, revulsion, meditative curiosity, rage, sleepy contentment, laughter become not a fixed structure but like the inner and outer worlds flapping like Einstein’s dimensions against each other as we walk between them.

It is the fluidity of virtual narratives that can bring the virtual to life.

Posted in Locative Music | 1 Comment

3DBARE

Music You Can Walk Inside:  3DBARE [thred-bair]

The 3D Binaural Audio Rendering Engine

You stand in an empty room, wearing a wireless headset.

Suddenly you are at the edge of an immense orchestra as they start to play: fantastical instruments – some apparently real, others bizarre or sweet in ways you have never heard.

Turning your head slightly to the right, your attention is attracted to the seeming sound of voices in another room, far away and yet impossibly clear.

Facing forwards again, the orchestra are in front and on both sides; you walk forwards, through the middle of the players.

Between many violins, playing the same melody, you can discern that each performance differs slightly from the next. On your right are 8 or 10 violins playing the main melody, to your left, a similar number playing a simpler line, slightly lower pitched.

A few paces more and the rhythmic stutters of the lower strings, first violas, then cellos, then basses enter your near-field hearing, while the violins are now clearly behind you, perceptibly further away than you have walked.

You are alone, in an empty space, wearing a wireless headset.

By tracking the listener’s position and orientation, our audio rendering engine is controlled to respond to the listener’s movement in three axes.

The experience of 3DBARE is like no other to have emerged from decades of experimentation with 3D audio: it allows you to walk inside a piece of music.

3DBARE permits investigation of the inner workings of a music composition while it is playing.

It allows the composer to create works of greater complexity than can be heard via speakers or in a live setting.

It is a tool for teachers, composers and listeners to explore the minute detail of complex music in the moment that it occurs. Each audition may be different, breathing life back into the crystallized format of a recording.

First demos taking place early in 2013: for more information and to hear it for yourself, get in touch!

Posted in Locative Music | Leave a comment

Free workshops and taking part in the largest locative music project in the world

Locative Music Workshops: “Take Me By The Hand”

Free events at various of Southampton’s green spaces: schools workshops and public demos about music spread across a landscape.

Visit our Eventbrite page to book a place and take part in the

Audio Portrait of a City

Posted in Locative Music | Leave a comment