Simulated Sense of Hearing


Written by Robert McIntyre

Table of Contents

1 Hearing

I want to be able to place ears in a similar manner to how I place the eyes. I want to be able to place ears in a unique spatial position, and receive as output at every tick the F.F.T. of whatever signals are happening at that point.

Hearing is one of the more difficult senses to simulate, because there is less support for obtaining the actual sound data that is processed by jMonkeyEngine3.

jMonkeyEngine's sound system works as follows:

  • jMonkeyEngine uses the AppSettings for the particular application to determine what sort of AudioRenderer should be used.
  • although some support is provided for multiple AudioRendering backends, jMonkeyEngine at the time of this writing will either pick no AudioRenderer at all, or the LwjglAudioRenderer
  • jMonkeyEngine tries to figure out what sort of system you're running and extracts the appropriate native libraries.
  • the LwjglAudioRenderer uses the LWJGL (LightWeight Java Game Library) bindings to interface with a C library called OpenAL
  • OpenAL calculates the 3D sound localization and feeds a stream of sound to any of various sound output devices with which it knows how to communicate.

A consequence of this is that there's no way to access the actual sound data produced by OpenAL. Even worse, OpenAL only supports one listener, which normally isn't a problem for games, but becomes a problem when trying to make multiple AI creatures that can each hear the world from a different perspective.

To make many AI creatures in jMonkeyEngine that can each hear the world from their own perspective, it is necessary to go all the way back to OpenAL and implement support for simulated hearing there.

2 Extending OpenAL

2.1 OpenAL Devices

OpenAL goes to great lengths to support many different systems, all with different sound capabilities and interfaces. It accomplishes this difficult task by providing code for many different sound backends in pseudo-objects called Devices. There's a device for the Linux Open Sound System and the Advanced Linux Sound Architecture, there's one for Direct Sound on Windows, there's even one for Solaris. OpenAL solves the problem of platform independence by providing all these Devices.

Wrapper libraries such as LWJGL are free to examine the system on which they are running and then select an appropriate device for that system.

There are also a few "special" devices that don't interface with any particular system. These include the Null Device, which doesn't do anything, and the Wave Device, which writes whatever sound it receives to a file, if everything has been set up correctly when configuring OpenAL.

Actual mixing of the sound data happens in the Devices, and they are the only point in the sound rendering process where this data is available.

Therefore, in order to support multiple listeners, and get the sound data in a form that the AIs can use, it is necessary to create a new Device, which supports this features.

2.2 The Send Device

Adding a device to OpenAL is rather tricky – there are five separate files in the OpenAL source tree that must be modified to do so. I've documented this process here for anyone who is interested.

Onward to that actual Device!

again, my objectives are:

  • Support Multiple Listeners from jMonkeyEngine3
  • Get access to the rendered sound data for further processing from clojure.

2.3 send.c

2.4 Header

#include "config.h"
#include <stdlib.h>
#include "alMain.h"
#include "AL/al.h"
#include "AL/alc.h"
#include "alSource.h"
#include <jni.h>

//////////////////// Summary

struct send_data;
struct context_data;

static void addContext(ALCdevice *, ALCcontext *);
static void syncContexts(ALCcontext *master, ALCcontext *slave);
static void syncSources(ALsource *master, ALsource *slave, 
                        ALCcontext *masterCtx, ALCcontext *slaveCtx);

static void syncSourcei(ALuint master, ALuint slave,
                        ALCcontext *masterCtx, ALCcontext *ctx2, ALenum param);
static void syncSourcef(ALuint master, ALuint slave,
                        ALCcontext *masterCtx, ALCcontext *ctx2, ALenum param);
static void syncSource3f(ALuint master, ALuint slave,
                        ALCcontext *masterCtx, ALCcontext *ctx2, ALenum param);

static void swapInContext(ALCdevice *, struct context_data *);
static void saveContext(ALCdevice *, struct context_data *);
static void limitContext(ALCdevice *, ALCcontext *);
static void unLimitContext(ALCdevice *);

static void init(ALCdevice *);
static void renderData(ALCdevice *, int samples);

#define UNUSED(x)  (void)(x)

The main idea behind the Send device is to take advantage of the fact that LWJGL only manages one context when using OpenAL. A context is like a container that holds samples and keeps track of where the listener is. In order to support multiple listeners, the Send device identifies the LWJGL context as the master context, and creates any number of slave contexts to represent additional listeners. Every time the device renders sound, it synchronizes every source from the master LWJGL context to the slave contexts. Then, it renders each context separately, using a different listener for each one. The rendered sound is made available via JNI to jMonkeyEngine.

To recap, the process is:

  • Set the LWJGL context as "master" in the init() method.
  • Create any number of additional contexts via addContext()
  • At every call to renderData() sync the master context with the slave contexts with syncContexts()
  • syncContexts() calls syncSources() to sync all the sources which are in the master context.
  • limitContext() and unLimitContext() make it possible to render only one context at a time.

2.5 Necessary State

////////////////////  State

typedef struct context_data {
  ALfloat ClickRemoval[MAXCHANNELS];
  ALfloat PendingClicks[MAXCHANNELS];
  ALvoid *renderBuffer;
  ALCcontext *ctx;
} context_data;

typedef struct send_data {
  ALuint size;
  context_data **contexts;
  ALuint numContexts;
  ALuint maxContexts;
} send_data;

Switching between contexts is not the normal operation of a Device, and one of the problems with doing so is that a Device normally keeps around a few pieces of state such as the ClickRemoval array above which will become corrupted if the contexts are not done in parallel. The solution is to create a copy of this normally global device state for each context, and copy it back and forth into and out of the actual device state whenever a context is rendered.

2.6 Synchronization Macros

////////////////////  Context Creation / Synchronization

  void NAME (ALuint sourceID1, ALuint sourceID2,        \
             ALCcontext *ctx1, ALCcontext *ctx2,        \
             ALenum param){                             \
    INIT_EXPR;                                          \
    ALCcontext *current = alcGetCurrentContext();       \
    alcMakeContextCurrent(ctx1);                        \
    GET_EXPR;                                           \
    alcMakeContextCurrent(ctx2);                        \
    SET_EXPR;                                           \
    alcMakeContextCurrent(current);                     \
#define MAKE_SYNC(NAME, TYPE, GET, SET)                 \
  _MAKE_SYNC(NAME,                                      \
             TYPE value,                                \
             GET(sourceID1, param, &value),             \
             SET(sourceID2, param, value))
#define MAKE_SYNC3(NAME, TYPE, GET, SET)                                \
  _MAKE_SYNC(NAME,                                                      \
             TYPE value1; TYPE value2; TYPE value3;,                    \
             GET(sourceID1, param, &value1, &value2, &value3),          \
             SET(sourceID2, param,  value1,  value2,  value3))

MAKE_SYNC( syncSourcei,  ALint,   alGetSourcei,  alSourcei);
MAKE_SYNC( syncSourcef,  ALfloat, alGetSourcef,  alSourcef);
MAKE_SYNC3(syncSource3i, ALint,   alGetSource3i, alSource3i);
MAKE_SYNC3(syncSource3f, ALfloat, alGetSource3f, alSource3f);

Setting the state of an OpenAL source is done with the alSourcei, alSourcef, alSource3i, and alSource3f functions. In order to completely synchronize two sources, it is necessary to use all of them. These macros help to condense the otherwise repetitive synchronization code involving these similar low-level OpenAL functions.

2.7 Source Synchronization

void syncSources(ALsource *masterSource, ALsource *slaveSource, 
                 ALCcontext *masterCtx, ALCcontext *slaveCtx){
  ALuint master = masterSource->source;
  ALuint slave = slaveSource->source;
  ALCcontext *current = alcGetCurrentContext();


  ALint source_type;
  alGetSourcei(master, AL_SOURCE_TYPE, &source_type);

  // Only static sources are currently synchronized! 
  if (AL_STATIC == source_type){
    ALint master_buffer;
    ALint slave_buffer;
    alGetSourcei(master, AL_BUFFER, &master_buffer);
    alGetSourcei(slave, AL_BUFFER, &slave_buffer);
    if (master_buffer != slave_buffer){
      alSourcei(slave, AL_BUFFER, master_buffer);
  // Synchronize the state of the two sources.
  ALint masterState;
  ALint slaveState;

  alGetSourcei(master, AL_SOURCE_STATE, &masterState);
  alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);

  if (masterState != slaveState){
    switch (masterState){
    case AL_INITIAL : alSourceRewind(slave); break;
    case AL_PLAYING : alSourcePlay(slave);   break;
    case AL_PAUSED  : alSourcePause(slave);  break;
    case AL_STOPPED : alSourceStop(slave);   break;
  // Restore whatever context was previously active.

This function is long because it has to exhaustively go through all the possible state that a source can have and make sure that it is the same between the master and slave sources. I'd like to take this moment to salute the OpenAL Reference Manual, which provides a very good description of OpenAL's internals.

2.8 Context Synchronization

void syncContexts(ALCcontext *master, ALCcontext *slave){
  /* If there aren't sufficient sources in slave to mirror 
     the sources in master, create them. */
  ALCcontext *current = alcGetCurrentContext();

  UIntMap *masterSourceMap = &(master->SourceMap);
  UIntMap *slaveSourceMap = &(slave->SourceMap);
  ALuint numMasterSources = masterSourceMap->size;
  ALuint numSlaveSources = slaveSourceMap->size;

  if (numSlaveSources < numMasterSources){
    ALuint numMissingSources = numMasterSources - numSlaveSources;
    ALuint newSources[numMissingSources];
    alGenSources(numMissingSources, newSources);

  /* Now, slave is guaranteed to have at least as many sources
     as master.  Sync each source from master to the corresponding
     source in slave. */
  int i;
  for(i = 0; i < masterSourceMap->size; i++){
                master, slave);

Most of the hard work in Context Synchronization is done in syncSources(). The only thing that syncContexts() has to worry about is automatically creating new sources whenever a slave context does not have the same number of sources as the master context.

2.9 Context Creation

static void addContext(ALCdevice *Device, ALCcontext *context){
  send_data *data = (send_data*)Device->ExtraData;
  // expand array if necessary
  if (data->numContexts >= data->maxContexts){
    ALuint newMaxContexts = data->maxContexts*2 + 1;
    data->contexts = realloc(data->contexts, newMaxContexts*sizeof(context_data));
    data->maxContexts = newMaxContexts;
  // create context_data and add it to the main array
  context_data *ctxData;
  ctxData = (context_data*)calloc(1, sizeof(*ctxData));
  ctxData->renderBuffer = 
    malloc(BytesFromDevFmt(Device->FmtType) * 
           Device->NumChan * Device->UpdateSize);
  ctxData->ctx = context;

  data->contexts[data->numContexts] = ctxData;

Here, the slave context is created, and it's data is stored in the device-wide ExtraData structure. The renderBuffer that is created here is where the rendered sound samples for this slave context will eventually go.

2.10 Context Switching

////////////////////  Context Switching 

/* A device brings along with it two pieces of state
 * which have to be swapped in and out with each context.
static void swapInContext(ALCdevice *Device, context_data *ctxData){
  memcpy(Device->ClickRemoval, ctxData->ClickRemoval, sizeof(ALfloat)*MAXCHANNELS);
  memcpy(Device->PendingClicks, ctxData->PendingClicks, sizeof(ALfloat)*MAXCHANNELS);

static void saveContext(ALCdevice *Device, context_data *ctxData){
  memcpy(ctxData->ClickRemoval, Device->ClickRemoval, sizeof(ALfloat)*MAXCHANNELS);
  memcpy(ctxData->PendingClicks, Device->PendingClicks, sizeof(ALfloat)*MAXCHANNELS);

static ALCcontext **currentContext;
static ALuint currentNumContext;

/* By default, all contexts are rendered at once for each call to aluMixData.
 * This function uses the internals of the ALCdevice struct to temporally 
 * cause aluMixData to only render the chosen context.
static void limitContext(ALCdevice *Device, ALCcontext *ctx){
  currentContext  = Device->Contexts;
  currentNumContext  = Device->NumContexts;
  Device->Contexts = &ctx;
  Device->NumContexts = 1;

static void unLimitContext(ALCdevice *Device){
  Device->Contexts = currentContext;
  Device->NumContexts = currentNumContext;

OpenAL normally renders all Contexts in parallel, outputting the whole result to the buffer. It does this by iterating over the Device->Contexts array and rendering each context to the buffer in turn. By temporally setting Device->NumContexts to 1 and adjusting the Device's context list to put the desired context-to-be-rendered into position 0, we can get trick OpenAL into rendering each slave context separate from all the others.

2.11 Main Device Loop

////////////////////   Main Device Loop

/* Establish the LWJGL context as the master context, which will
 * be synchronized to all the slave contexts
static void init(ALCdevice *Device){
  ALCcontext *masterContext = alcGetCurrentContext();
  addContext(Device, masterContext);

static void renderData(ALCdevice *Device, int samples){
  send_data *data = (send_data*)Device->ExtraData;
  ALCcontext *current = alcGetCurrentContext();

  ALuint i;
  for (i = 1; i < data->numContexts; i++){
    syncContexts(data->contexts[0]->ctx , data->contexts[i]->ctx);
  if ((ALuint) samples > Device->UpdateSize){
    printf("exceeding internal buffer size; dropping samples\n");
    printf("requested %d; available %d\n", samples, Device->UpdateSize);
    samples = (int) Device->UpdateSize;

  for (i = 0; i < data->numContexts; i++){
    context_data *ctxData = data->contexts[i];
    ALCcontext *ctx = ctxData->ctx;
    limitContext(Device, ctx);
    swapInContext(Device, ctxData);
    aluMixData(Device, ctxData->renderBuffer, samples);
    saveContext(Device, ctxData);

The main loop synchronizes the master LWJGL context with all the slave contexts, then walks each context, rendering just that context to it's audio-sample storage buffer.

2.12 JNI Methods

At this point, we have the ability to create multiple listeners by using the master/slave context trick, and the rendered audio data is waiting patiently in internal buffers, one for each listener. We need a way to transport this information to Java, and also a way to drive this device from Java. The following JNI interface code is inspired by the way LWJGL interfaces with OpenAL.

2.12.1 step

////////////////////   JNI Methods

#include "com_aurellem_send_AudioSend.h"

 * Class:     com_aurellem_send_AudioSend
 * Method:    nstep
 * Signature: (JI)V
JNIEXPORT void JNICALL Java_com_aurellem_send_AudioSend_nstep
(JNIEnv *env, jclass clazz, jlong device, jint samples){
  renderData((ALCdevice*)((intptr_t)device), samples);

This device, unlike most of the other devices in OpenAL, does not render sound unless asked. This enables the system to slow down or speed up depending on the needs of the AIs who are using it to listen. If the device tried to render samples in real-time, a complicated AI whose mind takes 100 seconds of computer time to simulate 1 second of AI-time would miss almost all of the sound in its environment.

2.12.2 getSamples

 * Class:     com_aurellem_send_AudioSend
 * Method:    ngetSamples
 * Signature: (JLjava/nio/ByteBuffer;III)V
JNIEXPORT void JNICALL Java_com_aurellem_send_AudioSend_ngetSamples
(JNIEnv *env, jclass clazz, jlong device, jobject buffer, jint position, 
 jint samples, jint n){

  ALvoid *buffer_address = 
    ((ALbyte *)(((char*)(*env)->GetDirectBufferAddress(env, buffer)) + position));
  ALCdevice *recorder = (ALCdevice*) ((intptr_t)device);
  send_data *data = (send_data*)recorder->ExtraData;
  if ((ALuint)n > data->numContexts){return;}
  memcpy(buffer_address, data->contexts[n]->renderBuffer, 
         BytesFromDevFmt(recorder->FmtType) * recorder->NumChan * samples);

This is the transport layer between C and Java that will eventually allow us to access rendered sound data from clojure.

2.12.3 Listener Management

addListener, setNthListenerf, and setNthListener3f are necessary to change the properties of any listener other than the master one, since only the listener of the current active context is affected by the normal OpenAL listener calls.

 * Class:     com_aurellem_send_AudioSend
 * Method:    naddListener
 * Signature: (J)V
JNIEXPORT void JNICALL Java_com_aurellem_send_AudioSend_naddListener
(JNIEnv *env, jclass clazz, jlong device){
  UNUSED(env); UNUSED(clazz);
  //printf("creating new context via naddListener\n");
  ALCdevice *Device = (ALCdevice*) ((intptr_t)device);
  ALCcontext *new = alcCreateContext(Device, NULL);
  addContext(Device, new);

 * Class:     com_aurellem_send_AudioSend
 * Method:    nsetNthListener3f
 * Signature: (IFFFJI)V
JNIEXPORT void JNICALL Java_com_aurellem_send_AudioSend_nsetNthListener3f
  (JNIEnv *env, jclass clazz, jint param, 
   jfloat v1, jfloat v2, jfloat v3, jlong device, jint contextNum){

  ALCdevice *Device = (ALCdevice*) ((intptr_t)device);
  send_data *data = (send_data*)Device->ExtraData;
  ALCcontext *current = alcGetCurrentContext();
  if ((ALuint)contextNum > data->numContexts){return;}
  alListener3f(param, v1, v2, v3);

 * Class:     com_aurellem_send_AudioSend
 * Method:    nsetNthListenerf
 * Signature: (IFJI)V
JNIEXPORT void JNICALL Java_com_aurellem_send_AudioSend_nsetNthListenerf
(JNIEnv *env, jclass clazz, jint param, jfloat v1, jlong device, 
 jint contextNum){

  ALCdevice *Device = (ALCdevice*) ((intptr_t)device);
  send_data *data = (send_data*)Device->ExtraData;
  ALCcontext *current = alcGetCurrentContext();
  if ((ALuint)contextNum > data->numContexts){return;}
  alListenerf(param, v1);

2.12.4 Initialization

initDevice is called from the Java side after LWJGL has created its context, and before any calls to addListener. It establishes the LWJGL context as the master context.

getAudioFormat is a convenience function that uses JNI to build up a javax.sound.sampled.AudioFormat object from data in the Device. This way, there is no ambiguity about what the bits created by step and returned by getSamples mean.

 * Class:     com_aurellem_send_AudioSend
 * Method:    ninitDevice
 * Signature: (J)V
JNIEXPORT void JNICALL Java_com_aurellem_send_AudioSend_ninitDevice
(JNIEnv *env, jclass clazz, jlong device){
  ALCdevice *Device = (ALCdevice*) ((intptr_t)device);

 * Class:     com_aurellem_send_AudioSend
 * Method:    ngetAudioFormat
 * Signature: (J)Ljavax/sound/sampled/AudioFormat;
JNIEXPORT jobject JNICALL Java_com_aurellem_send_AudioSend_ngetAudioFormat
(JNIEnv *env, jclass clazz, jlong device){
  jclass AudioFormatClass = 
    (*env)->FindClass(env, "javax/sound/sampled/AudioFormat");
  jmethodID AudioFormatConstructor = 
    (*env)->GetMethodID(env, AudioFormatClass, "<init>", "(FIIZZ)V");
  ALCdevice *Device = (ALCdevice*) ((intptr_t)device);
  int isSigned;
  switch (Device->FmtType)
    case DevFmtUByte: 
    case DevFmtUShort: isSigned = 0; break;  
    default : isSigned = 1;   
  float frequency = Device->Frequency;
  int bitsPerFrame = (8 * BytesFromDevFmt(Device->FmtType));
  int channels = Device->NumChan;
  jobject format = (*env)->
  return format;

2.13 Boring Device management stuff

This code is more-or-less copied verbatim from the other OpenAL backends. It's the basis for OpenAL's primitive object system.

////////////////////   Device Initialization / Management

static const ALCchar sendDevice[] = "Multiple Audio Send";

static ALCboolean send_open_playback(ALCdevice *device, 
 const ALCchar *deviceName)
  send_data *data;
  // stop any buffering for stdout, so that I can 
  // see the printf statements in my terminal immediately
  setbuf(stdout, NULL);

    deviceName = sendDevice;
  else if(strcmp(deviceName, sendDevice) != 0)
    return ALC_FALSE;
  data = (send_data*)calloc(1, sizeof(*data));
  device->szDeviceName = strdup(deviceName);
  device->ExtraData = data;
  return ALC_TRUE;

static void send_close_playback(ALCdevice *device)
  send_data *data = (send_data*)device->ExtraData;
  ALuint i;
  // Destroy all slave contexts. LWJGL will take care of 
  // its own context.
  for (i = 1; i < data->numContexts; i++){
    context_data *ctxData = data->contexts[i];
  device->ExtraData = NULL;

static ALCboolean send_reset_playback(ALCdevice *device)
  return ALC_TRUE;

static void send_stop_playback(ALCdevice *Device){

static const BackendFuncs send_funcs = {
  NULL,  /* These would be filled with functions to    */
  NULL,  /* handle capturing audio if we we into that  */
  NULL,  /* sort of thing...                           */

ALCboolean alc_send_init(BackendFuncs *func_list){
  *func_list = send_funcs;
  return ALC_TRUE;

void alc_send_deinit(void){}

void alc_send_probe(enum DevProbe type)
    case DEVICE_PROBE:

3 The Java interface, AudioSend

The Java interface to the Send Device follows naturally from the JNI definitions. It is included here for completeness. The only thing here of note is the deviceID. This is available from LWJGL, but to only way to get it is reflection. Unfortunately, there is no other way to control the Send device than to obtain a pointer to it.

package com.aurellem.send;

import java.nio.ByteBuffer;

import javax.sound.sampled.AudioFormat;

public class AudioSend {

    private final long deviceID;
    public AudioSend(long deviceID){
        this.deviceID = deviceID;
    /** This establishes the LWJGL context as the context which
     *  will be copies to all other contexts.  It must be called
     *  before any calls to <code>addListener();</code>
    public void initDevice(){
    public static native void ninitDevice(long device);
     * The send device does not automatically process sound.  This
     * step function will cause the desired number of samples to
     * be processed for each listener.  The results will then be
     * available via calls to <code>getSamples()</code> for each
     * listener.
     * @param samples
    public void step(int samples){
        nstep(this.deviceID, samples);}
    public static native void nstep(long device, int samples);

     * Retrieve the final rendered sound for a particular
     * listener.  <code>contextNum == 0</code> is the main LWJGL
     * context.
     * @param buffer
     * @param samples
     * @param contextNum
    public void getSamples(ByteBuffer buffer, 
                           int samples, int contextNum){
        ngetSamples(this.deviceID, buffer, 
                    buffer.position(), samples, contextNum);}
    public static native void 
        ngetSamples(long device, ByteBuffer buffer, 
                    int position, int samples, int contextNum);
     * Create an additional listener on the recorder device.  The
     * device itself will manage this listener and synchronize it
     * with the main LWJGL context. Processed sound samples for
     * this listener will be available via a call to
     * <code>getSamples()</code> with <code>contextNum</code>
     * equal to the number of times this method has been called.
    public void addListener(){naddListener(this.deviceID);}
    public static native void naddListener(long device);
     * This will internally call <code>alListener3f<code> in the
     * appropriate slave context and update that context's
     * listener's parameters. Calling this for a number greater
     * than the current number of slave contexts will have no
     * effect.
     * @param pname
     * @param v1
     * @param v2
     * @param v3
     * @param contextNum
    public void 
        setNthListener3f(int pname, float v1, 
                         float v2, float v3, int contextNum){
        nsetNthListener3f(pname, v1, v2, v3, 
                          this.deviceID, contextNum);}
    public static native void 
        nsetNthListener3f(int pname, float v1, float v2, 
                          float v3, long device, int contextNum);
     * This will internally call <code>alListenerf<code> in the
     * appropriate slave context and update that context's
     * listener's parameters. Calling this for a number greater
     * than the current number of slave contexts will have no
     * effect.
     * @param pname
     * @param v1
     * @param contextNum
    public void setNthListenerf(int pname, float v1, int contextNum){
        nsetNthListenerf(pname, v1, this.deviceID, contextNum);}
    public static native void 
        nsetNthListenerf(int pname, float v1, 
                         long device, int contextNum);
     * Retrieve the AudioFormat which the device is using.  This
     * format is itself derived from the OpenAL config file under
     * the "format" variable.
    public AudioFormat getAudioFormat(){
        return ngetAudioFormat(this.deviceID);}
    public static native AudioFormat ngetAudioFormat(long device);      

4 Finally, Ears in clojure!

Now that the infrastructure is complete (modulo a few patches to jMonkeyEngine3 to support accessing this modified version of OpenAL that are not worth discussing), the clojure ear abstraction is rather simple. Just as there were SceneProcessors for vision, there are now SoundProcessors for hearing.


import java.nio.ByteBuffer;

import javax.sound.sampled.AudioFormat;

public interface SoundProcessor {

     * Called when the SoundProcessor is being destroyed, and
     * there are no more samples to process.  This happens at the
     * latest when the Application is shutting down.
    void cleanup();
     * Called whenever there are new audio samples to process. The
     * audioSamples ByteBuffer contains 3D audio data rendered by
     * OpenAL.
     * @param audioSamples a ByteBuffer containing processed audio
     * samples
     * @param numSamples the number of samples, in bytes, that are valid
     * @param format the format of the audio samples in audioSamples
    void process(ByteBuffer audioSamples, int numSamples, AudioFormat format);
(ns cortex.hearing
  "Simulate the sense of hearing in jMonkeyEngine3. Enables multiple
  listeners at different positions in the same world. Passes vectors
  of floats in the range [-1.0 -- 1.0] in PCM format to any arbitrary
  {:author "Robert McIntyre"}
  (:use (cortex world util))
  (:import java.nio.ByteBuffer)
  (:import org.tritonus.share.sampled.FloatSampleTools)
  (:import javax.sound.sampled.AudioFormat))
(defn sound-processor
  "Deals with converting ByteBuffers into Vectors of floats so that
  the continuation functions can be defined in terms of immutable
  (proxy [SoundProcessor] []
    (cleanup [])
      [#^ByteBuffer audioSamples numSamples #^AudioFormat audioFormat]
      (let [bytes  (byte-array numSamples)
            floats (float-array numSamples)]
        (.get audioSamples bytes 0 numSamples)
         bytes 0 floats 0
         (/ numSamples (.getFrameSize audioFormat)) audioFormat)
         (vec floats))))))

(defn add-ear 
  "Add an ear to the world.  The continuation function will be called
  on the FFT or the sounds which the ear hears in the given
  timeframe. Sound is 3D."
  [world listener continuation]
  (let [renderer (.getAudioRenderer world)]
    (.addListener renderer listener)
    (.registerSoundProcessor renderer listener
                             (sound-processor continuation))

5 Example

(ns cortex.test.hearing
  (:use (cortex world util hearing))
  (:import ( AudioNode Listener))
  (:import com.jme3.scene.Node

(defn setup-fn [world]
  (let [listener (Listener.)]
    (add-ear world listener #(println-repl (nth % 0)))))
(defn play-sound [node world value]
  (if (not value)
      (.playSource (.getAudioRenderer world) node))))

(defn test-basic-hearing []
   (let [node1 (AudioNode. (asset-manager) "Sounds/pure.wav" false false)]
      {"key-space" (partial play-sound node1)}

(defn test-advanced-hearing
  "Testing hearing:
   You should see a blue sphere flying around several
   cubes.  As the sphere approaches each cube, it turns
  (doto (com.aurellem.capture.examples.Advanced.)
     (doto (AppSettings. true)
       (.setAudioRenderer "Send")))
    (.setShowSettings false)
    (.setPauseOnLostFocus false)))

This extremely basic program prints out the first sample it encounters at every time stamp. You can see the rendered sound being printed at the REPL.

  • As a bonus, this method of capturing audio for AI can also be used to capture perfect audio from a jMonkeyEngine application, for use in demos and the like.

Date: 2011-12-11 03:43:14 UTC

Author: Robert McIntyre

Org version 7.7 with Emacs version 23

Validate XHTML 1.0