Streaming environment - with stream-change

Oak · March 9, 2008, 9:11pm

I am incorporating sound into our game at the moment, and we have a lot of ambient sounds that set the environment. These sounds change as the focus of the user changes, and they are mutually exclusive. All of the tracks have to be streamed as they are quite long (1-5 minutes, encoded with OGG). I have now tried to implement this using the MusicQueue, using the EnvironmentPool, and using my own attempt to manipulate the tracks. Nothing so far has succeeded since I constantly run in to the following problems:

The tracks are restarted upon switching back to them (this is not very nice since it is easy to recognize the start of the track and the user then experiences the same sounds (not the same mood) every time he enters some part of the game).
The fading does not work due to hard coded values like 1.0f for volume (though I have fixed this in JME by adding "global" volumes to MusicQueue and EnvironmentPool).
I get everything running by not stopping/pausing the tracks but just fading them to 0 volume. But in our game we have about 12 ambient sound-tracks which take a lot of CPU to process, and effectively the user only hears one of them (2-3 during fast switching in the interface leading to some cross fading)
The cross fading does not really take effect (or is overruled somewhere else in the system in some other thread that I don't know has been started) and they tracks just stack up until my ears hurt and my CPU burns :-o

Now, does anyone have a similar problem - or better yet has anyone implemented such an mutually-exclusive-streaming-environment-solution already?

I Think we will most likely end up coding this whole thing, but considering how the OpenAL tracks and players work in JME we will most likely have to build our own OpenAL-based audio-system since we do not have enough control over things as it is (a lot of methods on the tracks invoke other methods on the player and visa versa - resulting in things like threads starting up and shutting down). Is there design-changes coming to JME2.0 that we should consider? And would such a solution be of interest to the JME-community?

MrCoder · March 9, 2008, 10:54pm

not of any help, but we had big problems with similar things back at jadestone, so this area is something that needs an upgrade for 2.0…

basixs · March 9, 2008, 10:57pm

HAHAHAHA

Okay Mr. Coder, how much is 2.0 gonna cost us??

Oak · March 10, 2008, 10:17am

I have just added an option on the AudioSystem controlling if new threads are created to update the streams of if the streams are updates as part of the call to AudionSystem.update(). Also added an option indicating what the interval should be in milliseconds (as opposed to the current hard coded value of 200).

I will post a patch for review by others once I get a little further down the road. Until then, I hope that others will post their comments/wishes to the audio-system as it seems like this part of JME does not have a lot of developer-attention - yet it is very important for a game to have great sound and music.

Oak · March 10, 2008, 12:03pm

Ok, now I have a working implementation which is quite robust (we can switch environments on mouse-moves - resulting in many switches before cross fading has completed). It took a few fixes in the AudioSystem as well as adding a new class similar to EnvironmentalPool.

Here is a patch for the audio-folder based on JME from CVS (10/22/07 07:20PM).

Index: AudioSystem.java =================================================================== --- AudioSystem.java (revision 272) +++ AudioSystem.java (revision 448) @@ -50,6 +50,10 @@ private MusicTrackQueue musicQueue = new MusicTrackQueue(); private EnvironmentalPool envPool = new EnvironmentalPool(); private float unitsPerMeter = 10; + /** Indicates if threads are to be created for every streaming player. */ + private boolean createStreamThreads = true; + /** Indicates the interval in milliseconds that streams should be updated with. */ + private long streamUpdateInterval = 200; /** * Singleton access to the audio system. FIXME: Currently hardcoded to @@ -105,7 +109,24 @@ public void setUnitsPerMeter(float toMeterValue) { this.unitsPerMeter = toMeterValue; } + + public boolean getCreateStreamThreads() { + return createStreamThreads; + } + + public void setCreateStreamThreads(boolean createStreamThreads) { + this.createStreamThreads = createStreamThreads; + } + + public long getStreamUpdateInterval() { + return streamUpdateInterval; + } + public void setStreamUpdateInterval(long streamUpdateInterval) { + this.streamUpdateInterval = streamUpdateInterval; + } + + public void cleanup() { system = null; } Index: AudioTrack.java =================================================================== --- AudioTrack.java (revision 272) +++ AudioTrack.java (revision 448) @@ -152,7 +152,9 @@ } public void addTrackStateListener(TrackStateListener listener) { - trackListeners.add(listener); + if(!trackListeners.contains(listener)) { + trackListeners.add(listener); + } } public void removeTrackStateListener(TrackStateListener listener) { @@ -181,23 +183,29 @@ } public void setVolume(float volume) { - if (volume > 1.0f) - volume = 1.0f; - this.volume = volume; - player.setVolume(volume); + this.volume = volume < 0.0f ? 0.0f : volume > 1.0f ? 1.0f : volume; + player.setVolume(this.volume); } public void fadeOut(float time) { - targetVolume = 0; - volumeChangeRate = (volume - targetVolume) / time; + fadeTo(time, 0); } - public void fadeIn(float time, float maxVolume) { + public void fadeIn(float time, float targetVolume) { setVolume(0); - targetVolume = maxVolume; - volumeChangeRate = maxVolume / time; + fadeTo(time, targetVolume); } + public void fadeTo(float time, float newvolume) { + targetVolume = newvolume < 0.0f ? 0.0f : newvolume > 1.0f ? 1.0f : newvolume; + if(volume != targetVolume) { + volumeChangeRate = (targetVolume - volume) / time; + } else { + volumeChangeRate = 0.0f; + fireFinishedFade(); + } + } + public AudioPlayer getPlayer() { return player; } @@ -262,17 +270,15 @@ dt = FastMath.FLT_EPSILON; // Do volume changes: - if (volume != targetVolume) { - if (volume < targetVolume) { - volume += volumeChangeRate * dt; - if (volume > targetVolume) volume = targetVolume; + if (volume != targetVolume && volumeChangeRate != 0.0f) { + float newvolume = volume + volumeChangeRate * dt; + if (volumeChangeRate > 0.0f) { + if (newvolume > targetVolume) newvolume = targetVolume; } else { - volume -= volumeChangeRate * dt; - if (volume < targetVolume) volume = targetVolume; + if (newvolume < targetVolume) newvolume = targetVolume; } - if (volume < 0 || volume > 1) volume = targetVolume; - setVolume(volume); - if (volume == targetVolume) { + setVolume(newvolume); + if (newvolume == targetVolume) { fireFinishedFade(); } } Index: MusicTrackQueue.java =================================================================== --- MusicTrackQueue.java (revision 272) +++ MusicTrackQueue.java (revision 448) @@ -65,6 +65,7 @@ private float crossfadeoutTime = 3.5f; private float crossfadeinTime = 3.5f; + private float volume = 1.0f; public MusicTrackQueue() { } @@ -137,9 +138,9 @@ AudioTrack track = tracks.get(currentTrack); if (crossfadeinTime > 0) - track.fadeIn(crossfadeinTime, track.getTargetVolume() > 0 ? track.getTargetVolume() : 1.0f); + track.fadeIn(crossfadeinTime, track.getTargetVolume() > 0 ? track.getTargetVolume() : getVolume()); else - track.setVolume(track.getTargetVolume() > 0 ? track.getTargetVolume() : 1.0f); + track.setVolume(track.getTargetVolume() > 0 ? track.getTargetVolume() : getVolume()); if (!track.isPlaying()) track.play(); } @@ -373,6 +374,14 @@ return isPlaying; } + public void setVolume(float volume) { + this.volume = volume; + } + + public float getVolume() { + return volume; + } + public void fadeOutAndClear(float fadeTime) { // fade out the current track. AudioTrack t = getCurrentTrack(); Index: EnvironmentalPool.java =================================================================== --- EnvironmentalPool.java (revision 272) +++ EnvironmentalPool.java (revision 448) @@ -53,6 +53,7 @@ private float crossfadeoutTime = 3.5f; private float crossfadeinTime = 3.5f; + private float volume = 1.0f; public EnvironmentalPool() { } @@ -97,7 +98,7 @@ if (t.isEnabled() && !t.isActive()) { t.stop(); if (crossfadeinTime > 0) - t.fadeIn(crossfadeinTime, 1.0f); + t.fadeIn(crossfadeinTime, getVolume()); t.play(); } else if (!t.isEnabled() && t.isActive()) { if (crossfadeoutTime > 0) { @@ -108,14 +109,14 @@ public void trackFinishedFade(AudioTrack track) { track.removeTrackStateListener(this); track.stop(); - track.setVolume(1.0f); - track.setTargetVolume(1.0f); + track.setVolume(getVolume()); + track.setTargetVolume(getVolume()); } @Override public void trackStopped(AudioTrack track) { track.removeTrackStateListener(this); - track.setVolume(1.0f); - track.setTargetVolume(1.0f); + track.setVolume(getVolume()); + track.setTargetVolume(getVolume()); } }); } else @@ -165,6 +166,14 @@ this.crossfadeoutTime = crossfadeoutTime; } + public void setVolume(float volume) { + this.volume = volume; + } + + public float getVolume() { + return volume; + } + public void fadeOutAndClear(float fadeTime) { // remove any listeners listListeners.clear(); Index: SoundEnvironment.java =================================================================== --- SoundEnvironment.java (revision 0) +++ SoundEnvironment.java (revision 448) @@ -0,0 +1,101 @@ +package com.jmex.audio; + +import java.net.URL; +import java.util.Hashtable; +import java.util.logging.Logger; + +import com.jme.math.FastMath; +import com.jmex.audio.AudioTrack.TrackType; +import com.jmex.audio.event.TrackStateListener; +import com.jmex.audio.util.AudioDebugging; + + +/** + * Much like the EnvironmentalPool but with only one sound playing at a time, using crossfade for transitions + * and keeping track of all players current location to avoid playing the start of the environment-tracks again + * and again. + * + * @author Emanuel Greisen <jme@emanuelgreisen.dk> + */ +public class SoundEnvironment +{ + final static Logger logger = Logger.getLogger(SoundEnvironment.class.getName()); + private float crossFadeTime; + private float volume; + private AudioTrack current_track; + private TrackStateListener pause_when_fade_complete_listener = new TrackStateListener() + { + public void trackFinishedFade(AudioTrack track) + { + logger.fine("Track.trackFinishedFade("+track.getVolume()+")"); + if(track.getVolume() <= 0.001f) + { + track.pause(); + track.removeTrackStateListener(this); + } + } + + public void trackPaused(AudioTrack track) + { + logger.fine("Track.paused():"+AudioDebugging.printSoundFileName(track)); + } + + public void trackPlayed(AudioTrack track) + { + logger.fine("Track.played():"+AudioDebugging.printSoundFileName(track)); + } + + public void trackStopped(AudioTrack track) + { + logger.fine("Track.stopped():"+AudioDebugging.printSoundFileName(track)); + } + }; + + + + public SoundEnvironment(float crossFadeTime, float volume) + { + this.crossFadeTime = crossFadeTime; + this.volume = volume; + } + + public void setVolume(float volume) + { + this.volume = FastMath.clamp(volume, 0.0f, 1.0f); + if(current_track != null) + { + current_track.fadeTo(0.2f, this.volume); + } + } + + /** + * Sets the current track for the environment. + * + * @param track + */ + public void setCurrentTrack(AudioTrack track) + { + if(track == current_track) + return; // We are already playing this track + + // Fade out old track + if(current_track != null) + { + current_track.fadeTo(crossFadeTime, 0.0f); + } + + current_track = track; + + // Fade in new track (and set some important stuff) + if(current_track != null) + { + current_track.setType(TrackType.ENVIRONMENT); + current_track.setLooping(true); + current_track.addTrackStateListener(pause_when_fade_complete_listener); + current_track.setEnabled(true); + current_track.setVolume(0); + current_track.fadeTo(crossFadeTime, volume); + current_track.play(); + } + } +} Index: util/AudioDebugging.java =================================================================== --- util/AudioDebugging.java (revision 0) +++ util/AudioDebugging.java (revision 448) @@ -0,0 +1,15 @@ +package com.jmex.audio.util; + +import java.io.File; + +import com.jmex.audio.AudioTrack; + +public class AudioDebugging +{ + public static String printSoundFileName(AudioTrack track) + { + if(track == null || track.getResource() == null) + return "null"; + return new File(track.getResource().getFile()).getName(); + } +} Index: openal/OpenALStreamedAudioPlayer.java =================================================================== --- openal/OpenALStreamedAudioPlayer.java (revision 272) +++ openal/OpenALStreamedAudioPlayer.java (revision 448) @@ -163,7 +163,15 @@ AL10.alSourcei(source.getId(), AL10.AL_SOURCE_RELATIVE, getTrack() .isRelative() ? AL10.AL_TRUE : AL10.AL_FALSE); - playInNewThread(200); + if(AudioSystem.getSystem().getCreateStreamThreads()) { + playInNewThread(); + } else { + if (playStream()) { + ((OpenALSystem)AudioSystem.getSystem()).addStreamedPlayer(this); + } else { + logger.warning("Could not play stream: "+getTrack().getResource()); + } + } } } @@ -209,10 +217,10 @@ * at which interval should the thread call update, in * milliseconds. */ - public boolean playInNewThread(long updateIntervalMillis) { + public boolean playInNewThread() { try { if (playStream()) { - playerThread = new PlayerThread(updateIntervalMillis); + playerThread = new PlayerThread(); playerThread.start(); return true; } @@ -314,12 +322,16 @@ } else if (getStream().getDepth() == 16) { format = (mono ? AL10.AL_FORMAT_MONO16 : AL10.AL_FORMAT_STEREO16); - } else return false; - + } else { + return false; + } + AL10.alBufferData(buffer, format, dataBuffer, getStream() .getBitRate()); + return true; } + if (isLoop() && getTrack().isEnabled()) { setStream(getStream().makeNew()); return stream(buffer); @@ -346,37 +358,53 @@ * XXX: I am considering abolishing these one-per-sound threads. */ class PlayerThread extends Thread { - // at what interval update is called. - long interval; /** Creates the PlayerThread */ - PlayerThread(long interval) { - this.interval = interval; + PlayerThread() { setDaemon(true); } /** Calls update at an interval */ public void run() { try { - while (!isStopped && update()) { - sleep(interval); + if(threadUpdate()) { + sleep(AudioSystem.getSystem().getStreamUpdateInterval()); } - while (isActive()) { - sleep(interval); - } - OpenALStreamedAudioPlayer.this.stop(); - } catch (Exception e) { -// e.printStackTrace(); + } catch (Exception e) { + e.printStackTrace(); } } } + + /** + * This is what the driver-thread should call every now and then to keep data + * coming to the audio-track. + * + * @return <code>true</code> if the thread should continue running. + * @throws IOException + */ + public boolean threadUpdate() { + try { + if(!isStopped && update()) { + return true; + } + + if(isActive()) { + return true; + } + } catch(IOException e) { + e.printStackTrace(); + } + OpenALStreamedAudioPlayer.this.stop(); + return false; + } @Override public void applyTrackProperties() { OpenALPropertyTool.applyProperties(this, source); } - @Override + @Override public void updateTrackPlacement() { Vector3f pos = getTrack().getWorldPosition(); Vector3f vel = getTrack().getCurrVelocity(); Index: openal/OpenALSystem.java =================================================================== --- openal/OpenALSystem.java (revision 272) +++ openal/OpenALSystem.java (revision 448) @@ -32,10 +32,12 @@ package com.jmex.audio.openal; +import java.io.File; import java.io.IOException; import java.net.URL; import java.nio.IntBuffer; import java.util.Collections; +import java.util.Iterator; import java.util.LinkedHashMap; import java.util.LinkedList; import java.util.Map; @@ -68,6 +70,8 @@ .75f, true)); private long held = 0L; private long lastTime = System.currentTimeMillis(); + private LinkedList<OpenALStreamedAudioPlayer> streamedPlayers = new LinkedList<OpenALStreamedAudioPlayer>(); + private long lastStreamPlayerTime = System.currentTimeMillis(); public OpenALSystem() { ear = new OpenALEar(); @@ -109,6 +113,25 @@ long thisTime = System.currentTimeMillis(); float dt = (thisTime - lastTime) / 1000f; lastTime = thisTime; + + // Update any stream-players? + if(!getCreateStreamThreads()) { + if(lastStreamPlayerTime + getStreamUpdateInterval() < thisTime) { + lastStreamPlayerTime = thisTime; + Iterator<OpenALStreamedAudioPlayer> player_iterator = streamedPlayers.iterator(); + OpenALStreamedAudioPlayer player = null; + while(player_iterator.hasNext()) { + player = player_iterator.next(); + //System.out.println("Updating["+player.isPlaying()+"]:"+new File(player.getTrack().getResource().getFile()).getName()+":"+player.getVolume()+"/"+player.getTrack().getVolume()); + //if(player.getTrack().getVolume() != player.getTrack().getTargetVolume()) { + // System.out.println("Fading from "+player.getTrack().getVolume()+" to "+player.getTrack().getTargetVolume()); + //} + if(!player.threadUpdate()) { + player_iterator.remove(); + } + } + } + } try { for (int x = 0; x < MAX_SOURCES; x++) { @@ -278,4 +301,12 @@ public void setSpeedOfSound(float unitsPerSecond) { AL10.alDopplerVelocity(unitsPerSecond); } + + public void addStreamedPlayer(OpenALStreamedAudioPlayer player) { + synchronized (this) { + if(!streamedPlayers.contains(player)) { + streamedPlayers.add(player); + } + } + } } No newline at end of file Index: openal/OpenALUtil.java =================================================================== --- openal/OpenALUtil.java (revision 0) +++ openal/OpenALUtil.java (revision 448) @@ -0,0 +1,41 @@ +package com.jmex.audio.openal; + +import java.nio.IntBuffer; + +import org.lwjgl.BufferUtils; +import org.lwjgl.openal.AL; +import org.lwjgl.openal.AL10; +import org.lwjgl.openal.ALC10; +import org.lwjgl.openal.ALCcontext; +import org.lwjgl.openal.ALCdevice; + +public class OpenALUtil +{ + private OpenALUtil() { + // no c'tor + } + + public static void printOpenALInfo() + { + ALCdevice device = AL.getDevice(); + ALCcontext context = AL.getContext(); + System.out.println("Device: "+device); + System.out.println("Context: "+context); + + // Device? + System.out.println("ALC_DEVICE_SPECIFIER:"+ALC10.alcGetString(device, ALC10.ALC_DEVICE_SPECIFIER)); + System.out.println("ALC_DEFAULT_DEVICE_SPECIFIER:"+ALC10.alcGetString(device, ALC10.ALC_DEFAULT_DEVICE_SPECIFIER)); + + // Settings + IntBuffer integerdata = BufferUtils.createIntBuffer(1); + ALC10.alcGetInteger(device, ALC10.ALC_ATTRIBUTES_SIZE, integerdata); + integerdata = BufferUtils.createIntBuffer(integerdata.get(0)); + ALC10.alcGetInteger(device, ALC10.ALC_ALL_ATTRIBUTES, integerdata); + integerdata.rewind(); + while(integerdata.position() != integerdata.capacity()) + { + System.out.println("I:"+integerdata.get()); + } + //System.out.println(AL10.alGetString()); + } +}