Audio Sprites (and fixes for iOS)

I recently had to work on a project for iOS that required that sound play on particular actions being performed. The problem is that iOS and HTML5 has been seriously oversold by Apple and the devices are pretty poor in comparison to the desktop. Audio and video are particularly poor, so to solve my problem I used an audio sprite, a technique that was similar to CSS sprites, just for audio.

The Problems

I’ll first explain some of the problems I encountered, because these issues are particular to iOS – but could be common to mobile devices.

  1. iPhones don’t like playing too much audio at once, it gets very choppy.
  2. iPads don’t play more than one audio stream at once. There’s some good info on 24 Ways, in the State of Audio – Early Findings – sadly this was published after I discovered this for myself!
  3. iOS won’t download the audio unless the user initiates the action.
  4. There’s about a 1/2 second delay before iOS is able to play the audio – this is because the audio object (in iOS, not HTML5) is being created.

Given some of these problems, multiple audio files wasn’t going to solve my problem, so I needed to create a sprite for my audio and work with that in iOS.

Audio Sprite

If you think CSS sprite, just with audio, then you’ll already know that each playable part of a sprite should be the same size. That said, you could program your app to have variable size sprites – but I just think it’s easier to build and develop with if they’re the same size.

So if your audio is 1 second long, and you want 10 audio clips to play, you need a sprite that’s 10 seconds long. Not rocket science.

One thing I did notice: using Audacity to create the sprite, shifted all the audio. This may have been me messing things up, but it’s something I consistently noticed, so I adjusted for it in the code using a lead time (something that’s included in the final source code). Not ideal though, really you want each sprite to start on the 10s modulus 1s increment.

Audio Sprite Track

The concept of playing a sprite is simple. You pass the index of the audio you want to play, and the track moves the playhead to index times sprite length and starts playing. You also need to know when to stop the sprite, so that’s the start time + sprite length, and there’s a setInterval that’s constantly watching for when to stop.

Here’s what the track code would look like (note that we’ve not fixed all the iOS issues yet):

function Track(src, spriteLength) {
  var audio = document.createElement('audio');

  audio.src = src;
  audio.autobuffer = true;
  audio.load(); // force the audio to start loading...doesn't work in iOS

  this.audio = audio;
  this.spriteLength = spriteLength;
}

Track.prototype.play = function (position) {
  var track = this,
      audio = this.audio, // the audio element with our sprite loaded
      length = this.spriteLength, // the length of the individual audio clip
      time = position * length,
      nextTime = time + length;

  audio.pause();
  audio.currentTime = time;
  audio.play();

  // clear any stop monitoring that was in place already
  clearInterval(track.timer);
  track.timer = setInterval(function () {
    if (audio.currentTime >= nextTime) {
      audio.pause();
      clearInterval(track.timer);
    }
  }, 10);
};

Fixing the iOS issues

The biggest problem with iOS is that the audio hasn’t loaded at all. So when we try to set audio.currentTime = time a fatal JavaScript error will be thrown because the audio doesn’t have that length loaded and all the code will go to crap.

So we need to wrap it with a try/catch:

try {
  audio.pause();
  audio.currentTime = time;
  audio.play();      
} catch (e) {
  // what now?
}

But now we’ve wrapped it in a try/catch, what do we do? We could play the audio and keep try/catching to move the currentTime but the problem is that the audio will start playing in the wrong place.

What we need to do is try to play the audio, and as soon as it has loaded the data for the audio, pause it, move to the right position and then start playing again. This is the tricky bit.

From experimenting with iOS, I found that the progress event was the one that told me I had enough data to fast forward before the user heard any sound. I would have expected this to be canplay or canplaythrough – but alas, this is iOS and all is not what you’d expect!

So if the catch in our code fires, we need to listen for the progress and then try to set the currentTime. So I’m changing the catch to this:

try {
  audio.currentTime = time;
  audio.play();
} catch (e) {
  track.updateCallback = function () {
    track.updateCallback = null;
    audio.currentTime = time;
    audio.play();
  };
  audio.play();
}

Now inside the Track object function, I’ve also adding the following code:

var track = this;

var progress = function () {
  audio.removeEventListener('progress', progress, false);
  if (track.updateCallback !== null) track.updateCallback();
};

audio.addEventListener('progress', progress, false);
track.updateCallback = null;

This means that we’re listening for the progress event, and when it fires for the first time, we remove the handler, and if there’s a callback, we’ll call it. The progress is triggered by trying to play. So now we’ve got it jumping to the right point, but we still have the 1/2 second delay.

That we’re going to try to fix by preloading the audio secretly in the background. Since the audio can only be loaded when the user clicks, let’s listen for that at the top level of the document and kick off the audio loading. However, the only way you can start the audio loading is by playing it. So that’s what we’ll do, but once the play event fires, we’ll pause it right away to stop the audio from playing back, but forcing it to move on to the progress event.

Again inside the new track object, we add the code to try to load the audio:

var force = function () {
  audio.pause();
  audio.removeEventListener('play', force, false);
};
audio.addEventListener('play', force, false);

var click = document.ontouchstart === undefined ? 'click' : 'touchstart';
var kickoff = function () {
  audio.play();
  document.documentElement.removeEventListener(click, kickoff, true);
};
document.documentElement.addEventListener(click, kickoff, true);

Now we’re try to forcibly load the audio as early as possible, which will trigger the progress event, which in turn will allow us to set the currentTime correctly in our audio sprite.

The completed code

I’ve also implemented a simple player to allow you to utilise multiple tracks (something that works very well on the desktop) or you can use a single track object directly. The completed code, with a few more comments and attempts to mute the audio has been included in a gist: HTML5 audio sprites

24 Responses to “Audio Sprites (and fixes for iOS)”

  1. This is a really clever idea – obvious now I’ve seen it, but not something I had thought of when I’d run into similar issues. (I gave up!)

    I agree wholeheartedly about the oversell of HTML5 in iOS – stuff is supported, but not always in a easy to use way. Was this iOS 4.2? It seems that lots has been fixed/added in that build.

  2. You are using the autobuffer property. This property was replaced by the more “feature-rich” preload attribut (includes content and IDL attribute). autobuffer was only supported by FF3.6. preload is partially supported by Safari 5.0+ for Mac (not for Win) and fully by FF4.0. So to be more futureproof, you should add preload = “auto”.

    I think we have to wait long time, till Apple will implement the preload attribute for iOS, most developers don’t really understand this property and do set this to often to “auto”/”" (“false” / false also means “auto”/”" in Safari).

    Chrome has implemented the preload-interface, but not the preload-feature (Feature detection is impossible…).

    Did you try loadstart or loadedmetadata, instead of progress?

    regards
    alex

  3. @Alex – yep, I tried all the events (I’ve got a script that listens for all the events) and lets me know exactly when I’ve got enough data. Note that progress fires before loadedmetatdata – loadstart fires first on iOS, but immediately suspends. Hense why I need to force the playing.

  4. Very nice implementation. Strangely enough, I had been thinking about audio sprites ages ago (even calling them that), but then never had a chance to do anything with that idea…so you, sir, win one internets.

  5. [...] inheritance paradigm by using delegation instead Glittery mouse (JS1k entry) – it sparkles! Audio Sprites (and fixes for iOS) – Remy Sharp explains the ins and outs of playing audio on iOS Tangram (Chinese) is a [...]

  6. Nice stuff. I like your solution to setting the currentTime on unplayed media. I’d been using a timeout to retry after 100ms, which has worked well for the past year or so. Your solution is much more elegant.

    I wrote a media event inspector for audio and video. Maybe of use to people interested in the events.
    http://www.jplayer.org/HTML5.Media.Event.Inspector/

  7. [...] pre-cache effects so they can be played on demand. Remy Sharp has investigated this issue: see Audio Sprites — a technique for loading multiple effects in a single audio [...]

  8. Crazy useful! Thanks so much for posting this. I’m of the opinion that HTML5′s biggest hurdles at the moment are audio and performance, in that order. Louder, faster! :)

  9. Hey, Remy, just an FYI, I used http://wavepad.en.softonic.com/ and it was really easy to create my sprite!

    Thanks,
    Atg

  10. Remy, this is a wonderful solution. I’m trying to get this working on IOS 4.2. Initially my sounds play correctly, but subsequent plays don’t seek to the correct location (currentTime never changes). Would it be possible for you to post a demo with a few buttons that each play a different track? That way I would at least know if the problem is on my side (which seems likely). Thanks!

  11. Hey, Remy, it appears the “var click” line is missing from your Gist?

    var click = document.ontouchstart === undefined ? 'click' : 'touchstart';

    Also, do you happen to have a working example, because I can get the above to work fine in a desktop Safari, but still get no sounds on the iPad…

    thanks,
    Atg

  12. Remy, this is fantastic. Thank you for your efforts on this. Although I can get sound to play, it usually only works when I specify there is only 1 track. And then, the sound doesn’t stop correctly.

    Also, would it be more efficient to handle some of the event listeners outside the Track object definition? As it is, wouldn’t handlers be added N times for each of the N tracks? Can you explain why that is necessary?

    Thanks again!

  13. Remy, thank you so very much for your efforts. I finally got it working on the iPad. At first I tried using three different audio files, but that was causing a lot of problems, so I just concatenated the audio files into one, and did it that way.

    Another weird quirk that came up was when I had sound that was in 2.5 second intervals. That didn’t seem to work well for some reason.

    Also I’ll need to figure out a way to get the sound to load and notify the user that it’s ready to use PRIOR to them clicking for the sound.

    Great work!!

  14. Hi Remy,

    I’d like to echo the other posters in my congratulations to you on such a great article. I’ve just started an iPad-oriented HTML5 audio project, and this blog post was the first one I saw that talked about your Problem #4 above (the 500 ms delay for IOS object creation). This has been driving me crazy, as I couldn’t see any browser-side activity to explain it. What technique did you use to determine this was the problem?

    Also, I’ve noticed that the delay is exacerbated if accessing the audio over a lower-bandwidth 3G connection vs. WiFi. Even after the audio has been loaded/cached (i.e. during a replay), the Audio.play() command seems to need to tickle the server with no visible sign it’s doing so. Have you found the same to be true in your experiments? Any idea what it’s doing? Wouldn’t these additional delays also afflict an audio-sprite solution like you’ve described here?

    Thanks again!

  15. Thanks for putting this together, Remy! I’m working on an iPad app and was thinking sound would be nice, and a quick google and here’s a working solution put together already. And it works great (iOS issues aside)! (Well, it works great after adding the line Aaron mentions…putting it at line 37 seems to fix things…you might want to update the source files with that).

  16. Regarding HTML5 hurdles, not many are talking about timing/synchronization problems in iOs but that is my number one concern at the moment. When setting up timers and events, delays up to hundreds and even thousands of milliseconds are not unusual.
    For a game, it can mean a total disaster when it comes to user experience.

    Second to that, is lack of good sound support, I mean sound being a major thing in modern games, you expect it to be there and to be good. Not one sound at a time with bad synchro on when it starts..

    Flash solves those two probs beautifully. Apple should try to use Flash more maybe they’d start to love it :-). At least they could learn something, regarding what is needed for HTML5 to really make it usable to the majority of developers.
    Why not use multimedia timers, with high system prio, in Safari for HTML pages with multmedia content instead of those low-prio timers it currently use?

    But then of course, if HTML5 was really good, who would feel the need to develop xcode apps? :-) Of course Apple doesn’t want to “lose” developers to HTML5 from their native app developer community. So, HTML5 will probably never be implemented in a great way in propriety devices like Apples, that is one of the major thumbs up for Google/Android, who don’t impose that kind of restrictive thinking but in the opposite way give full freedom with the best support in technology and devices. Android = freedom! Oh, time to go back to work.

  17. well, this is only working partially for me.

    sounds get played only the very first time. and then they dont stop after a track. that means the rest of the whole sprite is played.

    as others have said, a sample page with a working example would be great!

    anyway, keep up the good work! i would love to see an improved version!

  18. [...] much all brilliant ideas I have, I decided to search for this one before I dive in. Turned out Remy Sharp has already talked about this. So I knew it was possible and wanted to check the server-side or things. (Remy is amazing, by the [...]

  19. Does anyone happen to have a working example? I’m evidently missing something. Thanks!

  20. I am using this for a game. Sometimes, just after loading, the audio sprite doesn’t stop and continue playing all the clips.

  21. Hello! first: thank you very much for this!!!

    I have a question though…
    How can i make he system switch between the mp3 and ogg files automatically?

    greetings !

  22. [...] Add audio. Contrary to what I said at the beginning of this article, this isn’t impossible, just a bit of a headache. One technique is to use audio sprites (kind of like CSS image sprites); Remy Sharp breaks it down. [...]

  23. thanks for your code.
    but I can only constructing one object at all ? And I had been try modified the inside class code ‘audio’ instead of various but fail for construct multiple object purpose.

  24. [...] audio on other browsers (Internet Explorer, and Firefox for now).  It also made it easy to use audio sprites, which makes the pages load quicker and work much better in Safari and iOS — where things were [...]

Leave a Reply
Not required

CODE: Please escape code and wrap in <pre><code>, doing so will automatically syntax highlight