I’ve taken a liking to a video game called osu! over the past few months. It’s a rhythm game where you use move your mouse to circles that appear with the beat, and click (or press a key) at the right time. It looks something like this:
The key of this game is that the “beatmaps” (a song plus notes to hit) are user-submitted. There are thousands of them, and the difficulty curve is very long - I’ve been playing for 10 months and I’m only maybe 70% of the way up the difficulty curve. It’s also a competitive game, which leads to a lot more fun, where each user tries to complete maps a little bit better than everyone else can. You can see on the left in that video - this is a very good player who earned the #1 rank during this play.
- Decompress osu beatmap archives
- Decode the music with Web Audio
- Decode the osu! beatmap format
- Play the map!
In case you don’t have any osz files hanging around, try out this one, which is the one from the video above.
osu!web and the future
This part of the blog post is for non-technical readers, mostly osu players. osu!web is pretty cool, and I want to make it even better. My current plans are just to make it a beatmap viewer, and I’m working now on achieving that goal. I have to finish sliders and add spinners, and eventually work on things like storyboards. Playing background videos is not in the cards because of limitations with HTML5 video.
Eventually, I’d like to make it possible to link to a certain time in a certain map, or in a certain replay. Oh yeah, I want to make it support replays, too. If I get replays working, though, then I don’t see any reason not to let players try the maps out in their web browsers, too. Keep an eye out!
This project is only possible thanks to a whole bunch of new web technologies that have been stabilizing in the past year or so. The source code is on Github if you want to check it out.
- A number of “tracks” - osu files that define notes and such for various difficulties
- The song (mp3) and optionally a video background (avi)
- Assets - a background image and optionally a skin (like a Minecraft texture pack)
We then load the *.osu files and decode them. They look similar to ini files or Unix config files. Here’s a snippet:
[General] AudioFilename: MuryokuP - Sweet Sweet Cendrillon Drug.mp3 AudioLeadIn: 1000 PreviewTime: 69853 # snip [Metadata] Title:Sweet Sweet Cendrillon Drug TitleUnicode:Sweet Sweet Cendrillon Drug Artist:MuryokuP ArtistUnicode:MuryokuP Creator:Smoothie Version:Cendrillon # snip [HitObjects] 104,308,1246,5,0,0:0:0:0: 68,240,1553,1,0,0:0:0:0: 68,164,1861,1,0,0:0:0:0: 104,96,2169,1,0,0:0:0:0: 172,60,2476,2,0,P|256:48|340:60,1,170,0|0,0:0|0:0,0:0:0:0: 404,104,3399,5,0,0:0:0:0: # snip
This is decoded by
some sections (like
[Metadata]), it just puts each entry into a dict that you
can pull from later. It does more for things like hit objects, and understands
which of these lines is a slider versus a hit circle versus a spinner and so on.
I sneakily loaded a beatmap in the background in your browser as you were
reading. If you want to check it out, open up your console and play with the
track object. Ignore all the disqus errors, they’re irrelevant.
Enter stage: Web Audio
Web Audio had a bit of a rocky development cycle, what with Chrome thinking it’s special and implementing a completely different standard from everyone else. Things have settled by now, and I can start playing with it 😁 Bonus: Mozilla finally added mp3 support to all platforms, including Linux (which my dev machine runs).
The osz file includes an mp3, which we extract into an ArrayBuffer, and load into a Web Audio context. This is super cool and totally would not have been possible even a few months ago - kudos to the teams implementing all this exciting stuff in the browsers.
That’s about all we’re doing with Web Audio right now. I do add a gain node so that you can control the volume with your mouse wheel. In the future, we can get more creative by:
- Adding support for HT/DT mods
- Adding support for NC
Enter stage: PIXI
Once we’ve decoded the beatmap and loaded the audio, we can play it. After briefly showing the user a difficulty selection, we jump into rendering the map. For this, I’ve decided to use PIXI.js, which gives us a really nice API to use on top of WebGL with a canvas fallback for when WebGL is not available. I was originally just using canvas, but it wasn’t very performant, so I went looking for a 2D WebGL framework and found PIXI. It’s pretty cool.
First, we iterate over all of the hit objects on the beatmap and generate sprites for them:
This is all done before we start playing. We consider the timestamp in the music that the hit is scheduled for, and then we place all of the hit objects into an array and start the song. See code for createHitCircle, which puts together a bunch of sprites for each hit cirlce and sets their alpha to zero. See also createSlider, which is more complicated (I’ll go into detail later).
Each frame, we get the current time from the Web Audio layer, and we run a function that updates a list of upcoming hit objects:
I adopted this pattern early on for performance reasons. During each frame’s rendering step, we only have the sprites and such loaded for hit objects in the near future. This saves a lot of time. PIXI has all of these sprites loaded and draws them for us each frame. During each frame, all we have to do is update them:
This is passed in the current timestamp in the song, and based on this we are able to do some simple math to calculate how much alpha each note should have, as well as the scale of the approach circle (which tells you when to click the note):
I’ve left out sliders, which again are pretty complicated. We’ll get to them after you look at this screenshot again:
All of these hit objects are having their alpha and approach circle scale adjusted each frame by the above method. Since we’re basing this on the timestamp of the map, a convenient side effect is that we can pass in any time to see what the map should look like at that time.
The hardest thing so far has been rendering sliders, which are hit objects that you’re meant to click and hold as you move across the “slider”. They look like this:
The golden circle is the area you need to keep your mouse in if you want to pass this slider. Sliders are defined as a series of curves. There are a few kinds:
- Linear sliders (not curves)
- Catmull sliders
- Bezier sliders
For now I’ve only done bezier sliders. I give many thanks to opsu, which I learned a lot of useful stuff about sliders from. Each slider is currently generated using the now-deprecated “peppysliders” method, where the sprite is repeated along the curve several times. If you look carefully as a slider fades out, you can notice that this is the case.
The newer style of sliders involves rendering them with a custom shader. This should be possible with PIXI, but I haven’t done any research on them yet. Again, I expect to be able to draw a lot of knowledge from reading the opsu source code.
I left out the initializer for sliders earlier, because it’s long and complicated. I’ll include it here so you can see how this goes:
As you can see, there are many more moving pieces here. The important part is the curve:
In the curve code, a series of points along each curve are generated for us to place sprites at. These are precomputed like all other hit objects to save time during playback. However, the render updater is still quite complicated:
Much of this is the same as the hit circle updater, since we have a similar hit
circle at the start of the slider that needs to update in a similar fashion.
However, we also have to move the rolling ball and the follow circle along the
slider as the song progresses. This involves calling out to the curve code to
figure out what point is (
current_time / slider_end) along the length of the
slider. We put the ball there, and we also ask for the point at (
0.01) / slider_end) and make the ball rotate to face that direction.
That’s the bulk of the work neccessary to make an osu renderer. I’ll have to add spinners once I feel like the slider code is complete, and a friend is working on adding hit sounds (sound effects that play when you correctly hit a note). The biggest problem he’s facing is that Web Audio has no good solution for low-latency audio playback. On my side of things, though, everything is going great. PIXI was a really good choice - it’s an easy to use API and the WebGL frontend is fast as hell. osu!web plays a map with performance that compares to the performance of osu! native.