I gave a presentation this last Friday at WebGL Camp 4, the slides of which are online now. I had a great time, met some awesome developers, and saw a lot of things that got me really excited about the future of WebGL. I highly encourage anyone that is interested in WebGL to try and make it to WebGL Camp 6!
During my talk I was able to show what I think may be the first public demo of compressed textures in WebGL! The demo isn't terribly impressive, it simply displays a DXT5 texture loaded from a DDS file, but it shows off the required code effectively enough.
media.tojicode.com/webgl-samples/dds.html
(Warning: That demo will only work on machines that support DXT5 compression. That should be most desktops, but the majority of mobile or tablet devices will be out of luck! You'll also need to be running a fairly new Chrome dev channel build)
Yay! I got a textured cube on screen! Surely I'm the first person ever to do this!
Okay, yeah... it's not all that impressive. The key here is the potential that it provides. Compressed textures have been an integral part of 3D games and many other 3D applications on the desktop, console, and mobile platforms that they've become something of an invisible, pervasive optimization that everyone tends to take for granted. Up until now, however, they've been something that's been left out of WebGL (not without reason, they're tricky to get right). The fact that we're gaining the ability to use them now is, in my view, something of a benchmark of the maturity of the standard.
So what exactly do we mean by compressed texture? If you're not already familiar with the concept from a prior life as a game developer it can be a bit confusing to really grok what we're referring to and why it matters. After all, JPEGs and PNGs are compressed images, right? What's different here?
(Note: If you already know the answers to the above questions just skip to the "Implementation" section)
Theory
A good place to start it to look at what WebGL does with a normal texture. Let's say we've got a texture that's 1024x1024 pixels. The textures from iOS RAGE are a good example:
This texture, as it's used in my demo, is saved as a JPEG image and is 187k. That's not too bad, all things considered! For download times, certainly, it's great! But what about when we create a texture out of it? What happens then?
Your graphics card doesn't know how to read the JPEG file format, and even if it did we wouldn't want to make it do so. Decompressing a format like JPEG is slow compared to reading, say, a BMP and doesn't account very well for random access of texel data. (A texel is just a pixel of texture data.) So when we call gl.texImage2d with an image, what it actually does in the background is completely decompress the image and send the decompressed version to the graphics card. This is more readily apparent in OpenGL as used in a native language like C because it actually forces you to do the decompression yourself before providing it any texture data. WebGL is very, very kind to us developers in that regard.
The decompressed image data is basically just an array of RGB or RGBA values (think BMP without the header). Each color channel takes 1 byte, so each texel of an RGB texture is 24 bits, and each texel of an RGBA texture is 32 bits. This means that it's really easy to figure out how much RAM a given texture will take up on your graphics card. Given the texture from above:
1024 (height) * 1024 (width) * 3 (bytes per pixel) == 3,145,728 bytes == 3MB!
So that nice, compact little 187k texture is more than 16 times bigger when it reaches your GPU! Ouch! This has two practical side effects.
- Texture upload speeds can suffer because we're sending a lot of data (this was the topic of my presentation)
- Video memory can fill up pretty fast. Especially on mobile devices that won't have a lot of wiggle room to begin with.
At this point, you'd normally have to start looking at your game resources and say "Well, sorry artists. We just don't have the room to put all of these texture in the scene. We've got to make them all a quarter of the size." And your artists will now hate you for downscaling their lovingly crafted pixels.
Unless...
Cue texture compression!
Texture compression is, at it's core, algorithms built into your graphics card that let them decompress specific texture formats on the fly. This means that the texture actually stays compressed in your video memory and is only decompressed when your shader does a texture lookup. That may sound slow, but it's not. You can only use very specific formats for the compressed textures, and those are formats that are designed for speed rather than super high compression rates. For example, let's look at one of the more popular formats: DXT5 (The one used in the demo from above.)
DXT5 offers a fixed 4:1 compression ratio. This means that for every 1 pixel of an uncompressed texture, DXT5 can store 4. Applying that to our above 1024x1024 texture, this would translate to 768k (1/4th of the uncompressed version). That also happens to be the same amount of space that a 512x512 uncompressed texture would take. Seems like a pretty clear win, right? Unfortunately it's not without it's complications.
The compression for most formats is lossy (though in different ways than JPEG), but typically the visual artifacts of the compression are offset by the fact that you can fit four times as much texture data into the same space! Ask yourself this: Would you rather have a 512x512 texture that's completely accurate or a 1024x1024 texture that's got some minor artifacts? I know which one I would choose! Of course, the nice thing is that the developer gets to choose which route they want to go: Do you need four times as much detail in your scene, or do you want your scene to fit in a fourth the memory?
Also on the list of downsides, obviously 768k is much larger than the 182k JPEG version, so (counterintuitively) compressed textures will probably not help you out much when it comes to download times, which is unfortunate. Just one more thing to consider in the course of your development.
A final hitch in the whole compressed texture thing is that not every device supports every compressed texture format. For example: Most desktop systems will support S3 compression (that's your DXT1-5 textures), but your iOS devices won't. They support PVR compression. The two methods are similar in usage, but have different properties in terms of compression, artifacts, and performance. Those aren't the only formats either. There's a lot of them out there! It's hard to argue that one is "better" than the other (though there's plenty of people who try, usually the hardware manufacturers), and on the developers end you simply have to use whatever your platform can understand.
That does make things difficult for WebGL, however. Since pretty much any device has a browser nowadays, and many of those will probably have WebGL in the future, it means that a WebGL dev that wants to use compressed textures will have to keep three or four variants of their textures around, query the device for which one's it supports, and then download the one that meets the platforms needs. Sounds fun, right?
That being said, in many cases the upsides will still outweigh the downs, and it's probably worth the pain on our end as developers to deliver a richer experience to the users.
So, now that we know all about what a compressed texture is, how do we use them?
Implementation
First, a big fat disclaimer: These APIs are all very new and very experimental. While I don't expect the overall concepts to change to much, the exact details may shift around a bit as the feature settles down. I will do my best to keep this post updated in the future with the correct API calls, but don't kill me if they're out of date from time to time. Also, I'm pretty much only looking at Webkit browsers here. I have no idea what the other browsers are going to try and do in this space, but I would imagine that it shouldn't be too terribly different than the methods described here.DXT5 offers a fixed 4:1 compression ratio. This means that for every 1 pixel of an uncompressed texture, DXT5 can store 4. Applying that to our above 1024x1024 texture, this would translate to 768k (1/4th of the uncompressed version). That also happens to be the same amount of space that a 512x512 uncompressed texture would take. Seems like a pretty clear win, right? Unfortunately it's not without it's complications.
The compression for most formats is lossy (though in different ways than JPEG), but typically the visual artifacts of the compression are offset by the fact that you can fit four times as much texture data into the same space! Ask yourself this: Would you rather have a 512x512 texture that's completely accurate or a 1024x1024 texture that's got some minor artifacts? I know which one I would choose! Of course, the nice thing is that the developer gets to choose which route they want to go: Do you need four times as much detail in your scene, or do you want your scene to fit in a fourth the memory?
Also on the list of downsides, obviously 768k is much larger than the 182k JPEG version, so (counterintuitively) compressed textures will probably not help you out much when it comes to download times, which is unfortunate. Just one more thing to consider in the course of your development.
A final hitch in the whole compressed texture thing is that not every device supports every compressed texture format. For example: Most desktop systems will support S3 compression (that's your DXT1-5 textures), but your iOS devices won't. They support PVR compression. The two methods are similar in usage, but have different properties in terms of compression, artifacts, and performance. Those aren't the only formats either. There's a lot of them out there! It's hard to argue that one is "better" than the other (though there's plenty of people who try, usually the hardware manufacturers), and on the developers end you simply have to use whatever your platform can understand.
That does make things difficult for WebGL, however. Since pretty much any device has a browser nowadays, and many of those will probably have WebGL in the future, it means that a WebGL dev that wants to use compressed textures will have to keep three or four variants of their textures around, query the device for which one's it supports, and then download the one that meets the platforms needs. Sounds fun, right?
That being said, in many cases the upsides will still outweigh the downs, and it's probably worth the pain on our end as developers to deliver a richer experience to the users.
So, now that we know all about what a compressed texture is, how do we use them?
Implementation
Compressed textures are exposed to WebGL as an extension. To anyone that's familiar with extensions in desktop OpenGL, that may be a cringe inducing statement, but WebGL actually makes working with extensions fairly painless!
[Updated: You used to query for texture compression as a single extension. It has since been decided that it makes more sense to query for support of individual textures, which is what this line now does]
var ct = gl.getExtension("WEBKIT_WEBGL_compressed_texture_s3tc");
If the extension is supported, ct will be an object that contains all the functions and enumerations for the extension. If the extension is not supported, you'll get back NULL. Not too bad, right? From this point on, anything that needs to be done for compressed textures will happen on the ct object rather than the gl object.
Unfortunately the contents of the ct object aren't documented anywhere that I know yet, with the exception of looking at the code used to implement it. [EDIT: Brendan pointed out in the comments that the spec for this extension is available] I got most of my information from this Webkit bug. Lucky for us, there's only a few functions that we need to look at.
First is the enumerations for the texture types. This list could conceivably grow, but as implemented right now the following symbols are available from the extension queried above:
- COMPRESSED_RGB_S3TC_DXT1_EXT
- COMPRESSED_RGBA_S3TC_DXT1_EXT
- COMPRESSED_RGBA_S3TC_DXT3_EXT
- COMPRESSED_RGBA_S3TC_DXT5_EXT
Each of these represents a compressed texture type that may be supported. It should be pretty apparent from the names which is which. Just because