All posts by Admin

Kevin is the author of OpenGL ES 2 for Android: A Quick-Start Guide. He also has extensive experience in Android development.
Shark

Check out These Cool Apps from an Early Student of Learn OpenGL ES!

An early student of Learn OpenGL ES, Miguel, has been working on some neat apps lately. Miguel was one of the first readers at Learn OpenGL ES, and we’ve exchanged at least a few emails back and forth about different topics, such as working with the camera through matrices. We were both learning a lot back then, and that sort of interaction really helped to strengthen my own personal knowledge of OpenGL.

Miguel hasn’t stopped learning, and he recently sent me over a link to his live wallpaper, “Shark 3D Kosmos”. I’m really happy with the progress he’s made so far; read on to learn more!

Shark 3D Kosmos Live Wallpaper

This wallpaper features a scary and realistic shark swimming on your home screen! The shark can be rendered in different modes, including holo mode, and you can select from several different camera modes, including one where the shark swims straight at you! The live wallpaper even includes features to save battery usage.

Link: https://play.google.com/store/apps/details?id=ksm.plylab.ksmwpp

Kosmos Galaxy 3D

This is a neat dynamic simulation of the attraction between the center of a galaxy and its surrounding stars. You can move the simulation around and add galaxies in realtime.

Link: https://play.google.com/store/apps/details?id=pleyasLab.KSMglx

I’m happy with how far Miguel has come since the first lessons here at Learn OpenGL ES. If you’ve also learned from the site and would like to showcase your work, let me know!

Share
Android Lesson Eight: An Introduction to Index Buffer Objects (IBOs)

Android Lesson Eight: An Introduction to Index Buffer Objects (IBOs)

Android Lesson Eight: An Introduction to Index Buffer Objects (IBOs)In our last lesson, we learned how to use vertex buffer objects on Android. We learned about the difference between client-side memory and GPU-dedicated memory, and the difference between storing texture, position and normal data in separate buffers, or altogether in one buffer. We also learned how to work around Froyo’s broken OpenGL ES 2.0 bindings.

In this lesson, we’ll learn about index buffer objects, and go over a practical example of how to use them. Here’s what we’re going to cover:

  • The difference between using vertex buffer objects only, and using vertex buffer objects together with index buffer objects.
  • How to join together triangle strips using degenerate triangles, and render an entire height map in a single rendering call.

Let’s get started with the fundamental difference between vertex buffer objects and index buffer objects:

Vertex buffer objects and index buffer objects

In the previous lesson, we learned that a vertex buffer object is simply an array of vertex data which is directly rendered by OpenGL. We can use separate buffers for each attribute, such as positions and colors, or we can use a single buffer and interleave all of the data together. Contemporary articles suggest interleaving the data and making sure it’s aligned to 4-byte boundaries for better performance.

The downside to vertex buffers come when we use many of the same vertices over and over again. For example, a heightmap can be broken down into a series of triangle strips. Since each neighbour strip shares one row of vertices, we will end up repeating a lot of vertices with a vertex buffer.

Diagram of a heightmap using triangle strips and a vertex buffer object.

You can see a vertex buffer object containing the vertices for two triangle strip rows. The order of vertices is shown, and the triangles defined by these vertices is also shown, when drawn using glDrawArrays(GL_TRIANGLE_STRIP, …).  In this example, we’re assuming that each row of triangles gets sent as a separate call to glDrawArrays(). Each vertex contains data as follows:

vertexBuffer = {
    // Position - Vertex 1
    0, 0, 0,
    // Color
    1,1,1
    // Normal
    0,0,1,
    // Position - Vertex 2
    1, 0, 0,
    ... 
}

As you can see, the middle row of vertices needs to be sent twice, and this will also happen for each additional row of the height map. As our height map gets larger, our vertex buffer could end up having to repeat a lot of position, color, and normal data and consume a lot of additional memory.

How can we improve on this state of affairs? We can use an index buffer object. Instead of repeating vertices in the vertex buffer, we’ll define each vertex once, and only once. We’ll refer to these vertices using offsets into this vertex buffer, and when we need to reuse a vertex, we’ll repeat the offset instead of repeating the entire vertex. Here’s a visual illustration of the new vertex buffer:

Vertex buffer object for use with an index buffer object.

Notice that our vertices are no longer linked together into triangles. We’ll no longer pass our vertex buffer object directly; instead, we’ll use an index buffer to tie the vertices together. The index buffer will contain only the offsets to our vertex buffer object. If we wanted to draw a triangle strip using the above buffer, our index buffer would contain data as follows:

indexBuffer = {
    1, 6, 2, 7, 3, 8, ...
}

That’s how we’ll link everything together. When we repeat the middle row of vertices, we only repeat the number, instead of repeating the entire block of data. We’ll draw the index buffer with a call to glDrawElements(GL_TRIANGLE_STRIP, …).

Linking together triangle strips with degenerate triangles

The examples above assumed that we would render each row of the heightmap with a separate call to glDrawArrays() or glDrawElements(). How can we link up each row to the next? After all, the end of the first row is all the way on the right, and the beginning of the second row on the left. How do we link the two?

We don’t need to get fancy and start drawing right to left or anything like that. We can instead use what’s known as a degenerate triangle. A degenerate triangle is a triangle that has no area, and when the GPU encounters such triangles, it will simply skip over them.

Let’s look at our index buffer again:

indexBuffer = {
    1, 6, 2, 7, 3, 8, 4, 9, ...
}

When drawing with GL_TRIANGLE_STRIP, OpenGL will build triangles by taking each set of three vertices, advancing by one vertex for each triangle. Every subsequent triangle shares two vertices with the previous triangle. For example, here are the sets of vertices that would be grouped into triangles:

  • Triangle 1 = 1, 6, 2
  • Triangle 2 = 6, 2, 7
  • Triangle 3 = 2, 7, 3
  • Triangle 4 = 7, 3, 8
  • Triangle 5 = 3, 8, 4
  • Triangle 6 = 8, 4, 9

OpenGL also maintains a specific order, or winding when building the triangles. The order of the first 3 vertices determines the order for the rest. If the first triangle is counter-clockwise, the rest will be counter-clockwise, too. OpenGL does this by swapping the first two vertices of every even triangle (swapped vertices are in bold):

  • Triangle 1 = 1, 6, 2
  • Triangle 2 = 2, 6, 7
  • Triangle 3 = 2, 7, 3
  • Triangle 4 = 3, 7, 8
  • Triangle 5 = 3, 8, 4
  • Triangle 6 = 4, 8, 9

Let’s show the entire index buffer, including the missing link between each row of the height map:

indexBuffer = {
    1, 6, 2, 7, 3, 8, 4, 9, 5, 10, ..., 6, 11, 7, 12, 8, 13, 9, 14, 10, 15
}

What do we need to put in between, in order to link up the triangles? We’ll need an even number of new triangles in order to preserve the winding. We can do this by repeating the last vertex of the first row, and the first vertex of the second row. Here’s the new index buffer below, with the duplicated vertices in bold:

indexBuffer = {
    1, 6, 2, 7, 3, 8, 4, 9, 5, 10, 10, 6, 6, 11, 7, 12, 8, 13, 9, 14, 10, 15
}

Here’s what the new sequence of triangles looks like:

  • Triangle 8 = 5, 9, 10
  • Triangle 9 (degenerate) = 5, 10, 10
  • Triangle 10 (degenerate) = 10, 10, 6
  • Triangle 11 (degenerate) = 10, 6, 6
  • Triangle 12 (degenerate) = 6, 6, 11
  • Triangle 13 = 6, 11, 7

By repeating the last vertex and the first vertex, we created four degenerate triangles that will be skipped, and linked the first row of the height map with the second. We could link an arbitrary number of rows this way and draw the entire heightmap with one call to glDrawElements(). Let’s take a look at this visually:

Index buffer object for height map with degenerate triangles.

The degenerate triangles link each row with the next row.

Degenerate triangles done the wrong way

We need to repeat both the last vertex of the first row and the first of the second row. What would happen if we didn’t? Let’s say we only repeated one vertex:

indexBuffer = {
    1, 6, 2, 7, 3, 8, 4, 9, 5, 10, 10, 6, 11, 7, 12, 8, 13, 9, 14, 10, 15
}

Here’s what the sequence of triangles would look like:

  • Triangle 8 = 5, 9, 10
  • Triangle 9 (degenerate) = 5, 10, 10
  • Triangle 10 (degenerate) = 10, 10, 6
  • Triangle 11 = 10, 6, 11
  • Triangle 12 = 11, 6, 7

Triangle 11 starts at the right and cuts all the way across to the left, which isn’t what we wanted to happen. The winding is now also incorrect for the next row of triangles, since 3 new triangles were inserted, swapping even and odd.

A practical example

Let’s walk through the code to make it happen. I highly recommend heading over and reading Android Lesson Seven: An Introduction to Vertex Buffer Objects (VBOs) before continuing.

Do you remember those graph calculators, that could draw parabolas and other stuff on the screen? In this example, we’re going to draw a 3d parabola using a height map. We’ll walk through all of the code to build and draw the height map. First, let’s get started with the definitions:

class HeightMap {
	static final int SIZE_PER_SIDE = 32;
	static final float MIN_POSITION = -5f;
	static final float POSITION_RANGE = 10f;

	final int[] vbo = new int[1];
	final int[] ibo = new int[1];

	int indexCount;
  • We’ll set the height map to 32 units per side, for a total of 1,024 vertices and 1,922 triangles, not including degenerate triangles (The total number of triangles in a height map is equal to (2 * (units_per_side – 1)2)). The height map positions will range from -5 to +5.
  • The OpenGL reference to our vertex buffer object and index buffer object will go in vbo and ibo, respectively.
  • indexCount will hold the total number of generated indices.
Building the vertex data

Let’s look at the code that will build the vertex buffer object. Remember that we still need a place to hold all of our vertices, and that each vertex will be defined once, and only once.

HeightMap() {
	try {
		final int floatsPerVertex = POSITION_DATA_SIZE_IN_ELEMENTS + NORMAL_DATA_SIZE_IN_ELEMENTS
				+ COLOR_DATA_SIZE_IN_ELEMENTS;
		final int xLength = SIZE_PER_SIDE;
		final int yLength = SIZE_PER_SIDE;

		final float[] heightMapVertexData = new float[xLength * yLength * floatsPerVertex];

		int offset = 0;

		// First, build the data for the vertex buffer
		for (int y = 0; y < yLength; y++) {
			for (int x = 0; x < xLength; x++) {
				final float xRatio = x / (float) (xLength - 1);

				// Build our heightmap from the top down, so that our triangles are 
				// counter-clockwise.
				final float yRatio = 1f - (y / (float) (yLength - 1));

				final float xPosition = MIN_POSITION + (xRatio * POSITION_RANGE);
				final float yPosition = MIN_POSITION + (yRatio * POSITION_RANGE);

				...
			}
		}

This bit of code sets up the loop to generate the vertices. Before we can send data to OpenGL, we need to build it in Java’s memory, so we create a floating point array to hold the height map vertices. Each vertex will hold enough floats to contain all of the position, normal, and color information.

Inside the loop, we calculate a ratio between 0 and 1 for x and y. This xRatio and yRatio will then be used to calculate the current position for the next vertex.

Let’s take a look at the actual calculations within the loop:

// Position
heightMapVertexData[offset++] = xPosition;
heightMapVertexData[offset++] = yPosition;
heightMapVertexData[offset++] = ((xPosition * xPosition) + (yPosition * yPosition)) / 10f;

First up is the position. Since this is a 3D parabola, we’ll calculate the Z as X2 + Y2. We divide the result by 10 so the resulting parabola isn’t so steep.

// Cheap normal using a derivative of the function.
// The slope for X will be 2X, for Y will be 2Y.
// Divide by 10 since the position's Z is also divided by 10.
final float xSlope = (2 * xPosition) / 10f;
final float ySlope = (2 * yPosition) / 10f;

// Calculate the normal using the cross product of the slopes.
final float[] planeVectorX = {1f, 0f, xSlope};
final float[] planeVectorY = {0f, 1f, ySlope};
final float[] normalVector = {
		(planeVectorX[1] * planeVectorY[2]) - (planeVectorX[2] * planeVectorY[1]),
		(planeVectorX[2] * planeVectorY[0]) - (planeVectorX[0] * planeVectorY[2]),
		(planeVectorX[0] * planeVectorY[1]) - (planeVectorX[1] * planeVectorY[0])};

// Normalize the normal
final float length = Matrix.length(normalVector[0], normalVector[1], normalVector[2]);

heightMapVertexData[offset++] = normalVector[0] / length;
heightMapVertexData[offset++] = normalVector[1] / length;
heightMapVertexData[offset++] = normalVector[2] / length;

Next up is the normal calculation. As you’ll remember from our lesson on lighting, the normal will be used to calculate lighting. The normal of a surface is defined as a vector perpendicular to the tangent plane at that particular point. In other words, the normal should be an arrow pointing straight away from the surface. Here’s an visual example for a parabola:

Parabola with normals

The first thing we need is the tangent of the surface. Using a bit of calculus, (don’t worry, I didn’t remember it either and went and searched for an online calculator ;)) we know that we can get the tangent, or the slope from the derivative of the function. Since our function is X2 + Y2, our slopes will therefore be 2X and 2Y. We scale the slopes down by 10, since for the position we had also scaled the result of the function down by 10.

To calculate the normal, we create two vectors for each slope to define a plane, and we calculate the cross product to get the normal, which is the perpendicular vector.

We then normalize the normal by calculating its length, and dividing each component by the length. This ensures that the overall length will be equal to 1.

// Add some fancy colors.
heightMapVertexData[offset++] = xRatio;
heightMapVertexData[offset++] = yRatio;
heightMapVertexData[offset++] = 0.5f;
heightMapVertexData[offset++] = 1f;

Finally, we’ll set some fancy colors. Red will scale from 0 to 1 across the X axis, and green will scale from 0 to 1 across the Y axis. We add a bit of blue to brighten things up, and assign 1 to alpha.

Building the index data

The next step is to link all of these vertices together, using the index buffer.

// Now build the index data
final int numStripsRequired = yLength - 1;
final int numDegensRequired = 2 * (numStripsRequired - 1);
final int verticesPerStrip = 2 * xLength;

final short[] heightMapIndexData = new short[(verticesPerStrip * numStripsRequired)
		+ numDegensRequired];

offset = 0;

for (int y = 0; y < yLength - 1; y++) { 	 	if (y > 0) {
		// Degenerate begin: repeat first vertex
		heightMapIndexData[offset++] = (short) (y * yLength);
	}

	for (int x = 0; x < xLength; x++) {
		// One part of the strip
		heightMapIndexData[offset++] = (short) ((y * yLength) + x);
		heightMapIndexData[offset++] = (short) (((y + 1) * yLength) + x);
	}

	if (y < yLength - 2) {
		// Degenerate end: repeat last vertex
		heightMapIndexData[offset++] = (short) (((y + 1) * yLength) + (xLength - 1));
	}
}

indexCount = heightMapIndexData.length;

In OpenGL ES 2, an index buffer needs to be an array of unsigned bytes or shorts, so we use shorts here. We’ll read from two rows of vertices at a time, and build a triangle strip using those vertices. If we’re on the second or subsequent rows, we’ll duplicate the first vertex of that row, and if we’re on any row but the last, we’ll also duplicate the last vertex of that row. This is so we can link the rows together using degenerate triangles as described earlier.

We’re just assigning offsets here. Imagine we had the height map as shown earlier:

Vertex buffer object for use with an index buffer object.

With the index buffer, we want to end up with something like this:

Index buffer object for height map with degenerate triangles.

Our buffer will contain data like this:

heightMapIndexData = {1, 6, 2, 7, 3, 8, 4, 9, 5, 10, 10, 6, 6, 11, 7, 12, 8, 13, 9, 14, 10, 15}

Just keep in mind that although our examples start with 1, and go on to 2, 3, etc… in the actual code, our arrays should be 0-based and start with 0.

Uploading data into vertex and index buffer objects

The next step is to copy the data from Dalvik’s heap to a direct buffer on the native heap:

final FloatBuffer heightMapVertexDataBuffer = ByteBuffer
	.allocateDirect(heightMapVertexData.length * BYTES_PER_FLOAT).order(ByteOrder.nativeOrder())
	.asFloatBuffer();
heightMapVertexDataBuffer.put(heightMapVertexData).position(0);

final ShortBuffer heightMapIndexDataBuffer = ByteBuffer
	.allocateDirect(heightMapIndexData.length * BYTES_PER_SHORT).order(ByteOrder.nativeOrder())
	.asShortBuffer();
heightMapIndexDataBuffer.put(heightMapIndexData).position(0);

Remember, the index data needs to go in a short buffer or a byte buffer. Now we can create OpenGL buffers, and upload our data into the buffers:

GLES20.glGenBuffers(1, vbo, 0);
GLES20.glGenBuffers(1, ibo, 0);

if (vbo[0] > 0 && ibo[0] > 0) {
	GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, vbo[0]);
	GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, heightMapVertexDataBuffer.capacity()
							* BYTES_PER_FLOAT, heightMapVertexDataBuffer, GLES20.GL_STATIC_DRAW);

	GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, ibo[0]);
	GLES20.glBufferData(GLES20.GL_ELEMENT_ARRAY_BUFFER, heightMapIndexDataBuffer.capacity()
							* BYTES_PER_SHORT, heightMapIndexDataBuffer, GLES20.GL_STATIC_DRAW);

	GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
	GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, 0);
} else {
	errorHandler.handleError(ErrorType.BUFFER_CREATION_ERROR, "glGenBuffers");
}

We use GL_ARRAY_BUFFER to specify our vertex data, and GL_ELEMENT_ARRAY_BUFFER to specify our index data.

Drawing the height map

Much of the code to draw the height map will be similar as in previous lessons. I won’t cover the matrix setup code here; instead, we’ll just look at the calls to bind the array data and draw using the index buffer:

GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, vbo[0]);

// Bind Attributes
glEs20.glVertexAttribPointer(positionAttribute, POSITION_DATA_SIZE_IN_ELEMENTS, GLES20.GL_FLOAT,
		false, STRIDE, 0);
GLES20.glEnableVertexAttribArray(positionAttribute);

glEs20.glVertexAttribPointer(normalAttribute, NORMAL_DATA_SIZE_IN_ELEMENTS, GLES20.GL_FLOAT,
		false, STRIDE, POSITION_DATA_SIZE_IN_ELEMENTS * BYTES_PER_FLOAT);
GLES20.glEnableVertexAttribArray(normalAttribute);

glEs20.glVertexAttribPointer(colorAttribute, COLOR_DATA_SIZE_IN_ELEMENTS, GLES20.GL_FLOAT,
		false, STRIDE, (POSITION_DATA_SIZE_IN_ELEMENTS + NORMAL_DATA_SIZE_IN_ELEMENTS)
		* BYTES_PER_FLOAT);
GLES20.glEnableVertexAttribArray(colorAttribute);

// Draw
GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, ibo[0]);
glEs20.glDrawElements(GLES20.GL_TRIANGLE_STRIP, indexCount, GLES20.GL_UNSIGNED_SHORT, 0);

GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, 0);

In the previous lesson, we had to introduce a custom OpenGL binding to properly use VBOs, as they’re broken in Froyo. We’ll also need to use this binding for IBOs as well. Just like in the previous lesson, we bind our position, normal, and color data in our vertex buffer to the matching attribute in the shader, taking care to pass in the proper stride and start offset of each attribute. Both the stride and the start offset are defined in terms of bytes.

The main difference is when it’s time to draw. We call glDrawElements() instead of glDrawArrays(), and we pass in the index count and data type. OpenGL ES 2.0 only accepts GL_UNSIGNED_SHORT and GL_UNSIGNED_BYTE, so we have to make sure that we define our index data using shorts or bytes.

Rendering and lighting two-sided triangles

For this lesson, we don’t enable GL_CULL_FACE so that both the front and the back sides of triangles are visible. We need to make a slight change to our fragment shader code so that lighting works properly for our two-sided triangles:

float diffuse;

if (gl_FrontFacing) {
    diffuse = max(dot(v_Normal, lightVector), 0.0);
} else {
    diffuse = max(dot(-v_Normal, lightVector), 0.0);
}

We use the special variable gl_FrontFacing to find out if the current fragment is part of a front-facing or back-facing triangle. If it’s front-facing, then no need to do anything special: the code is the same as before. If it’s back-facing, then we simply invert the surface normal (since the back side of a triangle faces the opposite direction) and proceed with the calculations as before.

Further exercises

When would it be a downside to use index buffers? Remember, when we use index buffers the GPU has to do an additional fetch into the vertex buffer object, so we’re introducing an additional step.

Wrapping up

The full source code for this lesson can be downloaded from the project site on GitHub. A compiled version of the lesson can also be downloaded directly from the Android Market:

QR code for link to the app on the Android Market.

Thanks for stopping by, and please feel free to check out the code and share your comments below.

Enhanced by Zemanta
Share

Android Emulator Now Supports Native OpenGL ES2.0!

For those of you who may have missed it, the Android team recently released an upgraded emulator image that has support for native OpenGL ES 2.0! That’s right; not only is it now supported, but the calls are translated to desktop OpenGL so they can be accelerated by your native graphics layer. It’s still better to test on an actual device, but this is a great boon for those who want to try out their app on different form factors without having to go out and buy a bunch of tablets and devices.

Check out the video:

Three notes about the GPU emulation

  1. First, you need to edit your emulator image, go down to the hardware section, and add “GPU Emulation” and set it to true.
  2. Second, there’s a bug with the emulator such that this line: “final boolean supportsEs2 = configurationInfo.reqGlEsVersion >= 0×20000;” does not work. It will always return false. You can add “|| Build.FINGERPRINT.startsWith(“generic”)” or simply comment out these checks and assume that OpenGL ES 2 is supported, when running on the emulator.
  3. Finally, if it crashes with “no config found”, try adding this line before the call to “setRenderer(…)”: “glSurfaceView.setEGLConfigChooser(8 , 8, 8, 8, 16, 0);”

A quick roundup

Hope that everyone is enjoying the gradually improving weather out there.

Share

OpenGL ES Roundup, March 9, 2012

Thank yous and mentions

A big thank you goes to the following guys who have recently kindly featured Learn OpenGL ES:

Thanks again; you all have my deepest gratitude!

State of WebGL on Android

The following browsers now support WebGL on Android:

Using Vertex Buffer Objects on Froyo (Android 2.2, SDK 8)

Bindings can be downloaded from here:

I cover how to use these bindings in Android Lesson Seven: An Introduction to Vertex Buffer Objects (VBOs). Note: These bindings are NOT needed if you don’t mind excluding a quarter of the market (as of the time of this writing) and target Gingerbread and higher.

The new iPad

In non-Android related news, Apple just released the third generation iPad, known as the “new iPad”, with a 2048×1536 “retina display”. Here are the specs:

Apple A5 SoC (system on a chip):

Judging by initial benchmarks, this thing just screams. iOS had never had the “Android lag” problem seen so often on Android phones and tablets, and this new tablet will only continue that trend. To be fair, throw enough hardware at the problem and maybe even Android will become lag-free.

What are your thoughts on the new iPad? Interested in picking it up? I love competition, because the more Apple, Android, and Microsoft duke it out, the better it is for us consumers! So long as nobody ends up dominating the market.

Have a good weekend, all!

Enhanced by Zemanta
Share
A screenshot of Lesson Seven, showing the grid of cubes.

Android Lesson Seven: An Introduction to Vertex Buffer Objects (VBOs)

A screenshot of Lesson Seven, showing the grid of cubes.In this lesson, we’ll introduce vertex buffer objects (VBOs), how to define them, and how to use them. Here is what we are going to cover:

  • How to define and render from vertex buffer objects.
  • The difference between using a single buffer with all the data packed in, or multiple buffers.
  • Problems and pitfalls, and what to do about them.

What are vertex buffer objects, and why use them?

Up until now, all of our lessons have been storing our object data in client-side memory, only transferring it into the GPU at render time. This is fine when there is not a lot of data to transfer, but as our scenes get more complex with more objects and triangles, this can impose an extra cost on the CPU and memory usage. What can we do about this? We can use vertex buffer objects. Instead of transferring vertex information from client memory every frame, the information will be transferred once and rendering will then be done from this graphics memory cache.

Assumptions and prerequisites

Please read Android Lesson One: Getting Started for an intro on how to upload the vertices from client-side memory. This understanding of how OpenGL ES works with the vertex arrays will be crucial to understanding this lesson.

Understanding client-side buffers in more detail

Once you understand how to render using client-side memory, it’s actually not too hard to switch to using VBOs. The main difference is that there is an additional step to upload the data into graphics memory, and an additional call to bind to this buffer when rendering.

This lesson has been setup to use four different modes:

  • Client side, separate buffers.
  • Client side, packed buffer.
  • Vertex buffer object, separate buffers.
  • Vertex buffer object, packed buffers.

Whether we are using vertex buffer objects or not, we need to first store our data in a client-side direct buffer. Recall from lesson one that OpenGL ES is a native system library, whereas Dalvik Java runs in a virtual machine. To bridge the gap, we need to use a set of special buffer classes to allocate memory on the native heap and make it accessible to OpenGL:

// Java array.
float[] cubePositions;
...
// Floating-point buffer
final FloatBuffer cubePositionsBuffer;
...

// Allocate a direct block of memory on the native heap,
// size in bytes is equal to cubePositions.length * BYTES_PER_FLOAT.
// BYTES_PER_FLOAT is equal to 4, since a float is 32-bits, or 4 bytes.
cubePositionsBuffer = ByteBuffer.allocateDirect(cubePositions.length * BYTES_PER_FLOAT)

// Floats can be in big-endian or little-endian order.
// We want the same as the native platform.
.order(ByteOrder.nativeOrder())

// Give us a floating-point view on this byte buffer.
.asFloatBuffer();

Transferring data from the Java heap to the native heap is then a matter of a couple calls:

// Copy data from the Java heap to the native heap.
cubePositionsBuffer.put(cubePositions)

// Reset the buffer position to the beginning of the buffer.
.position(0);

What is the purpose of the buffer position? Normally, Java does not give us a way to specify arbitrary locations in memory using pointer arithmetic. However, setting the position of the buffer is functionally equivalent to changing the value of a pointer to a block of memory. By changing the position, we can pass arbitrary memory locations within our buffer to OpenGL calls. This will come in handy when we work with packed buffers.

Once the data is on the native heap, we no longer need to keep the float[] array around, and we can let the garbage collector clean it up.

Rendering with client-side buffers is straightforward to setup. We just need to enable using vertex arrays on that attribute, and pass a pointer to our data:

// Pass in the position information
GLES20.glEnableVertexAttribArray(mPositionHandle);
GLES20.glVertexAttribPointer(mPositionHandle, POSITION_DATA_SIZE,
	GLES20.GL_FLOAT, false, 0, mCubePositions);

Explanation of the parameters to glVertexAttribPointer:

  • mPositionHandle: The OpenGL index of the position attribute of our shader program.
  • POSITION_DATA_SIZE: How many elements (floats) define this attribute.
  • GL_FLOAT: The type of each element.
  • false: Should fixed-point data be normalized? Not applicable since we are using floating-point data.
  • 0: The stride. Set to 0 to mean that the positions should be read sequentially.
  • mCubePositions: The pointer to our buffer, containing all of the positional data.
Working with packed buffers

Working with packed buffers is very similar, except that instead of using a buffer each for positions, normals, etc… one buffer will contain all of this data. The difference looks like this:

Using separate buffers

positions = X,Y,Z, X, Y, Z, X, Y, Z, …
colors = R, G, B, A, R, G, B, A, …
textureCoordinates = S, T, S, T, S, T, …

Using a packed buffer

buffer = X, Y, Z, R, G, B, A, S, T, …

The advantage to using packed buffers is that it should be more efficient for the GPU to render, since all of the information needed to render a triangle is located within the same block of memory. The disadvantage is that it may be more difficult and slower to update, if you are using dynamic data.

When we use packed buffers, we need to change our rendering calls in a couple of ways. First, we need to tell OpenGL the stride, or how many bytes define a vertex.

final int stride = (POSITION_DATA_SIZE + NORMAL_DATA_SIZE + TEXTURE_COORDINATE_DATA_SIZE)
	* BYTES_PER_FLOAT;

// Pass in the position information
mCubeBuffer.position(0);
GLES20.glEnableVertexAttribArray(mPositionHandle);
GLES20.glVertexAttribPointer(mPositionHandle, POSITION_DATA_SIZE,
	GLES20.GL_FLOAT, false, stride, mCubeBuffer);

// Pass in the normal information
mCubeBuffer.position(POSITION_DATA_SIZE);
GLES20.glEnableVertexAttribArray(mNormalHandle);
GLES20.glVertexAttribPointer(mNormalHandle, NORMAL_DATA_SIZE,
	GLES20.GL_FLOAT, false, stride, mCubeBuffer);
...

The stride tells OpenGL ES how far it needs to go to find the same attribute for the next vertex. For example, if element 0 is the beginning of the position for the first vertex, and there are 8 elements per vertex, then the stride will be equal to 8 elements, or 32 bytes. The position for the next vertex will be found at element 8, and the next vertex after that at element 16, and so on.

Keep in mind that the value of the stride passed to glVertexAttribPointer should be in bytes, not elements, so remember to do that conversion.

Notice that we also change the start position of the buffer when we switch from specifying the positions to the normals. This is the pointer arithmetic I was referring to before, and this is how we can do it in Java when working with OpenGL ES. We’re still working with the same buffer, mCubeBuffer, but we tell OpenGL to start reading in the normals at the first element after the position. Again, we pass in the stride to tell OpenGL that the next normal will be found 8 elements or 32 bytes later.

Dalvik and memory on the native heap

If you allocate a lot of memory on the native heap and release it, you will probably run into the beloved OutOfMemoryError, sooner or later. There are a couple of reasons behind that:

  1. You might think that you’ve released the memory by letting the reference go out of scope, but native memory seems to take a few extra GC cycles to be completely cleaned up, and Dalvik will throw an exception if there is not enough free memory available and the native memory has not yet been released.
  2. The native heap can become fragmented. Calls to allocateDirect() will inexplicably fail, even though there appears to be plenty of memory available. Sometimes it helps to make a smaller allocation, free it, and then try the larger allocation again.

What can you do about these problems? Not much, other than hoping that Google improves the behaviour of Dalvik in future editions (they’ve added a largeHeap parameter to 3.0+), or manage the heap yourself by doing your allocations in native code or allocating a huge block upfront, and spinning off buffers based off of that.

Moving to vertex buffer objects

Now that we’ve reviewed working with client-side buffers, let’s move on to vertex buffer objects! First, we need to review a few very important points:

1. Buffers must be created within a valid OpenGL context.

This might seem like an obvious point, but it’s just a reminder that you have to wait until onSurfaceCreated(), and you have to take care that the OpenGL ES calls are done on the GL thread. See this document: OpenGL ES Programming Guide for iOS. It might be written for iOS, but the behaviour of OpenGL ES is similar on Android.

2. Improper use of vertex buffer objects will crash the graphics driver.

You need to be careful with the data you pass around when you use vertex buffer objects. Improper values will cause a native crash in the OpenGL ES system library or in the graphics driver library. On my Nexus S, some games freeze up my phone completely or cause it to reboot, because the graphics driver is crashing on their commands. Not all crashes will lock up your device, but at a minimum you will not see the “This application has stopped working” dialogue. Your activity will restart without warning, and the only info you’ll get might be a native debug trace in the logs.

3. The OpenGL ES bindings are broken on Froyo (2.2), and incomplete/unavailable in earlier versions.

This is the most unfortunate and most important point to consider. For some reason, Google really dropped the ball when it comes to OpenGL ES 2 support on Froyo. The mappings are incomplete, and several crucial functions needed to use vertex buffer objects are unavailable and cannot be used from Java code, at least with the standard SDK.

I don’t know if it’s because they didn’t run their unit tests, or if the developer was sloppy with their code generation tools, or if everyone was on 8 cups of coffee and burning the midnight oil to get things out the door. I don’t know why the API is broken, but the fact is that it’s broken.

There are three solutions to this problem:

  1. Target Gingerbread (2.3) and higher.
  2. Don’t use vertex buffer objects.
  3. Use your own Java Native Interface (JNI) library to interface with the native OpenGL ES system libraries.

I find option 1 to be unacceptable, since a full quarter of devices out there still run on Froyo as of the time of this writing. Option 2 works, but is kind of silly.

The option I recommend, and that I have decided to go with, is to use your own JNI bindings. For this lesson I have decided to go with the bindings generously provided by the guys who created libgdx, a cross-platform game development library licensed under the Apache License 2.0. You need to use the following files to make it work:

  • /libs/armeabi/libandroidgl20.so
  • /libs/armeabi-v7a/libandroidgl20.so
  • src/com/badlogic/gdx/backends/android/AndroidGL20.java
  • src/com/badlogic/gdx/graphics/GL20.java
  • src/com/badlogic/gdx/graphics/GLCommon.java

You might notice that this excludes Android platforms that do not run on ARM, and you’d be right. It would probably be possible to compile your own bindings for those platforms if you want to have VBO support on Froyo, though that is out of the scope of this lesson.

Using the bindings is as simple as these lines of code:

AndroidGL20 mGlEs20 = new AndroidGL20();
...
mGlEs20.glVertexAttribPointer(mPositionHandle, POSITION_DATA_SIZE, GLES20.GL_FLOAT, false, 0, 0);
...

You only need to call the custom binding where the SDK-provided binding is incomplete. I use the custom bindings to fill in the holes where the official one is missing functions.

Uploading vertex data to the GPU.

To upload data to the GPU, we need to follow the same steps in creating a client-side buffer as before:

...
cubePositionsBuffer = ByteBuffer.allocateDirect(cubePositions.length * BYTES_PER_FLOAT)
.order(ByteOrder.nativeOrder()).asFloatBuffer();
cubePositionsBuffer.put(cubePositions).position(0);
...

Once we have the client-side buffer, we can create a vertex buffer object and upload data from client memory to the GPU with the following commands:

// First, generate as many buffers as we need.
// This will give us the OpenGL handles for these buffers.
final int buffers[] = new int[3];
GLES20.glGenBuffers(3, buffers, 0);

// Bind to the buffer. Future commands will affect this buffer specifically.
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, buffers[0]);

// Transfer data from client memory to the buffer.
// We can release the client memory after this call.
GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, cubePositionsBuffer.capacity() * BYTES_PER_FLOAT,
	cubePositionsBuffer, GLES20.GL_STATIC_DRAW);

// IMPORTANT: Unbind from the buffer when we're done with it.
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);

Once data has been uploaded to OpenGL ES, we can release the client-side memory as we no longer need to keep it around. Here is an explanation of glBufferData:

  • GL_ARRAY_BUFFER: This buffer contains an array of vertex data.
  • cubePositionsBuffer.capacity() * BYTES_PER_FLOAT: The number of bytes this buffer should contain.
  • cubePositionsBuffer: The source that will be copied to this vertex buffer object.
  • GL_STATIC_DRAW: The buffer will not be updated dynamically.

Our call to glVertexAttribPointer looks a little bit different, as the last parameter is now an offset rather than a pointer to our client-side memory:

// Pass in the position information
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mCubePositionsBufferIdx);
GLES20.glEnableVertexAttribArray(mPositionHandle);
mGlEs20.glVertexAttribPointer(mPositionHandle, POSITION_DATA_SIZE, GLES20.GL_FLOAT, false, 0, 0);
...

Like before, we bind to the buffer, then enable the vertex array. Since the buffer is already bound, we only need to tell OpenGL the offset to start at when reading from the buffer. Since we are using separate buffers, we pass in an offset of 0. Notice also that we are using our custom binding to call glVertexAttribPointer, since the official SDK is missing this specific function call.

Once we are done drawing with our buffer, we should unbind from it:

GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);

When we no longer want to keep our buffers around, we can free the memory:

final int[] buffersToDelete = new int[] { mCubePositionsBufferIdx, mCubeNormalsBufferIdx,
	mCubeTexCoordsBufferIdx };
GLES20.glDeleteBuffers(buffersToDelete.length, buffersToDelete, 0);
Packed vertex buffer objects

We can also use a single, packed vertex buffer object to hold all of our vertex data. The creation of a packed buffer is the same as above, with the only difference being that we start from a packed client-side buffer. Rendering from the packed buffer is also the same, except we need to pass in a stride and an offset, like when using packed buffers in client-side memory:

final int stride = (POSITION_DATA_SIZE + NORMAL_DATA_SIZE + TEXTURE_COORDINATE_DATA_SIZE)
	* BYTES_PER_FLOAT;

// Pass in the position information
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mCubeBufferIdx);
GLES20.glEnableVertexAttribArray(mPositionHandle);
mGlEs20.glVertexAttribPointer(mPositionHandle, POSITION_DATA_SIZE, 
	GLES20.GL_FLOAT, false, stride, 0);

// Pass in the normal information
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mCubeBufferIdx);
GLES20.glEnableVertexAttribArray(mNormalHandle);
mGlEs20.glVertexAttribPointer(mNormalHandle, NORMAL_DATA_SIZE, 
	GLES20.GL_FLOAT, false, stride, POSITION_DATA_SIZE * BYTES_PER_FLOAT);
...

Notice that the offset needs to be specified in bytes. The same considerations of unbinding and deleting the buffer apply, as before.

Putting it all together

This lesson is setup so that it builds a cube of cubes, with the same number of cubes in each dimension. It will build a cube of cubes between 1x1x1 cubes, and 16x16x16 cubes. Since each cube shares the same normal and texture data, this data will be copied repeatedly when we initialize our client-side buffer. All of the cubes will end up inside the same buffer objects.

You can view the code for the lesson and view an example of rendering with and without VBOs, and with and without packed buffers. Check the code to see how some of the following was handled:

  • Posting events from the OpenGL thread back to the main UI thread, via runOnUiThread.
  • Generating the vertex data asynchronously.
  • Handling out of memory errors.
  • We removed the call to glEnable(GL_TEXTURE_2D), since that is actually an invalid enum on OpenGL ES 2. This is a hold over from the fixed pipeline days; In OpenGL ES 2 this stuff is handled by shaders, so no need to use a glEnable/glDisable.
  • How to render using different paths, without adding too many if statements and conditions.
Further exercises

When would you use vertex buffers and when is it better to stream data from client memory? What are some of the drawbacks of using vertex buffer objects? How would you improve the asynchronous loading code?

Wrapping up

The full source code for this lesson can be downloaded from the project site on GitHub. A compiled version of the lesson can also be downloaded directly from the Android Market:

QR code for link to the app on the Android Market.

Thanks for stopping by, and please feel free to check out the code and share your comments below. A special thanks goes out again to the guys at libgdx for generously providing the source code and libraries for their OpenGL ES 2 bindings for Android 2.2!

Enhanced by Zemanta
Share

Rotating An Object With Touch Events

Rotating an object in 3D is a neat way of letting your users interact with the scene, but the math can be tricky to get right. In this article, I’ll take a look at a simple way to rotate an object based on the touch events, and how to work around the main drawback of this method.

Simple rotations.

This is the easiest way to rotate an object based on touch movement. Here is example pseudocode:

Matrix.setIdentity(modelMatrix);

... do other translations here ...

Matrix.rotateX(totalYMovement);
Matrix.rotateY(totalXMovement);

This is done every frame.

To rotate an object up or down, we rotate it around the X-axis, and to rotate an object left or right, we rotate it around the Y axis. We could also rotate an object around the Z axis if we wanted it to spin around.

How to make the rotation appear relative to the user’s point of view.

The main problem with the simple way of rotating is that the object is being rotated in relation to itself, instead of in relation to the user’s point of view. If you rotate left and right from a point of zero rotation, the cube will rotate as you expect, but what if you then rotate it up or down 180 degrees? Trying to rotate the cube left or right will now rotate it in the opposite direction!

One easy way to work around this problem is to keep a second matrix around that will store all of the accumulated rotations.

Here’s what we need to do:

  1. Every frame, calculate the delta between the last position of the pointer, and the current position of the pointer. This delta will be used to rotate our accumulated rotation matrix.
  2. Use this matrix to rotate the cube.

What this means is that drags left, right, up, and down will always move the cube in the direction that we expect.

Android Code

The code examples here are written for Android, but can easily be adapted to any platform running OpenGL ES. The code is based on Android Lesson Six: An Introduction to Texture Filtering.

In LessonSixGLSurfaceView.java, we declare a few member variables:

private float mPreviousX;
private float mPreviousY;

private float mDensity;

We will store the previous pointer position each frame, so that we can calculate the relative movement left, right, up, or down. We also store the screen density so that drags across the screen can move the object a consistent amount across devices, regardless of the pixel density.

Here’s how to get the pixel density:

final DisplayMetrics displayMetrics = new DisplayMetrics();
getWindowManager().getDefaultDisplay().getMetrics(displayMetrics);
density = displayMetrics.density

Then we add our touch event handler to our custom GLSurfaceView:

public boolean onTouchEvent(MotionEvent event)
{
	if (event != null)
	{
		float x = event.getX();
		float y = event.getY();

		if (event.getAction() == MotionEvent.ACTION_MOVE)
		{
			if (mRenderer != null)
			{
				float deltaX = (x - mPreviousX) / mDensity / 2f;
				float deltaY = (y - mPreviousY) / mDensity / 2f;

				mRenderer.mDeltaX += deltaX;
				mRenderer.mDeltaY += deltaY;
			}
		}

		mPreviousX = x;
		mPreviousY = y;

		return true;
	}
	else
	{
		return super.onTouchEvent(event);
	}
}

Every frame, we compare the current pointer position with the previous, and use that to calculate the delta offset. We then divide that delta offset by the pixel density and a slowing factor of 2.0f to get our final delta values. We apply those directly to the renderer to a couple of public variables that we have also declared as volatile, so that they can be updated between threads.

Remember, on Android, the OpenGL renderer runs in a different thread than the UI event handler thread, and there is a slim chance that the other thread fires in-between the X and Y variable assignments (there are also additional points of contention with the += syntax). I have left the code like this to bring up this point; as an exercise for the reader I leave it to you to add synchronized statements around the public variable read and write pairs instead of using volatile variables.

First, let’s add a couple of matrices and initialize them:

/** Store the accumulated rotation. */
private final float[] mAccumulatedRotation = new float[16];

/** Store the current rotation. */
private final float[] mCurrentRotation = new float[16];
@Override
public void onSurfaceCreated(GL10 glUnused, EGLConfig config)
{

    ...

    // Initialize the accumulated rotation matrix
     Matrix.setIdentityM(mAccumulatedRotation, 0);
}

Here’s what our matrix code looks like in the onDrawFrame method:

// Draw a cube.
// Translate the cube into the screen.
Matrix.setIdentityM(mModelMatrix, 0);
Matrix.translateM(mModelMatrix, 0, 0.0f, 0.8f, -3.5f);

// Set a matrix that contains the current rotation.
Matrix.setIdentityM(mCurrentRotation, 0);
Matrix.rotateM(mCurrentRotation, 0, mDeltaX, 0.0f, 1.0f, 0.0f);
Matrix.rotateM(mCurrentRotation, 0, mDeltaY, 1.0f, 0.0f, 0.0f);
mDeltaX = 0.0f;
mDeltaY = 0.0f;

// Multiply the current rotation by the accumulated rotation, and then set the accumulated
// rotation to the result.
Matrix.multiplyMM(mTemporaryMatrix, 0, mCurrentRotation, 0, mAccumulatedRotation, 0);
System.arraycopy(mTemporaryMatrix, 0, mAccumulatedRotation, 0, 16);

// Rotate the cube taking the overall rotation into account.
Matrix.multiplyMM(mTemporaryMatrix, 0, mModelMatrix, 0, mAccumulatedRotation, 0);
System.arraycopy(mTemporaryMatrix, 0, mModelMatrix, 0, 16);
  1. First we translate the cube.
  2. Then we build a matrix that will contain the current amount of rotation, between this frame and the preceding frame.
  3. We then multiply this matrix with the accumulated rotation, and assign the accumulated rotation to the result. The accumulated rotation contains the result of all of our rotations since the beginning.
  4. Now that we’ve updated the accumulated rotation matrix with the most recent rotation, we finally rotate the cube by multiplying the model matrix with our rotation matrix, and then we set the model matrix to the result.

The above code might look a bit confusion due to the placement of the variables, so remember the definitions:

public static void multiplyMM (float[] result, int resultOffset, float[] lhs, int lhsOffset, float[] rhs, int rhsOffset)

public static void arraycopy (Object src, int srcPos, Object dst, int dstPos, int length)

Note the position of source and destination for each method call.

Trouble spots and pitfalls
  • The accumulated matrix should be set to identity once when initialized, and should not be reset to identity each frame.
  • Previous pointer positions must also be set on pointer down events, not only on pointer move events.
  • Watch the order of parameters, and also watch out for corrupting your matrices. Android’s Matrix.multiplyMM states that “the result element values are undefined if the result elements overlap either the lhs or rhs elements.” Use temporary matrices to avoid this problem.
WebGL examples

The example on the left uses the simplest method of rotating, while the example on the right uses the accumulated rotations matrix.

Your browser does not support the canvas tag. Your browser does not support the canvas tag.
Further exercises

What are the drawbacks of using a matrix to hold accumulated rotations and updating it every frame based on the movement delta for that frame? What other ways of rotation are there? Try experimenting, and see what else you can come up with!

  
Share
Basic blending (additive blending of RGB cubes).

Listening to Android Touch Events, and Acting on Them

Basic blending (additive blending of RGB cubes).

Basic blending.

We started listening to touch events in Android Lesson Five: An Introduction to Blending, and in that lesson, we listened to touch events and used them to change our OpenGL state.

To listen to touch events, you first need to subclass GLSurfaceView and create your own custom view. In that view, you create a default constructor that calls the superclass, create a new method to take in a specific renderer (LessonFiveRenderer in this case) instead of the generic interface, and override onTouchEvent(). We pass in a concrete renderer class, because we will be calling specific methods on that class in the onTouchEvent() method.

On Android, the OpenGL rendering is done in a separate thread, so we’ll also look at how we can safely dispatch these calls from the main UI thread that is listening to the touch events, over to the separate renderer thread.

public class LessonFiveGLSurfaceView extends GLSurfaceView
{
	private LessonFiveRenderer mRenderer;

	public LessonFiveGLSurfaceView(Context context)
	{
		super(context);
	}

	@Override
	public boolean onTouchEvent(MotionEvent event)
	{
		if (event != null)
		{
			if (event.getAction() == MotionEvent.ACTION_DOWN)
			{
				if (mRenderer != null)
				{
					// Ensure we call switchMode() on the OpenGL thread.
					// queueEvent() is a method of GLSurfaceView that will do this for us.
					queueEvent(new Runnable()
					{
						@Override
						public void run()
						{
							mRenderer.switchMode();
						}
					});

					return true;
				}
			}
		}

		return super.onTouchEvent(event);
	}

	// Hides superclass method.
	public void setRenderer(LessonFiveRenderer renderer)
	{
		mRenderer = renderer;
		super.setRenderer(renderer);
	}
}

And the implementation of switchMode() in LessonFiveRenderer:

public void switchMode()
{
	mBlending = !mBlending;

	if (mBlending)
	{
		// No culling of back faces
		GLES20.glDisable(GLES20.GL_CULL_FACE);

		// No depth testing
		GLES20.glDisable(GLES20.GL_DEPTH_TEST);

		// Enable blending
		GLES20.glEnable(GLES20.GL_BLEND);
		GLES20.glBlendFunc(GLES20.GL_ONE, GLES20.GL_ONE);
	}
	else
	{
		// Cull back faces
		GLES20.glEnable(GLES20.GL_CULL_FACE);

		// Enable depth testing
		GLES20.glEnable(GLES20.GL_DEPTH_TEST);

		// Disable blending
		GLES20.glDisable(GLES20.GL_BLEND);
	}
}

Let’s look a little bit more closely at LessonFiveGLSurfaceView::onTouchEvent(). Something important to remember is that touch events run on the UI thread, while GLSurfaceView creates the OpenGL ES context in a separate thread, which means that our renderer’s callbacks also run in a separate thread. This is an important point to remember, because we can’t call OpenGL from another thread and just expect things to work.

Thankfully, the guys that wrote GLSurfaceView also thought of this, and provided a queueEvent() method that you can use to call stuff on the OpenGL thread. So, when we want to turn blending on and off by tapping the screen, we make sure that we’re calling the OpenGL stuff on the right thread by using queueEvent() in the UI thread.

Further exercises

How would you listen to keyboard events, or other system events, and show an update in the OpenGL context?

Enhanced by Zemanta
Share
trilinear-filtering

Android Lesson Six: An Introduction to Texture Filtering

In this lesson, we will introduce the different types of basic texture filtering modes and how to use them, including nearest-neighbour filtering, bilinear filtering, and trilinear filtering using mipmaps.

You’ll learn how to make your textures appear more smooth, as well as the drawbacks that come from smoothing. There are also different ways of rotating an object, one of which is used in this lesson.

Assumptions and prerequisites

It’s highly recommended to understand the basics of texture mapping in OpenGL ES, covered in the lesson Android Lesson Four: Introducing Basic Texturing.

What is texture filtering?

Textures in OpenGL are made up of arrays of elements known as texels, which contain colour and alpha values. This corresponds with the display, which is made up of a bunch of pixels and displays a different colour at each point. In OpenGL, textures are applied to triangles and drawn on the screen, so these textures can be drawn in various sizes and orientation. The texture filtering options in OpenGL tell it how to filter the texels onto the pixels of the device, depending on the case.

There are three cases:

  • Each texel maps onto more than one pixel. This is known as magnification.
  • Each texel maps exactly onto one pixel. Filtering doesn’t apply in this case.
  • Each texel maps onto less than one pixel. This is known as minification.

OpenGL lets us assign a filter for both magnification and minification, and lets us use nearest-neighbour, bilinear filtering, or trilinear filtering. I will explain what these mean further below.

Magnification and minification

Here is a visualization of both magnification and minification with nearest-neighbour rendering, using the cute Android that shows up when you have your USB connected to your Android device:

Magnification

As you can see, the texels of the image are easily visible, as they now cover many of the pixels on your display.

Minification

With minification, many of the details are lost as many of the texels cannot be rendered onto the limited pixels available.

Texture filtering modes

Bilinear interpolation

The texels of a texture are clearly visible as large squares in the magnification example when no interpolation between the texel values are done. When rendering is done in nearest-neighbour mode, the pixel is assigned the value of the nearest texel.

The rendering quality can be dramatically improved by switching to bilinear interpolation. Instead of assigning the values of a group of pixels to the same nearby texel value, these values will instead be linearly interpolated between the neighbouring four texels. Each pixel will be smoothed out and the resulting image will look much smoother:

Some blockiness is still apparent, but the image looks much smoother than before. People who played 3D games back in the days before 3D accelerated cards came out will remember that this was the defining feature between a software-rendered game and a hardware-accelerated game: software-rendered games simply did not have the processing budget to do smoothing, so everything appeared blocky and jagged. Things suddenly got smooth once people starting using graphics accelerators.

English: bilinear interpolation, upscaling of ...

Image via Wikipedia

Bilinear interpolation is mostly useful for magnification. It can also be used for minification, but beyond a certain point and we run into the same problem that we are trying to cram far too many texels onto the same pixel. OpenGL will only use at most 4 texels to render a pixel, so a lot of information is still being lost.

If we look at a detailed texture with bilinear interpolation being applied, it will look very noisy when we see it moving in the distance, since a different set of texels will be selected each frame.

Mipmapping

How can we minify textures without introducing noise and use all of the texels? This can be done by generating a set of optimized textures at different sizes which we can then use at runtime. Since these textures are pre-generated, they can be filtered using more expensive techniques that use all of the texels, and at runtime OpenGL will select the most appropriate level based on the final size of the texture on the screen.

The resulting image can have more detail, less noise, and look better overall. Although a bit more memory will be used, rendering can also be faster, as the smaller levels can be more easily kept in the GPU’s texture cache. Let’s take a closer look at the resulting image at 1/8th of its original size, using bilinear filtering without and with mipmaps; the image has been expanded for clarity:

Bilinear filtering without mipmaps

Bilinear filtering with mipmaps

The version using mipmaps has vastly more detail. Because of the pre-processing of the image into separate levels, all of the texels end up getting used in the final image.

Trilinear filtering

When using mipmaps with bilinear filtering, sometimes a noticeable jump or line can be seen in the rendered scene where OpenGL switches between different mipmap levels of the texture. This will be pointed out a bit further below when comparing the different OpenGL texture filtering modes.

Trilinear filtering solves this problem by also interpolating between the different mipmap levels, so that a total of 8 texels will be used to interpolate the final pixel value, resulting in a smoother image.

OpenGL texture filtering modes

OpenGL has two parameters that can be set:

  • GL_TEXTURE_MIN_FILTER
  • GL_TEXTURE_MAG_FILTER

These correspond to the minification and magnification described earlier. GL_TEXTURE_MIN_FILTER accepts the following options:

  • GL_NEAREST
  • GL_LINEAR
  • GL_NEAREST_MIPMAP_NEAREST
  • GL_NEAREST_MIPMAP_LINEAR
  • GL_LINEAR_MIPMAP_NEAREST
  • GL_LINEAR_MIPMAP_LINEAR

GL_TEXTURE_MAG_FILTER accepts the following options:

  • GL_NEAREST
  • GL_LINEAR

GL_NEAREST corresponds to nearest-neighbour rendering, GL_LINEAR corresponds to bilinear filtering, GL_LINEAR_MIPMAP_NEAREST corresponds to bilinear filtering with mipmaps, and GL_LINEAR_MIPMAP_LINEAR corresponds to trilinear filtering. Graphical examples and further explanation of the most common options are visible further down in this lesson.

How to set a texture filtering mode

We first need to bind the texture, then we can set the appropriate filter parameter on that texture:

GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureHandle);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, filter);

How to generate mipmaps

This is really easy. After loading the texture into OpenGL (See Android Lesson Four: Introducing Basic Texturing for more information on how to do this), while the texture is still bound, we can simply call:

GLES20.glGenerateMipmap(GLES20.GL_TEXTURE_2D);

This will generate all of the mipmap levels for us, and these levels will get automatically used depending on the texture filter set.

How does it look?

Here are some screenshots of the most common combinations available. The effects are more dramatic when you see it in motion, so I recommend downloading the app and giving it a shot.

Nearest-neighbour rendering

This mode is reminiscent of older software-rendered 3D games.

GL_TEXTURE_MIN_FILTER = GL_NEAREST
GL_TEXTURE_MAG_FILTER = GL_NEAREST

Bilinear filtering, with mipmaps

This mode was used by many of the first games that supported 3D acceleration and is an efficient way of smoothing textures on Android phones today.

GL_TEXTURE_MIN_FILTER = GL_LINEAR_MIPMAP_NEAREST
GL_TEXTURE_MAG_FILTER = GL_LINEAR

It’s hard to see on this static image, but when things are in motion, you might notice horizontal bands where the rendered pixels switch between mipmap levels.

Trilinear filtering

This mode improves on the render quality of bilinear filtering with mipmaps, by interpolating between the mipmap levels.

GL_TEXTURE_MIN_FILTER = GL_LINEAR_MIPMAP_LINEAR
GL_TEXTURE_MAG_FILTER = GL_LINEAR

The pixels are completely smoothed between near and far distances; in fact, the textures may now appear too smooth at oblique angles. Anisotropic filtering is a more advanced technique that is supported by some mobile GPUs and can be used to improve the final results beyond what trilinear filtering can deliver.

Further exercises

What sort of effects can you achieve with the other modes? For example, when would you use something like GL_NEAREST_MIPMAP_LINEAR?

Wrapping up

The full source code for this lesson can be downloaded from the project site on GitHub.

A compiled version of the lesson can also be downloaded directly from the Android Market:

QR code for link to the app on the Android Market.

I hope you enjoyed this lesson, and thanks for stopping by!

Enhanced by Zemanta
Share

OpenGL ES Roundup, February 12, 2012

a single vanilla ice cream sandwich

Image via Wikipedia

If you haven’t checked it out yet, I recommend taking a look at the new Android Design website. There are a lot of resources and interesting information on developing attractive apps for Ice Cream Sandwich, Android’s newest platform. With Ice Cream Sandwich comes new changes, such as the deprecation of the menu bar.

Roundup

Question for you, dear reader: What do you all think about the current site design? I’d love to hear your thoughts and feedback, both positive and negative.

Enhanced by Zemanta
Share

Google Celebrates 10 Billion Downloads on the Market: Several Great Apps Available for Only 10 Cents Each!

Android Market

Image via Wikipedia

Android is booming very rapidly, and the market share has gone from nothing to dominating the market, within only a couple of years. I still remember when just a bit more than a year ago, competitors still thought that Android was only going to fill a niche market, and that it was “clunky” and “slow”. Well, its open nature has led to a rapid explosion of devices with Android, and rapidly improving speeds are getting rid of the Android “lag”.

From the Android Developer’s Blog:

The Android Market has recently crossed 10 billion downloads, and Google is celebrating:
“One billion is a pretty big number by any measurement. However, when it’s describing the speed at which something is growing, it’s simply amazing. This past weekend, thanks to Android users around the world, Android Market exceeded 10 billion app downloads—with a growth rate of one billion app downloads per month. We can’t wait to see where this accelerating growth takes us in 2012.”

“To celebrate this milestone, we partnered with some of the Android developers who contributed to this milestone to make a bunch of great Android apps available at an amazing price. Starting today for the next 10 days, we’ll have a new set of awesome apps available each day for only 10 cents each. Today, we are starting with Asphalt 6 HDColor & Draw for KidsEndomondo Sports Tracker ProFieldrunners HDGreat Little War GameMinecraftPaper CameraSketchbook MobileSoundhound Infinity and Swiftkey X.”

For 10 cents, why not try out a couple of apps? It’s amazing to see just how the quality of apps has improved over the past year, especially games. Unfortunately my Nexus S did reboot while playing one of the games, so the stability still isn’t all there, but I definitely do see an improvement in quality as compared to last year.

Check out the post for more details. It sounds like there will be a new set of apps at 10 cents each day!

Note: If you don’t see the apps at 10 cents on your phone, try going to the Android Market from your PC, instead. I had the same issue, where the app was listed at $4.99 on my phone and at 10 cents on the web. If you have a Google account linked to your phone, then you can just purchase from the web and Google will automatically send the app to your phone.

Enhanced by Zemanta
Share