How to Use OpenGL ES 2 in an Android Live Wallpaper

I’ve recently been looking into creating my first Live Wallpaper for Android as an opportunity to learn more about OpenGL ES and hopefully create something neat at the same time.

Throughout our entire series of Android tutorials, we’ve been using GLSurfaceView to take care of the hard work of threading and initialization for us; imagine my surprise when I learned that there is no equivalent for live wallpapers!

What we’re going to cover in this lesson

  • Adding multiple live wallpapers to a single APK.
  • Adding a thumbnail and description to the live wallpaper.
  • Using a GLSurfaceView to drive the live wallpaper.
  • Adapting the internals of GLSurfaceView to implement our own initialization and threading.
  • Allowing the user to switch between the two.

If you want to do OpenGL in a live wallpaper, it seems that there are a few alternatives:

  1. Do everything yourself. This might be a preferred option for some, but I won’t be exploring this in any more detail.
  2. Use GLSurfaceView, and adapt it to a live wallpaper. This lets us reuse a lot of code, but since this class was written to be used inside of an activity as a view, it might behave a little weirdly if we try to use it in a live wallpaper.
  3. Re-implement the features of GLSurfaceView. Robert Green has generously written up a re-implementation based on GLSurfaceView, and many have successfully based their wallpapers off of this code, so this might be one way to go. One caveat that I do have is that we need to duplicate a lot of the code and functionality inside GLSurfaceView, and we won’t be able to benefit from updates or variations between different versions of Android.

Since this is a research article, we’ll take a look at both options two and three. To start off, we’ll first look at adapting the existing GLSurfaceView for use within a live wallpaper. Later, we’ll see how we can use Robert Green’s work for rendering with OpenGL ES 2.0.

Here are the two main approaches we’re going to look at:

  • Approach 1: Using GLSurfaceView inside a live wallpaper
  • Approach 2: Using a custom live wallpaper based on the internals of GLSurfaceView

We’ll also look at combining the two approaches and allowing the user to select between the two. Let’s begin with the first approach: using GLSurfaceView.

Approach One: Using GLSurfaceView inside a live wallpaper

The first evidence I found that it was possible to use GLSurfaceView in a live wallpaper was from these files, by Ben Gruver:

I found these through this stack overflow question: http://stackoverflow.com/questions/4998533/android-live-wallpapers-with-opengl-es-2-0

Full credits to Ben Gruver for showing us one way that it could be done; in this article, we’ll work with our own adaptation of GLSurfaceView, with an emphasis on using it for OpenGL ES 2.0.

1. Creating the wallpaper service

Android Live Wallpapers are built off of WallpaperService, so the first thing we’ll do is create a new class called GLWallpaperService. Each service is really just a wrapper around an engine, which is subclassed from WallpaperService.Engine. This engine handles the actual lifecycle events of the live wallpaper.

We’ll be working with the Android OpenGL tutorials as a base, which can be downloaded from GitHub.

  1. Start off by creating a new package to contain the live wallpaper classes. For this lesson, I’ll be using com.learnopengles.android.livewallpaper.
  2. Inside this package, create a new class called GLWallpaperService.
Creating a new wallpaper service subclass

At the beginning of our new class, GLWallpaperService, extend WallpaperService:

public abstract class GLWallpaperService extends WallpaperService {
Creating a new wallpaper engine subclass

We’ll then create a new engine, so let’s also extend WallpaperService.Engine as an inner class:

public class GLEngine extends Engine {
Creating a custom subclass of GLSurfaceView

Inside the engine, create a new subclass of GLSurfaceView, called WallpaperGLSurfaceView:

class WallpaperGLSurfaceView extends GLSurfaceView {
	private static final String TAG = "WallpaperGLSurfaceView";

	WallpaperGLSurfaceView(Context context) {
		super(context);
	}

	@Override
	public SurfaceHolder getHolder() {
		return getSurfaceHolder();
	}

	public void onDestroy() {
		super.onDetachedFromWindow();
	}
}

We’ll be using this special subclass of GLSurfaceView to handle initializing OpenGL. The two methods to take note of are getHolder() and onDestroy(). Let’s take a look at these two methods in more detail:

getHolder()
Normally, GLSurfaceView calls getHolder() to get the surface holder provided by its superclass, SurfaceView, which assumes that it’s part of an activity as part of the content view. Since we’re driving a live wallpaper, we don’t want to use this surface. Instead, we override getHolder() to call [WallpaperService.Engine].getSurfaceHolder() to get the surface that’s associated with the live wallpaper.
onDestroy()
We need a way of telling the GLSurfaceView when it’s no longer valid. There’s no onDestroy(), but there is an onDetachedFromWindow() that we can use. We create our own onDestroy() to call onDetachedFromWindow().

Now that we have our extended GLSurfaceView in place, we can add a few lifecycle events and helper methods to round out a basic implementation:

Adding lifecycle events to the engine
private static final String TAG = "GLEngine";

private WallpaperGLSurfaceView glSurfaceView;
private boolean rendererHasBeenSet;

@Override
public void onCreate(SurfaceHolder surfaceHolder) {
	super.onCreate(surfaceHolder);
	glSurfaceView = new WallpaperGLSurfaceView(GLWallpaperService.this);
}

We initialize glSurfaceView in onCreate(). Remember that these methods should be placed inside of our inner class, GLEngine, and not at the same level as the wallpaper service. Let’s add a lifecycle event to handle visibility:

@Override
public void onVisibilityChanged(boolean visible) {
	super.onVisibilityChanged(visible);

	if (rendererHasBeenSet) {
		if (visible) {
			glSurfaceView.onResume();
		} else {
			glSurfaceView.onPause();			
		}
	}
}

Since we don’t have activity onPause() and onResume() events, we hook into the wallpaper engine’s onVisibilityChanged() event, instead. If the live wallpaper’s visible, we tell glSurfaceView to resume, and if it’s no longer visible, we tell it to pause.

We do check if the glSurfaceView is ready, since we shouldn’t call these methods unless a renderer has been set.

Let’s round out the life cycle events:

@Override
public void onDestroy() {
	super.onDestroy();
	glSurfaceView.onDestroy();
}

When the live wallpaper is destroyed, we tell glSurfaceView to stop rendering, using the custom onDestroy() method we defined earlier.

Adding helper methods to the engine

There’s also a couple of helper methods we’ll want to call from subclasses. Let’s define them now:

protected void setRenderer(Renderer renderer) {
	glSurfaceView.setRenderer(renderer);
	rendererHasBeenSet = true;
}

protected void setEGLContextClientVersion(int version) {
	glSurfaceView.setEGLContextClientVersion(version);
}

protected void setPreserveEGLContextOnPause(boolean preserve) {
	glSurfaceView.setPreserveEGLContextOnPause(preserve);
}

These methods simply wrap the glSurfaceView, so that we can call them from our subclasses.

2. Initializing OpenGL ES 2.0

The next step is to create a subclass that is derived from the GLWallpaperService that we’ve just created; this subclass will initialize OpenGL ES 2.0 and also initialize a custom renderer.

Create a new class, OpenGLES2WallpaperService, and add the following code:

public abstract class OpenGLES2WallpaperService extends GLWallpaperService {
	@Override
	public Engine onCreateEngine() {
		return new OpenGLES2Engine();
	}

	class OpenGLES2Engine extends GLWallpaperService.GLEngine {

		@Override
		public void onCreate(SurfaceHolder surfaceHolder) {
			super.onCreate(surfaceHolder);

			// Check if the system supports OpenGL ES 2.0.
			final ActivityManager activityManager =
				(ActivityManager) getSystemService(Context.ACTIVITY_SERVICE);
			final ConfigurationInfo configurationInfo =
				activityManager.getDeviceConfigurationInfo();
			final boolean supportsEs2 =
				configurationInfo.reqGlEsVersion >= 0x20000;

			if (supportsEs2)
			{
				// Request an OpenGL ES 2.0 compatible context.
				setEGLContextClientVersion(2);

				// On Honeycomb+ devices, this improves the performance when
				// leaving and resuming the live wallpaper.
				setPreserveEGLContextOnPause(true);

				// Set the renderer to our user-defined renderer.
				setRenderer(getNewRenderer());
			}
			else
			{
				// This is where you could create an OpenGL ES 1.x compatible
				// renderer if you wanted to support both ES 1 and ES 2.
				return;
			}
		}
	}

	abstract Renderer getNewRenderer();
}

As you can see, this is very similar to the OpenGL ES 2.0 initialization code that we’ve added to all of our activities in the Android tutorials. Calling setPreserveEGLContextOnPause() will ask the GLSurfaceView to preserve the OpenGL context, which will speed up pausing and restarting the live wallpaper by holding onto the surface.

3. Initializing a custom renderer

To round out the Java code, let’s add a final subclass, LessonThreeWallpaperService, which is going to use the renderer from lesson three as our live wallpaper:

public class LessonThreeWallpaperService extends OpenGLES2WallpaperService {
	@Override
	Renderer getNewRenderer() {
		return new LessonThreeRenderer();
	}
}

4. Adding the wallpaper to the manifest

Before we can install the live wallpaper on a device, we need to add it to the manifest and create a configuration file for it. Add the following to AndroidManifest.xml:

<service
    android:name=".livewallpaper.LessonThreeWallpaperService"
    android:label="@string/lesson_three_wallpaper_1"
    android:permission="android.permission.BIND_WALLPAPER" >
    <intent-filter>
        <action android:name="android.service.wallpaper.WallpaperService" />
    </intent-filter>

    <meta-data
        android:name="android.service.wallpaper"
        android:resource="@xml/wallpaper" />
</service>

This tells Android that this service is a live wallpaper, with the label set to @string/lesson_three_wallpaper_1. Let’s define a couple extra strings in strings.xml as follows:

<string name="lesson_three_wallpaper_1">Using GLSurfaceView</string>
<string name="lesson_three_wallpaper_2">Using re-implementation of GLSurfaceView</string>
<string name="lesson_three_wallpaper_3">Switching implementations</string>

We also need to create a wallpaper definition file. Create a new file, /res/xml/wallpaper.xml, and add the following:

<wallpaper xmlns:android="http://schemas.android.com/apk/res/android"
    android:thumbnail="@drawable/ic_lesson_three" />

This tells Android that this wallpaper should use this thumbnail when the user looks at the list of live wallpapers.

5. Build the application and view the wallpaper.

Now we can build the app and check out the live wallpaper. Install the app to your device, then go to your live wallpapers. You should see the following:

After selecting the live wallpaper, you should see a preview of lesson 3 on the screen:

Caveats with GLSurfaceView

Here are some of the caveats I can think of off of the top of my head; since this is a research article, I’d love to get your feedback.

  • I currently don’t understand enough about how SurfaceView works to see if using a GLSurfaceView uses additional resources in allocating Surfaces that never get used, since we’re using the live wallpaper’s surface.
  • SurfaceView, from which GLSurfaceView is based, sets the surface to PixelFormat.RGB_565 by default, whereas the WallpaperService sets PixelFormat.RGBX_8888 by default (checked against the ICS sources).

Approach Two: Using a custom live wallpaper based on the internals of GLSurfaceView

[Note: The wallpaper has changed since this was first written, as Mark Guerra and other contributors have extended Robert Green’s work at this GitHub repository: https://github.com/GLWallpaperService/GLWallpaperService. The rest of this section is no longer required for adding support for OpenGL ES 2.0.]

As the second part of this research article, we’ll also be creating a live wallpaper with Robert Green and Mark Guerra’s adaptation of the code from GLSurfaceView. For this, you’ll want to download GLWallpaperService from http://git.io/gbhjmQ. Since we already have a GLWallpaperService, let’s create a new package to contain this class, and let’s call it com.learnopengles.android.rbgrnlivewallpaper.

After you have copied the class from GitHub, be sure to update the package name and any imports.

1. Adding support for OpenGL ES 2.0.

The first thing to note with this wallpaper service is that it doesn’t seem to have support for OpenGL ES 2.0. We’ll add this support by using the source code for GLSurfaceView as a base.

Open up the GLWallPaperService we just downloaded, and let’s make the following changes:

Adding mEGLContextClientVersion

The first thing we’ll do is add a new variable called mEGLContextClientVersion. Add the following to the beginning of GLEngine:

private int mEGLContextClientVersion;
Adding a method to set the EGL context client version

The next thing we’ll need to do is adapt setEGLContextClientVersion() from GLSurfaceView. Add the following method before setRenderer():

public void setEGLContextClientVersion(int version) {
    checkRenderThreadState();
    mEGLContextClientVersion = version;
}
Updating DefaultContextFactory

The next thing we need to do is to update createContext() inside the class DefaultContextFactory. Update the interface EGLContextFactory and the class DefaultContextFactory as follows:

interface EGLContextFactory {
	EGLContext createContext(EGL10 egl, EGLDisplay display, EGLConfig eglConfig, int eglContextClientVersion);

	void destroyContext(EGL10 egl, EGLDisplay display, EGLContext context);
}

class DefaultContextFactory implements EGLContextFactory {
	private int EGL_CONTEXT_CLIENT_VERSION = 0x3098;

	public EGLContext createContext(EGL10 egl, EGLDisplay display, EGLConfig config, int eglContextClientVersion) {
        int[] attrib_list = {EGL_CONTEXT_CLIENT_VERSION, eglContextClientVersion,
                EGL10.EGL_NONE };

        return egl.eglCreateContext(display, config, EGL10.EGL_NO_CONTEXT,
        		eglContextClientVersion != 0 ? attrib_list : null);
    }
Updating EglHelper

We’ll also need to fix the call to createContext(). Find the call to mEGLContextFactory.createContext() which is located inside the class EglHelper, and update it as follows:

mEglContext = mEGLContextFactory.createContext(mEgl, mEglDisplay,
			mEglConfig, mEGLContextClientVersion);

Since mEGLContextClientVersion has not been defined in this scope, add it to the beginning of EglHelper, and pass it in the constructor as follows:

private int mEGLContextClientVersion;

public EglHelper(EGLConfigChooser chooser,
		EGLContextFactory contextFactory,
		EGLWindowSurfaceFactory surfaceFactory, GLWrapper wrapper,
		int eglContextClientVersion) {
	...
	this.mEGLContextClientVersion = eglContextClientVersion;
}
Updating GLThread

Now GLThread doesn’t compile, so we’ll need to define mEGLContextClientVersion in that scope, too. Add the following code:

private int mEGLContextClientVersion;

GLThread(GLWallpaperService.Renderer renderer, EGLConfigChooser chooser,
		EGLContextFactory contextFactory,
		EGLWindowSurfaceFactory surfaceFactory, GLWrapper wrapper,
		int eglContextClientVersion) {
		...
		this.mEGLContextClientVersion = eglContextClientVersion;
	}

Now we can update the call to new EglHelper as follows:

mEglHelper = new EglHelper(mEGLConfigChooser, mEGLContextFactory,
				mEGLWindowSurfaceFactory, mGLWrapper, mEGLContextClientVersion);
Updating setRenderer()

Now that we’ve added the variables at the right scope levels, we can now adjust the call to new GLThread, and pass in the mEGLContextClientVersion from GLEngine. Update the call to new GLThread in setRenderer() as follows:

mGLThread = new GLThread(renderer, mEGLConfigChooser,
			mEGLContextFactory, mEGLWindowSurfaceFactory, mGLWrapper,
			mEGLContextClientVersion);
Updating BaseConfigChooser

There’s one more change we need to do before we can use setEGLContextClientVersion(). Update the class BaseConfigChooser as follows:

protected int mEGLContextClientVersion;

public BaseConfigChooser(int[] configSpec, int eglContextClientVersion) {
    this.mEGLContextClientVersion = eglContextClientVersion;
    mConfigSpec = filterConfigSpec(configSpec);
}

...

private int[] filterConfigSpec(int[] configSpec) {
    if (mEGLContextClientVersion != 2) {
        return configSpec;
    }
    /* We know none of the subclasses define EGL_RENDERABLE_TYPE.
     * And we know the configSpec is well formed.
     */
    int len = configSpec.length;
    int[] newConfigSpec = new int[len + 2];
    System.arraycopy(configSpec, 0, newConfigSpec, 0, len-1);
    newConfigSpec[len-1] = EGL10.EGL_RENDERABLE_TYPE;
    newConfigSpec[len] = 4; /* EGL_OPENGL_ES2_BIT */
    newConfigSpec[len+1] = EGL10.EGL_NONE;
    return newConfigSpec;
}
Updating ComponentSizeChooser

We’ll need to scope this new variable, so let’s update the constructor for ComponentSizeChooser:

public ComponentSizeChooser(int redSize, int greenSize, int blueSize,
		int alphaSize, int depthSize, int stencilSize, int eglContextClientVersion) {
	super(new int[] { EGL10.EGL_RED_SIZE, redSize,
			EGL10.EGL_GREEN_SIZE, greenSize, EGL10.EGL_BLUE_SIZE,
			blueSize, EGL10.EGL_ALPHA_SIZE, alphaSize,
			EGL10.EGL_DEPTH_SIZE, depthSize, EGL10.EGL_STENCIL_SIZE,
			stencilSize, EGL10.EGL_NONE }, eglContextClientVersion);
Updating SimpleEGLConfigChooser

We’ll need to keep scoping this variable in, so update SimpleEGLConfigChooser as follows:

public static class SimpleEGLConfigChooser extends ComponentSizeChooser {
	public SimpleEGLConfigChooser(boolean withDepthBuffer, int eglContextClientVersion) {
		super(4, 4, 4, 0, withDepthBuffer ? 16 : 0, 0, eglContextClientVersion);
Updating methods in GLEngine

Now that we’ve added the scoping, we’ll have to update our calls from our methods in GLEngine as follows:

mEGLConfigChooser = new SimpleEGLConfigChooser(true, mEGLContextClientVersion);
...
setEGLConfigChooser(new SimpleEGLConfigChooser(needDepth, mEGLContextClientVersion));
...
setEGLConfigChooser(new ComponentSizeChooser(redSize, greenSize,
					blueSize, alphaSize, depthSize, stencilSize, mEGLContextClientVersion));

2. Initializing OpenGL ES 2.0.

Now that we’ve updated GLWallpaperView to add support for OpenGL ES 2.0, we can now subclass it to initialize OpenGL ES 2.0. Let’s copy the same OpenGLES2WallpaperService from before into our new package, com.learnopengles.android.rbgrnlivewallpaper:

public abstract class OpenGLES2WallpaperService extends GLWallpaperService {
	@Override
	public Engine onCreateEngine() {
		return new OpenGLES2Engine();
	}

	class OpenGLES2Engine extends GLWallpaperService.GLEngine {

		@Override
		public void onCreate(SurfaceHolder surfaceHolder) {
			super.onCreate(surfaceHolder);

			// Check if the system supports OpenGL ES 2.0.
			final ActivityManager activityManager =
				(ActivityManager) getSystemService(Context.ACTIVITY_SERVICE);
			final ConfigurationInfo configurationInfo =
				activityManager.getDeviceConfigurationInfo();
			final boolean supportsEs2 =
				configurationInfo.reqGlEsVersion >= 0x20000;

			if (supportsEs2)
			{
				// Request an OpenGL ES 2.0 compatible context.
				setEGLContextClientVersion(2);

				// Set the renderer to our user-defined renderer.
				setRenderer(getNewRenderer());
			}
			else
			{
				// This is where you could create an OpenGL ES 1.x compatible
				// renderer if you wanted to support both ES 1 and ES 2.
				return;
			}
		}
	}

	abstract Renderer getNewRenderer();
}

3. Adding a renderer.

We can also copy LessonThreeWallpaperService from before, and use it as is:

public class LessonThreeWallpaperService extends OpenGLES2WallpaperService {
	@Override
	Renderer getNewRenderer() {
		return new LessonThreeRenderer();
	}
}

We have a compile error because GLWallpaperService doesn’t use the same renderer interface. Let’s go back to it and delete the following lines:

public interface Renderer extends GLSurfaceView.Renderer {

}

We’ll need to update some class references, so replace all instances of GLWallpaperService.Renderer with Renderer, and optimize imports. Our code should now compile.

4. Updating the manifest.

As before, we’ll need to update the manifest to add in the new live wallpaper. Add the following to AndroidManifest.xml:

<service
    android:name=".rbgrnlivewallpaper.LessonThreeWallpaperService"
    android:label="@string/lesson_three_wallpaper_2"
    android:permission="android.permission.BIND_WALLPAPER" >
    <intent-filter>
        <action android:name="android.service.wallpaper.WallpaperService" />
    </intent-filter>

    <meta-data
        android:name="android.service.wallpaper"
        android:resource="@xml/wallpaper" />
</service>

5. Build the application and view the wallpaper.

If everything went well, then you should now see two live wallpapers in the list, corresponding to the two that we’ve created:

When we select one of the wallpapers, we should see the following:

Taking the best of approaches one & two: switching between implementations in a single live wallpaper

We’ve now shown that we can use either implementation as a backing for our live wallpaper: either by using a GLSurfaceView directly with a few slight modifications, or by adapting Robert Green’s work (itself based off of the internals of GLSurfaceView) to support OpenGL ES 2.0.

What if we could switch between implementations, based on a user toggle? That could be a useful debugging feature: if ever a user has an issue with one implementation, they could always try the other.

To do this, we need a new service that will return either the first engine we created or the second, based on the configured setting. Let’s create a new package called com.learnopengles.android.switchinglivewallpaper, and let’s create a new class called, you guessed it, GLWallpaperService.

1. Updating GLWallpaperService to support switching

The first thing we’ll do is copy the entire class body of GLWallpaperService from com.learnopengles.android.livewallpaper into the new GLWallpaperService. The only difference is that we’ll rename the engine to GLSurfaceViewEngine.

The second thing we’ll do is copy the entire class body of GLWallpaperService from com.learnopengles.android.rbgrnlivewallpaper into the new GLWallpaperService; let’s call that engine RbgrnGLEngine.

We can update each engine to implement the following interface, so that we won’t have to duplicate code afterwards:

interface OpenGLEngine {
	void setEGLContextClientVersion(int version);

	void setRenderer(Renderer renderer);
}

The code is too long to paste in its entirety, so you can double check against the GitHub code here: https://github.com/learnopengles/Learn-OpenGLES-Tutorials/blob/master/android/AndroidOpenGLESLessons/src/com/learnopengles/android/switchinglivewallpaper/GLWallpaperService.java

2. Creating a new OpenGLES2WallpaperService to support switching

In com.learnopengles.android.switchinglivewallpaper, create a new OpenGLES2WallpaperService with the following code:

public abstract class OpenGLES2WallpaperService extends GLWallpaperService {
	private static final String SETTINGS_KEY = "use_gl_surface_view";

	@Override
	public Engine onCreateEngine() {
		if (PreferenceManager.getDefaultSharedPreferences(
				OpenGLES2WallpaperService.this).getBoolean(SETTINGS_KEY, true)) {
			return new ES2GLSurfaceViewEngine();
		} else {
			return new ES2RgbrnGLEngine();
		}
	}

	class ES2GLSurfaceViewEngine extends GLWallpaperService.GLSurfaceViewEngine {

		@Override
		public void onCreate(SurfaceHolder surfaceHolder) {
			super.onCreate(surfaceHolder);
			init(this);
		}
	}

	class ES2RgbrnGLEngine extends GLWallpaperService.RgbrnGLEngine {

		@Override
		public void onCreate(SurfaceHolder surfaceHolder) {
			super.onCreate(surfaceHolder);
			init(this);
		}
	}

	void init(OpenGLEngine engine) {
		// Check if the system supports OpenGL ES 2.0.
		final ActivityManager activityManager = (ActivityManager) getSystemService(Context.ACTIVITY_SERVICE);
		final ConfigurationInfo configurationInfo = activityManager
				.getDeviceConfigurationInfo();
		final boolean supportsEs2 = configurationInfo.reqGlEsVersion >= 0x20000;

		if (supportsEs2) {
			// Request an OpenGL ES 2.0 compatible context.
			engine.setEGLContextClientVersion(2);

			// Set the renderer to our user-defined renderer.
			engine.setRenderer(getNewRenderer());
		} else {
			// This is where you could create an OpenGL ES 1.x compatible
			// renderer if you wanted to support both ES 1 and ES 2.
			return;
		}
	}

	abstract Renderer getNewRenderer();
}

As you can see, this is similar to the previous versions, except that it’s been updated to support switching. We can copy LessonThreeWallpaperService over to the new package as-is.

3. Adding a settings activity

The next thing we want to do is add a settings activity. Create WallpaperSettings as follows:

public class WallpaperSettings extends PreferenceActivity {
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		addPreferencesFromResource(R.xml.preferences);
	}
}
Creating the settings XML

We also need to create an XML for the settings. Create preferences.xml under /res/xml:

<?xml version="1.0" encoding="utf-8"?>
<PreferenceScreen xmlns:android="http://schemas.android.com/apk/res/android" >
    <CheckBoxPreference
        android:key="use_gl_surface_view"
        android:title="Use GLSurfaceView"
        android:defaultValue="true"/>
</PreferenceScreen>
Creating a new wallpaper XML

We’ll also need to create a new wallpaper.xml. Copy the existing wallpaper.xml to switching_wallpaper.xml, and update its content as follows:

<?xml version="1.0" encoding="utf-8"?>
<wallpaper xmlns:android="http://schemas.android.com/apk/res/android"
    android:settingsActivity="com.learnopengles.android.switchinglivewallpaper.WallpaperSettings"
    android:thumbnail="@drawable/ic_lesson_three" />
Updating the manifest

We’ll now need to update AndroidManifest.xml. Add the following:

<service
    android:name=".switchinglivewallpaper.LessonThreeWallpaperService"
    android:label="@string/lesson_three_wallpaper_3"
    android:permission="android.permission.BIND_WALLPAPER" >
    <intent-filter>
        <action android:name="android.service.wallpaper.WallpaperService" />
    </intent-filter>

    <meta-data
        android:name="android.service.wallpaper"
        android:resource="@xml/switching_wallpaper" />
</service>
<activity
    android:name=".switchinglivewallpaper.WallpaperSettings"
    android:exported="true"
    android:label="@string/lesson_three_wallpaper_3"
    android:permission="android.permission.BIND_WALLPAPER"
    android:theme="@style/WallpaperSettingsLight" >
</activity>
Adding new themes

We defined a new theme, so let’s add the following style definitions:

/res/values/styles.xml

https://github.com/learnopengles/Learn-OpenGLES-Tutorials/blob/master/android/AndroidOpenGLESLessons/res/values/styles.xml

/res/values-v11/styles.xml

https://github.com/learnopengles/Learn-OpenGLES-Tutorials/blob/master/android/AndroidOpenGLESLessons/res/values-v11/styles.xml

/res/values-v14/styles.xml

https://github.com/learnopengles/Learn-OpenGLES-Tutorials/blob/master/android/AndroidOpenGLESLessons/res/values-v14/styles.xml

These styles are adapted from the Theme.WallpaperSettings and Theme.Light.WallpaperSettings included with Android, but also updated so that they use the Holo theme on later versions of Android.

If you’re having trouble building the project after adding the new styles, change your build target to a newer version of Android (note that the app will still work on older versions, as they will simply ignore the XML).

4. Viewing the new live wallpaper

We can now build and run the application, and we’ll see our new live wallpaper in the list:

If we select the new one, “Switching implementations”, and open up the settings, we’ll see something like this:

We can use the toggle to change implementations; the toggle won’t take effect until we go back to the list of live wallpapers and then go back in to view the preview.

Wrapping up

I’d love to hear from those who have actually implemented live wallpapers, as well as from Robert Green and the Android guys if they ever come across this post! I just got this working with a few hours of research and development, but when it comes to this, I have nowhere near the same level of experience as some of the experts out there. I’d love to hear from you about the upsides and downsides of both using GLSurfaceView and of rolling out a custom derivative implementation based on the internals of GLSurfaceView.

Further exercises

Is there a better way to handle switching between implementations? How would you do it? Also, what would you do to continue animating the wallpaper even when the settings screen is displayed on top? Maybe when the settings screen appears you can grab a reference to the wallpaper and manually call its onVisibilityChanged()? You could also check for isPreview() directly in onVisibilityChanged(), but that will lead to problems when you exit the preview window.

I defined the settings key string in two places (once as a constant in Java, and once as an XML string); what would you do to avoid this duplication?

Get the source

The full source code for this lesson can be downloaded from the project site on GitHub. A compiled version of the lesson can also be downloaded directly from the Android Market:

QR code for link to the app on the Android Market.

Thanks for stopping by, and please feel free to check out the code and share your comments below.

Enhanced by Zemanta

Rotating An Object With Touch Events

Rotating an object in 3D is a neat way of letting your users interact with the scene, but the math can be tricky to get right. In this article, I’ll take a look at a simple way to rotate an object based on the touch events, and how to work around the main drawback of this method.

Simple rotations.

This is the easiest way to rotate an object based on touch movement. Here is example pseudocode:

Matrix.setIdentity(modelMatrix);

... do other translations here ...

Matrix.rotateX(totalYMovement);
Matrix.rotateY(totalXMovement);

This is done every frame.

To rotate an object up or down, we rotate it around the X-axis, and to rotate an object left or right, we rotate it around the Y axis. We could also rotate an object around the Z axis if we wanted it to spin around.

How to make the rotation appear relative to the user’s point of view.

The main problem with the simple way of rotating is that the object is being rotated in relation to itself, instead of in relation to the user’s point of view. If you rotate left and right from a point of zero rotation, the cube will rotate as you expect, but what if you then rotate it up or down 180 degrees? Trying to rotate the cube left or right will now rotate it in the opposite direction!

One easy way to work around this problem is to keep a second matrix around that will store all of the accumulated rotations.

Here’s what we need to do:

  1. Every frame, calculate the delta between the last position of the pointer, and the current position of the pointer. This delta will be used to rotate our accumulated rotation matrix.
  2. Use this matrix to rotate the cube.

What this means is that drags left, right, up, and down will always move the cube in the direction that we expect.

Android Code

The code examples here are written for Android, but can easily be adapted to any platform running OpenGL ES. The code is based on Android Lesson Six: An Introduction to Texture Filtering.

In LessonSixGLSurfaceView.java, we declare a few member variables:

private float mPreviousX;
private float mPreviousY;

private float mDensity;

We will store the previous pointer position each frame, so that we can calculate the relative movement left, right, up, or down. We also store the screen density so that drags across the screen can move the object a consistent amount across devices, regardless of the pixel density.

Here’s how to get the pixel density:

final DisplayMetrics displayMetrics = new DisplayMetrics();
getWindowManager().getDefaultDisplay().getMetrics(displayMetrics);
density = displayMetrics.density

Then we add our touch event handler to our custom GLSurfaceView:

public boolean onTouchEvent(MotionEvent event)
{
	if (event != null)
	{
		float x = event.getX();
		float y = event.getY();

		if (event.getAction() == MotionEvent.ACTION_MOVE)
		{
			if (mRenderer != null)
			{
				float deltaX = (x - mPreviousX) / mDensity / 2f;
				float deltaY = (y - mPreviousY) / mDensity / 2f;

				mRenderer.mDeltaX += deltaX;
				mRenderer.mDeltaY += deltaY;
			}
		}

		mPreviousX = x;
		mPreviousY = y;

		return true;
	}
	else
	{
		return super.onTouchEvent(event);
	}
}

Every frame, we compare the current pointer position with the previous, and use that to calculate the delta offset. We then divide that delta offset by the pixel density and a slowing factor of 2.0f to get our final delta values. We apply those directly to the renderer to a couple of public variables that we have also declared as volatile, so that they can be updated between threads.

Remember, on Android, the OpenGL renderer runs in a different thread than the UI event handler thread, and there is a slim chance that the other thread fires in-between the X and Y variable assignments (there are also additional points of contention with the += syntax). I have left the code like this to bring up this point; as an exercise for the reader I leave it to you to add synchronized statements around the public variable read and write pairs instead of using volatile variables.

First, let’s add a couple of matrices and initialize them:

/** Store the accumulated rotation. */
private final float[] mAccumulatedRotation = new float[16];

/** Store the current rotation. */
private final float[] mCurrentRotation = new float[16];
@Override
public void onSurfaceCreated(GL10 glUnused, EGLConfig config)
{

    ...

    // Initialize the accumulated rotation matrix
     Matrix.setIdentityM(mAccumulatedRotation, 0);
}

Here’s what our matrix code looks like in the onDrawFrame method:

// Draw a cube.
// Translate the cube into the screen.
Matrix.setIdentityM(mModelMatrix, 0);
Matrix.translateM(mModelMatrix, 0, 0.0f, 0.8f, -3.5f);

// Set a matrix that contains the current rotation.
Matrix.setIdentityM(mCurrentRotation, 0);
Matrix.rotateM(mCurrentRotation, 0, mDeltaX, 0.0f, 1.0f, 0.0f);
Matrix.rotateM(mCurrentRotation, 0, mDeltaY, 1.0f, 0.0f, 0.0f);
mDeltaX = 0.0f;
mDeltaY = 0.0f;

// Multiply the current rotation by the accumulated rotation, and then set the accumulated
// rotation to the result.
Matrix.multiplyMM(mTemporaryMatrix, 0, mCurrentRotation, 0, mAccumulatedRotation, 0);
System.arraycopy(mTemporaryMatrix, 0, mAccumulatedRotation, 0, 16);

// Rotate the cube taking the overall rotation into account.
Matrix.multiplyMM(mTemporaryMatrix, 0, mModelMatrix, 0, mAccumulatedRotation, 0);
System.arraycopy(mTemporaryMatrix, 0, mModelMatrix, 0, 16);
  1. First we translate the cube.
  2. Then we build a matrix that will contain the current amount of rotation, between this frame and the preceding frame.
  3. We then multiply this matrix with the accumulated rotation, and assign the accumulated rotation to the result. The accumulated rotation contains the result of all of our rotations since the beginning.
  4. Now that we’ve updated the accumulated rotation matrix with the most recent rotation, we finally rotate the cube by multiplying the model matrix with our rotation matrix, and then we set the model matrix to the result.

The above code might look a bit confusion due to the placement of the variables, so remember the definitions:

public static void multiplyMM (float[] result, int resultOffset, float[] lhs, int lhsOffset, float[] rhs, int rhsOffset)

public static void arraycopy (Object src, int srcPos, Object dst, int dstPos, int length)

Note the position of source and destination for each method call.

Trouble spots and pitfalls
  • The accumulated matrix should be set to identity once when initialized, and should not be reset to identity each frame.
  • Previous pointer positions must also be set on pointer down events, not only on pointer move events.
  • Watch the order of parameters, and also watch out for corrupting your matrices. Android’s Matrix.multiplyMM states that “the result element values are undefined if the result elements overlap either the lhs or rhs elements.” Use temporary matrices to avoid this problem.
WebGL examples

The example on the left uses the simplest method of rotating, while the example on the right uses the accumulated rotations matrix.

Your browser does not support the canvas tag. Your browser does not support the canvas tag.
Further exercises

What are the drawbacks of using a matrix to hold accumulated rotations and updating it every frame based on the movement delta for that frame? What other ways of rotation are there? Try experimenting, and see what else you can come up with!

  

Listening to Android Touch Events, and Acting on Them

Basic blending (additive blending of RGB cubes).
Basic blending.

We started listening to touch events in Android Lesson Five: An Introduction to Blending, and in that lesson, we listened to touch events and used them to change our OpenGL state.

To listen to touch events, you first need to subclass GLSurfaceView and create your own custom view. In that view, you create a default constructor that calls the superclass, create a new method to take in a specific renderer (LessonFiveRenderer in this case) instead of the generic interface, and override onTouchEvent(). We pass in a concrete renderer class, because we will be calling specific methods on that class in the onTouchEvent() method.

On Android, the OpenGL rendering is done in a separate thread, so we’ll also look at how we can safely dispatch these calls from the main UI thread that is listening to the touch events, over to the separate renderer thread.

public class LessonFiveGLSurfaceView extends GLSurfaceView
{
	private LessonFiveRenderer mRenderer;

	public LessonFiveGLSurfaceView(Context context)
	{
		super(context);
	}

	@Override
	public boolean onTouchEvent(MotionEvent event)
	{
		if (event != null)
		{
			if (event.getAction() == MotionEvent.ACTION_DOWN)
			{
				if (mRenderer != null)
				{
					// Ensure we call switchMode() on the OpenGL thread.
					// queueEvent() is a method of GLSurfaceView that will do this for us.
					queueEvent(new Runnable()
					{
						@Override
						public void run()
						{
							mRenderer.switchMode();
						}
					});

					return true;
				}
			}
		}

		return super.onTouchEvent(event);
	}

	// Hides superclass method.
	public void setRenderer(LessonFiveRenderer renderer)
	{
		mRenderer = renderer;
		super.setRenderer(renderer);
	}
}

And the implementation of switchMode() in LessonFiveRenderer:

public void switchMode()
{
	mBlending = !mBlending;

	if (mBlending)
	{
		// No culling of back faces
		GLES20.glDisable(GLES20.GL_CULL_FACE);

		// No depth testing
		GLES20.glDisable(GLES20.GL_DEPTH_TEST);

		// Enable blending
		GLES20.glEnable(GLES20.GL_BLEND);
		GLES20.glBlendFunc(GLES20.GL_ONE, GLES20.GL_ONE);
	}
	else
	{
		// Cull back faces
		GLES20.glEnable(GLES20.GL_CULL_FACE);

		// Enable depth testing
		GLES20.glEnable(GLES20.GL_DEPTH_TEST);

		// Disable blending
		GLES20.glDisable(GLES20.GL_BLEND);
	}
}

Let’s look a little bit more closely at LessonFiveGLSurfaceView::onTouchEvent(). Something important to remember is that touch events run on the UI thread, while GLSurfaceView creates the OpenGL ES context in a separate thread, which means that our renderer’s callbacks also run in a separate thread. This is an important point to remember, because we can’t call OpenGL from another thread and just expect things to work.

Thankfully, the guys that wrote GLSurfaceView also thought of this, and provided a queueEvent() method that you can use to call stuff on the OpenGL thread. So, when we want to turn blending on and off by tapping the screen, we make sure that we’re calling the OpenGL stuff on the right thread by using queueEvent() in the UI thread.

Further exercises

How would you listen to keyboard events, or other system events, and show an update in the OpenGL context?

Enhanced by Zemanta

How to Embed Webgl into a WordPress Post

Embedding into WordPress

This information was originally part of WebGL Lesson One: Getting Started, but I thought it would be more useful if I also broke it out into a post on its own.

Embedding WebGL into WordPress can be a little tricky, but if you use the right tags and the HTML editor it can be done without too much trouble! You’ll need to insert a canvas in your post, script includes for any third-party libraries, and a script body for your main script (this can also be an include).

First, place a canvas tag where you would like the graphics to appear:

<pre><canvas id="canvas" width="550" height="375">Your browser does not support the canvas tag. This is a static example of what would be seen.</canvas></pre>

Then, include any necessary scripts:

<pre><script type="text/javascript" src="http://www.learnopengles.com/wordpress/wp-content/uploads/2011/06/webgl-utils.js"></script></pre>

Embed any additional scripts that you need, such as the main body for your WebGL script:

<pre><script type="text/javascript">
/**
* Lesson_one.js
*/

...

</script></pre>

The <pre> tag is important; otherwise WordPress will mangle your scripts and insert random paragraph tags and other stuff inside. Also, once you’ve inserted this code, you have to stick to using the HTML editor, as the visual editor will also mangle or delete your scripts.

If you’ve embedded your scripts correctly, then your WebGL canvas should appear, like below:

Your browser does not support the canvas tag. This is a static example of what would be seen.
WebGL lesson one, example of what would be seen in a browser.

Hope this helps out some people out there!