David Chandler and I just finished our session on cloud-enabling your Android applications at Google I/O 2013.

1. the video of the talk is posted you YouTube.
2. the source code for the demo published on github
   
3. the Geek Serendipity application we created is on the Play Store

Here is the Geek Serendipity app we built.  It enables you to share your Geekly interest and see what geeks are around you. In the bottom we are showing your lat long and a geohash which we use to find who is “near” you.

There are four things that are important to me when we  started this application.

1. Client Focused Dev – we didn’t want to manage servers, deal with security patches, or figure out availability zones. As an android guy, David wanted to focus 95% of my energy on building an amazing android app rather than writing server code.
2. Location Saved to Cloud – We need to share locations between users and that means we do need a cloud backend to share this state.
3. Update Geeks Nearby Live – We like android apps that living and active. If I look at the app while I drink my soy latte double cappuccino I’d like to see geeks pop in as they come near by.
4. Avoid typing password – We don’t want my users to have to remember and type in passwords on the phone. Tiny screen, tiny keyboards and passwords don’t mix.

At Google I/O we are announcing the Mobile Backend Starter. This is a starter application that makes it easy for you to get started building your mobile backend in the cloud! We have focused on making it a no server code solution. So you can ship a production application with no server coding required. Of course if you’d like to download the server code you can customize it as much as you need. We have also added GCM support and continuous queries so you can register a query that will call you back at anytime in the future when it is true. All this with built in Google Authentication.

With this starter application, you don’t need to write any server code to get the basic experience we are showing up and running. Right out of the box, you get two things. A backend that stores your data with App Engine and a front end in android that makes it trivially easily to access that data.

We have taken that very cool Mobile Backend Starter and extended it for our Geek Serendipity use case. For the data, we are storing the geohash of the location and last reported timestamp for each user. We are also usingGoogle Cloud Messaging for Android and the continuous queries to keep each client up to date with where all the geeks are. And Finally we are using Google Maps V2 to display the locations.

Part One — Basic Cloud Connected App

To get started, go to the new Google Cloud Console. This console will be the one stop shop for all your Google cloud developers needs from getting access to Google APIs such as Google Maps to accessing computing resources such as Google Compute Engine and App Engine.

Here is our new Cloud Console. From here I can create and manage all my cloud backends. From here you can create a new project. We give the project a name, then we have an app ID that is used for programmatic access. Notice this is our startup name generator… let’s see if we can get a good one… I like dauntless-bay-213, but I’ll change it to geek-serendipity.

Now I have a project, we need to mobile enable it. To do that I will just click deploy here on this mobile backend. What this does is install the mobile backend starter on the app engine application

Clicking on deploy, starts the magic!

Step number 2 is to check out the settings for this backend. By clicking on the settings button there we are navigated to the setting page for the mobile backend.

By default this is set to locked down, no http requests will be accepted. We will move it to Open, this will make it so that anyone unauthorized client can access my backend, so this mode is suggested only for development.

Step #3 is to download the source code for the android client. We have pre-built a sample guestbook application that is intended for you to use as a way to get started. At the talk, we used Android with the latest version of the ADT installed.

Just import this into eclipse (this works just as well in Android Studio).

Then select the unzipped directory we downloaded…

…and import this project.

As you can see we have a few libraries here, then a main guestbook application.

We can open up Const.java and paste in our appid we gave when we created this app.

And set the project Id to the one we gave when we created the app.

 public static final String PROJECT_ID = "geek-serendipity";;

As you can see here, we have given the project ID on the client and that is the

Now we can run it and enter a few messages.

If we flip over to the Cloud Console and look at the Datastore, we see our two messages

Wow, that is pretty cool… 5 mins into this demo we already have a cloud connected application that will store it’s state in the cloud so it works perfectly as users change devices.

What we have done so far is spin up a backend on app engine, download the client, add the project ID and run it.

Part Two — Push Notifications

We are going to want a vibrant app that does live updates. Let’s see how we can work live updates into this application. A naive way to do this would be poll every few minutes by calling back to the server and see if there are new items. While this is a common technique the biggest problem is battery life, network traffic and load on your server. Not to mention latency the user sees for new messages arriving.

Google has already established a channel to communicate to the Google apps on the device. Google+, GMail and Chat, etc all get called back from the server whenever there is new content for you. Google handles making sure that connection stays active with the minimum power usage and network traffic. Wouldn’t it be cool if your applications could use that that exact same channel…. You can! Google Cloud Messaging for Android is exposed for your applications to use for exactly this reason.

But the setup for using Google Cloud Messaging has been tricky. You need to setup your own server, handle and track device registration, do OAuth2 auth, REST calls back to the GCM service, etc. Let’s see how easy it is with the Mobile Backend Starter.

We need to do two things. First, on the server side we need to enable GCM and enter the Server API key into the Mobile Backend Starter setting page. The Server API key identifies who (which project) is making the calls to GCM to register devices and send notifications.

The second thing we need to do is paste in the project number into the Const.java file in the client project. This is used as part of device registration with GCM. It tells GCM which notifications to send to this device.

Let’s look at how we do this.

First, we enable Cloud Messaging for Android in the API List of the Cloud Console by selecting APIs, and then enabling Cloud Messaging for Android.

Then we go to the Mobile Backend Starter setting page and enable GCM. As you see it needs a API Key.

I can create this in API Access by going back into the API Listing page and register a new app. In this case, I am registering the mobile backend app… such that it can access the GCM API we just enabled.

Now I can just grab that Server API Key and paste it into the config page and I am done!

The last step is to add the project number to the Const.java file. We are in the middle of transitioning to the new cloud console, but it isn’t easy to see the project number from the new console yet. You need to bounce back to the old APIs console to see it. The easy way to do that is to click on

And finally, we need to paste the project number into the client project.

Now, we really need two devices to really test this out. If you are not lucky enough to have two, you can run them in virtual devices as well. If all is working well, when you click send on one a few seconds later you see the message show up on the other.

Part Three — Add Map

The sample app for our talk shows the locations of geeks on a map. You can create a new MapActivity in your project by following the instructions at Getting Started with Google Maps Android API v2. To enable backward compatibility with older versions of Android, the demo app uses the MapFragment from the Android support library. The layout looks like this:

The FrameLayout and TextView are not required for maps. We’re using the TextView overlay to show authentication status directly on the map. Here is our corresponding MapActivity:

package com.google.cloud.backend.android;

import android.os.Bundle;
import android.support.v4.app.FragmentActivity;
import android.view.Menu;
import android.widget.TextView;

import com.google.android.gms.maps.GoogleMap;
import com.google.android.gms.maps.SupportMapFragment;

public class MapActivity extends FragmentActivity {

	private GoogleMap mMap;

	private void setUpMapIfNeeded() {
		// Do a null check to confirm that we have not already instantiated the
		// map.
		if (mMap == null) {
			// Try to obtain the map from the SupportMapFragment.
			mMap = ((SupportMapFragment) getSupportFragmentManager()
					.findFragmentById(R.id.map)).getMap();
			// Check if we were successful in obtaining the map.
			if (mMap != null) {
				setUpMap();
			}
		}
	}

	private void setUpMap() {
		mMap.setMyLocationEnabled(true);
	}

	@Override
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		setContentView(R.layout.activity_map);
		setUpMapIfNeeded();
	}

	@Override
	protected void onResume() {
		super.onResume();
		setUpMapIfNeeded();
		TextView overlay = (TextView) findViewById(R.id.overlay);
		overlay.setText("Not signed in");
	}

}

Part Four — Store Location in the Cloud

To send our location to the cloud backend, we need to get notified of location updates, then make a request to our backend service. To get notified of location updates, we’ll register the MapActivity class as an OnMyLocationChangeListener. To do this, we call mMap.setOnMyLocationChangeListener(this) in the setUpMap() method and let the MapActivity class implement OnMyLocationChangeListener. The required method looks like this:

	private void setUpMap() {
		mMap.setMyLocationEnabled(true);
		mMap.setOnMyLocationChangeListener(this);
	}

	protected String myLocation;
	protected static boolean locSent;
	private static final Geohasher gh = new Geohasher();

	@Override
	public void onMyLocationChange(Location location) {
		this.myLocation = gh.encode(location);
		if (!locSent) {
			sendMyLocation();
		}
	}

To send our location to the server, we can use the features of the Mobile Backend Starter project. MBS provides a generic entity (CloudEntity) on which we can set arbitrary properties and a set of CRUD services that know how to send and receive it. To implement sendMyLocation(), we simply let our MapActivity extend CloudBackendActivity, then create a new CloudEntity and update the server:

	private void sendMyLocation() {
		CloudEntity self = new CloudEntity("Geek");
		self.put("interest", "Cloud");
		self.put("location", this.myLocation);
		getCloudBackend().update(self, new CloudCallbackHandler() {
			@Override
			public void onComplete(CloudEntity results) {
				locSent = true;
				drawMyMarker();
			}
		});
	}

Note: at the time of this writing, CloudBackendActivity extends the basic Activity class. To use it with a MapFragment and the support library, you’ll need to modify CloudBackendActivity to instead extend FragmentActivity from the support library.

Part Five — Add Authentication

We are now storing sensitive information (your location) up in the cloud and as such we want to be sure we have locked down access to our server to only our client.  We also want users to authenticate so that we know who they are even if they use different devices.   Authentication can be difficult on mobile devices.  Are you going to ask your users to type in a username and password on the little keyboard?  Will they even be able to remember a username and password for just your app?  what if they lose their password or it gets stolen?

The Mobile Backend Starter makes it very easy to use Google Authentication.   Nearly every Android phone has a Google account that your customers are already using for GMail, Chat, Google Now or other google services.     You can use that same account in our application no passwords for our users to remember, no admin headaches for you.

We will just go in and enable auth in the Mobile Backend Starter setting page.

There are three steps to enabling auth. The first step is to ensure that we have strong identity for the android client. We want to lock down access to our Cloud Endpoints (our REST API) to only our own Android client.

That is way the first box is asking for — the Android Client ID. The Android Client ID is how we uniquely identify your application run in an android device. It allows for secure access between your code on the device and your code in the cloud. One common way to do this is to create a “secret” and embed that into the client application. The problem with this approach is that it is easy to find that secret. How many of you have ever decompiled a class file? yea — not that hard. The secrets don’t work well if they are easy to find.

We are going to use a different approach. The same approach in fact that Google Play uses to uniquely identify an application to update. You might imagine it would be “bad” to update the wrong app. So we have developed a mechanism that is client-secret free to strongly identify an application.

To do this, let’s create a client ID for Android in the Cloud Console, in the API section, select Register App. This time we want to register the Android client, so we select Android and directly from Android.

In the Android Identification section it asks for a package name and SHA1 fingerprint. It is a cryptographically secure hash of the manifest we use to sign the android app. There is a handy keytool app (you can get that for Windows as well) and run it over your keystore. Eclipse users can find it in ~/.android/debug.keystore. For the debug.keystore, there is likely not a password, so just hit return there.

There that is our SHA1 hash… now we give that to the Cloud Console along with our package name (com.google.cloud.backend.android) to give us a key.

And we give that key to the Mobile Backend Starter.

OK, I mentioned there’d be three steps for enabling auth… That was step 1, restricting access to just our client. Now for step 2, establishing the client and the server are from the same developer so we can avoid the standard OAuth2 prompt. You may have seen an OAuth2 dialog when an application is asking for permission to, for example, access your data on Google+.

It makes perfect sense that in this case the application has to ask the user for permission to access their data stored in google+. But in our case there is no 3rd party here. Our users are using our app to access their data stored by us. There is really no reason to prompt the user in this case.

So to avoid this dialog, we need to establish that the same developer is building the client and the server. To do this we share a token between them. Notice this is NOT a secret. Feel free to publish this client ID. Even if someone has this token they will still not be able to access your server as the requests would not be coming from a signed application.

We already have a Web Server Application created for GCM, we just need to grab the OAuth2 client ID.

And add it to the Mobile Backend Starter.

We just add this to the backend and the client.. and set the client to be auth enabled..

The final step is to pass the user name from the client to the server. Luckily Android, App Engine and the Mobile Backend Starter all have built in support for Google Auth so it is very easy to have the same user on the client and server. No explicit code need to enable it.

Now you see we are giving a chance to select an account (because I have more than one). and NO password typing on the phone!

You can see the client has access to my user name…

And there we see in Datastore in cloud console you see the entities are now owned by users. We keep up with who created and who updated each entity.

Part Six — Continuous Queries

One powerful feature of the Mobile Backend Starter code is continuous queries. Built on the App Engine Prospective Search API, this capability notifies the mobile client via Google Cloud Messaging whenever query results change on the server. For our Geek Serendipity app, the process works like this:

1. The client sends an updated location to the server.
2. Prospective Search notifies the MBS App Engine code.
3. The MBS server sends a GCM message to the client.
4. The MBS client calls the original query for to get updated results.
5. The MBS client invokes the query handler with the results.

All of this happens transparently using a single feature of the app. The relevant code for the Geek Serendipity app is here:

private void queryGeeks() {
		CloudCallbackHandler<List> handler = new CloudCallbackHandler<List>() {
			@Override
			public void onComplete(List results) {
				drawMarkers(results);
			}

			@Override
			public void onError(IOException e) {
				Toast.makeText(getApplicationContext(), e.getMessage(),
						Toast.LENGTH_LONG).show();
			}
		};

		CloudQuery cq = new CloudQuery("Geek");
		cq.setLimit(50);
		cq.setSort(CloudEntity.PROP_UPDATED_AT, Order.DESC);
		cq.setScope(Scope.FUTURE_AND_PAST);
		getCloudBackend().list(cq, handler);
	}

The queryGeeks() method registers a continuous query to return the top 50 most recent location updates. By setting Scope.FUTURE_AND_PAST, the handler passed to getCloudBackend().list() will be invoked whenever there are updated results on the server.

Because our geek query is a continuous query, we only need to register it once, but we want to do that only after MBS has been initialized (GCM has been registered and MBS is ready for query subscriptions). In addition, if authentication is enabled on the MBS settings page, we want to wait until the user has been authenticated. MBS has a special hook that meets all these criteria. To use it, simply override the onPostCreate() method from CloudBackendActivity:

	@Override
	protected void onPostCreate() {
	    queryGeeks();
	}

Now our query will run immediately on initialization and thereafter whenever there are new geek location updates on the server.

Part Seven — Customizing the Backend

Well, we have just about got this application wrapped up. And that Mobile Backend Starter we used has served us really well. We have not had to do any server coding at all. And that is great. This is a great way to get started, but what if I need more? What if I need to write a some custom server code? Well with the Mobile Backend Starter you never hit a limitation or roadblock you can’t work around.

The backend we deployed at the start of this talk, is actually an app engine application that you can go and download yourself. All the source code is on github. You can download it, see how we implemented it, and customize it to meet your needs.

To show that off, let’s look at how to add some custom client logging to server. As Googlers, we **LOVE** to make data driven decisions. Data on how our app is used on the client can help us make it better if we can see it in aggregate over time. So let’s add some logic for the clients to log directly to the server and stick them in App Engine’s logs where then can be imported in BigQuery for analysis.

So basically we are going to add a new custom Endpoint to the App Engine app (ClientLoggingEndpoint) that stores data in the app engine logs store, then add a CloudLog class to android app that sends client logs up to the server.

Lets’ start by downloading the backend, now we will important it into eclipse in the same workspace as our android project. (Note, for this part you do need the Google Plugin for Eclipse installed).

Now, let’s add our custom method…

ClientLoggingEndpoint.java

@ApiMethod(
   name = "logs.log",
   path = "logs/log/{message}",
   httpMethod = HttpMethod.POST
)
public void logFromClient(@Named("message") String message) {
   log.info("client log: " + message);
   return ;
}

Now we need to be able to access this method from the client. Well that is pretty easy. Right click on Google menu and select Generate Cloud Endpoint Library.

Now, the library will be generated in the endpoints-libs director. You see we have a new directory there for our logging service.

Now we simply need to copy this into the android project (you can just drag it down into the endpoints-libs of the android project.

Now we just add this library to the java build path… We actually need the clientlogging-v1-generated-source to the build path.

OK, great. Now we just need to use this client library from the Android app. To make that easy, I created a simple CloudLog.java class to encapsulate access. The main method of which is below.

Notice we are using a thread to get the network calls off the main UI thread. We want to be sure that this logging does not in anyway negatively affect the experience for the user. No laggy UI while we are posting the logs to the server. Also note, I am sending each log directly to the server… a better pattern would likely to be to batch, but I will leave that as an exercise for the user.

	public void Log (final String message) {

			new Thread(new Runnable() {
			    public void run() {

					Clientlogging.Builder builder = new Clientlogging.Builder (
							   AndroidHttp.newCompatibleTransport(),
                                                           new GsonFactory(), null);
					Clientlogging service = builder.build();
					try {
						String msg = android_id +":"+ message;
						service.logs().log(msg).execute();
						android.util.Log.d("log", msg);

					} catch (IOException e) {
						// TODO Auto-generated catch block
						e.printStackTrace();
						android.util.Log.e("log", e.toString());

					}
			    }
			  }).start();
	}

Now we just need to make some changes to the main activity to call it in GeekWatchActivity.java.

First we created the CloudLog class, passing the unique ID of this device (do we can look at pre-device usage patterns later).

android_id = Secure.getString(getApplicationContext().getContentResolver(),
		                Secure.ANDROID_ID);

cloudLog = new CloudLog (android_id);

Then we add some logging to interesting places, here is one example:

        @Override
        public void onCameraChange(CameraPosition position) {
                LatLngBounds visibleBounds = mMap.getProjection().getVisibleRegion().latLngBounds;
                findGeeks(visibleBounds);
                cloudLog.Log("Camera Changed");
        }

Deploy the backend to App Engine and run the client app and use the app a bit. Logs are being written to the App Engine logs store.

Now that we have it in GAE logs, we can easily import this into BigQuery or other tool to get meaning and drive improvements.

Conclusion

Hopefully you’ll find that the Mobile Backend Starter project is an easy way to get started building a cloud back end for your Android app. The CloudEntity and corresponding CRUD endpoints allow you to update and query data in the cloud without having to write any server code, and continuous queries offer Google Cloud Messaging functionality right out of the box.

To learn more, please see these resources:

Google I/O 13 session video

Geek Serendipity source

Geek Serendipity application on Play Store

Xavier Ducrohet and I had a great time today demoing “BigDaddy” which is the codename for the Google Plugin for Eclipse 2.4 Beta that we released today.

I started off with the following products installed:

Eclipse Helios, Android Developments Tools and, of course the Google Plugin for Eclipse 2.4 beta.

• See the video of the talk: http://www.youtube.com/watch?v=M7SxNNC429U
• Download the complete source code for the demo: https://code.google.com/p/cloud-tasks-io
• See the demo live (but please only use test data): http://cloudtasksio.appspot.com.  For the Android client, browse to the bottom of this page on your phone to find the link to the APK.
• You can also check out the default Big Daddy application: http://bigdaddy-io.appspot.com/
• Access the slides http://www.slideshare.net/brada/androidandroid-app-engine-a-developers-dream-combination

Our goal is to create a task tracking application for Larry Page.  As he takes over as CEO, Larry has a lot of tasks that he needs to track and this app will help him (and the rest of us) track tasks.  ;-)

The basic architecture of the task app looks very similar to a wide range of applications

First, we have an Android application that talks, via RPC to an App Engine server where your business logic is and the data is stored in the app engine datastore. We also needed a standards based web client that can work from anywhere on the same data with the same business logic.  We built that out with GWT to keep with the same programing language and tools, but of course, any web client technology will work.   Finally, we use C2DM to do push notification when new tasks are added to the database.

OK, so let’s look at how we build this task tracking application.

The first step is to create a new App Engine Connected Android Project. This project gives you a very useful starting project with the plumbing code in libraries and the code that you may want to change and tweak generated into the project.   This project gives you a few  things:

1. An authenticated Hello world RPC service with the logic in app engine and clients in GWT and Android.

2. All the C2DM infrastructure .

a. App Engine based datastore for handing registration

b. Android client with default UI for picking a user and responding to C2DM messaging

c. GWT based client for sending C2DM pings through the app engine server.

Let’s take a look at how this works.

Creating the Starter Project

In Eclipse select File\New then Other.  Under the Android node you will find the “App Engine connected Android project”.

In the first page of the wizard, simply give your project a name and namespace.  From these the tool will create Android, App Engine and GWT projects.

note:  if you see the warning to “configure Android SDK” just click on the link and select the Android SDK with at least API level 8 (to support C2DM).  You will also likely want to select an a Google APIs enabled SDK to support the google authentication that we will be using.

The next page of the wizard asks for your C2DM role account.  You can sign up for a C2DM role account from this wizard.

A tip: You can auto fill this in if you put a file called c2dm.properties in your home directory with the format:

user=yourid@gmail.com
passwd=yourpassword

This wizard creates two projects… An Android client and App Engine server. Along with a shared code directory for storing code that is compiled into both projects.

To run this, you will want to connect your android phone via USB or create an virtual device via the Android SDK and AVD manager and launch it.  For the AVD I found targeting google API level 9 and 256 meg SD card more than sufficient.   I used a screen size of 9 to be small enough to manage, but still readable.  I also found that saving a snapshot once after boot then only launching from snapshot NOT having to the snap shot made testing on a clean image with a very fast start-up helpful.

You can verify you have your device connected by looking at the DDMS perspective in eclipse.  You should see your emulator up and the list of all the process.  This indicates that you are ready to go.

Running the Starter Project

Once you have a device connected, you right click on the Android project and select Debug As then Local App Engine Connected Android Project.

What this does is wire up the android application to talk to the local app engine dev app server.

In the emulator, we see the application pop up.   The first thing it asks you to do is log in (if you don’t see it, click options and then accounts).  This very important — many android devices have google accounts built in.  A coworker of mine recently told me he’d rather stand naked in the rain than type his password on the phone.   Using the google accounts makes it possible to authenticate without typing a password.  This will also enable authenticated RPCs and C2DM pings as you will see.

Once you connect, your device will be registered to receive C2DM messages.  In particular, the app engine application with store a unique code for pinging your device when a message is sent to you.  You can look in the app engine datastore to see exactly what the app engine application is storing for you.  Browse to http://127.0.0.1:8888/_ah/admin/ and select DataStore viewer.

As you can see here, we store the user’s email address and the unique device ID.  This allows us to send C2DM pings through the App Engine server later.

Back in the Android client, click on the Say Hello button, you will see a message back from the app engine server.  It calls the getMessage() functional found in the HelloWorldService.java file in the com.cloudtasks.server package.    Notice that it is return the user’s email address from the server.  That is fantastically cool.  It means that the app engine server knows the same user identity information as the android client.

 public static String getMessage() {

 UserService userService = UserServiceFactory.getUserService();
 User user = userService.getCurrentUser();

 String message;

 if (user == null) {

    message = "No one is logged in!\nSent from App Engine at " + new Date();

 } else {

    message = "Hello, " + user.getEmail() + "!\nSent from App Engine at " + new Date();

 }

 log.info("Returning message \"" + message + "\"");

 return message;

 }

Also, once that C2DM registration is successful, you get a C2DM ping.  This is unlikely the right user experience for your application when completed, but it is a good, clear way to start and is very easy to update.

Now, let’s take a look at the standards based web client that is written in GWT.   You can start this up by flipping to the Development Mode tab and click on the URL there.

That will launch a web browser.  If you have not installed the GWT plugin for development mode only you will need to.  Notice that in production it is just javascript no plugin required.

Now you will see the local app engine test log in experience.  This is because we have setup the default project to require authentication.  That is down in the App Engine project under war/web.xml.  Notice this section:

 <security-constraint>

 <web-resource-collection>

    <url-pattern>/*</url-pattern>

 </web-resource-collection>

 <auth-constraint>

    <role-name>*</role-name>

 </auth-constraint>

 </security-constraint>

This indicates that all urls in this project require authentication.

You could easily add another node if you wanted some unsecured access URLs for general information, etc.  See the App Engine documentation for more information. This functionality enables you to “log in” with any email address when running in test mode.  Notice no password is required because this is just a test model.  When deployed to app engine this will be replaced with the typical Google log in dialog.   See more information.   We log in with the same email address as we used on the android client.  That will be important later when we share data.

In the GWT application, we have the same  “Say Hello” button.  Click on it does the RPC round trip to the server.   Already we are sharing identity and business logic between the android client, the GWT web front end and the App engine based server.

Sending  a C2DM Ping

Now, let’s send a C2DM ping.

Notice I use the same email address as I logged into my Android phone with.

Then I flip over to the phone and presto… A C2DM message arrives.

So that is basically what is in the starter app.  A very solid way to get started!

Want to try it live yourself?   Check out the running app here:

http://bigdaddy-io.appspot.com/

You can install the APK from this site on your phone.

Be sure to log in to the android app before you test sending C2DM pings

Tasks Data Layer and RPC Layer

Now let’s focus on building out the custom logic for our task tracking application for Larry Page.

First we will need a task entity.  We will just create a new java class in the app engine server project in the com.cloudtasks.server package.

Here is a snippet of it.. a very simple java class with no data access logic.. Just fields and accessors.

@Entity

public class Task {

 private String name;

 private String notes;

 private Date dueDate;

 @Id

 @GeneratedValue(strategy = GenerationType.IDENTITY)

 private Long id;

 private String userId;

 private String emailAddress;

 public Task () {}

 public String getName() {

    return name;

 }

Now we need to create our Service class.  This is the place we want to write our shared business logic that can be called from the GWT client and the Android client via JSON over http.

We do this with the new RPC tooling wizard in GPE 2.4.  Right click on the com.cloudtasks.server project, select New and then Other and find the “RPC Service” under the Google directory.

As you can see, the wizard looks in your server project for all the entities (marked with an attribute of “@Entity” or “@PresistenceCapable”).   The wizard will create Query + Create, Update and Delete methods for each entity you select.  You can add more custom methods, add custom params, etc in the code.    The wizard will also generate custom code in the com.cloudtasks.shared directory for calling the this service.  We will see that code a bit later.

Now, let’s flush this out a bit.  One of the beautiful parts of this system is that by the time the caller gets to one of these methods you the serialisation, authentication and validation is complete.  In other words, all the plumbing code is gone.  You simply need to implement your storage logic.  You can use any store you want from some legacy system, to S3.  But of course we will show how to use App Engine’s Datastore.

I wrote a simple data layer that uses JDO to store entities.  I called the class DataStore.

First, let’s create an instance of the DataStore:

public class CloudTasksService {

   static DataStore db = new DataStore();

Then let’s handle implementing the queryTasks() method.  As you can see, it is pretty simple.

 public static List<Task> queryTasks() {

    return db.findAll();

 }

But let’s take a look at what that db.findAll() method does.

public List<Task> findAll() {

 PersistenceManager pm = PMF.get().getPersistenceManager();

 try {

    Query query = pm.newQuery("select from " + Task.class.getName()

                              + " where emailAddress=='" + getUserEmail() + "'");

     List<Task> list = (List<Task>) query.execute();

     if (list.size() == 0) { //workaround for known issue

     list.size();

 }

 return list;

 } finally {

    pm.close();

 }

}

Notice that query..  the App Engine datastore is a highly scalable no-sql database, but you can do sql-like queries over it.  In this case, we are looking at the task items, but only those where the emailAddress of the task matches that of the currently logged on user.  So even thought we are storing all the tasks in a single datastore, a given user can only ever access their own tasks.  Notice again how the common notion of identity between app engine and Android really helps us out.

Delete is pretty much what you’d expect.  We actually do delete the item from the datastore, but you could of course use some sort of tombstoning technique if you’d like.

 public static void deleteTask(Task task) {

    db.delete(task.getId());

 }

But update is more interesting

 public static Task updateTask(Task task) {

    task.setEmailAddress(DataStore.getUserEmail());

    return db.update(task);

 }

As you can see, here we are not just updating the task, but we are also tromping over whatever email address the user sent in.  We are doing this to ensure that no one else tries to submit a task with Larry Page’s email address.  Of course our clients would not allow that, but remeber this is just Json over http, so some rouge client cloud.

The implementation of getUserEmail() is also very basic… again leveraging the power app engine knowing about google identities.

 public static String getUserEmail() {

    UserService userService = UserServiceFactory.getUserService();

    User user = userService.getCurrentUser();

    return user.getEmail();

 }

The implementation of db.update() is also very basic.  The hard work is all handed by app engine.

public Task update(Task item) {

 PersistenceManager pm = PMF.get().getPersistenceManager();

 try {

    pm.makePersistent(item);

    return item;

 } finally {

    pm.close();

 }

 }

The implementation of create and read are also very simple.

public static Task createTask() {

    return db.update(new Task());

 }

 public static Task readTask(Long id) {

    return db.find(id);

 }

Notice in createTask() we round trip the task the datastore in order to get a unique ID for it, such that update can work later.

While this was very simple, it is also very scalable…  we have app engine customers storing hundreds of millions of entities using this same pattern.

Basic GWT Web Client

OK.. I think we are ready to build the Android app out, but before we do, we hand better do a quick test of these methods.  To do so, I will make some quick modifications to the default GWT web client to simply test each of these methods.

So in com.cloudtasks.client.CloudTasks.java I added buttons for Add, Update, Query and Delete.

From the Create button, we simply call the “createTask” method, this is the client proxy for the createTask method we implemented in the App Engine project.  We call it off the TaskRequest class which was generated by the RPC tooling wizard. Notice we use a TaskProxy, that is the client proxy for the Task class in the server project.

private void create() {

 TaskRequest request = taskRequestFactory.taskRequest();

 request.createTask().fire(new Receiver<TaskProxy>() {

    @Override

    public void onSuccess(TaskProxy task) {

       Window.alert("CREATE SUCCESS:(" + task.getId() + ")" );

       taskProxy = task;

   }

  });

}

Update looks just like you’d expect

private void update(TaskProxy task) {

 TaskRequest request = taskRequestFactory.taskRequest();

 taskProxy = request.edit(taskProxy);

 taskProxy.setName(getTaskName());

 request.updateTask(task).fire(new Receiver<TaskProxy>() {

    @Override

    public void onSuccess(TaskProxy task) {

    Window.alert("UPDATE SUCCESS:(" + task.getId() + "): " + task.getName());

    }

 });

 }

Query looks pretty much like you might expect:

 private void query() {

 taskRequestFactory.taskRequest().queryTasks().fire(

 new Receiver<List<TaskProxy>>() {

    @Override

    public void onSuccess(List<TaskProxy> taskList) {

       String names = "\n";

       for (TaskProxy task : taskList) {

          names += " (" + task.getId() + "): " + task.getName() + "\n";

       }

      Window.alert("QUERY SUCCESS: Count[" + taskList.size  + "] Values:" + names);

 }

 });

Running it after adding a few more tasks for Larry looks like:

Now that we have the basic data layer and RPC service complete, it is time for Xav to come up and implement the Android client.

Basic Android Client

We have seen a very basic GWT based client, but now let’s look at how to build out the Android client.  You will notice it looks very similar to the RPC calling code we have shown above.

The CloudTasksActivity.java class contains the main activity for the application.  In onCreate() we setup the list adapter.

 TaskApplication taskApplication = (TaskApplication) getApplication();

 adapter = taskApplication.getAdapter(this);

 listView.setAdapter(adapter);

Then on onStart() we fetch the tasks, passing -1 to indicate that we need to get all the tasks

protected void onStart() {

 super.onStart();

    // only fetch task on start if the registration has happened.

    SharedPreferences prefs = Util.getSharedPreferences(mContext);

    String deviceRegistrationID = prefs.getString(Util.DEVICE_REGISTRATION_ID, null);

    if (deviceRegistrationID != null) {

    fetchTasks(-1);

 }

 }

Then in CloudTaskActivity.java we  click off an async fetch.

public void fetchTasks(long id) {

    progressBar.setVisibility(View.VISIBLE);

    if (task != null) {

      task.cancel(true);

    }

    task = new AsyncFetchTask(this);

    task.execute(id);

 }

The AsyncFetchTask fires the RPC to get the task list.

protected List<TaskProxy> doInBackground(Long... arguments) {

 final List<TaskProxy> list = new ArrayList<TaskProxy>();

 CloudTasksRequestFactory factory = Util.getRequestFactory(activity,

 CloudTasksRequestFactory.class);

 factory.taskRequest().queryTasks().fire(new Receiver<List<TaskProxy>>() {

    @Override

    public void onSuccess(List<TaskProxy> arg0) {

     list.addAll(arg0);

    }

 });

  return list;

 }

The AsyncFetchTask fires the RPC to get the task list.

protected List<TaskProxy> doInBackground(Long... arguments) {

 final List<TaskProxy> list = new ArrayList<TaskProxy>();

 CloudTasksRequestFactory factory = Util.getRequestFactory(activity,

 CloudTasksRequestFactory.class);

 factory.taskRequest().queryTasks().fire(new Receiver<List<TaskProxy>>() {

    @Override

     public void onSuccess(List<TaskProxy> arg0) {

     list.addAll(arg0);

 }

 });

 return list;

 }

The AsyncFetchTask fires the RPC to get the task list.

protected List<TaskProxy> doInBackground(Long... arguments) {

 final List<TaskProxy> list = new ArrayList<TaskProxy>();

CloudTasksRequestFactory factory = Util.getRequestFactory(activity,

 CloudTasksRequestFactory.class);

 factory.taskRequest().queryTasks().fire(new Receiver<List<TaskProxy>>() {

    @Override

    public void onSuccess(List<TaskProxy> arg0) {

      list.addAll(arg0);

    }

 });

 return list;

 }

And we run it and it looks great!

Nice GWT Client

Wow, that Android client looks great, let’s see what we can do to make the GWT client look good as well.

A few updates to HTML and CSS in the CloudTasks-AppEngine\war folder and some updates to the GWT client code in CloudTasks-AppEngine\com.cloudtasks.client, we get a nice looking web client that can be accessed from anywhere.

One tip for GWT development…  You don’t need to stop and start GWT Development mode.  Simply hit refresh in the browser and see your changes.

Now, we add a new task, but that task does not show up in the Android client until the user manually refreshes it.    That is not great – Larry could easily miss tasks he adds via the web page.   We could make the Android client poll the server every few minutes for new tasks, but that is a huge battery and network drain on the device, and creates unnecessary load on the server.

Cloud to Device Messaging to the rescue!  C2DM allows you send a ping the android client to let it know there are new tasks to pick up.   You can’t send a lot of data over this channel, so we will just send the ID of the task to that was updated.

Luckily we have a very convenient place to add this logic on the server.  Let’s look at CloudTasksService.java in the CloudTasks-AppEngine\com.cloudtasks.server directory.

 public static Task updateTask(Task task) {

    task.setEmailAddress(DataStore.getUserEmail());

    task =  db.update(task);

    DataStore.sendC2DMUpdate(TaskChange.UPDATE +
    TaskChange.SEPARATOR + task.getId());

 return task;

 }

First, check out that sendC2DMUpdate() method.  The default project makes it very easy to send a C2DM message.

 public static void sendC2DMUpdate(String message) {

    UserService userService = UserServiceFactory.getUserService();

    User user = userService.getCurrentUser();

    ServletContext context = RequestFactoryServlet.getThreadLocalRequest().getSession().getServletContext();

    SendMessage.sendMessage(context, user.getEmail(), message);

}

We call the SendMessage class passing the email address of the owner of the task that was updated.  Recall that we verified this was not spoofed when we updated the task.

For the messaging, you can see we are using the TaskChange class that we define in the Shared directory.  This ensures that the App Engine server and Android client projects share the exact same definition of this class.

To run this, we need to reload the App Engine development time server by clicking the refresh button in the Development Mode tab.

Now, when we run this and add a new task, we get a ping on the Android client.

That is very cool, but the UI isn’t quite right here.  We really want to just trigger an RPC call for the task when the C2DM message arrives rather than notifying the user.   Luckily that is very easy to do.

In the Android project, in C2DMReceiver.java that the default project generated, we want to remove the notifications message and instead handling it programmatic. To do that we replace the onMessage() implementation as follows.

 public void onMessage(Context context, Intent intent) {

    TaskApplication app = (TaskApplication) getApplication();

    app.notifyListener(intent);

 }

The in TaskApplication.java we handle the notification. Notice we use the TaskChange class from the shared code directory that we also used in the App Engine project.   To handle it we simply call onTaskUpdated() passing the message which includes the ID of the item being updated.

 public void notifyListener(Intent intent) {

    if (listener != null) {

       Bundle extras = intent.getExtras();

       if (extras != null) {

          String message = (String) extras.get("message");

          String[] messages = message.split(Pattern.quote(TaskChange.SEPARATOR));

          listener.onTaskUpdated(messages[0], Long.parseLong(messages[1]));

       }

    }

 }

Finally, we do the actual update:

 public void onTaskUpdated(final String message, final long id) {

    runOnUiThread(new Runnable() {

    public void run() {

       if (TaskChange.UPDATE.equals(message)) {

         fetchTasks(id);

       }

   }

 });

Now we can see when I add a new task via the web front end, I get the wait icon while an async fetch of that task is done.

then magically the new task just shows up with the minimum of battery life and network.

And of course, this works two way… when you add a new task from the android phone, the browser app automatically updates without a refersh.

Deployment

The final step here is to deploy your application to the production app engine.  This is very easy to do from eclipse.

First you go to https://appengine.google.com/ and create an App ID.  Go ahead it is free to get started!

Next go to the Android project and find the Setup.java file and update the App name to match the one you just created.

public class Setup {

 /**

 * The AppEngine app name, used to construct the production service URL

 * below.

 */

 private static final String APP_NAME = "androidcloudtasks";

Then you click on the App Engine button and select the App Engine Project to deploy.

Next click on App Engine project settings and verify that the correct Application ID is set.   Notice you can also change the version string.   You can use things like “beta” or “testing” as well as numbers.  This helps with the staging, testing, release process.

Then you click deploy.   It might take a while for the project to build.  GWT creates very optimized  javascript for different classes of browsers.   Once it is done you should be able to browse to http://yourappid.appspot.com.

Notice that asks you to log in.   This is for real, unlike the test client.  So you need to give a valid google ID.

Next, App Engine verifies that the user is giving permission for your application to use their email address.

Finally, we see the application running.   Notice that you will not see any data at first as this is running against the live App Engine server.  You will need to add some tasks here… but once you do you can access them from any web browser anywhere and a nice rich client on your phone!

Summary

Wow – that was a whole load of stuff.   Just to review:

1. We created a new App Engine connected Android project with built in support for C2DM and RPC.

2. We add a data layer in App Engine for storing tasks

3. Exposed it over RPC

4. Built an Android client for viewing and updating tasks

5. Built a standards based GWT client for viewing and updating tasks

6. plumbed in C2DM for doing push notifications of new tasks

All in less than 45 mins!    Imagine what you can do!

We’d love your feedback on BigDaddy (GPE 2.4)…  Please post the the GWT forums or comment here.

Over the years I have collected a few tips on how to run software projects.  Many of these are from my days at Microsoft, but I hope to be adding some soon from my (new) days at Google.

• PM Tip #10: Interview Tips for Landing a Great Program Management Job
• PM tip# 11: Information is the Currency of Program Management
• PM Tip #14: Great teams have members that defy roles
• PM Tip #12: Don’t waste keystrokes
• PM Tip #21: Sanity Will Prevail
• PM Tip # 32: Nothing kills excitement like ambiguity
• PM Tip #57: Write the agenda on the board
• PM Tip #72 Instant feedback…
• PM Tip #73: The “Be More Visible” Sham
• PM Tip  #78: Reveal your stupidity

Some bonus material:

• Presentation tip: They Remember the Mistakes more than the Polish
• Building Consensus
• That is a full day’s work!
• The What and the How
• Thoughts on writing reviews…
• The Wisdom of Crowds: Rethinking consensus

“Going Google” is the phrase companies use when they choose to move over to Google Apps for some of their IT needs.  I will be "going Google" on a more personal level — I have accepted a position in Product Management at Google! We are still working on the details of exactly what I will be working on but it will be something in cloud/web development space and I will be based here in Kirkland/Seattle.

Why did I make this transition?

I needed a change personally, to learn something new, see the tech world from a fresh perspective. And what better place than Google! As I look at where things are going in the industry (and where I think they should go), it is clear that the web, and cloud+devices specifically, are the dominant trends of the future. Working at Google will give me an opportunity to have a deep and meaningful impact on that future.

Google has benefited more than any other company from the rapid growth of the web and as such they have a responsibility to help the web grow to the next level. I am excited to see what that will look like: from the ubiquity of the rich web with HTML5 to the maturing of the cloud as an integral part of every company’s IT portfolio to open, powerful personal devices to help us navigate it all anywhere. I am excited to work at Google to help make the future arrive sooner

As far as exactly what I will do – I don’t know yet. But I have a huge passion for building and releasing software and love the developer space. I am super excited that my first week at Google will be spent at Google I/O. This conference has been sold out for weeks, so I guess the only way you can attend now is to join Google .  If you are going, I’d love to connect.  I clearly have a lot to learn! 

This is obviously a HUGE change for me and I’d love to hear any advice or thoughts you have as I make the transition to being a Noogler (a new Google employee).

Only a few hours into my time off and I accomplished my first goal.   I finished reading Beggars in Spain today.   Clearly a great book that really made me think.   It has the basic scifi stuff I like such as a view of the future, genetic engineering, a new form of cheap energy and its effect on the economy.   

The book starts by proposing an interesting question:  are we wasting time sleeping.   Personally, I love to sleep, but it does feel like a waste of almost 1/3 of our lives sometimes.  

But Beggars in Spain is not about the scifi nor this “sleeper” question.  The books heart is about how the strong relate to the weak.  Do the weak have any claims on the strong?  Does a poor beggar in Spain have any right to ask for money or help from a the rich?    Oddly enough, I found my free-market economy, The Wealth of Nations, roots agreeing with one of the main characters that “the weak have no claims on the strong”.  But as they find out, this is a shallow argument.  My personal view is while the weak may have no claims on the strong, the strong DO have an obligation on the weak.  Not only because the weak may one day be strong (as the book argues) but because we are all created by God and of certain inalienable value. 

Enjoy!

Look for more information about what I will be doing next….