Nick Johnson
September 2009
, updated
January 2013
Introduction
With the introduction of the XMPP service to App Engine, it's now possible to write an App Engine app that communicates with users - or even other applications - over XMPP. XMPP is an instant-messaging protocol, used by Google Talk, Jabber, and other IM networks. In this article, we're going to walk through an example that covers all the basic functionality of App Engine's XMPP API.
For our example app, we're going to write the Amazing Crowd Guru. The Amazing Crowd Guru is a veritable oracle, who can answer any question you might pose it over XMPP. Writing an omniscient computer program is no small task, but thanks to a little behind-the-scenes trickery, we're going to get our users to do all the work of answering questions for us.
The basic sequence of events will go like this:
-
A user adds
[email protected]to their buddy list in Google Talk, or another XMPP client. - The user asks the Amazing Crowd Guru a question, by typing " /tellme Does a duck's quack echo? "
- Our code receives the question, stores it in the datastore as an unanswered question, then looks in the datastore for another unanswered question. If it finds one, it sends it back to the user, saying " While I'm thinking, perhaps you can answer me this: If a mole can dig a mole of holes, how many moles of holes can a mole of moles dig? "
- The user thinks a bit, and replies " A mole of moles can dig a mole of holes! "
- Our code receives the user's answer, stores it in the datastore alongside the original question, and then sends it back to the user who originally asked that question.
In addition to the basic flow above, we'll add a couple of enhancements: A user can type "/help" to ask the Guru for a quick overview of what they can type, and they can type "/askme", to be asked a question by the Guru, without having to ask one of their own. We'll also suspend questions for users who are offline, to ensure that answers are delivered to users who are available.
Getting Started
First, we need to set up a few basic settings. Create a directory for your app, and create your configuration file:
Python
app.yaml
application: crowdguru-hrd version: 1 runtime: python27 api_version: 1 threadsafe: true handlers: - url: /static static_dir: static - url: /.* script: guru.APPLICATION inbound_services: - xmpp_message - xmpp_presence libraries: - name: jinja2 version: latest
Java
war/WEB-INF/appengine-web.xml
<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<application>crowdguru</application>
<version>1</version>
<threadsafe>true</threadsafe>
<system-properties>
<property name="java.util.logging.config.file" value="WEB-INF/logging.properties" />
</system-properties>
<inbound-services>
<service>xmpp_message</service>
<service>xmpp_presence</service>
</inbound-services>
</appengine-web-app>
If you're planning to deploy your version of the app to App Engine,
you'll want to replace
crowdguru
with another app ID that
you've created.
Next, if you're building your application in Python, create a handler script
called
guru.py
and start it out with the necessary imports:
Python
guru.py
import datetime from google.appengine.api import datastore_types from google.appengine.api import xmpp from google.appengine.ext import ndb from google.appengine.ext.webapp import xmpp_handlers import webapp2 from webapp2_extras import jinja2
All of this should seem very familiar from writing previous apps, so we won't go into much detail describing it. Note that we haven't defined any models or handlers yet - we'll do that next.
Creating a Model
Before we can do anything useful, we'll need a datastore model class to
store questions and answers in. If you're building your application in Python, add the following to
guru.py
immediately after the import statements. If you're building in Java, create a new domain class,
Question.java
, with the following:
Python
guru.py
class Question(ndb.Model):
"""Model to hold questions that the Guru can answer."""
question = ndb.TextProperty(required=True)
asker = IMProperty(required=True)
asked = ndb.DateTimeProperty(required=True, auto_now_add=True)
suspended = ndb.BooleanProperty(required=True)
assignees = IMProperty(repeated=True)
last_assigned = ndb.DateTimeProperty()
answer = ndb.TextProperty(indexed=True)
answerer = IMProperty()
answered = ndb.DateTimeProperty()
Note:
The property class
IMProperty
is
defined in the source for this application.
Java
src/Question.java
// Java JDO
import com.google.appengine.api.datastore.Key;
import com.google.appengine.api.datastore.Text;
import java.util.Date;
import javax.jdo.annotations.IdGeneratorStrategy;
import javax.jdo.annotations.IdentityType;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;
@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class Question {
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private Key key;
@Persistent(defaultFetchGroup = "true")
private Text question;
@Persistent
private String asker;
@Persistent
private Date asked;
@Persistent(defaultFetchGroup = "true")
private Text answer;
@Persistent
private String answerer;
@Persistent
private Date answered;
@Persistent
private Boolean suspended;
public Key getKey() {
return key;
}
// ...
We've omitted a couple of helper methods here to save space, including several Python functions dealing with assigning questions to a user in a transaction-safe manner. If you want to see all the gory details, the source is freely available and linked from here .
Responding to XMPP messages
Finally, we're ready to tackle the meat of the problem: Receiving and responding to XMPP messages sent by users. As described in the introduction, we're using a pattern fairly common to 'bots': Users send us either plain text messages, or commands prefixed with a "/" character. Messages are interpreted based on the command specified and the (optional) argument.
The Python environment includes some convenience classes
to make writing XMPP bots easier, namely
CommandHandler
in
google.appengine.ext.webapp.xmpp_handlers
. This class
implements a "command dispatch" pattern: messages prefixed with "/foo"
are handled by a method named
foo_command
, messages without
a prefix are handled by
text_message
, and any messages with
unknown prefixes are handled by
unhandled_command
which
sends a message back to the sender saying "Unknown command".
While the Java environment does not have a special handler for dispatching XMPP commands, its XMPP API includes a handy
parseMessage
method for creating a new
Message
object from HTTP request data, and from there it's straightforward to inspect the body of the message to determine how to handle the command.
With these techniques, we can write the code to handle incoming requests to the Guru and send responses fairly simply:
Python
guru.py
def bare_jid(sender):
return sender.split('/')[0]
class XmppHandler(xmpp_handlers.CommandHandler):
"""Handler class for all XMPP activity."""
def tellme_command(self, message=None):
"""Handles /tellme requests, asking the Guru a question.
Args:
message: xmpp.Message: The message that was sent by the user.
"""
im_from = datastore_types.IM('xmpp', bare_jid(message.sender))
asked_question = Question.get_asked(im_from)
if asked_question:
# Already have a question
message.reply(WAIT_MSG)
else:
# Asking a question
asked_question = Question(question=message.arg, asker=im_from)
asked_question.put()
currently_answering = Question.get_answering(im_from)
if not currently_answering:
# Try and find one for them to answer
question = Question.assign_question(im_from)
if question:
message.reply(TELLME_MSG.format(question.question))
return
message.reply(PONDER_MSG)
Java
src/XmppReceiverServlet.java
import com.google.appengine.api.xmpp.JID;
import com.google.appengine.api.xmpp.Message;
import com.google.appengine.api.xmpp.MessageBuilder;
import com.google.appengine.api.xmpp.MessageType;
import com.google.appengine.api.xmpp.XMPPService;
import com.google.appengine.api.xmpp.XMPPServiceFactory;
import java.io.IOException;
import java.util.Date;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
/**
* Handler class for all XMPP messages.
*/
public class XmppReceiverServlet extends HttpServlet {
private static final XMPPService xmppService =
XMPPServiceFactory.getXMPPService();
@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
Message message = xmppService.parseMessage(req);
if (message.getBody().startsWith("/askme")) {
handleAskMeCommand(message);
} else if (message.getBody().startsWith("/tellme ")) {
String questionText = message.getBody().replaceFirst("/tellme ", "");
handleTellMeCommand(message, questionText);
} else if (message.getBody().startsWith("/")) {
handleUnrecognizedCommand(message);
} else {
handleAnswer(message);
}
}
/**
* Handles /tellme requests, asking the Guru a question.
*/
private void handleTellMeCommand(Message message, String questionText) {
QuestionService questionService = new QuestionService();
JID sender = message.getFromJid();
Question previouslyAsked = questionService.getAsked(sender);
if (previouslyAsked != null) {
// Already have a question, and they're not answering one
replyToMessage(message, "Please! One question at a time! You can ask " +
"me another once you have an answer to your current question.");
} else {
// Asking a question
Question question = new Question();
question.setQuestion(questionText);
question.setAsked(new Date());
question.setAsker(sender);
questionService.storeQuestion(question);
// Try and find one for them to answer
Question assigned = questionService.assignQuestion(sender);
if (assigned != null) {
replyToMessage(message, "While I'm thinking, perhaps you can " +
"answer me this: " + assigned.getQuestion());
} else {
replyToMessage(message, "Hmm. Let me think on that a bit.");
}
}
}
// ...
private void replyToMessage(Message message, String body) {
Message reply = new MessageBuilder()
.withRecipientJids(message.getFromJid())
.withMessageType(MessageType.NORMAL)
.withBody(body)
.build();
xmppService.sendMessage(reply);
}
}
This is the core functionality: how users ask the Guru a question. We've
left out the
get_asked()
(Python) and
getAsked
(Java) methods here for brevity; they are fairly straightforward methods
that query the datastore for the question the user asked (if any). If
you're interested, you can see the full, unmodified code, linked from
here
.
As you can see, the actual functionality of using XMPP is very straightforward. In Python, received messages result in a call to the appropriate function which is passed a message object. In Java, the message object is parsed directly from the posted request. The message object encapsulates information about the sender of the message, the JID (Jabber ID) it was sent to, and the body of the message. The Python implementation also provides convenience methods to parse the body of the message into command and argument portions (if it is of that form), and to reply to the message.
In this case, the only user we're sending messages to is the sender of the
message we're handling, so everything is very straightforward - we just call
message.reply()
with the text we want to send back to the user.
Next, we need a way for the user to send us their answer to a question. They do this by simply typing the answer, with no /command prefix:
Python
guru.py
def text_message(self, message=None):
"""Called when a message not prefixed by a /cmd is sent to the XMPP bot.
Args:
message: xmpp.Message: The message that was sent by the user.
"""
im_from = datastore_types.IM('xmpp', bare_jid(message.sender))
question = Question.get_answering(im_from)
if question:
other_assignees = question.assignees
other_assignees.remove(im_from)
# Answering a question
question.answer = message.arg
question.answerer = im_from
question.assignees = []
question.answered = datetime.datetime.now()
question.put()
# Send the answer to the asker
xmpp.send_message([question.asker.address],
ANSWER_INTRO_MSG.format(question.question))
xmpp.send_message([question.asker.address],
ANSWER_MSG.format(message.arg))
# Send acknowledgement to the answerer
asked_question = Question.get_asked(im_from)
if asked_question:
message.reply(TELLME_THANKS_MSG)
else:
message.reply(THANKS_MSG)
# Tell any other assignees their help is no longer required
if other_assignees:
xmpp.send_message([user.address for user in other_assignees],
SOMEONE_ANSWERED_MSG)
else:
self.unhandled_command(message)
Java
src/XmppReceiverServlet.java
// ...
/**
* Handles answers to questions we've asked the user.
*/
private void handleAnswer(Message message) {
QuestionService questionService = new QuestionService();
JID sender = message.getFromJid();
Question currentlyAnswering = questionService.getAnswering(sender);
if (currentlyAnswering != null) {
// Answering a question
currentlyAnswering.setAnswer(message.getBody());
currentlyAnswering.setAnswered(new Date());
currentlyAnswering.setAnswerer(sender);
questionService.storeQuestion(currentlyAnswering);
// Send the answer to the asker
sendMessage(currentlyAnswering.getAsker(), "I have thought long and " +
"hard, and concluded: " + currentlyAnswering.getAnswer());
// Send acknowledgment to the answerer
Question previouslyAsked = questionService.getAsked(sender);
if (previouslyAsked != null) {
replyToMessage(message, "Thank you for your wisdom. I'm still " +
"thinking about your question.");
} else {
replyToMessage(message, "Thank you for your wisdom.");
}
} else {
handleUnrecognizedCommand(message);
}
}
// ...
private void sendMessage(String recipient, String body) {
Message message = new MessageBuilder()
.withRecipientJids(new JID(recipient))
.withMessageType(MessageType.NORMAL)
.withBody(body)
.build();
xmppService.sendMessage(message);
}
This is a little longer than
tellme_command
, but still fairly easy to
follow. First we look up the question the user is answering (again, we've
omitted the
get_answering
(Python) and
getAnswering
(Java) methods for brevity). If they're not answering a
question, we call
unhandled_command()
(Python) or
handleUnrecognizedCommand
(Java), which prints help text.
Otherwise, we record the answer, send it back to the person who originally
asked the question, and send an acknowledgement to the answerer. Note the
bit where we send the answer to the original asker: This demonstrates using
xmpp.send_message
(Python) to send a message to a JID other than the sender of the
message we're currently handling. In Java, we've added a custom method for sending a message to an arbitrary sender.
If a user asks a question and then goes offline, we shouldn't bother answering their question until they're around to hear the answer. We can track this using presence handlers.
Python
guru.py
class XmppPresenceHandler(webapp2.RequestHandler):
"""Handler class for XMPP status updates."""
def post(self, status):
"""POST handler for XMPP presence.
Args:
status: A string which will be either available or unavailable
and will indicate the status of the user.
"""
sender = self.request.get('from')
im_from = datastore_types.IM('xmpp', bare_jid(sender))
suspend = (status == 'unavailable')
query = Question.query(Question.asker == im_from,
Question.answer == None,
Question.suspended == (not suspend))
question = query.get()
if question:
question.suspended = suspend
question.put()
Java
@Override
public void doPost(HttpServletRequest request, HttpServletResponse response)
throws IOException {
Presence presence = xmppService.parsePresence(request);
JID sender = presence.getFromJid();
QuestionService questionService = new QuestionService();
if (presence.getPresenceType() == PresenceType.AVAILABLE) {
questionService.setSuspended(sender, true);
} else if (presence.getPresenceType() == PresenceType.UNAVAILABLE) {
questionService.setSuspended(sender, false);
}
}
When we are notified that a user has gone offline, we search our database for an unanswered question that they have asked. If one exists, we mark it as "suspended" and don't give it to other users to answer. When the user comes back online, we un-suspend the question so it's available to be answered again.
Finally, it would be nice if users have a way to answer questions without having to ask one first; this way, dedicated users can help clear out any backlog of unanswered questions:
Python
guru.py
def askme_command(self, message=None):
"""Responds to the /askme command.
Args:
message: xmpp.Message: The message that was sent by the user.
"""
im_from = datastore_types.IM('xmpp', bare_jid(message.sender))
currently_answering = Question.get_answering(im_from)
question = Question.assign_question(im_from)
if question:
message.reply(TELLME_MSG.format(question.question))
else:
message.reply(EMPTYQ_MSG)
# Don't unassign their current question until we've picked a new one.
if currently_answering:
currently_answering.unassign(im_from)
Java
src/XmppReceiverServlet.java
// ...
/**
* Handles requests to be asked a question without providing one.
*/
private void handleAskMeCommand(Message message) {
QuestionService questionService = new QuestionService();
JID sender = message.getFromJid();
Question newlyAssigned = questionService.assignQuestion(sender);
if (newlyAssigned != null) {
replyToMessage(message, "While I'm thinking, perhaps you can answer " +
"me this: " + newlyAssigned.getQuestion());
} else {
replyToMessage(message, "Sorry, I don't have anything to ask you at " +
"the moment.");
}
}
// ...
This method is even simpler - grab a question and send it back to the user.
In fact, it's pretty much just a subset of the code we saw in the
tellme_command()
method.
There's one last thing we need to do to get this all working, of course -
hook it up to the serving infrastructure so it can serve requests.
In Python, a
CommandHandler
is a standard webapp
RequestHandler
subclass, so we can set it up as we would any
other handler. In Java, you can wire up the XMPP receiving servlet in your
deployment descriptor.
Python
guru.py
APPLICATION = webapp2.WSGIApplication([
('/_ah/xmpp/message/chat/', XmppHandler),
('/_ah/xmpp/presence/(available|unavailable)/', XmppPresenceHandler),
], debug=True)
Java
war/WEB-INF/web.xml
<?xml version="1.0" encoding="utf-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
version="2.5">
<servlet>
<servlet-name>xmppreceiver</servlet-name>
<servlet-class>com.google.appengine.demos.crowdguru.web.XmppReceiverServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>xmppreceiver</servlet-name>
<url-pattern>/_ah/xmpp/message/chat/</url-pattern>
</servlet-mapping>
</web-app>
The URL path
/_ah/xmpp/message/chat
is reserved for XMPP messages to be sent to.
Similarly, the paths
/_ah/xmpp/presence/available/
and
/_ah/xmpp/presence/unavailable
are reserved for XMPP presence notifications.
Presto! We have a working (and amusing) XMPP bot! For the full source code, including the bits we left out for space reasons, see the next section.
Source
The
Python source
for the full implementation of the
crowdguru
sample application
is available in our samples repository.