Please note that the contents of this offline web site may be out of date. To access the most recent documentation visit the online version .
Note that links that point to online resources are green in color and will open in a new window.
We would love it if you could give us feedback about this material by filling this form (You have to be online to fill it)



Using the Datastore

Learning objectives
  • Learn how to build and deploy an App Engine app, a simple guestbook
Prerequisites
Related

Amy Unruh, Dan Sanderson, Oct 2012
Google Developer Relations

Introduction

Storing data in a scalable web application can be tricky. A user could be interacting with any of dozens of web servers at a given time, and the user's next request could go to a different web server than the previous request. All web servers need to be interacting with data that is also spread out across dozens of machines, possibly in different locations around the world.

With Google App Engine, you don't have to worry about any of that. The App Engine Datastore is a schemaless object datastore providing robust, scalable storage for your web application, with the following features:

  • No planned downtime
  • Atomic transactions
  • High availability of reads and writes
  • Strong consistency for reads and ancestor queries
  • Eventual consistency for all other queries

The Python Datastore interface includes rich data modeling APIs and a SQL-like query language called GQL.

As part of App Engine's infrastructure, the Datastore takes care of all distribution, replication, and load balancing of data—and provides a powerful query engine and transactions as well. App Engine includes two data modeling APIs for Python, which use the Datastore behind the scenes. In this course, we'll use the newer of these, the NDB API ; this lesson shows you how to use it to persist and manage the guestbook application's data.

The High Replication Datastore and Entities

Before we look at the code, let's introduce some Datastore concepts.

App Engine's High Replication Datastore (HRD) uses the Paxos algorithm to replicate data across multiple data centers.

Data is written to the Datastore in objects known as entities . Each Datastore entity is of a particular kind , which categorizes the entity for the purpose of queries. Each entity has a key that uniquely identifies it. The key consists of the kind of the entity; an identifier for the individual entity; and an optional ancestor path locating the entity within the Datastore hierarchy. More specifically, an entity can optionally designate another entity as its parent; the first entity is a child of the parent entity. The entities in the Datastore thus form a hierarchically structured space similar to the directory structure of a file system. An entity's parent, parent's parent, and so on recursively, are its ancestors; its children, children's children, and so on, are its descendants. An entity without a parent is a root entity.

The Datastore is extremely resilient in the face of catastrophic failure, but its consistency guarantees may differ from what you're familiar with. An entity and its descendants are said to belong to the same entity group; the common ancestor's key is the group's parent key, which serves to identify the entire group. Queries over a single entity group, called ancestor queries , refer to the parent key. Entity groups are a unit of both consistency and transactionality: whereas queries over multiple entity groups may return stale, eventually consistent results, those limited to a single entity group always return up-to-date, strongly consistent results.

The code in this lesson organizes related entities into entity groups, and uses ancestor queries on those entity groups to return strongly consistent results. We’ll briefly discuss some application design considerations related to this organization. For more detailed information, see Structuring Data for Strong Consistency in the App Engine documentation.

A Complete Example Using the Datastore

Here is a new version of helloworld/main.py that stores greetings in the Datastore. The rest of this lesson and the next will walk through the new pieces.

import cgi
import urllib
import webapp2

from google.appengine.ext import ndb
from google.appengine.api import users

class Greeting(ndb.Model):
  """Models an individual guestbook entry with author, content, and date."""
  author = ndb.UserProperty()
  content = ndb.StringProperty()
  date = ndb.DateTimeProperty(auto_now_add=True)

  @classmethod
  def query_book(cls, ancestor_key):
    return cls.query(ancestor=ancestor_key).order(-cls.date)

class MainPage(webapp2.RequestHandler):
  def get(self):
    self.response.out.write('<html><body>')
    guestbook_name = self.request.get('guestbook_name')
    # There is no need to actually create the parent Book entity; we can
    # set it to be the parent of another entity without explicitly creating it
    ancestor_key = ndb.Key("Book", guestbook_name or "*notitle*")
    greetings = Greeting.query_book(ancestor_key).fetch(20)

    for greeting in greetings:
      if greeting.author:
        self.response.out.write(
            '<b>%s</b> wrote:' % greeting.author.nickname())
      else:
        self.response.out.write('An anonymous person wrote:')
      self.response.out.write('<blockquote>%s</blockquote>' %
                              cgi.escape(greeting.content))

    self.response.out.write("""
          <form action="/sign" method="post">
            <input type="hidden" value="%s" name="guestbook_name">
            <div><textarea name="content" rows="3" cols="60"></textarea></div>
            <div><input type="submit" value="Sign Guestbook"></div>
          </form>
          <hr>
          <form>Guestbook name: <input value="%s" name="guestbook_name">
          <input type="submit" value="switch"></form>
        </body>
      </html>""" % (cgi.escape(guestbook_name), cgi.escape(guestbook_name)))

class Guestbook(webapp2.RequestHandler):
  def post(self):
    # Set parent key on each greeting to ensure that each
    # guestbook's greetings are in the same entity group.
    guestbook_name = self.request.get('guestbook_name')
    # There is no need to actually create the parent Book entity; we can
    # set it to be the parent of another entity without explicitly creating it
    greeting = Greeting(parent=ndb.Key("Book", guestbook_name or "*notitle*"),
                        content = self.request.get('content'))
    if users.get_current_user():
      greeting.author = users.get_current_user()
    greeting.put()
    self.redirect('/?' + urllib.urlencode({'guestbook_name': guestbook_name}))

app = webapp2.WSGIApplication([
  ('/', MainPage),
  ('/sign', Guestbook)
])

Replace helloworld/main.py with this file, then reload http://localhost:8080/ in your browser. Post a few messages to verify that messages get stored and displayed correctly.

Now, let's look at what's going on with this new code.

Storing the Submitted Greetings

For the guestbook application, we want to store greetings posted by users. Each greeting includes the author's name, the message content, and the date and time the message was posted, so we can display messages in chronological order.

To use NDB , import the google.appengine.ext.ndb module:

from google.appengine.ext import ndb

class Greeting(ndb.Model):
  """Models an individual guestbook entry with author, content, and date."""
  author = ndb.UserProperty()
  content = ndb.StringProperty()
  date = ndb.DateTimeProperty(auto_now_add=True)

This code defines an NDB Greeting model with three properties: author , whose value is a User object; content , whose value is a string; and date , whose value is a datetime.datetime .

Some property constructors take parameters to further configure their behavior. Giving the ndb.DateTimeProperty constructor an auto_now_add=True parameter configures the model to automatically stamp new objects with the date and time the object is created, if the application doesn't otherwise provide a value. For a complete list of property types and their options, see the NDB Properties page.

Now that we have a data model for greetings, the application can use the model to create new Greeting objects and put them into the Datastore. The following new version of the Guestbook handler creates new greetings and saves them to the Datastore:

class Guestbook(webapp2.RequestHandler):
  def post(self):
    # Set the parent key on each greeting to ensure that each
    # guestbook's greetings are in the same entity group.
    guestbook_name = self.request.get('guestbook_name')
    # There is no need to actually create the parent Book entity; we can
    # set it to be the parent of another entity without explicitly creating it
    greeting = Greeting(parent=ndb.Key("Book", guestbook_name or "*notitle*"),
                        content = self.request.get('content'))
    if users.get_current_user():
      greeting.author = users.get_current_user()
    greeting.put()
    self.redirect('/?' + urllib.urlencode({'guestbook_name': guestbook_name}))

This new Guestbook handler creates a new Greeting object, then sets its author and content properties with the data posted by the user. The parent of the new object is a Datastore entity of kind Book , with ID generated from the guestbook name. (There is no need to actually create the parent Book entity; we can set it to be the parent of another entity without explicitly creating it). This parent Book entity is being used as a placeholder for transaction and consistency purposes. See the Transactions page in the documentation for more information. Objects that share a common ancestor form an entity group, so setting the same parent for each Greeting object assigns all greetings to the same entity group. We do this because querying in the High Replication Datastore is strongly consistent only within entity groups. This ensures that a user will always see a greeting immediately after it was written. We'll discuss consistency a bit further below.

The example code doesn't set the date property, so date is automatically set to the time the entity is saved, as we configured the model to do.

Finally, greeting.put() saves our new object to the Datastore. If we instead had been accessing an existing object, put() would have updated the existing object. Here, since we created this object with the model constructor, put() adds the new object to the Datastore.

Review and Summary

In this lesson, we've shown how to use the App Engine Datastore's NDB interface to persist Greeting objects to the Datastore.

In the next lesson , we'll show how to query for Greetings, and introduce Datastore indexes .

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.