Sign in

1. Products
2. Google Cloud Platform
3. Google App Engine
4. Go

Go

Feedback on this document

Entities, Properties, and Keys

Data objects in the App Engine Datastore are known as entities . An entity has one or more named properties , each of which can have one or more values. Entities of the same kind need not have the same properties, and an entity's values for a given property need not all be of the same data type. (If necessary, an application can establish and enforce such restrictions in its own data model.)

The Datastore supports a variety of data types for property values . These include, among others:

• Integers
• Floating-point numbers
• Strings
• Dates
• Binary data

Each entity in the Datastore has a key that uniquely identifies it. The key consists of the following components:

• The namespace of the entity, which allows for multitenancy
• The kind of the entity, which categorizes it for the purpose of Datastore queries
• An identifier for the individual entity, which can be either
  • a key name string
  • an integer numeric ID
• An optional ancestor path locating the entity within the Datastore hierarchy

An application has access only to entities it has created itself; it can't access data belonging to other applications. It can fetch an individual entity from the Datastore using the entity's key, or it can retrieve one or more entities by issuing a query based on the entities' keys or property values.

The Go App Engine SDK includes a package for representing Datastore entities as Go structs, and for storing and retrieving them in the Datastore.

The Datastore itself does not enforce any restrictions on the structure of entities, such as whether a given property has a value of a particular type; this task is left to the application.

Contents

1. Kinds and identifiers
2. Ancestor paths
3. Transactions and entity groups
4. Properties and value types
5. Working with entities
   1. Creating an entity
   2. Retrieving an entity
   3. Updating an entity
   4. Deleting an entity
   5. Batch operations
6. Understanding write costs

Kinds and identifiers

Each Datastore entity is of a particular kind, which categorizes the entity for the purpose of queries: for instance, a human resources application might represent each employee at a company with an entity of kind Employee . In the Go Datastore API, you specify an entity's kind when you create a datastore.Key . All kind names that begin with two underscores ( __ ) are reserved and may not be used.

The following example creates an entity of kind Employee , populates its property values, and saves it to the Datastore:

import (
    "time"

    "appengine/datastore"
)

type Employee struct {
    FirstName          string
    LastName           string
    HireDate           time.Time
    AttendedHRTraining bool
}

func f(...) {
    // ...
    employee := &Employee{
        FirstName: "Antonio",
        LastName:  "Salieri",
        HireDate:  time.Now()
    }
    employee.AttendedHRTraining = true

    key := datastore.NewIncompleteKey(c, "Employee", nil)
    _, err := datastore.Put(c, key, employee)
    // ...
}

The Employee type declares four fields for the data model: FirstName , LastName , HireDate , and AttendedHRTraining .

In addition to a kind, each entity has an identifier , assigned when the entity is created. Because it is part of the entity's key, the identifier is associated permanently with the entity and cannot be changed. It can be assigned in either of two ways:

• Your application can specify its own key name string for the entity.
• You can have the Datastore automatically assign the entity an integer numeric ID .

To assign an entity a key name, provide a non-empty stringID argument to datastore.NewKey :

// Create a key with a key name "asalieri".
key := datastore.NewKey(
    c,           // appengine.Context
    "Employee",  // Kind
    "asalieri",  // String ID; empty means no string ID
    0,           // Integer ID; if 0, generate automatically. Ignored if string ID specified.
    nil          // Parent Key; nil means no parent
)

To have the Datastore assign a numeric ID automatically, use an empty stringID argument:

// Create a key such as Employee:8261.
key := datastore.NewKey(c, "Employee", "", 0, nil)
// This is equivalent:
key := datastore.NewIncompleteKey(c, "Employee", nil)

Assigning identifiers

The Datastore can be configured to generate auto IDs using two different auto id policies :

• The default policy generates a random sequence of IDs that are approximately uniformly distributed. Each ID can be up to 16 decimal digits long.
• The legacy policy creates a sequence of non-consecutive smaller integer IDs.

If you want to display the entity IDs to the user, and/or depend upon their order, the best thing to do is use manual allocation.

Datastore generates a random sequence of IDs that are approximately uniformly distributed. Each ID can be up to 16 decimal digits long.

Ancestor paths

Entities in the Datastore form a hierarchically structured space similar to the directory structure of a file system. When you create an entity, you can optionally designate another entity as its parent; the new entity is a child of the parent entity (note that unlike in a file system, the parent entity need not actually exist). An entity without a parent is a root entity. The association between an entity and its parent is permanent, and cannot be changed once the entity is created. The Datastore will never assign the same numeric ID to two entities with the same parent, or to two root entities (those without a parent).

An entity's parent, parent's parent, and so on recursively, are its ancestors; its children, children's children, and so on, are its descendants. An entity and its descendants are said to belong to the same entity group. The sequence of entities beginning with a root entity and proceeding from parent to child, leading to a given entity, constitute that entity's ancestor path. The complete key identifying the entity consists of a sequence of kind-identifier pairs specifying its ancestor path and terminating with those of the entity itself:

[Person:GreatGrandpa, Person:Grandpa, Person:Dad, Person:Me]

For a root entity, the ancestor path is empty and the key consists solely of the entity's own kind and identifier:

[Person:GreatGrandpa]

To designate an entity's parent, use the parent argument to datastore.NewKey . The value of this argument should be the parent entity's key.. The following example creates an entity of kind Address and designates an Employee entity as its parent:

// Create Employee entity
employee := &Employee{...}
employeeKey, err := datastore.Put(c, datastore.NewIncompleteKey(c, "Employee", nil), employee)

// Use Employee as Address entity's parent
// and save Address entity to datastore
address = &Address{...}
addressKey := datastore.NewIncompleteKey(c, "Address", employeeKey)
_, err = datastore.Put(c, addressKey, address)

Transactions and entity groups

Every attempt to create, update, or delete an entity takes place in the context of a transaction . A single transaction can include any number of such operations. To maintain the consistency of the data, the transaction ensures that all of the operations it contains are applied to the Datastore as a unit or, if any of the operations fails, that none of them are applied.

A single transaction can apply to multiple entities, so long as the entities belong to a limited number (5) of entity groups. You need to take this limitation into account when designing your data model: the simplest approach is to determine which entities you need to be able to process in the same transaction. Then, when you create those entities, place them in the same entity group by declaring them with a common ancestor. They will then all be in the same entity group and you will always be able to update and read them transactionally.

Properties and value types

The data values associated with an entity consist of one or more properties. Each property has a name and one or more values. A property can have values of more than one type, and two entities can have values of different types for the same property. Properties can be indexed or unindexed (queries that order or filter on a property P will ignore entities where P is unindexed).

The following value types are supported:

When a query involves a property with values of mixed types, the Datastore uses a deterministic ordering based on the internal representations:

1. Null values
2. Fixed-point numbers
   • Integers
   • Dates and times
3. Boolean values
4. Strings

Because byte slices and long strings are not indexed, they have no ordering defined.

Working with entities

Applications can use the Datastore API to create, retrieve, update, and delete entities. If the application knows the complete key for an entity (or can derive it from its parent key, kind, and identifier), it can use the key to operate directly on the entity. An application can also obtain an entity's key as a result of a Datastore query; see the Datastore Queries page for more information.

Creating an entity

In Go, you create a new entity by constructing an instance of a Go struct, populating its fields, and calling datastore.Put to save it to the Datastore. You can specify the entity's key name by passing a non-empty stringID argument to datastore.NewKey :

employee := &Employee{
    FirstName: "Antonio",
    LastName:  "Salieri",
    HireDate:  time.Now(),
}
employee.AttendedHRTraining = true
key := datastore.NewKey(c, "Employee", "asalieri", 0, nil)
_, err := datastore.Put(c, key, employee)

If you provide a empty key name, or use datastore.NewIncompleteKey , the Datastore will automatically generate a numeric ID for the entity's key:

employee := &Employee{
    FirstName: "Antonio",
    LastName:  "Salieri",
    HireDate:  time.Now(),
}
employee.AttendedHRTraining = true
key := datastore.NewIncompleteKey(c, "Employee", nil)
_, err := datastore.Put(c, key, employee)

Retrieving an entity

To retrieve an entity identified by a given key, pass the *datastore.Key as an argument to the datastore.Get function. You can generate the *datastore.Key using the datastore.NewKey function.

employeeKey := datastore.NewKey(c, "Employee", "asalieri", 0, nil)
addressKey := datastore.NewKey(c, "Address", "", 1, employeeKey)
var addr Address
err := datastore.Get(c, addressKey, &addr)

datastore.Get populates an instance of the appropriate Go struct.

Updating an entity

To update an existing entity, modify the attributes of the struct, then call datastore.Put . The data overwrites the existing entity. The entire object is sent to the Datastore with every call to datastore.Put .

Deleting an entity

Given an entity's key, you can delete the entity with the datastore.Delete function:

key := datastore.NewKey(c, "Employee", "asalieri", 0, nil)
err := datastore.Delete(c, key)

Batch operations

datastore.Put , datastore.Get and datastore.Delete have bulk variants called datastore.PutMulti , datastore.GetMulti and datastore.DeleteMulti . They permit acting on multiple entities in a single Datastore call:

// A batch put.
_, err := datastore.PutMulti(c, []*datastore.Key{k1, k2, k3}, []interface{e1, e2, e3})

// A batch get.
var entities = make([]*T, 3)
err := datastore.GetMulti(c, []*datastore.Key{k1, k2, k3}, entities)

// A batch delete.
err := datastore.DeleteMulti(c, []*datastore.Key{k1, k2, k3})

Performing operations in batches does not affect the cost, regardless of the entity's size. A batch operation for two keys costs two reads, even if one of the keys did not exist. For example, it is more economical to do a keys-only query that retrieves 1000 keys, and then do a fetch on 500 of them, than to do a regular (not keys-only) query for all 1000 directly:

Query returning 1000 keys + fetching 500 entities:

$0.0000007 (base query cost) + $0.0001 (per-key query cost) + 0.00035 (entity fetch)
= $0.0004507

Fetching 1000 entities:

$0.0000007 (base query cost) + $0.0007 (per-entity query cost)
= $0.0007007

Understanding write costs

When your application executes a Datastore put operation, the Datastore must perform a number of writes to store the entity. Your application is charged for each of these writes. You can see how many writes will be required to store an entity by looking at the data viewer in the SDK Development Console. This section explains how these write costs are calculated.

Every entity requires a minimum of two writes to store: one for the entity itself and another for the built-in EntitiesByKind index, which is used by the query planner to service a variety of queries. In addition, the Datastore maintains two other built-in indexes, EntitiesByProperty and EntitiesByPropertyDesc , which provide efficient scans of entities by single property values in ascending and descending order, respectively. Each of an entity's indexed property values must be written to each of these indexes.

As an example, consider an entity with properties A , B , and C :

Key: 'Foo:1' (kind = 'Foo', id = 1, no parent)
A: 1, 2
B: null
C: 'this', 'that', 'theOther'

Assuming there are no composite indexes (see below) for entities of this kind, this entity requires 14 writes to store:

• 1 for the entity itself
• 1 for the EntitiesByKind index
• 4 for property A (2 for each of two values)
• 2 for property B (a null value still needs to be written)
• 6 for property C (2 for each of three values)

Composite indexes (those referring to multiple properties) require additional writes to maintain. Suppose you define the following composite index:

Kind: 'Foo'
A ▲, B ▼

where the triangles indicate the sort order for the specified properties: ascending for property A and descending for property B. Storing the entity defined above now takes an additional write to the composite index for every combination of A and B values:

( 1 , null ) ( 2 , null )

This adds 2 writes for the composite index, for a total of 1 + 1 + 4 + 2 + 6 + 2 = 16. Now add property C to the index:

Kind: 'Foo'
A ▲, B ▼, C ▼

Storing the same entity now requires a write to the composite index for each possible combination of A , B , and C values:

( 1 , null , 'this' ) ( 1 , null , 'that' ) ( 1 , null , 'theOther' )

( 2 , null , 'this' ) ( 2 , null , 'that' ) ( 2 , null , 'theOther' )

This brings the total number of writes to 1 + 1 + 4 + 2 + 6 + 6 = 20.

If a Datastore entity contains many multiple-valued properties, or if a single such property is referenced many times, the number of writes required to maintain the index can explode combinatorially. Such exploding indexes can be very expensive to maintain. For example, consider a composite index that includes ancestors:

Kind: 'Foo'
A ▲, B ▼, C ▼
Ancestor: True

Storing a simple entity with this index present takes the same number of writes as before. However, if the entity has ancestors, it requires a write for each possible combination of property values and ancestors , in addition to those for the entity itself. Thus an entity defined as

Key: 'GreatGrandpa:1/Grandpa:1/Dad:1/Foo:1' (kind = 'Foo', id = 1, parent = 'GreatGrandpa:1/Grandpa:1/Dad:1')
A: 1, 2
B: null
C: 'this', 'that', 'theOther'

would require a write to the composite index for each of the following combinations of properties and ancestors:

( 1 , null , 'this' , 'GreatGrandpa' ) ( 1 , null , 'this' , 'Grandpa' ) ( 1 , null , 'this' , 'Dad' ) ( 1 , null , 'this' , 'Foo' )

( 1 , null , 'that' , 'GreatGrandpa' ) ( 1 , null , 'that' , 'Grandpa' ) ( 1 , null , 'that' , 'Dad' ) ( 1 , null , 'that' , 'Foo' )

( 1 , null , 'theOther' , 'GreatGrandpa' ) ( 1 , null , 'theOther' , 'Grandpa' ) ( 1 , null , 'theOther' , 'Dad' ) ( 1 , null , 'theOther' , 'Foo' )

( 2 , null , 'this' , 'GreatGrandpa' ) ( 2 , null , 'this' , 'Grandpa' ) ( 2 , null , 'this' , 'Dad' ) ( 2 , null , 'this' , 'Foo' )

( 2 , null , 'that' , 'GreatGrandpa' ) ( 2 , null , 'that' , 'Grandpa' ) ( 2 , null , 'that' , 'Dad' ) ( 2 , null , 'that' , 'Foo' )

( 2 , null , 'theOther' , 'GreatGrandpa' ) ( 2 , null , 'theOther' , 'Grandpa' ) ( 2 , null , 'theOther' , 'Dad' ) ( 2 , null , 'theOther' , 'Foo' )

Storing this entity in the Datastore now requires 1 + 1 + 4 + 2 + 6 + 24 = 38 writes.