Sign in

1. Products
2. Google Cloud Platform
3. Google App Engine
4. Go

Go

Feedback on this document

Documents and Indexes

Python | Java | PHP | Go

See more...

The Search API provides a model for indexing documents that contain structured data. You can search an index, and organize and present search results. The API supports partial text matching on string fields. Documents and indexes are saved in a separate persistent store optimized for search operations. The Search API can index any number of documents. However, an index search can find no more than 10,000 matching documents. The App Engine Datastore may be more appropriate for applications that need to retrieve very large result sets.

1. Overview
2. Documents and fields
3. Creating a document
4. Working with an index
   • Putting documents in an index
   • Retrieving documents by docID
   • Searching for documents by their contents
   • Deleting documents from an index
   • Eventual consistency
5. Index schemas
6. Viewing indexes in the Admin Console
7. Search API quotas
8. Search API pricing

Overview

The Search API is based on four main concepts: documents, indexes, queries, and results.

Documents

A document is an object with a unique ID and a list of fields containing user data. Each field has a name and a type. There are several types of fields, identified by the kinds of values they contain:

• Atom Field - an indivisible character string
• Text Field - a plain text string that can be searched word by word
• HTML Field - a string that contains HTML markup tags, only the text outside the markup tags can be searched
• Number Field - a floating point number
• Time Field - a time.Time value (stored with millisecond precision)
• Geopoint Field - a data object with latitude and longitude coordinates

The maximum size of a document is 1 MB.

Indexes

An index stores documents for retrieval. You can retrieve a single document by its ID, a range of documents with consecutive IDs, or all the documents in an index. You can also search an index to retrieve documents that satisfy given criteria on fields and their values, specified as a query string. You can manage groups of documents by putting them into separate indexes.

There is no limit to the number of documents in an index, or the number of indexes you can use. However, the total size of all the documents in a single index cannot be more than 10GB.

Queries

To search an index, you construct a query, which has a query string, and possibly some additional options. A query string specifies conditions for the values of one or more document fields. When you search an index you get back only those documents in the index with fields that satisfy the query.

The simplest query, sometimes called a "global search" is a string that contains only field values. This search uses a string that searches for documents that contain the words "rose" and "water":

index.Search(c, "rose water", nil)

This one searches for documents with date fields that contain the date July 4, 1776, or text fields that include the string "1776-07-04":

index.Search(c, "1776-07-04", nil)

A query string can also be more specific. It can contain one or more terms, each naming a field and a constraint on the field's value. The exact form of a term depends on the type of the field. For instance, assuming there is a text field called "Product", and a number field called "Price", here's a query string with two terms:

// search for documents with pianos that cost less than $5000
index.Search(c, "Product = piano AND Price < 5000", nil)

Query options are not yet available in the Go runtime.

Search results

Additional training material

In additional to this documentation, you can read the two-part training class on the Search API at the Google Developer's Academy. (Although the class uses the Python API, you may find the additional discussion of the Search concepts useful.)

Documents and fields

Document identifier

Every document in an index must have a unique document identifier, or docID . The identifier can be used to retrieve a document from an index without performing a search. By default, the Search API automatically generates a docID when a document is created. You can also specify the docID yourself when you create a document. A docID must contain only visible, printable ASCII characters (ASCII codes 33 through 126 inclusive) and be no longer than 500 characters. A document identifier cannot begin with an exclamation point ('!'), and it can't begin and end with double underscores ("__").

While it is convenient to create readable, meaningful unique document identifiers, you cannot include the docID in a search. Consider this scenario: You have an index with documents that represent parts, using the part's serial number as the docID . It will be very efficient to retrieve the document for any single part, but it will be impossible to search for a range of serial numbers along with other field values, such as purchase date. Storing the serial number in an atom field solves the problem.

Document fields

A document contains fields that have a name , a type , and a single value of that type. Two or more fields can have the same name, but different types. For instance, you can define two fields with the name "age": one with a text type (the value "twenty-two"), the other with a number type (value 22).

Field names

There is a limit of 1000 unique field names over all the documents in an index. Note the limit is imposed on field names , not fields .

Field names are case sensitive and can only contain ASCII characters. They must start with a capital letter and can contain letters, digits, or underscore. A field name cannot be longer than 500 characters.

Multi-valued fields

A field can contain only one value, which must match the field's type. Field names do not have to be unique. A document can have multiple fields with the same name and same type, which is a way to represent a field with multiple values. (However, date and number fields with the same name can't be repeated.) A document can also contain multiple fields with the same name and different field types.

Field types

There are three kinds of fields that store character strings; collectively we refer to them as string fields :

• Text Field: A string with maximum length 1024**2 characters.
• HTML Field: An HTML-formatted string with maximum length 1024**2 characters.
• Atom Field: A string with maximum length 500 characters.

There are also three field types that store non-textual data:

• Number Field: A double precision floating point value between -2,147,483,647 and 2,147,483,647.
• Time Field - a time.Time value (stored with millisecond precision)
• Geopoint Field: A point on earth described by latitude and longitude coordinates

The string field types are Go's built-in string type and the search package's HTML and Atom types. Number fields are represented with Go's built-in float64 type, time fields use the time.Time type, and geopoint fields use the appengine package's GeoPoint type.

Special treatment of string and time fields

When a document with time, text, or HTML fields is added to an index, some special handling occurs. It's helpful to understand what's going on "under the hood" in order to use the Search API effectively.

Tokenizing string fields

When an HTML or text field is indexed, its contents are tokenized . The string is split into tokens wherever whitespace or special characters (punctuation marks, hash sign, etc.) appear. The index will include an entry for each token. This enables you to search for keywords and phrases comprising only part of a field's value. For instance, a search for "dark" will match a document with a text field containing the string "it was a dark and stormy night", and a search for "time" will match a document with a text field containing the string "this is a real-time system".

In HTML fields, text within markup tags is not tokenized, so a document with an HTML field containing "it was a <strong>dark</strong> night" will match a search for "night", but not for "strong". If you want to be able to search markup text, store it in a text field.

Atom fields are not tokenized. A document with an atom field that has the value "bad weather" will only match a search for the entire string "bad weather". It will not match a search for "bad" or "weather" alone.

Note that the underscore (_) and ampersand (&) characters do not break words, and non-western languages, like Japanese and Chinese, use other tokenization rules.

Time field accuracy

When you create a time field in a document you set its value to a time.Time . For the purpose of indexing and searching the time field, any time component is ignored and the date is converted to the number of days since 1/1/1970 UTC. This means that even though a time field can contain a precise time value a date query can only specify a time field value in the form yyyy-mm-dd. This also means the sorted order of time fields with the same date is not well-defined. While the time.Time type represents time with nanosecond precision, the Search API stores them with only millisecond precision.

Linking from a document to other resources

You can use a document's docID and other fields as links to other resources in your application. For example, if you use Blobstore you can associate the document with a specific blob by setting the docID or the value of an Atom field to the BlobKey of the data.

Creating a document

The following code sample shows how to create a document object. The User type specifies the document structure, and a User value is constructed in the usual way.

import (
    "fmt"
    "net/http"
    "time"

    "appengine"
    "appengine/search"
)

type User struct {
    Customer  string
    Comment   search.HTML
    Visits    float64
    LastVisit time.Time
    Birthday  time.Time
}

func putHandler(w http.ResponseWriter, r *http.Request) {
    id := "PA6-5000"
    user := &User{
        Name: "Joe Jackson",
        Comment: "this is <em>marked up</em> text",
        Visits: 7,
        LastVisit: time.Now(),
        Birthday: time.Date(1960, time.June, 19, 0, 0, 0, 0, nil),
    }
    // ...
}

Working with an index

Putting documents in an index

When you put a document into an index, the document is copied to persistent storage and each of its fields is indexed according to its name, type, and the docID .

The following code example shows how to access an Index and put a document into it.

    // ...
    c := appengine.NewContext(r)
    index, err := search.Open("users")
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    _, err = index.Put(c, id, user)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    fmt.Fprint(w, "OK")
}

When you put a document into an index and the index already contains a document with the same docID the new document replaces the old one. No warning is given. You can call Index.Get before creating or adding a document to an index to check whether a specific docID already exists.

The Put method returns a docID . If you did not specify the docID yourself, you can examine the result to discover the docID that was generated:

    id, err := index.Put(c, "", user)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    fmt.Fprint(w, id)

Note that creating an instance of the Index type does not guarantee that a persistent index actually exists. A persistent index is created the first time you add a document to it with the put method.

Updating documents

A document cannot be changed once you've added it to an index. You can't add or remove fields, or change a field's value. However, you can replace the document with a new document that has the same docID .

Retrieving documents by docID

Use the Index.Get method to retrieve a document from an index by its docID :

func getHandler(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)

    index, err := search.Open("users")
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    id := "PA6-5000"
    var user User
    _, err = index.Get(c, id, &user)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    fmt.Fprint(w, "Retrieved document: ", user)
}

Searching for documents by their contents

To retrieve documents from an index, you construct a query string and call Index.Search . Search returns an iterator that yields matching documents in order of decreasing rank.

func searchHandler(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)

    index, err := search.Open("myIndex")
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    for t := index.Search(c, "Product: piano AND Price < 5000", nil); ; {
        var doc Doc
        id, err := t.Next(&doc)
        if err == search.Done {
            break
        }
        if err != nil {
            fmt.Fprintf(w, "Search error: %v\n", err)
            break
        }
        fmt.Fprintf(w, "%s -> %#v\n", id, doc)
    }
}

Deleting documents from an index

You can delete documents in an index by specifying the docID of the document you wish to delete to the Index.Delete method.

func deleteHandler(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)

    index, err := search.Open("users"))
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    id := "PA6-5000"
    err = index.Delete(c, id)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    fmt.Fprint(w, "Deleted document: ", id)
}

Eventual consistency

When you put, update, or delete a document in an index, the change propagates across multiple data centers. This usually happens quickly, but the time it takes is variable. The Search API guarantees eventual consistency . This means that in some cases if you perform a search or retrieve one or more documents by id, the results may not reflect the most recent change.

Index schemas

Every index has a schema that shows all the field names and field types that appear in the documents it contains. You cannot define a schema yourself. Schemas are maintained dynamically; they are updated as documents are added to an index. A simple schema might look like this, in JSON-like form:

{'comment': ['TEXT'], 'date': ['DATE'], 'author': ['TEXT'], 'count': ['NUMBER']}

Each key in the dictionary is the name of a document field. The key's value is a list of the field types used with that field name. If you have used the same field name with different field types the schema will list more than one field type for a field name, like this:

{'ambiguous-integer': ['TEXT', 'NUMBER', 'ATOM']}

Once a field appears in a schema it can never be removed. There is no way to delete a field, even if the index no longer contains any documents with that particular field name.

A schema does not define a "class" in the object-programming sense. As far as the Search API is concerned, every document is unique and indexes can contain different kinds of documents. If you want to treat collections of objects with the same list of fields as instances of a class, that's an abstraction you must enforce in your code. For instance, you could insure that all documents with the same set of fields are kept in their own index. The index schema could be seen as the class definition, and each document in the index would be an instance of the class.

Viewing indexes in the Admin Console

You can view information about your application's indexes, and the documents they contain, by clicking the application's name in the App Engine Administration Console . In the sidebar section labeled Data , click the Text Search link to see a list of the application's indexes. Clicking an index name displays the documents that index contains. You'll see all the defined schema fields for the index; for each document with a field of that name, you'll see the field's value. You can also issue queries on the index data directly from the Administration Console.

Search API quotas

The Search API has several free quotas:

The Search API imposes these limits to ensure the reliability of the service. These apply to both free and paid apps:

API usage is counted in different ways depending on the type of call:

• Index.Search : There are separate quotas for simple and complex queries. A query is complex if its query string includes the name of a geopoint field or at least one OR or NOT boolean operator. A query is also complex if it uses query options to specify non-default sorting or scoring, field expressions, or snippets. Otherwise the query is simple.
• Index.Put : When you add documents to indexes the size of each document and the number of documents counts towards the indexing quota.
  
• All other Search API calls are counted based on the number of operations they involve:
• Index.Get : 1 operation counted for each document actually returned, or 1 operation if nothing is returned.
• Index.Delete : 1 operation counted for each document in the request, or 1 operation if the request is empty.

More information on quotas can be found on the Quotas page. When an app tries to exceed these amounts, an insufficient quota error is returned.

Note that although these limits are enforced by the minute, the Admin Console displays the daily totals for each. Customers with Silver, Gold, or Platinum support can request higher throughput limits by contacting their support representative.

Search API pricing

If you enable billing for your app you will be charged for additional usage beyond free quotas. The following charges apply to billed apps:

Additional information on pricing is on the Pricing page.