Using Push Queues in Go
In App Engine push queues, a task is a unit of work to be performed by the application. Each task is an object of the Task type . Each Task object contains an application-specific URL with a request handler for the task, and an optional data payload that parameterizes the task.
For example, consider a calendaring application that needs to notify an invitee, via email, that an event has been updated. The data payload for this task consists of the email address and name of the invitee, along with a description of the event. The webhook might live at
/app_worker/send_email
and contain a function that adds the relevant strings to an email template and sends the email. The app can create a separate task for each email it needs to send.
You can use push queues only within the App Engine environment; if you need to access App Engine tasks from outside of App Engine, use pull queues .
- Using push queues
- Push task execution
- Deferred tasks
- URL endpoints
- Push queues and the development server
- Push queues and backends
- Quotas and limits for push queues
Using push queues
A Go app sets up queues using a configuration file named
queue.yaml
(see
Go Task Queue Configuration
). If an app does not have a
queue.yaml
file, it has a queue named
default
with some default settings.
To enqueue a task, you call the taskqueue.Add function. The task consists of data for a request, including a URL path, parameters, HTTP headers, and an HTTP payload. It can also include the earliest time to execute the task (the default is as soon as possible) and a name for the task. The task is added to a queue, then performed by the Task Queue service as the queue is processed.
The following example defines a task handler (
worker
) that increments a counter in the datastore, mapped to the URL
/worker
. It also defines a user-accessible request handler that displays the current value of the counter for a
GET
request, and for a
POST
request enqueues a task and returns. It's difficult to visualize the execution of tasks with the user-accessible handler if the queue processes them too quickly. Therefore, the task in this example should run at a rate no greater than once per second.
package counter
import (
"html/template"
"net/http"
"appengine"
"appengine/datastore"
"appengine/taskqueue"
)
func init() {
http.HandleFunc("/", handler)
http.HandleFunc("/worker", worker)
}
type Counter struct {
Name string
Count int
}
func handler(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
if name := r.FormValue("name"); name != "" {
t := taskqueue.NewPOSTTask("/worker", map[string][]string{"name": {name}})
if _, err := taskqueue.Add(c, t, ""); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
q := datastore.NewQuery("Counter")
var counters []Counter
if _, err := q.GetAll(c, &counters); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if err := handlerTemplate.Execute(w, counters); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// OK
}
func worker(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
name := r.FormValue("name")
key := datastore.NewKey(c, "Counter", name, 0, nil)
var counter Counter
if err := datastore.Get(c, key, &counter); err == datastore.ErrNoSuchEntity {
counter.Name = name
} else if err != nil {
c.Errorf("%v", err)
return
}
counter.Count++
if _, err := datastore.Put(c, key, &counter); err != nil {
c.Errorf("%v", err)
}
}
var handlerTemplate = template.Must(template.New("handler").Parse(handlerHTML))
const handlerHTML = `
{{repeat .}}
<p>{{.Name}}: {{.Count}}</p>
{{end}}
<p>Start a new counter:</p>
<form action="/" method="POST">
<input type="text" name="name">
<input type="submit" value="Add">
</form>
`
Note that this example is not idempotent. It is possible for the task queue to execute a task more than once. In this case, the counter is incremented each time the task is run, possibly skewing the results.
Push task execution
App Engine executes push tasks by sending HTTP requests to your app. Specifying a programmatic asynchronous callback as an HTTP request is sometimes called a web hook . The web hook model enables efficient parallel processing.
The task's URL determines the handler for the task and the module that runs the handler.
The handler is determined by the path part of the URL (the forward-slash separated string following the hostname), which is specified by using the
Path
field of the
Task struct
. The path must be relative and local to your application's root directory.
The module (or frontend or backend) and version in which the handler runs is determined by: