Please note that the contents of this offline web site may be out of date. To access the most recent documentation visit the online version .
Note that links that point to online resources are green in color and will open in a new window.
We would love it if you could give us feedback about this material by filling this form (You have to be online to fill it)



Using Pull Queues in Go

Pull queues allow you to design your own system to consume App Engine tasks. The task consumer can be part of your App Engine app (such as a backend ) or a system outside of App Engine (using the Task Queue REST API ). The task consumer leases a specific number of tasks for a specific duration, then processes and deletes them before the lease ends.

Using pull queues requires your application to handle some functions that are automated in push queues:

  • Your application needs to scale the number of workers based on processing volume. If your application does not handle scaling, you risk wasting computing resources if there are no tasks to process; you also risk latency if you have too many tasks to process.
  • Your application also needs to explicitly delete tasks after processing. In push queues, App Engine deletes the tasks for you. If your application does not delete pull queue tasks after processing, another worker might re-process the task. This wastes computing resources and risks errors if tasks are not idempotent .

Pull queues require a specific configuration in queue.yaml . For more information, please see Defining Pull Queues on the Task Queue configuration page.

The following sections describe the process of enqueuing, leasing, and deleting tasks using pull queues.

  1. Pull queue overview
  2. Pulling tasks within App Engine
  3. Pulling tasks to a backend
  4. Pulling tasks from outside App Engine
  5. Quotas and limits for pull queues

Pull queue overview

Pull queues allow a task consumer to process tasks outside of App Engine's default task processing system. If the task consumer is a part of your App Engine app, you can manipulate tasks using simple API calls from the appengine/taskqueue package .

The process works like this:

  1. The task consumer leases tasks, either via the Task Queue API (if the consumer is internal to App Engine) or the Task Queue REST API (if the consumer is external to App Engine).
  2. App Engine sends task data to the consumer.
  3. The consumer processes the tasks. If the task fails to execute before the lease expires, the consumer can lease it again. This counts as a retry attempt, and you can configure the maximum number of retry attempts before the system deletes the task.
  4. Once a task executes successfully, the task consumer must delete it.
  5. The task consumer is responsible for scaling instances based on processing volume.

Pulling tasks within App Engine

You can use pull queues within the App Engine environment using simple API calls to add tasks to a pull queue, lease them, and delete them after processing.

Before you begin, make sure to configure the pull queue in queue.yaml .

Adding tasks to a pull queue

To add tasks to a pull queue, simply get the queue using the queue name defined in queue.yaml , and set the Method to PULL . The following example enqueues tasks in a pull queue named pull-queue : 

import "appengine/taskqueue"

t := &taskqueue.Task{
    Payload: []byte("hello world"),
    Method:  "PULL",
}
err := taskqueue.Add(c, t, "pull-queue")

Leasing tasks

Once you have added tasks to a pull queue, you can lease one or more tasks using taskqueue.Lease . There may be a short delay before tasks recently added using taskqueue.Add become available via taskqueue.Lease . When you request a lease, you specify the number of tasks to lease (up to a maximum of 1,000 tasks) and the duration of the lease in seconds (up to a maximum of one week). The lease duration needs to be long enough to ensure that the slowest task will have time to finish before the lease period expires. You can extend a task lease using taskqueue.ModifyLease .

Leasing a task makes it unavailable for processing by another worker, and it remains unavailable until the lease expires. If you lease an individual task, the API selects the task from the front of the queue. If no such task is available, an empty list is returned. 

import "appengine/taskqueue"

tasks, err := taskqueue.Lease(c, 100, "pull-queue", 3600)

Experimental!

Task Tagging is an experimental, innovative, and rapidly changing new feature for Google App Engine. Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes to Task Tagging. We will inform the community when this feature is no longer experimental.


Not all tasks are alike; your code can "tag" tasks and then choose tasks to lease by tag. The tag acts as a filter. 

import "appengine/taskqueue"

_, err := taskqueue.Add(c, &taskqueue.Task{Payload: []byte("parse1"), Method: "PULL", Tag: "parse"})
_, err := taskqueue.Add(c, &taskqueue.Task{Payload: []byte("parse2"), Method: "PULL", Tag: "parse"})
_, err := taskqueue.Add(c, &taskqueue.Task{Payload: []byte("render1"), Method: "PULL", Tag: "render"})
_, err := taskqueue.Add(c, &taskqueue.Task{Payload: []byte("render2"), Method: "PULL", Tag: "render"})

// leases render tasks, but not parse
tasks, err := taskqueue.LeaseByTag(c, 100, "pull-queue", 3600, "render")

// Leases up to 100 tasks that have same tag.
// Tag is that of "oldest" task by ETA.
tasks, err := taskqueue.LeaseByTag(c, 100, "pull-queue", 3600, "")

Deleting tasks

In general, once a worker completes a task, it needs to delete the task from the queue. If you see tasks remaining in a queue after a worker finishes processing them, it is likely that the worker failed; in this case, the tasks need to be processed by another worker.

You can delete a list of tasks, such as that returned by taskqueue.Lease , by passing it to taskqueue.DeleteMulti : 

import "appengine/taskqueue"

tasks, err := taskqueue.Lease(c, 100, "pull-queue", 3600)
// Perform some work with the tasks here

taskqueue.DeleteMulti(c, tasks, "pull-queue")

Pulling tasks to a backend

You can use App Engine Backends as workers to lease and process pull queue tasks. Backends allow you to process more work without having to worry about request deadlines and other restrictions normally imposed by App Engine. Using backends with pull queues gives you processing efficiencies by allowing you to batch task processing using leases.

For more information about using backends, check out the Backends documentation .

Pulling tasks from outside App Engine

If you need to use pull queues from outside App Engine, you must use the Task Queue REST API . The REST API is a Google web service accessible at a globally-unique URI of the form: 

https://www.googleapis.com/taskqueue/v1beta2/projects/taskqueues

Google provides the following client libraries that you can use to call the Task Queue methods remotely:

In the tables below, the first column shows each library's stage of development (note that some are in early stages), and links to documentation for the library. The second column links to available samples for each library.

Documentation Samples
Google APIs Client Library for Java (rc) Java samples
Google APIs Client Library for JavaScript (beta) JavaScript samples
Google APIs Client Library for .NET .NET samples
Google APIs Client Library for Objective-C Objective-C samples
Google APIs Client Library for PHP (beta) PHP samples
Google APIs Client Library for Python Python samples

These early-stage libraries are also available:

Documentation Samples
Google APIs Client Library for Go (alpha) Go samples
Google APIs Client Library for Node.js (alpha) Node.js samples
Google APIs Client Library for Ruby (alpha) Ruby samples

In addition to the Google-supported client libraries listed in the table above, a set of third-party Dart libraries (with samples ) is available for the Dart language .

Prerequisites

The REST API uses OAuth as the authorization mechanism. When you configure your pull queue, make sure that your queue.yaml file supplies the email addresses of the users that can access the queue using the REST API. The OAuth scope for all methods is https://www.googleapis.com/auth/taskqueue .

Quotas and limits for pull queues

Enqueuing a task counts counts toward the following quotas:

  • Task Queue Stored Task Count
  • Task Queue API Calls
  • Task Queue Stored Task Bytes

Leasing a task counts toward the following quotas:

  • Task Queue API Calls
  • Outgoing Bandwidth (if using the REST API)

The Task Queue Stored Task Bytes quota is configurable in queue.yaml by setting total_storage_limit . This quota counts towards your Stored Data (billable) quota.

The following limits apply to the use of pull queues:

Pull Queue Limits
Maximum task size 1MB
Maximum number of active queues (not including the default queue) Free apps: 10 queues, Billed apps: 100 queues
Maximum countdown/ETA for a task 30 days from the current date and time
Maximum number of tasks that can be added in a batch 100 tasks
Maximum number of tasks that can be added in a transaction 5 tasks
Maximum number of tasks that you can lease in a single operation 1000 tasks
Maximum payload size when leasing a batch of tasks 32MB (1MB when using the REST API)

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License , and code samples are licensed under the Apache 2.0 License . For details, see our Site Policies .

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.