With the Task Queue API, applications can perform work outside of a user request, initiated by a user request. If an app needs to execute some background work, it can use the Task Queue API to organize that work into small, discrete units, called tasks . The app adds tasks to task queues to be executed later.
App Engine provides two different queue configurations:
- Push queues process tasks based on the processing rate configured in the queue definition . App Engine automatically scales processing capacity to match your queue configuration and processing volume, and also deletes tasks after processing. Push queues are the default.
- Pull queues allow a task consumer (either your application or code external to your application) to lease tasks at a specific time for processing within a specific timeframe. Pull queues give you more control over when tasks are processed, and also allow you to integrate your application with non-App-Engine code using the experimental Task Queue REST API . When using pull queues, your application needs to handle scaling of instances based on processing volume, and also needs to delete tasks after processing.
This page provides the basic concepts common to both types of queues. Once you've understood the basics, you can check out the queue configuration page , push queue overview , and pull queue overview to see how to configure and use these two types of queues.
Task Queue concepts
Tasks queues are an efficient and powerful tool for background processing; they allow your application to define tasks, add them to a queue, and then use the queue to process them in aggregate. You name queues and
configure their properties
in a configuration file named
queue.xml
.
Push queues function only within the App Engine environment. These queues are the best choice for applications whose tasks work only with App Engine tools and services. With push queues, you simply configure a queue and add tasks to it. App Engine handles the rest. Push queues are easier to implement, but are restricted to use within App Engine. For more information about push queues and examples of how to use them, see Using Push Queues .
If you want to use a different system to consume tasks, pull queues are your best choice. In pull queues, a task consumer (either in your App Engine application, a backend , or code outside of App Engine) leases a specific number of tasks from a specific queue for a specific timeframe. After leasing tasks from a pull queue, the task consumer is responsible for deleting them. If you are consuming tasks from within App Engine, you can use calls from the com.google.appengine.api.taskqueue package . If you are consuming tasks from outside of App Engine, you need to use the Task Queue REST API . Pull queues give you more power and flexibility over when and where tasks are processed, but they require you to handle scaling of workers based on processing volume. Your task consumer also needs to delete tasks after processing.
In summary, push queues allow you to process tasks within App Engine at a steady rate and App Engine scales computing resources according to the number of tasks in your queue. Pull queues allow an alternate task consumer to process tasks at a specific time, either in or outside App Engine, but your application needs to scale workers based on processing volume, as well as delete tasks after processing.
You can read more about these two types of queues in the Using Push Queues and Using Pull Queues .
The default queue
For convenience, App Engine provides a default push queue for each application (there is no default pull queue). If you do not name a queue for a task, App Engine automatically inserts it into the default queue. You can use this queue immediately without any additional configuration. All modules and versions of the same application share the same default task queue.
The default queue is preconfigured with a throughput rate of 5 task invocations per second. If you want to change the preconfigured settings, simply define a queue named
default
in
queue.xml
. Code may always insert new tasks into the default queue, but if you wish to disable execution of these tasks, you may do so by clicking the Pause Queue button in the
Task Queues
tab of the Administration Console.
Named queues
While the default queue makes it easy to enqueue tasks with no configuration, you can also create custom queues by defining them in
queue.xml
. Custom queues allow you to more effectively handle task processing by grouping similar types of tasks. You can control the processing rate—and a number of other settings—based specifically on the type of task in each queue.
All versions of an application share the same named task queues.
For more information about configuring queues in
queue.xml
, please see
Task Queue Configuration
.
Task queues in the Administration Console
You can manage task queues for an application using the Task Queue tab of the Administration Console . The Task Queue tab lists all of the queues in the application. Clicking on a queue name brings up the Task Queue Details page where you can see all of the tasks scheduled to run in a queue and you can manually delete individual tasks or purge every task from a queue. This is useful if a task in a push queue cannot be completed successfully and is stuck waiting to be retried. You can also pause and resume a queue on this page.
You can view details of individual tasks by clicking the task name from the list of tasks on the Task Queue Details page. This page allows you to debug why a task did not run successfully. You can also see information about the previous run of the task as well as the task's body.
Checking task queue statistics
You can view queue statistics in the Console to determine performance and status. However, you can also access queue statistics programmatically using the QueueStatistics class.
Task concepts
A
task
is a unit of work to be performed by the application. Each task is an object of the
TaskOptions
class. Each Task object contains an endpoint (with a request handler for the task and an optional data payload that parameterizes the task). You can enqueue push tasks to a queue defined in
queue.xml
. Push tasks and pull tasks are defined differently; see the
Using Push Queues
and
Using Pull Queues
for specific usage details.
Task names
In addition to a task's contents, you can declare a task's name. Once a task with name N is written, any subsequent attempts to insert a task named N fail.
While this is generally true, task names do not provide an absolute guarantee of once-only semantics. In rare cases, multiple calls to create a task of the same name may succeed. It's also possible in exceptional cases for a task to run more than once—even if it was only created once.
All project, queue, and task names must be a combination of one or more digits, letters a–z, underscores, and/or dashes, satisfying the following regular expression:
[0-9a-zA-Z\-\_]+
Task names may be up to 500 characters long.
If a push task is created successfully, it will eventually be deleted (at most seven days after the task successfully executes). Once deleted, its name can be reused.
If a pull task is created successfully, your application needs to delete the task after processing. The system may take up to seven days to recognize that a task has been deleted; during this time, the task name remains unavailable. Attempting to create another task during this time with the same name will result in an "item exists" error. The system offers no method to determine if deleted task names are still in the system. To avoid these issues, we recommend that you let App Engine generate the task name automatically.
Tasks within transactions
You can enqueue a task as part of a datastore transaction, such that the task is only enqueued—and guaranteed to be enqueued—if the transaction is committed successfully. Tasks added within a transaction are considered to be a part of it and have the same level of isolation and consistency .
An application cannot insert more than five transactional tasks into task queues during a single transaction. Transactional tasks must not have user-specified names.
The following code sample demonstrates how to insert transactional tasks into a push queue as part of a datastore transaction:
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Queue queue = QueueFactory.getDefaultQueue();
try {
Transaction txn = ds.beginTransaction();
// ...
queue.add(TaskOptions.Builder.withUrl("/path/to/my/worker"));
// ...
txn.commit();
} catch (DatastoreFailureException e) {
}
Deleting tasks
You can delete tasks through the Administration Console or programatically. It may take several hours to reclaim the quota used by purged queues. It takes the system about 20 seconds to notice that the queue has been purged. In push queues, tasks continue to execute during this time.
You have two ways to delete tasks:
- Using the Administration Console .
- Programmatically, using purge() to delete all tasks from the specified queue, or using deleteTask() to delete an individual task:
// Purge entire queue...
Queue queue = QueueFactory.getQueue("foo");
queue.purge();
// Delete an individual task...
Queue q = QueueFactory.getQueue("queue1");
q.deleteTask("foo")
Asynchronous operations
If you want to make asynchronous calls to a task queue, you use the asynchronous methods provided by the
Queue
class. Call
get
on the returned Future to force the request to complete. When asynchronously adding tasks in a transaction, you should call
get()
on the Future before committing the transaction to ensure that the request has finished.