Object generation numbers enable users to uniquely identify data resources and apply optimistic concurrency to guarantee atomicity of their multi-step transactions.
Contents
- Generation Numbers
- Optimistic Concurrency with Preconditions
- Read-Modify-Write
- Parallel Upload
- Preconditions for ETags
- XML API
- JSON API
Generations
Even without Object Versioning enabled, all Google Cloud Storage Objects have generation numbers and meta-generation numbers. The generation number is a monotonically increasing integer that updates when the data associated with an object name changes. Thus an object name and generation number uniquely identify a specific data resource that can be deleted, but never replaced with different data.
Similarly, meta-generation numbers monotonically increase whenever metadata changes, but since object meta-generation numbers reset to one for each new object generation, they are only meaningful when paired with a generation number.
Buckets also maintain a meta-generation number enabling users to uniquely identify a bucket metadata state. Since buckets have no payload data, and thus no generation numbers, their meta-generation numbers are meaningful on their own.
Optimistic Concurrency with Preconditions
To protect against race conditions where independent processes corrupt each other by making unexpected writes to the same object(s), Google Cloud Storage supports preconditions that guarantee to only execute an operation if the (meta-)generation number(s) of the resource(s) relevant to the operation match (or don’t match) what you expect. Important applications of preconditions are read-modify-write and parallel upload.
Read-Modify-Write
A common pattern for updating a resource’s data or metadata involves reading its current state, applying modifications locally, and sending the modified resource to Google Cloud Storage for writing. This can be precarious if two or more independent processes attempt the sequence simultaneously.
Specifically, if two processes read the same initial state, one will end up overwriting the other’s update. We can prevent this race condition by conditioning the write operations on the generation numbers that we read in the first step, and adding retry logic to handle the case where two processes end up colliding.
For example, if two processes are attempting to add an ACL entry to an object, the write operation that sends updated ACLs to Google Cloud Storage should condition on the generation and meta-generation numbers read in the first step, and if there’s a conflicting process, the second write would fail with response code 412 - Precondition Failed thus allowing that process to restart the read-modify-write cycle.
Parallel Upload
Parallel uploads are accomplished by dividing a file into multiple pieces, uploading those pieces to temporary objects simultaneously, composing the new object from its parts, and deleting the temporary objects. A race condition arises if an independent process inadvertently uses the same name for one or more of its temporary objects, and one of the two processes ends up using a corrupted component.
We avoid this race condition by keeping the generation numbers returned at the end of each upload and applying them as component preconditions in the compose operation.
Preconditions for ETags
Preconditions are also available for ETags. Note that XML API ETags for non-composite objects change only when content changes while ETags for composite objects and JSON API resources change whenever the content or metadata changes. For more information about ETags, see Hashes and ETags: Best Practices .
XML API
In the XML API, generation and meta-generation numbers are exposed via the x-goog-generation and x-goog-metageneration response headers. To condition an operation on the single relevant resource or the resource being written to, see the HTTP Headers Reference for the appropriate request header. Copy is a function of two objects, so a distinct set of headers prefixed with x-goog-copy-source-if- are available to condition copies on the source object.
Preconditions for compose components are supplied in the XML request body via
the
<IfGenerationMatch>
tag. For an example request using a
component precondition, see
Object Composition with the XML API
.
In the XML API, ETags (and their corresponding HTTP
If-Match
and
If-None-Match
headers) are only supported for object resources.
JSON API
The JSON API supports HTTP 1.1 ETags and the corresponding HTTP
If-Match
and
If-None-Match
headers for all
resources, including buckets, objects, and ACLs. An ETag is returned as part
of the response header whenever a resource is returned, as well as included in
the resource itself.
Generation and meta-generation numbers are exposed as part of the bucket and object resources. Generation and meta-generation preconditions may be specified with query parameters on the requested (destination) resource for any bucket or object operation.
Two object operations (copy and compose) also use source objects. For copy , there is a separately named set of query parameters for preconditions on the source object. For compose , preconditions for source objects are supplied in the JSON request body via the objectPreconditions property.
Generation and meta-generation preconditions are not accepted for ACL operations; use the access-control entry resource ETag instead. This can be found inside each access-control entry resource, which is also accessible from the containing object or bucket resource.