Sign in

1. Products
2. Google Cloud Platform
3. Google Cloud Storage

Google Cloud Storage

Feedback on this document

Hashes and ETags: Best Practices

Google Cloud Storage encourages users to validate the data they transfer to/from their buckets using either CRC32c or MD5 checksums. This section describes best practices for performing these validations.

Contents

1. Using Hashes for Integrity Checking
2. CRC32C
3. MD5
4. ETags
5. Validation
6. XML API
7. JSON API

Using Hashes for Integrity Checking

There are a variety of ways that data can be corrupted while uploading to or downloading from the Cloud:

• Noisy network links
• Memory errors on client or server computers, or routers along the path
• Software bugs (e.g., in a library that customers use)

To protect against data corruption, Google Cloud Storage supports two types of hashes: CRC32C and MD5 (described below). Google recommends that customers use CRC32C for all cases, as described in the Validation section below. Customers that prefer MD5 can use that hash, but that hash is not supported for composite objects or range GETs.

CRC32C

All GCS objects have a CRC32c hash. CRC32C is a 32 bit Cyclic Redundancy Check (CRC) based on the Castagnoli polynomial. This CRC is described by the IETF specification for SCTP . Libraries for computing CRC32c include Boost for C++, crcmod for Python, and digest-crc for Ruby. Java users can find an implementation of the algorithm in the GoogleCloudPlatform crc32c Java project .

MD5

Google Cloud Storage supports an MD5 hash for non-composite objects. This hash only applies to a complete object, so it cannot be used to integrity check partial downloads caused by performing a range GET.

ETags

Historically, an object’s MD5 was expressed via the ETag header. This behavior will continue for non-composite objects in the XML API, but since composite objects don’t support MD5 hashes, users should make no assumptions about those ETags except that they will change whenever the underlying data changes, per the IETF specification for HTTP/1.1 .

Validation

A download integrity check can be performed by hashing downloaded data on the fly and comparing your results to server-supplied hashes. You should discard downloaded data with incorrect hash values, and you should use retry logic to avoid potentially expensive infinite loops.

Google Cloud Storage supports server-side validation for uploads, but client-side validation is also a common approach. If your application has already computed the object’s MD5 or CRC32c prior to starting the upload, you can supply it with the upload request, and Google Cloud Storage will only create the object if the hash you provided matches the value we calculated.

Alternatively, users can choose to perform client-side validation by issuing a request for the new object’s metadata, comparing the reported hash value, and deleting the object in case of a mismatch. To avoid race conditions where independent processes delete or overwrite each other’s data, we also recommend that you use object generations and preconditions .

In the case of parallel uploads using object composition , users should perform an integrity check for each component upload and then use component preconditions with their compose requests to protect against race conditions. Object composition offers no server-side MD5 validation, so users who wish to perform an end-to-end integrity check should apply client-side validation to the new composite object.

At the end of each copy operation, the gsutil cp and rsync commands validate that the checksum of the local file matches that of the checksum of the object stored in Google Cloud Storage. If it does not, gsutil will delete the invalid copy and print a warning message. This very rarely happens. If it does, please contact [email protected] .

XML API

In the XML API, base64-encoded MD5 and CRC32c hashes are exposed and accepted via the x-goog-hash header . In the past, MD5s were used as object ETags, but we recommend that users avoid assuming this since composite objects will use opaque ETag values that make no guarantees outside of changing when the object changes.

Server-side upload validation can be performed by supplying locally computed hashes via the x-goog-hash request header. Additionally, the MD5 can be supplied using the standard HTTP Content-MD5 header .

JSON API

In the JSON API, the objects resource md5Hash and crc32c properties contain base64-encoded MD5 and CRC32c hashes, respectively. Supplying either metadata field with an objects insert request will trigger server-side validation for the new object.

Back to top