Overview

Google Cloud Storage provides a cyclic redundancy check (CRC) header that allows clients to verify the integrity of object contents. For non-composite objects GCS also provides an MD5 header to allow clients to verify object integrity, but for composite objects only the CRC is available. gsutil automatically performs integrity checks on all uploads and downloads. Additionally, you can use the “gsutil hash” command to calculate a CRC for any local file.

The CRC variant used by Google Cloud Storage is called CRC32C (Castagnoli), which is not available in the standard Python distribution. The implementation of CRC32C used by gsutil is provided by a third-party Python module called crcmod .

The crcmod module contains a pure-Python implementation of CRC32C, but using it results in very poor performance. A Python C extension is also provided by crcmod, which requires compiling into a binary module for use. gsutil ships with a precompiled crcmod C extension for Mac OS X; for other platforms, see the installation instructions below.

At the end of each copy operation, the gsutil cp and rsync commands validate that the checksum of the source file/object matches the checksum of the destination file/object. If the checksums do not match, gsutil will delete the invalid copy and print a warning message. This very rarely happens, but if it does, please contact gs-team @ google . com .

Configuration

To determine if the compiled version of crcmod is available in your Python environment, you can inspect the output of the gsutil version command for the “compiled crcmod” entry::

$ gsutil version -l
...
compiled crcmod: True
...

If your crcmod library is compiled to a native binary, this value will be True. If using the pure-Python version, the value will be False.

To control gsutil’s behavior in response to crcmod’s status, you can set the “check_hashes” configuration variable. For details on this variable, see the surrounding comments in your gsutil configuration file. If check_hashes is not present in your configuration file, rerun gsutil config to regenerate the file.

Installation

sudo yum install gcc python-devel python-setuptools
sudo easy_install -U pip
sudo pip uninstall crcmod
sudo pip install -U crcmod

sudo apt-get install gcc python-dev python-setuptools
sudo easy_install -U pip
sudo pip uninstall crcmod
sudo pip install -U crcmod

sudo easy_install -U pip
sudo pip install -U crcmod