This pages discusses how to install gsutil, a tool that enables you to access Google Cloud Storage from the command-line.
Contents
- System Requirements
- How to Install gsutil
- How to Set Up Credentials to Access Protected Data
- How to Convert gsutil to Use OAuth 2.0
- How to Use boto with Google Cloud Storage
- boto Configuration File
System requirements
gsutil runs on Linux/Unix, Mac OS, and Windows. To use gsutil, you must have Python 2.6.x or 2.7.x installed on your computer. gsutil does not currently run on Python 3.x. Python is installed by default on most distributions of Linux and Mac OS X. Python is not installed by default on Windows; you must install Python before you can run gsutil on Windows.
Warning:
Please be aware that a tool also named
gsutil
, by
GrandStream BudgeTone, is pre-installed on some Linux OS distributions.
If you run this command instead of the Google Cloud Storage gsutil it will
likely print an error message like
"Choose one of -b, -d, -e, or -r to do something"
.
If this happens, you can either move the Google Cloud Storage version of gsutil
to the front of your PATH environment variable, or you
can specify the full path when attempting to run Google Cloud Storage
gsutil (e.g., /home/users/joan/gsutil/gsutil ls).
The following procedure helps you install gsutil on Windows so that you can run gsutil at the Windows command prompt.
How to install gsutil
The following instructions help you install gsutil but does not discuss how to authenticate to the Google Cloud Storage service. You do not need to authenticate to the Google Cloud Storage service unless you want to access protected data, in which case, you should review the How to Set Up Credentials to Access Protected Data section below.
Installing from tar or zip archive
This is the officially supported installation method for gsutil.Linux and Mac OS X
-
You can download gsutil as
gsutil.tar.gz
or as
gsutil.zip
.
It doesn't matter which of these you use; however, these installation instructions
assume you are using
gsutil.tar.gz
.Whichever format you download, gsutil is bundled in a single archive. Save the archive in a convenient location.
-
Extract the archive files.
Open a shell window, change directories to where you downloaded gsutil, and run the following command:
tar xfz gsutil.tar.gz -C $HOME
This installs gsutil in your home directory and creates a directory named gsutil . The gsutil directory should contain the gsutil tool, a gslib directory, and a third_party directory that contains several directories, including the boto library .
If you want to install gsutil in a different location, replace
$HOME
with another location on your computer. -
Add gsutil to your PATH environment variable.
Add the following line to your ~/.bashrc file on Linux or to your ~/.bash_profile on Mac OS X:
export PATH=${PATH}:$HOME/gsutil
If you did not install gsutil in your home directory, be sure to change
$HOME/gsutil
so that it matches the location of gsutil. -
Restart your shell or terminal program.
In some cases you might have to restart your shell or terminal program in order for the PATH environment variable to take effect.
That's it. You're ready to start using gsutil. To see a listing of gsutil commands, type
gsutil
at the command prompt.
Updating gsutil
Gsutil has a built-in update command that checks for the latest gsutil release and offers to let you automatically update to it if it differs from the version you have currently installed. Simply run the following command:
gsutil update
Windows
Note: The following procedure helps you install gsutil on Windows so that you can run gsutil at the Windows command prompt.
-
Download and install
Python
.
Install a version of Python that will work with gsutil (see System Requirements above). We recommend that you install a 32-bit version of Python and that you install Python in its default location (for example,
C:\Python27
). -
Download
gsutil
.
gsutil is bundled in a single archive named gsutil.zip. Save the archive in a convenient location, such as
C:\
. -
Extract the archive files.
We recommend that you extract the archive files into your root directory. This should create a directory named gsutil in your root directory (for example,
C:\gsutil
). The gsutil directory should contain the gsutil tool, a gslib directory, and a third_party directory that contains several directories, including the boto library. -
Start a command prompt window and run
python gsutil
to get a list of gsutil commands.
That's it. You're ready to start using gsutil.
Note:
To run gsutil commands on Windows you must first call the Python interpreter. For example, to see a listing of gsutil commands you must type
python gsutil
at the command prompt. Also, if you run gsutil outside of the
C:\gsutil
directory you must provide the explicit path to the gsutil executable. For example, to run gsutil in the
C:\Users\Admin\My Documents
directory you must type
python C:\gsutil\gsutil
at the command prompt.
Updating gsutil
gsutil has a built-in update command that checks for the latest gsutil release and offers to let you automatically update to it if it differs from the version you have currently installed. Simply run the following command:
python gsutil update
Facilitating the use of gsutil
The following steps show how to set environmental variables permanently in the Advanced System Properties. This can facilitate the use of the tool.
-
Associate file names with
.py
extension to the Python interpreter. -
Rename
gsutil
togsutil.py
. This enables command line execution. -
Add the
gsutil
directory absolute path to thePATH
environmental variable. -
Create or append
gsutil\boto
absolute path to thePYTHONPATH
environment variable. -
If it does not exist, create a
HOME
environmental variable to locate the tool and set the proper access rights (tryecho %USERPROFILE%
in a command window). -
In a command line console execute
gsutil.py ls
to create the.boto
config file. Any valid command will work.
Now you can use the python command to run gsutil from a command line console.
To learn how to use gsutil, see Getting Started: Using the gsutil Tool exercise, or simply run "gsutil help".
Installing from the Python package index (PyPi)
Experimental!
Installing gsutil via PyPi is an experimental and rapidly changing new feature for Google Cloud Storage. Unfortunately, this means that we may make backwards-incompatible changes. We will inform the community when this feature is no longer experimental.
CentOS, RHEL, and Fedora
-
Install required system packages.
Several packages are required to successfully install gsutil from PyPi. You can install them with the following command:
sudo yum install gcc openssl-devel python-devel python-setuptools libffi-devel
-
Install pip.
We recommend using the pip installer. You can install it with the following command:
sudo yum install python-pip
If you are using CentOS or RHEL and yum can't find the package
python-pip
, it's likely because you do not have the Extra Packages for Enterprise Linux (EPEL) repository source enabled. To enable it, follow these instructions . -
Install gsutil from PyPi.
To install gsutil from PyPi, use the following command:
sudo pip install gsutil
That's it. You're ready to start using gsutil. To see a listing of gsutil commands, type
gsutil
at the command prompt.
Updating gsutil
The gsutil update command is not available when installing via PyPi. To update gsutil, use the pip installer:
sudo pip install -U gsutil
Debian and Ubuntu
-
Install required system packages.
Several packages are required to successfully install gsutil from PyPi. You can install them with the following command:
sudo apt-get install gcc python-dev python-setuptools libffi-dev
-
Install pip.
We recommend using the pip installer. You can install it with the following command:
sudo apt-get install python-pip
-
Install gsutil from PyPi.
To install gsutil from PyPi, use the following command:
sudo pip install gsutil
That's it. You're ready to start using gsutil. To see a listing of gsutil commands, type
gsutil
at the command prompt.
Updating gsutil
The gsutil update command is not available when installing via PyPi. To update gsutil, use the pip installer:
sudo pip install -U gsutil
Mac OS X
-
Install pip.
We recommend using the pip installer. You can install it with the following command:
sudo easy_install -U pip
-
Install gsutil from PyPi.
To install gsutil from PyPi, use the following command:
sudo pip install gsutil
That's it. You're ready to start using gsutil. To see a listing of gsutil commands, type
gsutil
at the command prompt.
Updating gsutil
The gsutil update command is not available when installing via PyPi. To update gsutil, use the pip installer:
sudo pip install -U gsutil
Windows
Installing gsutil from PyPi is not support on Windows.
How to Set Up Credentials to Access Protected Data
Installing gsutil allows you to download and access publicly-accessible objects, but in order to access protected data or write to a protected bucket, you need to set up credentials to access the Google Cloud Storage service. For example, if someone else has created a Google Cloud Storage account and uploaded data that is only accessible to you or other specific individuals, you need to set up your credentials to the Google Cloud Storage service to be able to access this data.
Authenticate to Google Cloud Storage on Linux and OS X
- Open a new command prompt instance.
-
Run
gsutil config
at the command prompt.The first time you run this command, gsutil prints a URL to the authorization page where you can authorize gsutil to access Google Cloud Storage. The gsutil dialog looks similar to the following:
This script will create a boto config file at <your_homedir>/.boto containing your credentials, based on your responses to the following questions. Please navigate your browser to the following URL: <http://urlto/authorization/dialog> In your browser you should see a page that requests you to authorize gsutil to access Google Cloud Storage on your behalf. After you approve, an authorization code will be displayed. Enter the authorization code:
Note: If you want gsutil to launch a browser window for you, type
gsutil config -b
instead ofgsutil config
. However, if you are using SSH to connect to a remote machine, the-b
option may not work.
-
Copy and paste the URL into a browser window.
If prompted, sign in with the Google account associated with your Google Cloud Storage data in the browser window.
-
Click the Allow Access button.
This authorizes gsutil to access Google Cloud Storage.
- Copy the authorization code that appears on the next page into the gsutil prompt and press Enter.
-
Log in to the Google Developers Console to find a project ID you can specify as the default project.
After you provide an authorization code, gsutil prints the prompt below and prints the URL to the Google Developers Console . Copy and paste the URL into a browser window.
Please navigate your browser to the following URL: https://console.developers.google.com/ What is your project-id?
-
Copy and paste the project ID into gsutil.
Choose a project to set as the default for gsutil by clicking on the Overview tab on the left, copying the project ID and entering the project ID into the gsutil prompt. If you were given a project ID, you can enter it here. You can only set one project ID for gsutil at a time.
You can change the project ID for gsutil at any time by editing the boto configuration file .
That's it. You're ready to start using gsutil. To see a listing of gsutil commands, type
gsutil
at the command prompt.
Authenticate to Google Cloud Storage on Windows
Note: The following procedure helps you authenticate gsutil on Windows.
- Start a command prompt window.
-
Switch to the
C:\gsutil
directory. -
Run
python gsutil config
at the command prompt.The first time you run this command, gsutil prints a URL to the authorization page where you can authorize gsutil to access Google Cloud Storage. The gsutil dialog looks similar to the following:
This script will create a boto config file at <your_homedir>/.boto containing your credentials, based on your responses to the following questions. Please navigate your browser to the following URL: <http://urlto/authorization/dialog> In your browser you should see a page that requests you to authorize gsutil to access Google Cloud Storage on your behalf. After you approve, an authorization code will be displayed. Enter the authorization code:
Note: If you want gsutil to launch a browser window for you, type
python gsutil config -b
instead ofpython gsutil config
. However, if you are using SSH to connect to a remote machine, the-b
option may not work.
-
Copy and paste the URL into a browser window.
If prompted, sign in with the Google account associated with your Google Cloud Storage data in the browser window.
-
Click the Allow Access button.
This authorizes gsutil to access Google Cloud Storage.
- Copy the authorization code that appears on the next page into the gsutil prompt.
-
Log in to the Google Developers Console to find a project ID you can specify as the default project.
After you provide an authorization code, gsutil prints the prompt below and prints the URL to the Google Developers Console . Copy and paste the URL into a browser window.
Please navigate your browser to the following URL: https://console.developers.google.com/ What is your project-id?
-
Copy and paste the project ID into gsutil.
Choose a project to set as default for gsutil by clicking on the Overview tab on the left, copying the project ID and entering the project ID into the gsutil prompt. If you were given a project ID, you can enter it here. You can only set one project ID for gsutil at a time.
You can change the project ID for gsutil at any time by editing the boto configuration file .
That's it. You're ready to start using gsutil.
Note:
To run gsutil commands on Windows you must first call the Python interpreter. For example, to see a listing of gsutil commands you must type
python gsutil
at the command prompt. Also, if you run gsutil outside of the
C:\gsutil
directory you must provide the explicit path to the gsutil executable. For example, to run gsutil in the
C:\Users\Admin\My Documents
directory you must type
python C:\gsutil\gsutil
at the command prompt.
To learn how to use gsutil, see the Hello Google Cloud Storage! exercises.
How to convert gsutil to use OAuth 2.0
If you are currently using HMAC authentication and developer keys with gsutil, we recommend that you switch to using OAuth 2.0 authentication instead. Follow these instructions to set up OAuth 2.0 for gsutil.
- Download the latest version of gsutil.
-
Extract the archive files to the directory where gsutil is currently installed.
tar xfz gsutil.tar.gz -C $HOME
If gsutil is currently installed in another directory, replace
$HOME
with that location. The new gsutil files will replace the existing files.For Windows, you will need to download an extraction tool to extract the archive files. The Windows operating system does not provide an extraction utility for .tar files.
-
Run
gsutil config
at the command prompt.gsutil prints a URL to the authorization page where you can authorize gsutil to access Google Cloud Storage. The gsutil dialog looks similar to the following:
Backing up existing config file '<your_homedir>/.boto' to '<your_homedir>/.boto.bak'... This script will create a boto config file at <your_homedir>/.boto containing your credentials, based on your responses to the following questions. Please navigate your browser to the following URL: <http://urlto/authorization/dialog> In your browser you should see a page that requests you to authorize gsutil to access Google Cloud Storage on your behalf. After you approve, an authorization code will be displayed. Enter the authorization code:
Note: If you want gsutil to launch a browser window for you, type
gsutil config -b
instead ofgsutil config
. However, if you are using SSH to connect to a remote machine, the-b
option may not work.If gsutil shows an error message like the following:
CommandException: Cannot back up existing config file '<your_homedir>/.boto': backup file exists ('<your_homedir>.boto.bak').
gsutil could not create a new configuration file because there are two existing configuration files. You will need to move or rename one file in order for gsutil to continue the configuration process.
It is also possible to express a different scope for gsutil by including the following flags with the
gsutil config
command:-
Request token with read-only access:
-r
-
Request token with read-write access:
-w
-
Request token with full-control (including changing Access Control Lists (ACLs):
-f
-
Request a token with scope for other Google Services:
-s <scope>
Note: Generally, gsutil does not have the ability to make requests to other Google APIs. However, you can use this option in combination with the
-r, -w,
or-f
flags to obtain a token that can access both Google Cloud Storage and another Google API. To find the specific scope strings for an API, refer to the API's documentation.
-
Request token with read-only access:
-
Copy and paste the URL into a browser window.
If prompted, sign in with the Google account associated with your Google Cloud Storage data in the browser window.
-
Click the Allow Access button.
This authorizes gsutil to access Google Cloud Storage.
- Copy the authorization code that appears on the next page into the gsutil prompt.
-
Log in to the Google Developers Console to find a project ID you can specify as the default project.
After you provide an authorization code, gsutil prints the prompt below and prints the URL to the Google Developers Console . Copy and paste the URL into a browser window.
Please navigate your browser to the following URL: https://console.developers.google.com/ What is your project-id?
-
Copy and paste the project ID into gsutil.
Choose a project to set as default for gsutil by clicking on the Overview tab on the left, copying the project ID and entering the project ID into the gsutil prompt.
You can change the project ID for gsutil at any time by editing the boto configuration file . When you use the XML or JSON API, you must provide a project ID for certain operations (such as creating buckets). gsutil allows you to specify a default project ID in the boto configuration file, so you don't need to explicitly specify the project ID on operations when you work with the default project.
Congratulations, gsutil is now set up to use OAuth 2.0 authentication!
How to use boto with Google Cloud Storage
To use boto with Google Cloud Storage, we recommend that you configure boto to use OAuth 2.0 authentication:
Linux and Mac OS X
-
Set up your boto configuration file to use OAuth 2.0.
You can do so using gsutil and running the command
gsutil config
. If you don't have gsutil installed, see the gsutil installation instructions . -
In your boto configuration file, make sure that the
https_validate_certificates
flag in the [Boto] section is not set toFalse
.This implements server certificate validation. For information about the location of the configuration file, see boto configuration file .
-
Include the gsutil, boto, and third_party libraries in your PYTHONPATH.
Add the following line to your .bashrc or .bash_profile.
export PYTHONPATH=${PYTHONPATH}:$HOME/gsutil/third_party/boto:$HOME/gsutil
If you installed gsutil in a directory other than your home directory, replace
$HOME
with the directory where your gsutil directory is located. -
Import the oauth2_plugin.
Run the following command to import the oauth2_plugin:
from gslib.third_party.oauth2_plugin import oauth2_plugin
That's it! You can now use boto with OAuth 2.0 authentication. The following example uses boto with OAuth 2.0 authentication to request the contents of a text file named rose.text:
from boto import storage_uri from gslib.third_party.oauth2_plugin import oauth2_plugin content = storage_uri('gs://pub/shakespeare/rose.txt').get_contents_as_string() print(content)
Windows
-
Set up your boto configuration file set up to use OAuth 2.0.
You can do so using gsutil and running the command
python gsutil config
. If you don't have gsutil installed, see the gsutil installation instructions . -
In your boto configuration file, make sure that the
https_validate_certificates
flag in the [Boto] section is not set toFalse
.This implements server certificate validation. For information about the location of the configuration file, see boto configuration file .
-
Include the gsutil, boto, and third_party libraries in your PYTHONPATH.
- Open System properties in Control Panel, click the Advanced tab, and then click Environment Variables .
-
Create a new variable named PYTHONPATH and assign it the following value:
C:\gsutil\third_party\boto
. - Reboot for the environment variable change to take effect.
-
Import the oauth2_plugin.
Run the following command to import the oauth2_plugin:
from gslib.third_party.oauth2_plugin import oauth2_plugin
Thats it! You can now use boto with OAuth 2.0 authentication. The following example uses boto with OAuth 2.0 authentication to request the contents of a text file named rose.text:
from boto import storage_uri from gslib.third_party.oauth2_plugin import oauth2_plugin content = storage_uri('gs://pub/shakespeare/rose.txt').get_contents_as_string() print(content)
boto configuration file
The boto/gsutil configuration file specifies values that control how
gsutil behaves, as well as credentials and OAuth2 settings. You typically work
with the configuration file indirectly by using the
gsutil config
command.
In some situations, you might need to edit the file directly, for example,
to change the default project ID. For more information about what the boto configuration file
contains, see
gsutil config
.
The default location for the
boto configuration file is in the user home directory, ~/.boto, for Linux and Mac OS X,
and in %HOMEDRIVE%%HOMEPATH%, for Windows.
You can get the location of the configuration file by
running the command
gsutil version -l
.
You can override where the config file is used by setting the BOTO_CONFIG environment variable. You can also set up a path of boto config files to load. This could be useful, for example, in data sharing and collaboration scenarios.