The Usage Export feature allows you to export detailed reports of your Google Developers Console project's Compute Engine usage to a Google Cloud Storage bucket. These detailed reports are CSV files with information about usage at the resource level, such as the running time of a virtual machine instance, the storage space of a persistent disk, and information about other Compute Engine features.
The usage export feature is useful for users who want to better understand their Compute Engine usage. Users who are interested in billing information can use the billing export feature to download billing data.
Contents
Overview
When you enable the usage export feature, Compute Engine delivers two types of reports to the Google Cloud Storage bucket you specified, for which you must also be an owner. The two types of reports are:
-
Daily usage reports
These reports are delivered daily and include usage data from the latest 24 hour period. Each report is a separate file that contains immutable data from the last period; if there are inaccuracies in a report, the data is corrected in the next daily report.
Daily usage reports have the following name format:
<report_prefix>_<numeric_project_id>_<YYYYMMDD>.csv
-
Monthly rollup report
A single monthly rollup report is delivered daily, which contains monthly usage data for that project up to that day. The monthly usage report is overwritten each day with new data that reflects the monthly usage of resources up to that date. There will only be one monthly usage data file per project, per month.
Monthly rollup reports have following name format:
<report_prefix>_<numeric_project_id>_<YYYYMM>.csv
Note the difference in date format, where the monthly rollup reports are dated using the year and month (
YYYYMM
), and the daily usage reports are dated using the year, month, and date (YYYYMMDD
).
All usage reports are delivered in
comma-separated values (CSV)
format
and usage report files are prefixed using
<report_prefix>
. The
<report_prefix>
is a customizable value chosen by the user. If you don't specify a report prefix,
the prefix
usage_gce
will be used by default. All dates are given in Pacific time (PST).
Supported metrics
Daily usage reports provide usage information about the following resources:
- Virtual machines
- Persistent disks
- Images
- Snapshots
- Static IP addresses
- Load balancer
Each resource is described using the following metrics:
Metric Name | Metric Properties |
---|---|
Report Date |
|
MeasurementId |
|
Quantity |
|
Unit |
|
Resource URI |
|
ResourceId |
|
Location |
|
An example entry in the report would look like the following:
Report Date | MeasurementId | Quantity | Unit | Resource URI | Resource ID | Location |
---|---|---|---|---|---|---|
02/13/2014 |
com.google.cloud/services/compute-engine/VmimageN1Standard_1
|
86400 | seconds |
http://www.googleapis.com/compute/v1/projects/myproject/zones/us-central1-a/instances/my-instance
|
16557630484 | us-central1-a |
Access control
When you enable the usage export feature for a Cloud Storage bucket, Compute Engine automatically adds itself to the bucket with write access in order to deliver usage reports. As long as Compute Engine has access to the bucket and the usage export feature is enabled, Compute Engine will continue to export usage reports to the specified Cloud Storage bucket.
You can identify that Compute Engine has access to a bucket if you see the following identity added to the bucket access control list:
[email protected]
Any user who is an owner of the project has full access to the Google Cloud Storage bucket. Other users, such as writers and readers, have different degrees of access to the bucket. To learn about ACLs for a bucket, read the Cloud Storage access control documentation.
If you disable the usage export feature, Compute Engine will automatically remove write access from Compute Engine to the bucket. If you modify the permissions on the [email protected] account and then disable the usage export feature, Compute Engine will disable the usage export feature but won't remove the account from the project access list. You can choose to remove the account manually if desired.
Prerequisites
Before you can start using Compute Engine usage export, you must sign up for Google Cloud Storage if you haven't already.
Setting up usage export
When you first enable the usage export feature, the first report will be sent the following day, detailing the previous day's usage. Afterwards, you will receive reports in 24 hour intervals.
To set up the usage export feature, you can enable the feature in the
gcloud compute
tool. When you enable the feature, you must define two properties:
-
The Google Cloud Storage bucket where you would like your reports to be delivered.
You can select any Cloud Storage bucket for which you are an owner, including buckets that are from different projects. This bucket must exist before you can start exporting reports and you must have owner access to the bucket. Google Cloud Storage charges for usage, so you should review the Cloud Storage pricesheet for information on how you might incur charges for the service.
Any user who has read access to the Cloud Storage bucket will be able to view the usage reports in the bucket. Any user who has write access to the bucket will be able to create, view, and modify existing files in the bucket. For more information, see the Access control section.
-
The report prefix for your files.
You can specify the report prefix to use for your usage reports. Your usage reports will have file names that contain this prefix. For example, specifying "my-cool-project-report" as your report prefix results in a file name similar to the format
my-cool-project-report_1234567890_20131230.csv
. If you do not specify a report prefix, the default prefixusage_gce
is used.
After you decide on these two properties, you can enable the usage export feature in the following ways:
Enable usage export in gcloud compute
In
gcloud compute
, use the
gcloud compute project-info set-usage-bucket
command to enable this feature:
$ gcloud compute project-info set-usage-bucket --bucket URI [--prefix PREFIX]
The URI to the bucket can be in the format
gs://<bucket-name>
or
https://storage.googleapis.com/<bucket-name>
.
Enabling usage export in the Developer Console
- Log into the Google Developers Console .
- Select the project for which you want to enable the usage export feature.
- On the left-hand menu, click Settings .
- Scroll down to the Compute Engine usage export section.
-
Fill in the fields asking for a
Bucket name
and
Report prefix
,
if desired. If you leave the report prefix empty, the default
prefix
usage_gce
is used. - Click Save .
Enabling usage export in the API
Client libraries
Using the client libraries, set the
setUsageExportBucket()
method in the
Projects
collection to enable usage export. For example, in the Python
Client Library, you can enable the feature like so:
def setUsageExport(gce_service, auth_http):
body = {
"bucketName": "https://storage.googleapis.com/usage-export-sample",
"reportNamePrefix": "exampleprefix"
}
request = gce_service.projects().setUsageExportBucket(project=PROJECT_ID, body=body)
response = request.execute(http=auth_http)
print response
HTTP
You can also directly make a HTTP request to the
setUsageExportBucket
method as well. The following example uses the
httplib2
library to enable
usage export:
#!/usr/bin/python
import urllib
import argparse
import logging
import sys
import random
from json import load,dumps
...
PROJECT_ID = "myproject"
API_VERSION = "v1"
API_URL = "https://www.googleapis.com/compute/" + API_VERSION + "/projects/" + PROJECT_ID
OAUTH_FILE = "oauth-dev.dat"
def main(argv):
logging.basicConfig(level=logging.INFO)
parser = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
parents=[tools.argparser])
# Parse the command-line flags.
flags = parser.parse_args(argv[1:])
http = httplib2.Http()
# Add code to authenticate to the service
....
url = API_URL + "/setUsageExportBucket"
body = {
"bucketName": "https://storage.googleapis.com/usage-export-sample",
"reportNamePrefix": "exampleprefix"
}
formattedBody = dumps(body)
resp,content = http.request(url,
"POST",
headers=listOfHeaders,
body=formattedBody)
print str(content)
if __name__ == "__main__":
main(sys.argv)
For more information, see the API reference documentation.
Checking if usage export is enabled
You can check on a project's usage export settings by getting information about the project:
$ gcloud compute project-info describe
Look for the
usageExportLocation
field:
+-------------------------+----------------------------------------------------+ | name | my-project-id | | description | | | creation-time | 2012-10-18T16:31:52.308-07:00 | | usage | | | snapshots | 1.0/1000.0 | | networks | 2.0/2.0 | | firewalls | 3.0/10.0 | |... | | | usageExportLocation | | | bucketName | https://storage.googleapis.com/usage-export-sample | | reportNamePrefix | | +-------------------------+----------------------------------------------------+
Disabling usage export
When you disable usage reports, Compute Engine automatically removes write access for Compute Engine to your Cloud Storage bucket and discontinues sending any new reports.
Disable usage export in gcloud compute
In
gcloud compute
, you can disable the usage export feature by using the
gcloud compute project-info set-usage-bucket
command with an empty
--bucket
flag:
$ gcloud compute project-info set-usage-bucket --bucket
Disabling usage export in the API
Client libraries
To use the client libraries, make a request to the
setUsageExportBucket()
method, with a blank bucket name. For example, in the Python Client Library,
you can disable the feature like so:
def disableUsageExport(gce_service, auth_http):
body = {
"bucketName": "",
}
request = gce_service.projects().setUsageExportBucket(project=PROJECT_ID, body=body)
response = request.execute(http=auth_http)
print response
HTTP
You can also directly make a HTTP request to the
setUsageExportBucket
method as well. The following example uses the httplib2 library to make a
request to disable the usage export bucket:
#!/usr/bin/python
import urllib
import argparse
import logging
import sys
import random
from json import load,dumps
...
PROJECT_ID = "myproject"
API_VERSION = "v1"
API_URL = "https://www.googleapis.com/compute/" + API_VERSION + "/projects/" + PROJECT_ID
OAUTH_FILE = "oauth-dev.dat"
def main(argv):
logging.basicConfig(level=logging.INFO)
parser = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
parents=[tools.argparser])
# Parse the command-line flags.
flags = parser.parse_args(argv[1:])
http = httplib2.Http()
# Add code to authenticate to the service
....
url = API_URL + "/setUsageExportBucket"
body = {
"bucketName": ""
}
formattedBody = dumps(body)
resp,content = http.request(url,
"POST",
headers=listOfHeaders,
body=formattedBody)
print str(content)
if __name__ == "__main__":
main(sys.argv)
Disable usage export in Developers Console
- Log into the Google Developers Console .
- Select the project for which you want to disable this feature.
- Click Settings in the left-hand menu.
- Scroll down to the Compute Engine usage export section.
- Click Disable usage export .