Before you can query your data, you first need to load it into BigQuery. You can bulk load the data by using a job , or stream records individually .

Load jobs support two data sources:

1. Objects in Google Cloud Storage
2. Data sent with the job or streaming insert

Loaded data can be added to a new table, appended to a table, or can overwrite a table. Data can be represented as a flat or nested/repeated schema, as described in Data formats . Each individual load job can load data from multiple sources, configured with the sourceUris property .

It can be helpful to prepare the data before loading it into BigQuery, or transform the data if needed.

Contents

• Access control
• Quota policy
• Additional limits
• Loading data from Google Cloud Storage
• Loading data by sending a POST request
• Loading data from other Google services

Access control

Loading data into BigQuery requires the following access levels .

Back to top

Quota policy

The following limits apply for loading data into BigQuery .

• Daily limit: 1,000 load jobs per table per day (including failures), 10,000 load jobs per project per day (including failures)
• Maximum File Sizes:

  File Type	Compressed	Uncompressed
  CSV	1 GB	With new-lines in strings: 4 GB Without new-lines in strings: 1 TB
  JSON	1 GB	1 TB

• Maximum size per load job: 1 TB across all input files
• Maximum number of files per load job: 10,000
• There are several additional limits that are specific to BigQuery's supported data formats. For more information, see preparing data for BigQuery .

Back to top

Additional limits

The following additional limits apply for loading data into BigQuery.

• Maximum columns per table: 10,000
• Data format limits : Depending on which format you use to load your data, additional limits may apply. For more information, see Data formats .

Back to top

Loading data from Google Cloud Storage

To load data from Google Cloud Storage:

1. Upload your data to Google Cloud Storage.

   The easiest way to upload your data to Google Cloud Storage is to use the Google Developers Console . Be sure that you upload your data to a project the BigQuery service is activated on.

2. Create a load job pointing to the source data in Google Cloud Storage. The source URIs must be fully-qualified, in the format gs://<bucket>/<object> .
3. Check the job status.

   Call jobs.get( <jobId> ) with the ID of the job returned by the initial request, and check for status.state = DONE ; if the status.errorResult property is present, the request failed, and that object will include information describing what went wrong. If the request failed, no table will have been created or data added. If status.errorResult is absent, the job finished successfully, although there might have been some non-fatal errors, such as problems importing a few rows. Non-fatal errors are listed in the returned job object's status.errors property.

Example

The following Python client example loads CSV data from a Google Cloud Storage bucket and prints the results on the command line.

# Python example
# Loads the table from Google Cloud Storage and prints the table.
def loadTable(service, projectId, datasetId, targetTableId, sourceCSV):
  try:
    jobCollection = service.jobs()
    jobData = {
      'projectId': projectId,
      'configuration': {
          'load': {
            'sourceUris': [sourceCSV],
            'schema': {
              'fields': [
                {
                  'name': 'Name',
                  'type': 'STRING'
                },
                {
                  'name': 'Age',
                  'type': 'INTEGER'
                },
                {
                  'name': 'Weight',
                  'type': 'FLOAT'
                },
                {
                  'name': 'IsMagic',
                  'type': 'BOOLEAN'
                }
              ]
            },
            'destinationTable': {
              'projectId': projectId,
              'datasetId': datasetId,
              'tableId': targetTableId
            },
          }
        }
      }

    insertResponse = jobCollection.insert(projectId=projectId,
                                         body=jobData).execute()

    # Ping for status until it is done, with a short pause between calls.
    import time
    while True:
      job = jobCollection.get(projectId=projectId,
                                 jobId=insertResponse['jobReference']['jobId']).execute()
      if 'DONE' == job['status']['state']:
          print 'Done Loading!'
          return

      print 'Waiting for loading to complete...'
      time.sleep(10)

    if 'errorResult' in job['status']:
      print 'Error loading table: ', pprint.pprint(job)
      return

  except HttpError as err:
    print 'Error in loadTable: ', pprint.pprint(err.resp)

Back to top

Loading data by sending a POST request

You can load data into BigQuery by sending a multipart/related POST request to the following URL:

https://www.googleapis.com/upload/bigquery/v2/projects/<projectId>/jobs

The request body must be in the following format:

--xxx
Content-Type: application/json; charset=UTF-8

{
  "configuration": {
    "load": {
      "sourceFormat": "<required for JSON files>",
      "schema": {
        "fields": [
          {"name":"f1", "type":"STRING"},
          {"name":"f2", "type":"INTEGER"}
        ]
      },
      "destinationTable": {
        "projectId": "projectId",
        "datasetId": "datasetId",
        "tableId": "tableId"
      }
    }
  }
}
--xxx
Content-Type: application/octet-stream

<your data>
--xxx--

The request body contains several key parts:

• Encapsulation Boundary (e.g. "xxx")

  When you create a multipart request that contains different sets of data in a single body, you need to specify a boundary parameter to indicate where each dataset starts and end. The boundary string must be prefixed with two dashes. To indicate the end of the entire body, you must specify the boundary string in the following format:

  --<boundary>--

  You must also specify this boundary string in the overall document's Content-Type header in the following format:

  Content-Type: multipart/related; boundary=xxx

  Note that the boundary string cannot occur within the actual data of the request.

• Load job configuration

  The request body contains two parts. The load configuration should be the first part of the request, with a Content-Type header of application/json . This section creates a load job using the JSON fields described in the job resource documentation.

• The data of the request

  The second part of the request contains data to be loaded into BigQuery. The part must always have a Content-Type header of application/octet-stream .