Getting Started
Before running through the steps below, make sure that:
- You have activated your project for GCS and App Engine, including creating at least one GCS bucket.
- You have downloaded the client lib distribution and unzipped it.
- You have installed and configured the latest App Engine Java SDK.
Running the LocalExample.java Sample (Eclipse)
LocalExample.java is a non-deployable sample useful for quick testing and investigation of the GCS functionality. It has no UI components other than Eclipse console output. ( GCS client library deployable samples with UI are also available.)
To run LocalExample.java in Eclipse:
- Start Eclipse.
- In Eclipse, click Windows > Preferences > Google > App Engine and then click Add . (On Mac OS X: Eclipse > Preferences > Google > App Engine .)
- Follow the prompts to supply the install path of the App Engine SDK, then click OK .
-
In the
Files
menu, click
Files > New > Java Project
and create a project
named
LocalExample
with a package name ofcom.google.appengine.demos
. -
Select the project in the Package Explorer pane and click
Files > New >
Class
and give the class the name
LocalExample
with a package name ofcom.google.appengine.demos
. - Copy the contents of the LocalExampleJava source into that class file.
- Select the project again in the Package Explorer pane and right-click, then click Properties > Java Build Path .
-
In the
Libraries
tab, click
Add External Jars
. You must add the following
JARs:
-
appengine-gcs-client.jar
from wherever you installed the GCS client library -
guava-15.0.jar
from wherever you installed the GCS client library -
joda-time-2.3.jar
from wherever you installed the GCS client library -
appengine-testing.jar
from the App Engine install subdirectory/lib/testing
. -
appengine-api.jar
from the App Engine install subdirectory/lib/impl
. -
appengine-api-stubs.jar
from the App Engine install subdirectory/lib/impl
.
-
- Build, then Run As > Java Application .
-
You should see the following output in the Eclipse Console pane:
Investigating the LocalExample.java Sample
LocalExample.java is described in detail below, to illustrate the use of the GCS client library.
Imports
There are a number of imports used that you might not need, or that are used to enable local testing. The followng snippet lists this sample's imports:
import com.google.appengine.tools.cloudstorage.GcsFileOptions; import com.google.appengine.tools.cloudstorage.GcsFilename; import com.google.appengine.tools.cloudstorage.GcsInputChannel; import com.google.appengine.tools.cloudstorage.GcsOutputChannel; import com.google.appengine.tools.cloudstorage.GcsService; import com.google.appengine.tools.cloudstorage.GcsServiceFactory; import com.google.appengine.tools.cloudstorage.RetryParams; import com.google.appengine.tools.development.testing.LocalBlobstoreServiceTestConfig; import com.google.appengine.tools.development.testing.LocalDatastoreServiceTestConfig; import com.google.appengine.tools.development.testing.LocalServiceTestHelper; import java.io.IOException; import java.io.ObjectInputStream; import java.io.ObjectOutputStream; import java.nio.ByteBuffer; import java.nio.channels.Channels; import java.util.Arrays; import java.util.HashMap; import java.util.Map;
These imports are briefly described below:
-
com.google.appengine.tools.cloudstorage.*
this lets us use the GCS client library. -
com.google.appengine.tools.development.testing.*
Only needed for local unit testing of certain App Engine features while running your app. -
java.io.ObjectInputStream
andjava.io.ObjectOutputStream
are used for reading and writing objects. -
java.nio.ByteBuffer
is used for unbuffered reads and writes. -
Although not shown in the snippets,
java.io.IOException
is needed for error handling. -
java.nio.channels.Channels
is used to convert the input or output channel to a stream. -
java.nio.channels.ReadableByteChannel
is used for the data being read in from GCS.
Creating a GcsService To Send Requests to GCS
To send and receive requests to GCS via this library, you need a GcsService instance:
private final GcsService gcsService = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
In the snippet, notice the use of
RetryParams
in the call to
createGcsService
. Supplied as shown,
createGcsService(RetryParams.getDefaultInstance())
, this sets the default
retry settings to be used for retries in the event
of request failures due to timeouts or other unexpected failures in accessing
GCS. To specify different values, such as for maximum number of retries, you
use
RetryParams.Builder
to change settings in a new
RetryParams
object
and supply it when you create
GcsService
. Note that once a
GcsService
object is created, its retry parameters cannot be changed.
You can create as many
GcsService
instances as you need. Each instance is
independent, immutable (and therefore threadsafe), and reusable. For example,
one can be used to write a file with one set of parameters, and another can read
a different file with different retry parameters at the same time.
A recommended practice is to use a separate (possibly static) instance in each of your classes that does I/O; this will help keep your classes independent and has very little overhead.
Writing Data to GCS
The following samples show how to write data to a file stored in GCS. Separate snippets are provided for writing serialized object data and byte arrays.
GCS files are not fully created until
close
is invoked.
In the snippets below,
close
does not appear in a
finally
block,
as is often done in Java. This is so that in the event of an exception,
any partly written data will automatically be cleaned up.
Writing an Object to a GCS File
Here is how to serialize an object to a GCS file. First, get a writable byte channel for GCS:
GcsOutputChannel outputChannel = gcsService.createOrReplace(fileName, GcsFileOptions.getDefaultInstance());
In the snippet,
GcsService.createOrReplace
is invoked with the
GcsFilename
as the first parameter. This object contains the name of the
GCS bucket to be used and name of the object. If there already is an object with
that same name in that bucket, and your app has write permissions to it, this
call will overwrite the existing file. (GCS does not support appending.) If
there is no file of that name, this call results in the creation of a new file.
The second parameter shown is
GcsFileOptions
. To supply default options,
use
GcsFileOptions.getDefaultInstance
. To supply your own settings, use
GcsFileOptions.Builder
to set options and create a
GcsFileOptions
to supply as the second parameter.
The file options are passed through to GCS to tell it the content type of the
file, any user metadata you want passed in the headers, the ACL to govern access to
the file, and so forth. If you don't supply a content type (
mimeType
), the file
will be served from GCS in the default MIME type, which is currently
binary/octet-stream
. If you don't specify an ACL, the access assigned to the
object will be the current
default object ACL
.
Note that all of the file options information can be obtained from an object
after a
close
without having to download the object itself, by calling
GcsService.getMetadata(fileName)
.
For more information on the above settings, visit the Google Cloud Storage documentation on ACLs and file options .
Now write the data using an output stream:
@SuppressWarnings("resource") ObjectOutputStream oout = new ObjectOutputStream(Channels.newOutputStream(outputChannel)); oout.writeObject(content); oout.close();
Writing a Byte Array to a GCS File
Here is how to write a byte array to a GCS file:
@SuppressWarnings("resource") GcsOutputChannel outputChannel = gcsService.createOrReplace(fileName, GcsFileOptions.getDefaultInstance()); outputChannel.write(ByteBuffer.wrap(content)); outputChannel.close();
For a description of the parameters used in the
createOrReplace
call, see the
description under
Writing an Object to a GCS File
.
Reading Data from GCS
The following samples show how to read data from a file stored in GCS. Separate snippets are provided for reading the GCS file into an object (serialize) and into a byte array.
Reading GCS Data into an Object
This approach is useful for streaming larger files into a buffer:
GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(fileName, 0, 1024 * 1024);
The call to GcsService.openPrefetchingReadChannel takes a GcsFilename , which contains the name of the GCS bucket to be read from and name of the object to be read. The second parameter is the number of bytes from the start of the file, with 0 as shown causing the read to start at the beginning of the file. If you supply some other starting point within the file, say, at byte 300, the read operation begins at byte 300 and continues to the end of the file, or until you stop reading. (The prefetch buffer will read ahead, though, so it will contain bytes past the point where you stop.)
Prefetching provides is a major performance advantage for most applications, because it allows processing part of the file while more data is being downloaded in the background in parallel.
The third parameter is the size of the prefetch buffer, set in the example to 1 megabyte.
Now, read the file from the channel:
try (ObjectInputStream oin = new ObjectInputStream(Channels.newInputStream(readChannel))) { return oin.readObject(); }
Reading GCS Data into a Byte Array
For smaller files, you could read the file all at once into a byte array:
int fileSize = (int) gcsService.getMetadata(fileName).getLength(); ByteBuffer result = ByteBuffer.allocate(fileSize); try (GcsInputChannel readChannel = gcsService.openReadChannel(fileName, 0)) { readChannel.read(result); }
In the snippet above, note the use of
java.nio.ByteBuffer
, especially the
setting of the buffer size to be equal to the size of the file being read in
from the channel.
Note, however, that for most applications, the prefered method is to stream the file (see Reading GCS Data into an Object ) because that doesn't require holding the all of the data in memory at once.
Deployable GCS Client Library Samples with UI
For deployable samples with UI, visit the source code for:
- GcsExampleServlet.java , which uploads/downloads files to/from GCS.
- PortOfFilesAPIGuestbookServlet.java , which is a port of a sample program that formerly used the deprecated Files API. For more information about migrating, see [Migrating from the Google Cloud Storage API][/appengine/docs/java/googlecloudstorageclient/migrate].
You'll need to build the samples before you can run them against the development server or deploy them. To build the samples:
-
Checkout the source code by invoking the following command from a terminal
window:
svn checkout http://appengine-gcs-client.googlecode.com/svn/trunk/ appengine-gcs-client-read-only
-
Change directory to
appengine-gcs-client-read-only/java
. -
Invoke
ant compile_example
, which will build the samples using thebuild.xml
in that directory. For more information about using Apache Ant, see Using Apache Ant . -
Run the dev server by invoking
/path/to/AppEngSDK/dev_appserver.sh /path/to/example/war
-
In your browser, visit
localhost:8080
. ForGCSExampleServlet
you should see something like this: