Getting started load testing your App Engine application

In the documentation for App Engine and in presentations we've given at Google I/O we have mentioned that you should ramp up slowly when load testing an application on App Engine. Ramping up too quickly won't give an accurate picture of how App Engine scales; you have to accomodate our load balancing code which determines how many instances of your application to spin up by watching how much traffic is directed to your application. That monitoring and adjustment takes time, therefore the need for not ramping up too quickly.

I've looked at various load testing tools and in the end wrote my own short script in Python which I use as a base for all my load testing. This isn't to say that what I have here is better for load testing than the available packages, please look at them and judge them against your own criteria. I'm most comfortable with Python and a skeleton script that can be tweaked for each test scenario is optimal for me.

The load_test.py script is very small and appears below:

#    Copyright 2009 Google Inc.
#
#    Licensed under the Apache License, Version 2.0 (the "License");
#    you may not use this file except in compliance with the License.
#    You may obtain a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS,
#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#    See the License for the specific language governing permissions and
#    limitations under the License.

"""Simple web application load testing script.

This is a simple web application load
testing skeleton script. Modify the code between !!!!!
to make the requests you want load tested.
"""


import httplib2
import random
import socket
import time
from threading import Event
from threading import Thread
from threading import current_thread
from urllib import urlencode

# Modify these values to control how the testing is done

# How many threads should be running at peak load.
NUM_THREADS = 10

# How many minutes the test should run with all threads active.
TIME_AT_PEAK_QPS = 10 # minutes

# How many seconds to wait between starting threads.
# Shouldn't be set below 30 seconds.
DELAY_BETWEEN_THREAD_START = 30 # seconds

quitevent = Event()

def threadproc():
    """This function is executed by each thread."""
    print "Thread started: %s" % current_thread().getName()
    h = httplib2.Http(timeout=30)
    while not quitevent.is_set():
        try:
            # HTTP requests to exercise the server go here
            # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
            resp, content = h.request(
                "http://sparklines.bitworking.info/spark.cgi?type=smooth&d;=88,84,82,92")
            if resp.status != 200:
                print "Response not OK"
            # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
        except socket.timeout:
            pass

    print "Thread finished: %s" % current_thread().getName()


if __name__ == "__main__":
    runtime = (TIME_AT_PEAK_QPS * 60 + DELAY_BETWEEN_THREAD_START * NUM_THREADS)
    print "Total runtime will be: %d seconds" % runtime
    threads = []
    try:
        for i in range(NUM_THREADS):
            t = Thread(target=threadproc)
            t.start()
            threads.append(t)
            time.sleep(DELAY_BETWEEN_THREAD_START)
        print "All threads running"
        time.sleep(TIME_AT_PEAK_QPS*60)
        print "Completed full time at peak qps, shutting down threads"
    except:
        print "Exception raised, shutting down threads"

    quitevent.set()
    time.sleep(3)
    for t in threads:
        t.join(1.0)
    print "Finished"

There are two constants at the top of the file that you can change to control how the script behaves.

TIME_AT_PEAK_QPS

How long, in minutes, should the test run with all threads going.

NUM_THREADS

How many threads should be started all together.

Note that you can control the number of threads you start, but the script doesn't do any work to determine how many queries per second were made against the app.

resp, content = h.request(
    "http://sparklines.bitworking.info/spark.cgi?type=smooth&d;=88,84,82,92")
if resp.status != 200:
print "Response not OK"

If we wanted to modify the script to generate random data to plot, we could modify that section of code like so:

query = {
    'type': 'smooth',
    'd': ",".join([ str(randint(0, 100)) for i in range(10) ])
}
uri = "http://sparklines.bitworking.info/spark.cgi" + "?" + urlencode(query)
resp, content = h.request(uri)
if resp.status != 200:
    print "Response not OK"

The examples so far all use GET requests, see the httplib2 documentation for examples dealing with forms.

Once you've run your tests you can look at how your application handled the load through the Admin Console. The results of running the load test with the default parameters agains the sparklines service are shown below. You will notice that with just 10 threads we were able to drive the traffic up to 80 QPS. Obviously your application will vary depending on how much work it is doing for each request and how much data is returned.

The code above is released under the Apache 2 license, so grab a copy and start modifying it to suit your needs, and happy load testing!