What is App Engine?

Google App Engine made a splash when it launched in the spring of 2008. It was different from most other cloud systems back in the day because it was neither IaaS (Infrastructure-as-a-Service, e.g., Amazon EC2) nor SaaS (Software-as-a-Service, e.g., Salesforce). App Engine provides an alternative to fixed applications (SaaS) and raw hardware (IaaS). App Engine's managed infrastructure provides a development platform to create apps, leveraging Google's infrastructure as a hosting platform.

The development-to-release cycle is minimized because high-level services that developers would normally have to build are already available via an App Engine API. A development server is provided to let users test their code (with certain limitations) before running in production. And finally, deployment is simplified as Google handles that all. Outside of setting up an account and billing structure, there is no machine setup or administration as Google takes care of all logistics there too. Even as your app is running with fluctuating network traffic, the App Engine system auto-scales up to allocate more instances of your app as needed, then similarly releases resources when no longer needed.

A web developer can now use Google's infrastructure, finely tuned for speed and massive scaling, instead of trying to build it themselves. In the past, developers would create an app, generally need a machine or web hosting service that could host a LAMP stack, administer each of the "L", "A", "M", and "P" components, and somehow made the app globally accessible. Moreover, developers were also generally responsible for the load-balancing, monitoring, and reporting of their systems, and to reiterate one of the most difficult and expensive things to build yourself: scaling. All of these are taken care of by App Engine.

Why does App Engine exist?

By now, you have a good idea as to why Google developed App Engine. To put it simply, to remove the burden of being a system administrator from the developer. Using a LAMP stack involves choosing a distribution of Linux, choosing of the kernel version, etc., configuring PHP and an Apache web server. There is also the need to run and manage a database server (MySQL or otherwise) and other aspects of a running system (monitoring, load-balancing, reporting). The list continues with managing user authentication, applying software patches and performing upgrades, each of which may break your app, bringing even more headaches for developers/sysadmins.

Other than the actual application, everything else is nearly orthogonal to the solution that developers are trying to create for their customers. App Engine attempts to handle these complexities to let you focus on your app(s). An app running on App Engine should be easy to build, manage, and scale.

PaaS and what isn't App Engine?

Some users confuse Google App Engine with Amazon's EC2 service. The problem is that this is an apples-to-oranges comparison. The two services operate at different cloud service levels, and each has its own pluses and minuses. With App Engine, you only need to worry about your application; Google takes care of hosting and running it for you. With EC2, you're responsible for the app, but also its database server, web server, operating system, monitoring, load-balancing, upgrades, etc. This is the reason why, typically, the costs for IaaS services run lower than those of PaaS services: with PaaS, you're "outsourcing" more work and responsibility. Such cost estimates are usually clouded by not considering the administration overhead of managing the infrastructure yourself. A better, apples-to-apples comparison would be EC2 to the Google Compute Engine IaaS service.

PaaS systems also differ from that of the SaaS layer above, as SaaS applications are fixed and must be taken directly from the respective cloud vendor. Unless you work with or at the vendor, you cannot change the SaaS application you use. It’s quite the opposite with PaaS systems because you (as a PaaS user) are the developer, building and maintaining the app, so the source code is your responsibility. One interesting perspective is that you can use a PaaS service to build and run SaaS apps!

Language Runtimes

App Engine lets engineers use familiar development tools and environments to build their applications. This includes the Python, Java, and Go languages. Because App Engine supports Java, a host of other languages which run on the Java Virtual Machine (JVM) are also supported... these include (but are not limited to): Scala, Ruby (JRuby), Groovy, PHP (Quercus), JavaScript (Rhino), and Python (Jython). (If you're curious about the Jython support (running Python on the Java runtime vs. the pure Python runtime), it's a great way to leverage a legacy Java codebase while developing new code in Python as Jython code can work with both Python and Java objects.)

Sandbox

Security is gravely important. Developers (typically) would not be interested in letting other applications/users get any kind of access to their application code or data. To ensure this, all App Engine applications run in a restricted environment known as a sandbox .

Because of the sandbox, applications can't execute certain actions. These include: open a local file for writing, open a socket connection, and make operating system calls. (There used to be more restrictions, but over time, the team has tried to bump up quotas and remove as many restrictions as they can. These don't make the airwaves as much as bad news does.)

Services

Any network developer would say, "Without being able to support two-way socket connections, you can't create useful applications!" The same may be true of the other restrictions. However, if you think about it carefully, why would you want use these lower-level OS features? "To make useful apps with!" You want to use outbound sockets to talk to other processes, and perhaps you may want inbound sockets to listen for service requests.

The good news is that the App Engine team knows what you want, so that's why the team has created a set of higher-level APIs/services for developers to use. Want your app to send and receive e-mail or instant messages? That's what the e-mail and XMPP APIs are for! Want to reach out to other web applications? Use the URLfetch service! Need Memcache? Google has a global Memcache API. Need a database? Google provides both its traditional NoSQL scalable datastore or access to the relational MySQL-compatible Google Cloud SQL service.

The list of all the services that are available to users changes quite often as new APIs are created. At the time of this writing, the following services/APIs are available to developers:

• App Identity
• Appstats
• Backends
• Backup, Restore, Copy, Delete (of Datastore)
• Blobstore
• Capabilities
• Channel
• Cloud SQL
• Cloud Storage
• Conversion
• Cron
• Datastore
• Denial-of-Service
• Federated Login (OpenID authentication)
• Files
• (Full-Text) Search
• Images
• LogService
• Mail
• MapReduce
• Matcher
• Memcache
• Namespaces/Multitenancy
• NDB (new database)
• OAuth (authorization)
• Pipeline
• Prospective Search
• Task Queue (Pull and Push)
• URLfetch
• Users (authentication)
• WarmUp
• XMPP

You can read more about most of these APIs in the official APIs docs pages . (Docs for the others are available but not on this page.) Also, The Google App Engine team is constantly adding new features, so keep your eyes on the Google App Engine blog for announcements on new and updated services and APIs.

Administration

One of the benefits in choosing to host your apps on PaaS systems is being freed from administration. However, this means giving up a few things... no longer do you have full access to your logs or be able to implement custom monitoring of your app (or your system). This is further impacted by the sandbox runtime environment mentioned above.

To make up for some of this lack of access to application and system information, Google has provided various tools for you to gain a better insight into your app, including its performance, traffic, error rate, etc.

The first tool is an administration console. (App Engine provides two “admin consoles” actually.) A fully-featured version is for your application running in production while the other one is a lightweight version for the development server. In Figures 1 and 2, you'll see images of both the production and SDK/development admin consoles.

Figure 1 : Google App Engine Administration Console

Figure 2 : Google App Engine SDK Admin Console

We'd like to point out that the above images are just a subset of the functionality that's available to you. The figures represent both Admin Consoles as they were several years ago. The team has added so many new features that the current incarnation of the dashboard includes far more features than are illustrated here. However, the general structure and information displayed is relatively the same.

Another tool is a general system status page, shown below in Figure 3. While it is not an indication of how any one particular app is doing, it does show what is going on with the system as a whole.

Figure 3 : Google App Engine Status Page

The final tool is Appstats. It is in the same class as a profiler but custom-made to help you determine inefficient ways your code may be interacting with App Engine services (rather than traditional profiling of code coverage, memory usage, function call metrics, program behavior, etc.). Its specific use is best described in App Engine team’s introductory blogpost :

“Appstats is a library for App Engine that enables you to profile your App Engine app’s performance, and see exactly what API calls your app is making for a given request, and how long they take. With the ability to get a quick overview of the performance of an entire page request, and to drill down to the details of an individual RPC call, it’s now easy to detect and eliminate redundancies and inefficiencies in the way your App Engine app works.”

In Figure 4 below, you can see how Appstats can find and highlight inefficiencies in your code. The blue bars represent wallclock time while the red bars indicate CPU and “service” time, time that is necessary for the system to complete your request.

Figure 4 : Application Performance via Appstats

You can dig into this particular use case to get the full details in the blogpost, but the result is that by finding and optimizing the calls your app makes to App Engine services (“RPC calls”), you can significantly improve the overall performance of your application, as shown below in Figure 5.

Figure 5 : Application Performance after modifications

Here are additional resources regarding Appstats which you may find useful:

1. Official Python documentation
2. Video by engineer Guido van Rossum
3. Official Java documentation

Applications (web & non-web)

While many applications running on Google App Engine are web-based apps, they are certainly not limited to those. App Engine is also a popular backend system for mobile apps. When developing such apps, it's much safer to store data in a distributed manner and not solely on devices which could get lost, stolen, or destroyed. Putting data in the cloud improves the user experience because recovery is simplified and users have more access to their data.

For example, the cloud is a great place for mobile app user info such as high scores, contacts, levels/badges, etc. If users lose their phone, they would only need to get a new phone and reinstall the application. All their data can be streamed from the cloud after that. Not only is recovery simplified, but it makes possible scenarios like users being able to pull up their leveling or high score info from the home computer upstairs in the evenings while their mobile phones charge downstairs. Again, cloud can be a tool to provide a better user experience!

When developing a backend for mobile applications, the same decision needs to be made on whether a company should host it themselves or take it to the cloud. Do you spend time and resources building out infrastructure, or is it better to leave it to companies that do this for a living and focus on the application instead?

Mobile phones only need to be able to make outbound HTTP requests to instantly connect to your App Engine backend. You can control the application-level protocol, authentication/authorization, and payload contents, so it's not any more complex than providing a similar backend for a traditional web app.

Migration to/from App Engine

The section title alone cannot convey all the aspects of this category when considering cloud vendors. It includes migration of applications to/from your target platform (in this case App Engine), ETL , backup and restoration of your datastore, and vendor lock-in. A more complete treatment would constitute a separate lesson, however we can describe the main points here.

Porting your applications to App Engine is made simpler by providing familiar development environments, namely Java, Python, PHP, and now Go. Java is the de facto standard in enterprise software development, and developers who have experience building Java servlets will find App Engine to be quite similar. In fact, Java App Engine’s APIs adhere as closely with existing JSR (Java Specification Request) standards as possible.

In addition to the servlet standard (JSR-154), App Engine supports the JDO and JPA database interfaces (or you can choose to use Objectify or the low-level interface directly). If you're not comfortable with NoSQL databases yet, you can use Google Cloud SQL, the MySQL-compatible relational cloud database. The App Engine URLfetch service works like the Java SE java.net.URL class, the App Engine Mail API should work just like the javax.mail (JSR-919) class, the App Engine Memcache API is nearly identical to using the javax.cache (JSR-107) class, etc. You can even use JSP for your web templates.

On the Python side, while Google ships a lightweight web framework (webapp/webapp2) for you to use, you aren't limited to it. You can also use: Django, web2py, Tipfy, Bottle, and Pyramid, to name some of the more well-known frameworks that work with Python. Furthermore, if you have a Django app and use the third-party Django-nonrel package (along with djangoappengine), you can move pure Django apps onto App Engine or off App Engine to a traditional hosting service supporting Django with no changes to your application code outside of configuration. For users choosing Cloud SQL instead of App Engine's traditional NoSQL datastore, you can use Django directly as there is an adapter specially written for Cloud SQL.

Next are the SDKs. For all supported runtimes, they are open source. This allow users to become familiar with the App Engine client-side libraries and possibly build their own APIs. In addition, if users desire to control the backend, they can use the SDK and the client APIs to come up with corresponding backend services. Not comfortable letting Google host and run your app(s)? This gives you an alternative. In fact, there are already two well-known App Engine backend projects: AppScale and TyphoonAE. Both claim they are 100% API-compatible, meaning that any App Engine app that Google can run, they should be able to as well.

Next, you have control over your data. App Engine makes it easy to back up, restore, or delete your datastore data, and to copy it to another application. Find out more about these features in the official docs . If using Google Cloud SQL , you can easily import or export your data using Google Cloud Storage as an intermediary. You can read more about that at the Cloud SQL docs on import/export .

Finally, with all the advantages of a PaaS system like Google App Engine, some developers may wonder about “vendor lock-in,” a situation describing how it may be difficult or impossible for companies to move their apps and/or data to similar or alternative systems. While Google would love you to stay as an App Engine customer for a long time, we recognize that having choices makes for a healthy and competitive ecosystem.

If Google is a cloud vendor and App Engine is its product, does vendor lock-in apply? Well, yes and no. Think of it this way: you use Google's infrastructure to avoid having to build it yourself. Arguably this is one of the most difficult and time-consuming things to do. So you can’t get something for nothing. The price you pay is that you need to integrate your app so that it connects to all of App Engine's components.

However, while we recommend that you code against Google App Engine's APIs, there are workarounds. Also, think back to why you want to use App Engine... for its robustness and scalability. We created our APIs so you could take advantage of Google’s infrastructure and not as a deliberate attempt to force you into using Google’s APIs.

By allowing alternative ways of accessing your data, using familiar development environments, following open standards, and distributing open source SDK libraries, Google fights vendor lock-in on your behalf. It may not be easy to migrate, but Google tries hard to ensure that you can move your apps onto or off of App Engine. But it doesn’t stop there... the team is continually innovating, listening to user input, and improving and simplifying App Engine services to further provide a development environment of choice for building your web (and non-web) apps. Finally, the best practices you’ll learn in creating great App Engine apps can be used for application development in general, regardless of the execution platform.

Other Important Features

The final two remarks here pertain mostly to enterprises. Google App Engine is proud to be compliant with global security standards. We are certified SAS 70 compliant as well as compliant for its replacements SSAE 16 and ISAE 3402.

Enterprises who are Google Apps customers will find App Engine to be an extremely useful tool. When you purchase Google Apps, we provide a default set of applications that help you run your business. In addition to the apps from Google, there are also many more built by third-party software firms that you may find compelling in the (Google Apps Marketplace](http://google.com/enterprise/marketplace). Where does App Engine fit in?

If none of the applications above meet your needs, you can use Google App Engine to build your own custom applications and roll them into your Google Apps domain and manage them from the same control panel as if you bought it from Google directly or from vendors in the Apps Marketplace. You'll have the "best of three worlds!"

Conclusion & Review

Q: What cloud service level does Google App Engine sit?

A: App Engine is a platform (PaaS) service.

Q: What does the entire App Engine stack offer you?

A: You can use App Engine's APIs to build and host SaaS applications. It's unique because it's both a set of APIs as well as a robust, scalable, application hosting service.

Q: What are some of App Engine's security features?

A: It has a sandbox which prevents other applications (and users) from accessing your code and your data. All applications run in that sandbox, keeping them isolated from affecting other users. Furthermore, Google App Engine is certified as SAS 70, SSAE 16, and ISAE 3402 compliant.

Q: How can you do interesting things on App Engine like network programming, sending/receiving e-mail or instant messages, talking to other applications online, etc., if you can only do outbound sockets?

A: App Engine provides higher-level APIs and services that you would implement yourself if you were given the ability to get around App Engine's restrictions. We provide these services not only for security, but to help ensure you concentrate on your solution rather than building lower-level services reinventing the wheel.

Q: What are some of App Engine's features, services, and/or APIs?

A: The primary APIs include the Datastore, Memcache, Users, Blobstore Task Queue, and URLfetch. Check out a more complete list above in the Services section .

Q: Is App Engine only useful for creating web apps?

A: No. App Engine is also a great platform to for developing mobile backends. You can read about one success story for a mobile developer who got up-to-speed and released both their mobile app along with its App Engine backend to the world in only 6 days after hearing about App Engine.

Q: How does Google feel about vendor lock-in?

A: As much as we would love to keep you as our customer for the long-term, we realize that requirements change, so we're working hard to ensure that you can move your application and/or data off of App Engine whenever you wish. We've open-sourced our SDKs, we've tried hard to adhere to existing standards, and there are other open source tools that play nicely with App Engine. There are even open source backend systems which aim for full compatibility with App Engine. In addition, there are a variety of tools which help you get all of your data into or out of App Engine.

Q: What's next?

A: Now that you’ve gotten sufficient background on what cloud computing is and where App Engine fits into the model, let’s get your hands a little bit dirty! The simplest and first examples is traditionally the “Hello World” regardless of whether you’re learning a new programming language or new web framework, so your introduction to App Engine won’t be any different.

Next Lesson : Getting Started with Python App Engine