• What is an unapplied write?
• Does my application have unapplied writes?
• Is my data corrupt?
• How do unapplied writes happen?
• Will this happen again?
• How do I work with the 'unapplied writes' data?
• I don't need the data, I want to just delete it.
• Is there anywhere that I can learn more about the low-level implementation of the Datastore and how all of this works?

What is an unapplied write?

Unapplied writes are writes to the App Engine Datastore that did not get replicated from the primary to the secondary Datastore. Unapplied writes do not impact the transactional consistency and have not corrupted application data. You can think of them as causing the mirror image between the primary and secondary Datastore to be out of sync.

Does my application have unapplied writes?

If your application has unapplied writes, the administrators listed in the Admin Console of your application will receive an email notification letting them know that they should take action. Alternatively, you can determine if your application has unapplied writes by looking at the Admin Console Datastore Viewer . If you see any Kinds in the Kind dropdown box with the prefix __unapplied_write__ , your application has unapplied writes that will need to be re-integrated.

Is my data corrupt?

No. Unapplied writes do not impact the transactional consistency of application data and have not resulted in corruption. Multiple writes to a single entity group within a transaction are all present; no partial writes are possible.

How do unapplied writes happen?

During a Datastore outage, it is possible for the primary datacenter to slow down or stop replicating data. As a result, a small amount of recently written data may be unavailable to applications when we move to the secondary datacenter. Because almost all applications have written new data to the Datastore since the outage, we cannot automatically apply the writes on the application's behalf. Application developers need to manually re-integrate the unapplied writes as appropriate for the application's data model.

Will it happen again?

Unapplied writes are caused by an unplanned Datastore outage in our primary datacenter. Based on the performance of our current infrastructure, we expect that this may happen less than once a year. The App Engine team is also actively working on an alternative Datastore configuration that will prevent this issue from happening at all, and we hope to make it available to applications in the near future.

How do I work with the unapplied writes?

There are three ways to work with your data:

1. Entity-by-entity with the Datastore Viewer
2. Programmatically using the Datastore API
3. Directly with the App Engine team

1. Entity-by-entity with the Datastore Viewer

The unapplied writes are stored within your application's Datastore. The Kind of the unapplied write has been prefixed with the string __unapplied_write__ so that it cannot conflict with any data or queries already happening within your application. The easiest way to view and re-integrate with this data is to simply view the entity in question in the Admin Console Datastore Viewer . If you wish to apply an unapplied write to your Datastore, select "Create a Copy" to create a new entity with the same values as the unapplied write. You can also modify any of the values of an entity before it's re-integrated. The Datastore viewer is appropriate for all applications with fewer than 50 unapplied writes.

2. Programatically using the Datastore API

You can also work with the data from within your application. The following code snippets below, in Python and Java, demonstrate how to apply an individual entity with a known key. You will need to add additional code to provide a list of keys.

Python developers can also use the pre-built fix method to reintegrate entities. Here is a downloadable code sample demonstrating how to do this in the context of a sample application.

# Take an unapplied write of type Greeting with id=4 and apply it.
from google.appengine.api import datastore

UNAPPLIED_WRITE_KIND_PREFIX = '__unapplied_write__'

my_kind = 'Greeting'
my_id = 4
key = datastore.Key().from_path(UNAPPLIED_WRITE_KIND_PREFIX + my_kind,
                                my_id)
unapplied_entity = datastore.Get(key)
# ... Do something with the entity ...
new_entity = datastore.Entity(my_kind, id=my_id)
new_entity.update(unapplied_entity)
datastore.Put(new_entity)

import com.google.appengine.api.datastore.Key;
import com.google.appengine.api.datastore.KeyFactory;
import com.google.appengine.api.datastore.Entity;

private static final String UNAPPLIED_WRITE_KIND_PREFIX =
  "__unapplied_write__";

// Take an unapplied write of kind Greeting with id=4 and apply it
String myKind = "Greeting";
int myId = 4;

Key unappliedKey =
  KeyFactory.createKey(UNAPPLIED_WRITE_KIND_PREFIX + myKind, myId);

Entity restoredEntity = new Entity(KeyFactory.createKey(myKind, myId));
restoredEntity.setPropertiesFrom(datastore.get(unappliedKey));
// ... Do something with restoredEntity ...

datastore.put(restoredEntity);

3. Directly with the App Engine team

We will email all applications affected by unapplied writes directly when we find them in the datastore, but you can also email the team directly to request assistance with re-integration. If your application has more than 200 entities, this is definitely the best way to handle your recovered data.

To request assistance, please email [email protected] with your application ID, Kind names, and what you'd like us to do with them (apply them to your current Datastore or delete them).

Note that we will prioritize requests based on the number of unapplied writes your application has. If your application has a small number of unapplied writes, we recommend you use one of the first two methods listed above. For issues not related to unapplied writes, please visit the App Engine support groups .

I don't need the data, I want to just delete it

You can delete the data in the Admin Console's Datastore Viewer with the delete button, by iterating over the entities and deleting them with the Datastore API, or by contacting the App Engine team with the above email address. Alternatively, you can leave the data in your datastore; it will not conflict with your production data. Keep in mind that this data does count against your storage budget so if you have a significant number of unapplied writes, you can save storage costs by proactively deleting the data with any of the above methods.

Is there anywhere that I can learn more about the low-level implementation of the Datastore and how all of this works?

Yes! Ryan Barrett, a member of the App Engine engineering team, has recorded two in-depth talks, including one that talks specifically about how App Engine replicates across datacenters: