AWS S3 Eventual Consistency Caveat

Today, I think I ran into one of the less known caveats with AWS S3 consistency model. S3 provides read-after-write consistency for new objects; so as long as a fresh object is being created, any subsequent reads from any S3 clients are guaranteed to see that read. However, if prior to creating a new object, a GetObjectRequest is done on the object key, the read-after-write consistency guarantee goes away.

In our system, we create new objects with a unique id and upload them to S3. The unique id is a 64-bit randomly generated integer. However, to prevent any accidental over-writes of existing objects, we first check if an object with the randomly generated id exists already. If it doesn’t, we create the object with the random id. However, this object existence check is done via a GetObjectRequest, so the immediate read-after-write consistency no longer applies. In fact, this is what we observed in our system. Even after a minute of the object being uploaded, it was not visible to S3 readers despite the upload being clearly successful. Seeing the eventuality of “eventual consistency” last for tens of seconds was definitely a first.

Reference:

https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel