Persistent Storage sounds like a tautology to me. Been used as I am (was?) to Hard Disks, USB keys, DVDs, and all other possible way to store information, it seems that storage is persistent by definition, and only failures, or human errors can cause non-persistent and catastrophic behavior . Well, in IaaS terminology, Storage comes in different flavors and can also be Ephemeral.
- Buckets (provided by Walrus),
- Elastic Block Storage (EBS) Volumes (provided by the Storage Controller),
- Ephemeral (provided by the Node Controller).
Our Storage Team, started a wiki to dig into the technical aspect of the different storage types: stay tune on GitHub for more in depth technical dive on how they have been implemented.
Of the above list, two are meant to be persistent (i.e. to persist across instance termination): Volumes and Buckets. Two provides the familiar block interface (i.e. they appear and are used as Hard Disks): Volumes and Ephemeral. One is designed to be massively scalable: Buckets. And one is meant to be temporary: Ephemeral.
Instances and Storage
Instances by default are sitting on Ephemeral Storage. Uploaded images (EMIs) are the master copies, and all instances will start as a fresh copy of the that very image. All changes made to the instance (e.g. packages installed, configuration, application data) will disappear once the instance terminates. Notice that the termination of an instance can be voluntary (i.e. the Cloud User issue a terminate-instance command) or accidental (e.g. the hardware running the instance fail, or the software within the instance fails badly). This kind of instances are called instance store.
Instances can also use Volumes for their root file system: they are aptly called boot from EBS instance. In this case, at instance creation, a Volume is cloned from a specific EMI snapshot. The instance will then have exclusive access to this Volume, throughout its lifetime, allowing for stopping and re-starting without loss of any changes made. The instance can be restarted wherever the Volume is available (i.e. EBS Storage is only available on a cluster basis, or availability zone), and its performance is driven by the Volume performance (e.g. network speed to a SAN, or DAS serviced by the Storage Controller).
The main different in Storage speed between instance store and boot from EBS, is a trade between speed of the local disk (in the case of instance store), and the speed of accessing a SAN (or DAS) across a network. Things gets more complicated when multiple instances competes for shared resource (e.g. common disks on a Node Controller, or network access to the SAN).
Ephemeral is served by the Node Controller. Sizing the physical storage subsystem for the expected number and type of instances is needed to ensure the full load can be achieved. Also, with the current multi-cores CPUs, quite a few instances can run on the same Node Controller. Too many concurrent disk requests can easily overwhelm the Node Controller's host, causing instances to time out, or unpredictable and erratic behavior: the storage subsystem needs to be properly tested for the expected concurrent load.