I blog posted about the importance of digital data storage in 2015, and reliable backups must be the next thing to put in place after setting up your computer. Reliable here often means redundant too. This may sound obvious and while many would say ‘of course! no need to tell me.’, many might still leave it too late and not vigorous enough to have regrets later. Just a few important facts are listed below. But before you need to consider these and below in details, you first need to decide if you want to have your data stored by yourself (your own storage solutions including local and remotely located, e.g. your own NAS), or choose to have your data stored by someone else (in the clouds). If you are considering or have decided on the latter, then read my other post . But most personal consumers/users typically have some local storage, so below may still be useful?
OK, on available data storage solutions (irrespective if you own/maintain them or others do that for you as in clouds):
- All data storage solutions (just like our human brain, in fact much more inferior) will NOT last forever- their lifespan are only measured in single-digit years or in some cases weeks! I’m not spreading paranoid, just read on if interested.
- All data solutions need exercising! Just like human brains. Sounds untrue?
- There is no single solution that fits all, and not the latest technology and fastest or most expensive one is always the best.
Next, let’s look at what factors should be considered when choosing (most are interlinked):-
- Lifespan,
- Reliability during useable life,
- speed – read/write, sequential or random,
- price,
- how important is the data,
- frequency of access/retrieval AND data write,
- the usage environment- portability, temperature, and humidity- these are related to the failure mechanisms of the storage media.
Now, let’s look at the available solutions now, note that I’m excluding the older generations of floppy disks and optical disks (CD, DVD). The former has vanished due to small capacity and unreliable, the latter while rather reliable and permanent, are also limited by their capacity and slow write speed for general consumer-grade use:
- Hard disk drives (HDD)- based on magnetic recording and has some physical access parts. They are typical 3.5″ and 2.5″ in physical sizes (the latter much more popular in consumer area), and with storage volume typically up to around 15TB (1TB=1,000GB); although in 2020, Seagate has announced a 20TB, and SDK an 80TB drive using HAMR (Heat assisted magnetic recording) technology. The typical lifespan of hard drives has been estimated to be up to 10 years, considerably less with poor maintenance. The typical speed is around 100MB/s (see above photo), and price (Sep 2020) is around US$20/TB (based on a 5TB external HDD).
- SSD (Solid State Drives)- it uses semiconductor chips, not magnetic media, to store data (more technically- it uses NAND flash to provide non-volatile, rewritable memory). The reported largest capacity to date is 100TB. Unlike hard drive, it does not have any moving parts and hence speeds are only typically limited by the input/output interfaces. They are considerably faster than hard drives, and hence have been used as the primary source of storage for smartphones and tablets. The typical speeds are around 500MB/s and can reach 1GB/sec for the latest NVMe interface, typical price ((Sep 2020)) is around US$150/TB (based on a 2TB NVMe SSD) . Apart from faster, SSD are usually smaller in physical size (especially those with NVMe interface – see the 3 items in the above photo), and lighter to carry around without the danger of data damage via drops, etc. Due to cost considerations, etc, SSD are often used as memory cache in conjunction with HDDs to speed up computer systems. SSD, while more rugged than hard drives, has typical lifespans less (surprise!)- typically around 5 to 10 years (despite some claimed theoretically lifespan of >200 years and some put it to a few weeks!- but these are extremes based on very extreme usage and assumptions). As expected, the actual lifespan of different types of SSD is a very complicated issue, but the fact that NAND (or flash type memory cells) can only be programmed and erased a limited number of times or P/E cycles, means that the more these read/write cycles are (intensive uses), the shorter the lifespan. But figures show that during its useable lifespan, SSD should have higher reliability than hard drives as their MTBF (mean time before failure) is typically higher.
- Various common interfaces/ standards and their speeds are: typical for hard drives- SATA has a throughput ceiling of 6.0 gigabits per second (Gbps), while SAS has a ceiling of 12 Gbps. Typical for SSD- PCIe (PCIe 3.0 was released in 2010 and delivers ~1 GB/sec per lane (bidirectional). With 4 lanes, PCIe 3.0 ‘s throughput would be 8 Gbps in both directions at once (duplex) for a total of 16 Gbps). NVMe is the latest PCIe interface protocol that has been designed to maximize the transfer speed between computer and SSD utilizing all the computer cores, etc. If you are interested in the fastest state of the art NVMe SSD, read this.
Maintenance and exercise?! – sounds ridiculous, but both HDD and SSD needs maintenance to maintain their lifespans and performance. Exercises mainly apply to storage devices not frequently used/accessed (like long term backups, etc).
Firstly, let’s look at the failure mechanisms of HDD- magnetic deterioration (Permanent magnets generally lose their field strength at the rate of 1% per year- Refreshing the stored data can reduce magnetic field deterioration), magnetic field corruption ( caused by nearby high power magnets, motors, and the sun); mechanical failure (most prone if HDD not powered on for long time due to motor, bearing and spindle failures), and environmental conditons (both high and low temperatures will shorten the lifespan of the heads, platters and bearings, – e.g. Western Digital recommends storing their hard drives between 12C to 32C). Hence regularly powering HDD up and reading/writing to it is mandatory to keeping good health and reaching their design lifespan. Some recommending moving the data around, e.g. by cloning from A to B, and then back from B to A, say once a year or once every two years, even and especially if you don’t use and access the HDD- i.e. never leave them idle and forget above them, because when one day don’t be surprised that to find it dead when you want to access it to retrieve data!
SSD failure mechanisms mainly related to the environment (high temperatures in particular) and infrequent usage/access. In addition to backing up your data, the simplest way to avoid data loss is to make sure any SSD that is placed in long-term storage, is powered on and used at least twice a year. There’s no reason to rewrite the data; simply powering the drive on and using it as you normally would for a few minutes should be sufficient to maintain data integrity.
Aside from exercising drives you have in long-term storage, the normal usage your drives in typical everyday use (the SSD inside your desktop and laptop computer and mobile phones) is more than sufficient to keep data intact and performance on par. Of course, having redundant backups is a MUST in every scenario!
Armed with the above basics, so which and how to choose? As depicted above, with so many factors to consider, there is no one solution that fits all even if money is not an issue. It mainly depends on the data you’re storing – are they critical (like emails, applications, databases, important docs), access frequency (for backups, or long term storage for say photos, videos, movies, ebooks, etc), and access speed required. Putting it in very simple terms, if access is frequent and speed is critical, then go for SSD. If you take it around frequently, then SSD and especially the latest NVMe external SSD (e.g. this super fast Sabrent 2TB external SSD is smaller than a credit card, and now only US$299, is very attractive), just make sure you access it frequently too. If for long term storage- go for old faithful HDD, and this US$105 Seagate 5TB portable ext HDD would be difficult to beat in value. For more details, this document (a 12 page mid-2019 white paper by Fujitsu) is an extremely useful guide. (Note that I receive nothing from Amazon, Seagate, Sarbrent or Fujitsu).
Finally, with the ever dropping prices for storage solutions (remember the prices for HDD and SSD say 5 years ago?- now for the price of a HDD then, you can easily afford a SSD of the same size now!). With that, there is no excuse for not paying for the most important element of your IT usage- I mean say for US$20 per TB for a HDD as a redundant backup (e.g.), would you rather stand to lose the valuable data and face the associated extreme regret and frustration?
Credits and Sources
Coughlin, T. (2020) 80 TB Hard Disk Drives, Forbes. Available at: https://www.forbes.com/sites/tomcoughlin/2020/02/12/80-tb-hard-disk-drives/#1d7ef37b48f7 (Accessed: 6 September 2020).
Garden, H. et al. (2012) How Solid-state Drives Work, HowStuffWorks. Available at: https://computer.howstuffworks.com/solid-state-drive.htm#:~:text=Known%20as%20a%20solid%2Dstate,when%20your%20computer%20is%20running. (Accessed: 6 September 2020).
Thornton, S. (2018) NVMe vs SATA: What’s the difference and which is faster?, Microcontrollertips.com. Available at: https://www.microcontrollertips.com/why-nvme-ssds-are-faster-than-sata-ssds/#:~:text=SSDs%20have%20a%20clear%20advantage,that%20SSDs%20are%20capable%20of. (Accessed: 6 September 2020).
Bauer, R. and Bauer, R. (2019) Are Solid State Drives / SSDs More Reliable Than HDDs?, Backblaze Blog | Cloud Storage & Cloud Backup. Available at: https://www.backblaze.com/blog/how-reliable-are-ssds/ (Accessed: 6 September 2020).
Nelson, T. (2018) Do HDDs or SSDs Need ‘Exercise’? The Rocket Yard Investigates, Rocket Yard – The OWC Blog. Available at: https://blog.macsales.com/43702-we-bet-you-didnt-know-that-your-hdds-or-ssds-may-need-exercise-too/ (Accessed: 6 September 2020).
Jacobi, J. (2020) NVMe SSDs: Everything you need to know about this insanely fast storage, PCWorld. Available at: https://www.pcworld.com/article/2899351/everything-you-need-to-know-about-nvme.html (Accessed: 6 September 2020).
White PaperFUJITSU Server PRIMERGY, HDD or SSD or NVMe drives for servers –what is more suitable?- https://sp.ts.fujitsu.com/dmsp/Publications/public/wp-py-hdd-ssd-en.pdf