Latest or all posts or last 15, 30, 90 or 180 days.
Upgrade the memory of your 2018 Mac mini up to 64GB
64GB for 2019 iMac 5K

$550 savings vs Apple!
64GB costs *less* than 32GB Apple!

Reader Comment: Hard Drive Error Rates

See my OWC storage wishlist.

Photographers and videographers need capacious storage, and thus should pay close attention to performance as needed, but most important of all: data integrity.

After reading Drives Full, Upgrading to OWC Thunderbay 6 with Six 14TB Drives (84TB total space), reader Tetsu E writes:

Regarding your choice of RAID 4 or 5, isn´t it dangerous to use such a big drives? The failure rate is quite big and the build times too.

Quote: " almost all SATA drives, and many SCSI drives, were spec'd with one Unrecoverable Read Error (URE) at 10^14. That's one URE every 12.5TB".

Rebuild times can take ages with big drives and the possibility of URE during rebuild is high and you can lose everything. Raid 10 is recommended on this situations, whats your opinion on this?

DIGLLOYD: the quote that Tetsu E references is dated and inaccurate and the assumptions are incorrect about the chances of URE during rebuild.

Small drives increase failure rates

For starters, smaller drives are likely to have higher error rates because they typically are not enterprise grade and/or are older models. Even if one uses twice as many drives of half the capacity with the same error rate, it would not change the error rate. Indeed, the chance of one drive of 12 failing (for mechanical reasons or otherwise) versus one drive in 6 is twice as high.

Error rate is stunningly low

Specifications for the Toshiba MG07ACA14TE 14TB hard drive state 1 unrecoverable read error per 10^16 bits (100 times better than 10^14), along with a MTBF of 2.5 million hours. Do I have the math right that this means a 1-bit error every 1250 terabytes? The URE rate for my 84TB setup is thus about 0.07 bit. Thus the chance of any critical corruption is so close to nil as to be absurdly low. Especially since only a portion of the capacity is used, and there will be plenty of spare blocks to map out bad ones.

10^16 / 8 / (1000*1000*1000*1000) = 1 bit error every 1250 terabytes.

SoftRAID rocks

SoftRAID is highly effective at warning about drive failures, and should a read error occur, SoftRAID can rebuild the RAID from parity information. Partial rebuilds are very fast, and typically take minutes, not a day. Plus SoftRAID is generally faster to much faster than hardware RAID.

Experience

Tip: buy new hard drives every two years or so, moving the older ones into backup duty—failure rates rise dramatically past year three. By year five, it’s russian roulette with your data.

I’ve used eight 12TB drives for a year and eight 8TB drives over two years. One 8TB drive has failed with no data loss due to using RAID-5 (I kept working and replaced the drive). Plus SoftRAID warned of the impending failure. I've NEVER lost a raid in the last decade.

Multiple and frequent backups are the ONLY true solution

Finally, for most everyone, the smart move is making multiple redundant backups rather than wasting a huge amount of storage. RAID is NOT a backup and physical loss (theft, fire, flood, electrical surge, etc) all pose a far greater risk to data than 1 bit every 1250 terabytes.

Barring mission-critical storage where downtime is unacceptable, I consider it a huge waste of space and performance to use RAID 1+0 (stripe of mirrors) rather than a simpler setup with frequent multiple redundant backups.

Validate data integrity

diglloydTools

If one actually is concerned about such remote possibilities (I am, slightly), run diglloydTools IntegrityChecker regularly on your original master data files just in case—IntegrityChecker will flag any changes in files (it cannot check the file system itself, only file data). See Data Integrity Over Time, and with OS Changes. Certainly data integrity verification should be done as a high priority for any major data transfer such as switching storage systems; see How to Safely Transfer Data or Verify Backups (IntegrityChecker).

Below, a recent validation of two of my main storage volumes (volumes Main and Archived) totaling 9.5TB (8.9TiB), using diglloyd IntegrityChecker (java version).

diglloyd-iMac:DIGLLOYD lloyd$ icj verify Main Archived
### RUNNING development version in /Master/diglloyd/DOMAINS/DIGLLOYD
# icj version 1.01 @ 2019-01-10 11:00
# Copyright 2018-2019 DIGLLOYD INC. All Rights Reserved
# Use of this software requires a license.http://macperformanceguide.com/Software-License.html
# Thu Jan 10 12:35:18 PST 2019

[2]={/Volumes/Main, /Volumes/Archived} Scanning folders...4668.7740.11971.17829.22819.25440 10979 ms to find 25440 folders

Loading hash data concurrently for 25440 folders...0.100.200.300 ... 25057.25326.25440 Loaded hash data for 25440 folders in 36423 ms 47468 ms to scan/load 25440 folders 158 ms to count files and sizes

Hashing 366962 files totaling 8887 GiB in 25440 folders. 0%: 68 files 706 MiB @ 335 MiB/sec, 00:02.107 . . . 99%: 366907 files 8887 GiB @ 275 MiB/sec, 09:11:26.417 Waiting for 12 files to finish... 100%: 366962 files 8887 GiB @ 275 MiB/sec, 09:11:28.644
============================================================================ FILE STATUS SUMMARY for 25440 folders 2019-01-10 21:47:35 ============================================================================ # With hash: 367554 files # Without hash: 0 # Missing : 0 # Hashed: 366961 (some files were empty) # Changed size: 0 # Changed date: 0 # Changed content + date, size unchanged: 0 # Total changed content: 0 # SUSPICIOUS: 0 icj done at Thu Jan 10 21:47:35 PST 2019 runtime 09:12:16.368
B&H Deal ZoneDeals by Brand/Category/Savings
Deals expire in 19 hours unless noted. Certain deals may last longer.
$3097 SAVE $200 = 6.0% Nikon D850 DSLR in Cameras: DSLR
$1797 SAVE $200 = 10.0% Pentax K-1 Mark II DSLR in Cameras: DSLR
$998 SAVE $400 = 28.0% Sony a7 II Mirrorless in Cameras: Mirrorless
$1798 SAVE $200 = 10.0% Sony a7R II Mirrorless in Cameras: Mirrorless
$2798 SAVE $400 = 12.0% Sony a7R III Mirrorless in Cameras: Mirrorless
$3998 SAVE $500 = 11.0% Sony a9 Mirrorless in Cameras: Mirrorless

diglloyd Inc. | FTC Disclosure | PRIVACY POLICY | Trademarks | Terms of Use
Contact | About Lloyd Chambers | Consulting | Photo Tours
RSS Feeds | Twitter
Copyright © 2019 diglloyd Inc, all rights reserved.