UPDATE Jan 13: my data is all intact, thanks to backups. The SoftRAID folks have reproduced the issue on their end. It is as yet unclear whether it is a SoftRAID bug or a macOS bug.
Update Jan 18: my data is all OK and restored. The bug is being taken very seriously as a top priorit by the SoftRAID folks, but as yet there is no finding on the root cause. I do expect that regardless of root cause, there will be an effort to fix it. As for myself, I have three 8TB SSDs that I am unable/afraid to put into “production” use on my 2019 Mac Pro, for fear of seeing them wiped out. I do have two of them mounted and present, but they are “canaries” for now.
I am not going to go into details at this time until I understand the cause, which kept corrupting volumes and involves complex hardware.
But I lost my main Work volume today, a 4-blade RAID-0 stripe. Total destruction with all data corrupted as IntegrityChecker instantly showed for every file (including its own files)—even though Carbon Copy Cloner thought all was well with 800GB of cloning.
Proof that backing up is not enough—data integrity should be proven, not assumed—verify your backups with IntegrityChecker.
Trying to redo the corrupted volume, it corrupted a different volume on other blades and vice versa. This fiasco involves 12 NVMe SSD blades and RAID-0 (though only 4 each together). It is as if the system jumbled the numbers of all the blades on the PCIe cards, and thus any effort to use any of them would toast everything involving any of the blades.
That's all I want to say at this time because I’m still trying to wrap my head around repeated volume corruption of freshly-created volumes. Maybe it’s a macOS Catalina (Crapalina) + 2019 Mac Pro thing and maybe it is hardware and maybe it is software and who the hell knows. But I am working with appropriate parties to track it down.
I love some aspects of the 2019 Mac Pro, but might I suggest waiting a few months for most people, both for Crapalina and the Mac Pro. New hardware and buggy new OS just smells like rotten fish. Apple should have made Mojave an option.
Fact is, macOS Crapalina has so many bugs it’s ridiculous. Tonight’s shock and awe of wanton data destruction might have nothing to do with Crapalina or the Mac Pro, but it might.
The good news is that I follow my own advice and backup many times over, and I had just backed up. So I am spending some hours restoring, and pulling out suspect hardware and will go from there, keeping some hardware out of the equation for now.
I am very glad I have the 2019 iMac 5K as a 2nd computer so that I can do something useful while fixing the mess on the Mac Pro.
It is also a very good thing to have a large (4TB or 8TB) internal Apple SSD (not just PCIe SSDs), because Shit Happens, a whole truck load of it tonight. So much so that I may ask B&H to let me return the 16-core Mac Pro with 4TB SSD for a 24-core with 8TB SSD.