See my Sigma mirrorless wish list.
There is a long discussion below, but I want to uplevel this to the key point as I see them now, pertaining to the Sigma sd Quattro H:
- Out-of-camera DNG files are huge because they are uncompressed, and also 12-bit vs the 14 bits of X3F format. However, it’s not clear that 14 bits helps in any way given the noisy sensor.
- Using DNG precludes ever using Sigma Photo Pro, which could be an issue, as one of my comparisons shows.
- Running Sigma sd Quattro H DNG files through Adobe DNG Converter cuts the size by more than 50% (via compression), and finished images are identical to before conversion.
- DNG converter destroys file dates during conversion, although the EXIF info still contains the original date.
Bottom line: Sigma shooters wishing to use Lightroom or Photoshop/ACR can save 50% or more in file size by running Sigma DNG files through Adobe DNG Converter, with no loss in image quality vs out-of-camera DNG.
In Sigma sd Quattro-H: Shoots X3F as Usual, or DNG With a Huge Size Penalty, I discussed the huge file size penalty for shooting DNG files instead of X3F format.
On a lark, I decided to convert DNG files using Adobe DNG Converter 188.8.131.521. That is, convert DNG from the camera to DNG. This should be a no-op, but instead it has a major effect.
Adobe DNG Converter shrinks DNG files from the Sigma sd Quattro H by more than 50%.
As shown at right, the file shrinks from 150MB to 65.5MB, a reduction of 56%. What is lost exactly? Obviously something went away. Or, maybe not, as the evidence shows no difference in the final image quality, zero difference.
This behavior raises the whole data loss concern I have always had about converting camera original camera raw files to DNG.
But could it all be just lossless compression starting from an uncompressed DNG? That is the claim by Eric C (towards end of post).
One might think that the savings is from removing an embedded original (e.g., an X3F). But Adobe DNG Converter 184.108.40.2061 states that there is no embedded original. Still, there might be—if the camera DNG is non-compliant and thus trips up the Adobe DNG Converter.
Roy P writes:
For several years now, Sigma has been trying to get Adobe to integrate its Foveon cameras into Lightroom, and even offered up its RAW conversion software. But Adobe has not cooperated because a lot of the operations in Lightroom are hard-coded to traditional Bayer matrix. To accommodate the Foveon would require too many touch points in the software (properly referred to as “bloatware”, coming from Adobe, but I digress.) Adobe can’t justify the cost of incorporating and maintaining all those changes – Sigma is simply not a big player in the camera business.
So Sigma had no option but to require its users either shoot JPEG or do a conversion to TIFF and use a convoluted workflow. It is less of an issue for people who use Photoshop as their workflow, but that is a minority. Most people use Lightroom, and for them, it’s a real pain. In my own case, I own all the three DPx Merrills, and I’d love to use them, but I use a Lightroom workflow and I’m so used to it that I find it a hassle to use the DPx Merrills – it has been more than a year since I used any of them, and I’m wondering why I even continue to have them.
It looks like for the Quattro-H, Sigma has decided to provide an optional DNG output, pretty much as the Leica S and M cameras do. This ensures compatibility with Lightroom, and helps Sigma gain more acceptance. Problem is, in doing this, I suspect Sigma may have had to sacrifice some IQ by essentially Bayerizing its Foveon data to make it DNG compatible. That would explain differences between processing TIFF files with Photoshop yielding in sharper images with better colors than shoving DNG files of the same scenes through Lightroom.
Another issue is, even for Bayer sensors, the DNG format might be suboptimal. A PhaseOne dealer once told me the DNG format degrades the image quality by limiting color space and dynamic range. He didn’t know exactly how, but the obvious suspect would be lossy compression – if certain fields are limited to certain number of bits or bytes, then by definition, some values would have to be truncated, or some spectral frequencies cut off. This guy’s claim was, RAW processing and developing in CaptureOne delivered better image quality than DNG and LR or Photoshop. He called DNG a consumer format and the “lowest common denominator”. As a PhaseOne guy, he might have been biased, but there could be some truth to it.
So between the two issues, converting to DNG could be losing a lot of the benefits of shooting with a Foveon camera. It may be best to use the Sigma software, even if the workflow becomes non-standard.
DIGLLOYD: The files before/after are identical in the resulting finished RGB image.
I’m not an expert on the ins and outs of raw format, but I’ve long recommended sticking with the native format that the camera shoots (not converting to DNG)—it’s a safe choice that is not really debatable. Whether or not DNG is inferior in generally is unclear to me, but it’s clear that DNG from the Sigma sd Quattro is 12 bits, and X3F is 14 bits, and the behavioral difference between X3F and DNG that I document is another downside of DNG over and above the huge hit in file size (though the conversion addresses that).
Philip S writes:
I think this part of Roy P’s comment is wrong, fed to him by an ignorant, or perhaps biased as he says, PhaseOne dealer.
1. The only “color space” that could be said to be “in" a native raw or DNG file is the camera RGB “space”. True, if you use ACR/Lightroom, CIE XYZ will be the bridge between camera RGB and your working RGB space (ProPhoto, Adobe, etc.). Given that CIE XYZ was developed specifically to include all colors that humans can see, it is not going to be a “limiting color space.” Empirically, the translation of camera RGB to XYZ is always done with “error” — sometimes described as camera sensors not satisfying the Luther condition perfectly.
We know what ACR/Lightroom does, but we don’t know what Sigma PhotoPro or CaptureOne do. It’s possible that neither uses CIE XYZ as the bridge between camera RGB and working RGB spaces. Or, that they are using different methods to estimate the camera-RGB-to-XYZ conversion matrix (if they are using XYZ). If Sigma PhotoPro does not use CIE XYZ, that could explain the color differences between X3F and DNG files. I stress that that is pure speculation, and doesn’t seem very likely to me. It’s also possible that Sigma made a conscious decision to render colors differently. The camera-produced DNGs presumably include camera-RGB-to-XYZ conversion matrices developed by Sigma. Otherwise, I don’t think they would work with ACR/Lightroom, or perhaps ACR/Lightroom would default to some generic camera profile. Something I think Sigma would not want to happen. Assuming the matrices are in the DNGs, it would be interesting to know if they change depending on the camera color mode setting (Standard, Neutral, Portrait, etc.). One would hope not, but one never knows.
2. Compression. See my preceding message describing what happens when an uncompressed ARW is converted to DNG. A limited test on a single image, but no degradation apparent even though file size is reduced by almost 50%. BTW, if I run that DNG through DNG converter again, there is NO additional compression.
3. In short, I can imagine several reasons why colors are different between X3F files processed through PhotoPro and DNG files processed through ACR. But, I’m skeptical that DNG-ifying is the real culprit, unless it’s the requirement to use CIE XYZ as the bridge between camera RGB and working RGB (or unless the compression is not truly loss-less).
DIGLLOYD: Obviously an uncompressed Sony ARW raw file is going to see a file size reduction, just as Nikon and Canon see, by default. But there is more going on I think.
Savings of 50% for image compression are rare, particularly well-exposed images whose high bits are non-zero (my files were optimally exposed and thus have no zeroed high bits to improve the compression). CaNikon typically achieves 30% or so for lossless compression (NEF and CR2).
What does RawDigger say? What about the processed images?
Something went missing in the conversion to the tune of 56% of the file size. One can make a weak argument for compression, but it does not seem credible to the tune of 56% given that the zip-compressed file is far larger.
CaNikon generally do not achieve more than about 35% in NEF and CR2 files. It just is not possible to save 56% on a highly detailed file, let alone one with a lot of low-level noise like the Sigma files. So the evidence strongly suggests that something is removed during DNG conversion (which does not preclude some savings from compression).
Still, what most users care about is the data that ACR will process and thus the finished RGB image. And as the RawDigger histograms show, there is no difference between the out-of-camera DNG and the converted DNG (toggle to compare).
So the RawDigger seems to be identical, but the final test is to calculate the pixel differences between raw conversions from the two files (…). I did so—no difference—none.
Eric C writes:
The reduction of ~50% (2:1) in file size that you're observing when applying the Adobe DNG Converter is normal and expected. This is because the original DNG files in this case are uncompressed. By default, Adobe DNG Converter will save the converted DNG output by applying lossless compression to the raw image data. This lossless compression typically saves around 2:1 compared to an uncompressed raw file.
The same compression ratio (around 2:1) also applies when converting uncompressed non-DNG raw files (for example, 14-bit uncompressed NEFs from a Nikon D810) to lossless-compressed DNG.
The reason you typically don't see 2:1 when comparing to Canon CR2 is because the mosaic raw data in Canon CR2 files are already lossless-compressed using Canon's own method. And similarly, many users shooting with Nikon cameras are actually using Nikon's own lossless compressed format (not uncompressed). Converting to DNG afterward may still save some extra space, for technical reasons (*).
(*) Lossless compression of raw image data is usually based on a so-called Huffman transform or tree. When saving out new DNGs, the DNG Converter analyzes the raw image data to (dynamically) configure the compression method to save the most space. This optimization step takes a little more time, but can sometimes squeeze out some more size savings.
The Huffman-based coding (aka, Lossless JPEG) is usually modified in a critical way for mosaic raw files: namely, that it takes advantage of the fact that in a mosaic pattern (such as Bayer), the predictors are based on the repeating color pattern (e.g., reds are used to predict reds, greens are used to predict greens, etc.). It's very difficult for a more general compression method that doesn't "know" this layout a priori to get similar compression ratios.
I am not sure about the increased file size in the "compressed off" case, but I suspect it's probably because the DNG Converter's uncompressed mode uses 16 bits (2 bytes) to represent each pixel, whereas the source Sigma DNG is probably using 14 bits to represent each pixel, and the adjacent bits are packed.
DIGLLOYD: Years ago, I filed several patents on compression, including delta-pixel compression for images. So my main issue here is working blind on what is being done in this case.
CaNikon files are lossless compressed unless one is foolish enough to choose uncompressed so yes it's a given that the big savings are not going to be achieved. But there may still be some space savings if indeed CaNikon use the inefficient algorithm of Huffman compression, and not Lempel-Ziv with pixel-delta preprocessing.
I’m more than glad to be proven wrong, but I can’t dispute the savings other than to say that 56% is unusually good for image files. My further thoughts:
- Nothing changes these two facts: a particular vendor’s RAW file processing software will not work with DNG (e.g., Nikon Capture NX, Canon Digital Professional, Sigma Photo Pro).
- If the original raw is not embedded then something is surely lost. See previous point.
- My understanding from Eric’s note is that CaNikon are dumb enough to use Huffman encoding instead of Lempel Ziv. It may be a CPU power / memory issue, I don't know, but it sucks to foist larger than necessary files totaling terabytes over time onto hapless users. Still, it’s better than uncompressed format.
- Why can gzip -9 save only 27% on the same file vs 56% for DNG. I presume it’s one of the delta-pixel offset optimizations (I patented such an approach), but I don’t know.
- When compression is disabled during DNG conversion (Custom...) and JPEG Preview=None, why does the output file increase in size from 150MB to 179MB? That is, 29MB larger.
A final note on DNG
I dislike DNG for two reasons:
- Using DNG in Photoshop/ACR modifies the DNG file every time a change is made. This approach means that a backup may have to copy many gigabytes instead of a few kilobytes of sidecar files. It also means that the file modifications dates are rewritten. It also entails the risk of file damage if anything goes wrong, though hopefully Adobe at least does that part correctly. Basically, I want my original raw files LEFT ALONE.
- Conversion of DNG files via DNG converter destroys the creation and modification dates. I consider this DATA LOSS. Date and time of shooting are highly relevant. Yes, I know they should be buried in there in the EXIF info in the DNG, but that’s just not good enough.
I wish Adobe would fix these nasty behaviors. The only workaround is to lock the DNG files—doing so causes Photoshop/ACR to create sidecar files as with any other non-DNG raw format.