Reader Comment: Sony A7R V Pixel Shift vs Demosaicing, Motion Correction, etc
Related: Bayer matrix color filter array, digital sensor, Glenn K, LibRaw, pixel shift, PixelShift2DNG, reader comment, Sony, Sony A7R, Sony A7R II, Sony A7R III, Sony A7R IV, Sony A7R V, Sony mirrorless, Sony Pixel Shift, true-color sensor or image
re: Sony pixel shift
Seemingly I have not done a good job explaining why pixel shift is so valuable. This exchange with a highly technical friend made me realize that. But as it turns out, his question is an insightful one.
With pixel shift, each pixel has a red and green and blue value—full information. No guesswork (demosaicing) need be done—the information is all there.
Versus Bayer matrix where all pixel are monochrome (red OR green OR blue), with half the sensor photosites being green, 1/4 red, and 1/4 blue. This is why the color and spatial resolution of Bayer sensors about half of one might hope for; the process of demosaicing must guess at the true R/G/B value for that pixel. The guessing is sophisticated, but actual details and colors of the capture are forever lost and non-guessable at the single-pixel level. Strongly red or blue subject matter fares especially poorly in resolution.
See Blur by Bayer Pattern Demosaicing.
In the past, we had no motion correction and all PixelShift2DNG (or Sony software) did was to create a file with a full RGB value (using values from 4 sequential captures), which requires no demosaicing. Ugly artifacts in areas of motion or lighting changes resulted, and thus a capture mode largely unusable for outdoor photography until today, with motion correction.
EXIF info, DNG from ARQ:
Image Width : 9600
Image Height : 6376 Bits Per Sample : 16 16 16
Photometric Interpretation : Linear Raw
Samples Per Pixel : 3
Black Level : 0 0 0 Strip Offsets : 187468
Rows Per Strip : 171
Strip Byte Counts : 131328
EXIF info, ARW: Exif Image Width : 9504
Exif Image Height : 6336
Bits Per Sample : 14 Photometric Interpretation : Color Filter Array Samples Per Pixel : 1 Black Level : 512 512 512 512
Strip Offsets : 13033472
Rows Per Strip : 6376
Strip Byte Counts : 122419200
Reader Glenn K writes:
Have you compared pixel shift vs single shot with the latter processed in the same Sony software as the pixel shift? Some of the differences could be ACR vs Sony.
DIGLLOYD: yes, I have, but it’s not a consideration.
All the ARQ format does is bundle together the 4 taken frames without processing other than motion correction, which essentially “paints in” areas of motion with those of a single frame or some equivalent.
But how Sony motion correction generates a full RGB value for each pixel in areas of motion is unspecified. There might be localized demosaicing in such motion areas (likely), or something smarter going on. But the end result is an ARQ file with a full RGB value for each pixel.
I have verified the non-motion areas against PixelShift2DNG as identical. LibRaw folks confirm that what PixelShift2DNG does is to create a full RGB pixel value for each pixel by averaging the two G channels for the G component and then taking the R and B values as-is. Each pixel then has a full R/G/B value. The resulting file then has three 14-bit values per pixel, one each for R/G/B.
Ah, I assumed you had to demosaic in the Sony software.
Generally not, but possibly in areas of motion. Adobe DNG converter is not doing motion correction in the ARQ => DNG process, so the “painted in” areas of motion correction clearly have already happened in the Sony software when creating the ARQ. Presumably by localized demosaicing, perhaps by something smarter.
So then ACR knows to just assemble an RGB image without any interpolation or demosaicing?
Precisely, and for some years now; the DNG format supports “true color” aka “full RGB” files, where each pixel has a full R/G/B value. Thus no Bayer matrix is involved, no demosaicing is needed or done, and the time to open such a file is much faster than single-shot files, since far less computation is needed with the CPU-intensive demosaicing step skipped.
