Image Sensor Shootout (... the myth of pixel-shifting)
Ever since the release of the Panasonic's AG-HVX200 camcorder, video geeks all over the world are discussing the benefits of pixel-shifting.
Advanced video-camcorders usually have three image sensors - one for each of the primary colors red, green and blue. Normally, each of the three sensors has the same resolution as the desired target video format, for example 720x480 pixels for NTSC or 1440x1080 pixels for HDV.
However, for a HD-camcorder the AG-HVX200 has low resolution chips (960x540) and claims to double resolution to full-HD (1920x1080) by shifting the green sensor side- and upwards by half a pixel relative to the red and blue sensors.
Advanced video-camcorders usually have three image sensors - one for each of the primary colors red, green and blue. Normally, each of the three sensors has the same resolution as the desired target video format, for example 720x480 pixels for NTSC or 1440x1080 pixels for HDV.
However, for a HD-camcorder the AG-HVX200 has low resolution chips (960x540) and claims to double resolution to full-HD (1920x1080) by shifting the green sensor side- and upwards by half a pixel relative to the red and blue sensors.
Look at the exemplary sensor on the right: The luminance of row 4 / column 4 is calculated by taking a quarter of the luminance of the green sensor B and red and blue sensors F: 709Y44 = 0.2126 x RedF + 0.7152 x GreenB + 0.0722 x BlueF. The weight factors (0.2126, 0.7152 and 0.0722) are standardized in ITU-R 709 and model the human image perception.
This trick obviously allows to calculate twice as many columns and rows of luminance information compared to unshifted sensors. But is the result comparable to an (unshifted) high resolution image sensor with 4 times as many sensor elements?
This trick obviously allows to calculate twice as many columns and rows of luminance information compared to unshifted sensors. But is the result comparable to an (unshifted) high resolution image sensor with 4 times as many sensor elements?
Pixel-shifting is not a new technology, though. It has been used in many professional cameras for years and is also used in semi-professional handheld-camcorders manufactured by Canon and others. It’s acknowledged benefits are improved color rendering and reduced need for alias filtering.
So what is the truth behind the myth?
To find out let's look at a small sensor with just 5x5 pixels. Although this sensor is obviously much too small to be useful in real world it still is big enough to demonstrate the principles.
Pixel-shifting shootout
So what is the truth behind the myth?
To find out let's look at a small sensor with just 5x5 pixels. Although this sensor is obviously much too small to be useful in real world it still is big enough to demonstrate the principles.
Here is a schematic representation of our sensor:
The numbers in the upper left corner of each sensor element represent the calculated luminance values (with 0 being full black and 1 being bright white).
In this example the sensor is seeing one black square exactly in the middle (0,0) with bright white on all other fields (1,0).
For illustration purposes there are gaps between the gray sensor elements.
In this example the sensor is seeing one black square exactly in the middle (0,0) with bright white on all other fields (1,0).
For illustration purposes there are gaps between the gray sensor elements.
On the right is the numerical representation of the exemplary image from above.
I have used a much higher image resolution (20x20) than the sensor resolution (5x5) to be able to explore the sensor behavior when the image is gradually moving.
I have used a much higher image resolution (20x20) than the sensor resolution (5x5) to be able to explore the sensor behavior when the image is gradually moving.
So now a graphical view of our first image:
And this is what the sensor will make of it:
No surprises ;-)
Let's try a more complex example.
We will shift the black dot slightly to the lower right so that it spans across several sensor elements:
We will shift the black dot slightly to the lower right so that it spans across several sensor elements:
The image now looks like this:
Due to the limited resolution of our sensor (5x5) it can not represent this image correctly. Instead, the luminance values will be mapped to the individual pixels like this:
Obviously, this does not work so very well.
Now we use the same example and see what the result would be with a pixel-shifted sensor.
To the right is a graphical representation of that sensor. Again, the sensor elements are in a 5x5 matrix. This time, however, the green elements are shifted sideways and upwards from the red and blue sensor elements.
To the right is a graphical representation of that sensor. Again, the sensor elements are in a 5x5 matrix. This time, however, the green elements are shifted sideways and upwards from the red and blue sensor elements.
We are using the same image as above:
Due to the spatial offset of the green sensor elements of the pixel-shifted sensor we can now calculate a 10x10 matrix of luminance values from the 5x5 matrix of sensor elements:
Both position and size of the black square are much closer to reality than they were with the unshifted sensor. A clear improvement. However there are some halos and the square is still pretty much distorted.
Now let's look at a 'true' high-resolution sensor with twice as many sensor-elements in both directions (10x10):
For the third time we are using the same image:
And here are the luminance-values of the high-res sensor:
The square renders a little larger than in reality but it is located at the right spot, has the correct shape and there are no halos around the image.
A clear winner.
A clear winner.
Out of curiosity, let's see what a Bayer-pattern sensor would do to this image. Such sensors are used in nearly all single-sensor cameras (DSLRs and low-end camcorders).
Such a sensor combines all three primary colors within it's sensor array. The green pixels are usually included twice as often as the red and blue pixels due to the color selectivity of our eyes.
In our 5x5 matrix there are 12,5 green pixels and 6 1/4 red and blue pixels each.
Such a sensor combines all three primary colors within it's sensor array. The green pixels are usually included twice as often as the red and blue pixels due to the color selectivity of our eyes.
In our 5x5 matrix there are 12,5 green pixels and 6 1/4 red and blue pixels each.
For the last time our shifted square as the original image:
Rendering of a Bayer-pattern sensor:
Again, not a surprise: The output of this sensor is by far the worst. Bayer-pattern sensors need much higher pixel density than the desired resolution in the final image. After all, three colors are combined in just one chip.
As all of this was done with a simple EXCEL-Sheet. I invite you to download the file and do your own experiments.
More images
For a fair comparison many more images need to be examined. So I have created a video with moving black squares of different sizes.
The video-screen is divided in four quadrants: The upper left part shows the original image - a moving black square on solid white background. The square comes in three different sizes as the video progresses: At first, it’s just as large as one pixel of the 5x5 array (like the one in our example). The second square is one quarter that size (i.e. just as large as a 10x10 sensor’s pixel) and the third and final square is even smaller (again one quarter of the previous size).
The other three quarters of the video-screen show the output of the 5x5 sensor, the pixel-shifted 5x5 sensor and the 10x10 sensor.
Now compare for yourself: Can the pixel shifted sensor compete with the high-resolution sensor? Is it at least performing any better than an unshifted sensor?
Download the Sensor-Shootout Video (640x480, wmv, 2.9 MB).
Here's what I found:
Low-Res Sensor
|
Low-Res Sensor
(pixel-shifted) |
High-Res Sensor
|
|||
|---|---|---|---|---|---|
Footprint
|
Varies from 4 to 16 squares |
Varies from 4 to 13 squares |
Varies from 4 to 12 squares |
||
Contrast
|
Good contrast |
Most images with low contrast |
High contrast all over |
||
Shape
|
Some images |
Some images distorted |
Mostly a square |
||
Overall score
|
4 Points |
5 Points |
9 Points |
||
Low-Res Sensor
|
Low-Res Sensor
(pixel-shifted) |
High-Res Sensor
|
|||
|---|---|---|---|---|---|
Footprint
|
Varies from 4 to 8 squares |
Varies from 2 to 4 squares |
Varies from 1 to 4 squares |
||
Contrast
|
Contrast varies from very low to |
Contrast is always low |
Good contrast |
||
Shape
|
Sometimes a square, sometimes |
Just like low-res |
Just like low-res |
||
Overall score
|
3 Points |
5 Points |
8 Points |
||
Low-Res Sensor
|
Low-Res Sensor
(pixel-shifted) |
High-Res Sensor
|
|||
|---|---|---|---|---|---|
Footprint
|
Contrast below visibility |
Contrast below visibility |
Too big; one square |
||
Contrast
|
Contrast below visibility |
Contrast below visibility |
Good but could be better |
||
Shape
|
Contrast below visibility |
Contrast below visibility |
Always the correct shape |
||
Overall score
|
0 Points |
0 Points |
6 Points |
||
Conclusion
This doesn't come as a big surprise: The high-res sensor wins hands-down with 23 points over both the low-res sensor (7 points) and the the pixel-shifted low-res sensor (10 points).
Pixel-shifted sensors do provide an improvement over unshifted sensors yet they don't get anywhere near high-resolution sensors.
Would you buy a Ferrari with a 1.2 liter / 100 HP engine? Even if the sales-guy told you it had chip tuning and was just as good as the original 5 liter / 400 HP engine?
So when your are looking for a HD-camera make sure it has the proper sensor resolution and don't get fooled by the myth of pixel-shifting.