JP2002027487A - System for reproducing image from color image information of bayer type color arrangement - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent a horizontal or vertical high frequency region image component from being missed or having a false color by regarding it as aliasing occurring when a sampling theorem is not satisfied and setting some assumption on the image in a so-called digital camera, a TV camera for imaging through a Bayer type CCD sensor or an apparatus for reproducing image information of such a color arrangement. SOLUTION: For a low chroma image region, the horizontal or vertical high frequency region image component of a green g signal beyond the sampling theorem (or close to the limit) is restored adaptively. Based on that signal, a red r signal and a blue b signal are interpolated uniquely from horizontally adjacent pixels, vertically adjacent pixels and four corner pixels depending on its pixel arranging position. Horizontal or vertical high frequency region component is restored and false color is reduced simultaneously. Although it has no effect on an oblique direction high frequency region component, that component is scarce in the natural field and the aperture effect of a sensor pixel is expected.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION In a TV camera, a digital still camera, and the like (hereinafter collectively referred to as a camera), a so-called C
CD sensors, especially Bayer-type CCD sensors, are widely used. Figure 2 shows the pixel arrangement in this sensor.
Here, x and y indicate the horizontal and vertical positions on the sensor. That is, pixels of visually important components (usually a green signal, hereinafter referred to as a green signal g) are arranged in an offset manner. Further, the subsequent components (usually a red signal and a blue signal, hereinafter referred to as a red signal r and a blue signal b) are arranged to be thinned out to 1/2.

When a camera is constructed using this sensor, pixels surrounding a pixel where there is no color cell corresponding to a certain color (hereinafter referred to as an interpolation pixel for that color) (hereinafter referred to as the color pixel) are used. Is referred to as the sensor pixel for).

[0003] From the standpoint of two-dimensional signal processing techniques, these processes are demodulations from subsampled values without a prefilter. If the original image (strictly speaking, an image formed on the sensor surface) contains high-frequency components, the sampling theorem is not satisfied. For this reason, aliasing occurs, and the fine pattern becomes another frequency component, resulting in a false color.

The present invention relates to these improvements. Here, the case of a camera using a CCD will be described. However, in general, the present invention can be applied to a case where a color image is reproduced from image information represented by the same color information arrangement.

[0005] Here, matters necessary for future explanation will be outlined. Now, the repetition [horizontal, vertical] frequency when all the pixels of red r, green g, and blue b are considered together is (μ ₀ ,
ν ₀ ). For red signal r and blue signal b, the cell is 1/2.
Because it is the decimation (sub-sample), the sampling frequency is _{(μ 0/2, ν 0} /2). As for the green signal g,
This is known as offset sampling. These sampling frequencies are indicated by circles in Fig. 3. ◎ mark is the center of the basis component.

FIG. 3 shows a red signal r (x, y) and a green signal g.
(X, y) and the frequency spectrum R (μ, ν), G (μ, ν), B (μ, ν) of each signal of the blue signal b (x, y) are shown. Here, μ and ν represent the corresponding horizontal and vertical frequencies. Note that the spatial value (pixel value) is represented by a lowercase r,
The frequency components are represented by capital letters R, G, and B.

Next, three types of high frequency components are defined as shown in FIG. As a result of the sampling, the frequency spectrum of the high-frequency component that is converted to direct current is indicated by a circle in FIG. The horizontal and vertical high frequency components of the green signal component (marked by X in FIGS. 9 and 3) do not become DC components but degenerate as described later.

In the CCD sensor, each color signal component is read as a discrete value by each color sensor, that is, sampled as described above. Here, the above high-frequency components do not satisfy the sampling theorem, so they are aliased.

[0009]

2. Description of the Related Art Many inventions and conference presentations have been made in the past in order to improve the disappearance and false color of such high-frequency components and to obtain a color signal as much as possible from the output of a CCD sensor.

In terms of aiming at eliminating the above high-definition components and solving false colors, the technically closest to the object of the present invention is as follows.
It is presumed to be Patent No. 2931520 "Color separation circuit of single-plate color video camera" (patent holder: Sanyo Electric Co., Ltd .; hereinafter referred to as a cited patent).

According to this method, a first interpolation circuit suitable for a case where the horizontal correlation is strong, and a second interpolation circuit suitable for a case where the vertical correlation is strong, for three signals of a red signal r, a green signal g and a blue signal b. The interpolation circuit is switched according to the result of determination of the correlation value described later. Hereafter, this “switching” means not only on / off switching (hard switching), but also continuous switching or smooth switching (soft switching) depending on the value for controlling switching (correlation value in this case). The case is included.

The above cited patent seems to have the following two problems.

First, in the method of determining the direction using only the green g component (FIG. 7 of the cited patent), the above-mentioned high-frequency component (horizontal, vertical, and oblique) cannot be detected. By aliasing,
The horizontal high-frequency component and the vertical high-frequency component degenerate to the same frequency component and cannot be distinguished. This is because the oblique high-frequency components are converted to DC components and low-frequency components.

Next, the cited patent uses three components of red r, green g, and blue b, and suppresses the detected correlation value where the degree of color saturation is high (FIG. 15 of the cited patent). . However, there is a contradiction in this concept. Even in achromatic regions, false colors make them seem chromatic at first glance.

Further, in the case of half of the above cases, interpolation is difficult. That is, as a result of the judgment,
For example, even if it is attempted to horizontally interpolate the red signal r of the green g cell pixel, if the pixels on both sides in the horizontal direction are blue b cell pixels, it is not possible to simply horizontal interpolate, and it is necessary to add vertical interpolation.

As described above, the horizontal / vertical judgment is impossible. Further, it is impossible to perform directional interpolation on the red signal r and the blue signal b.

[Problems to be solved by the invention]

Already

2. Description of the Related Art
It is solved by (2).

(1) Broadband demodulation of green signal g: As described above, the green signal g is offset-sampled. As a result, as is well known, both the horizontal high-frequency component and the vertical high-frequency component (marked by x) are degenerated. In other words, completely different two-dimensional frequency components are converted into the same two-dimensional frequency by sampling. The signal is reproduced by interpolating this in the direction of high correlation. The following assumptions are made in this judgment.

(2) Demodulation of red signal r and blue signal b: Red signal r and blue signal b are interpolated only by the correlation between the nearest neighbor cell of the same color and green signal g without directional interpolation. . This includes horizontal interpolation, vertical interpolation, and 4
There is a corner interpolation, which uses the closest pixel value and is independent of the horizontal / vertical correlation results above.

In other words, in the present invention, only the green component g is switched by the output of the correlation judgment circuit and interpolation is performed in the present invention. For the red signal r and the blue signal b, no switching based on the determination is performed.
The output is uniquely determined by the value of the peripheral pixel of b and the green signal g. As described above, in the cited patent, interpolation of two systems may not be easily performed, but there is no problem in the present invention.

As a result, the circuit is reduced. By the way, in the cited patent, FIG.
As shown in FIG. 16 and the like, two series of interpolation circuits are provided for a red signal component, a green signal component, and a blue signal component, one for horizontal and one for vertical. And these two
The output of the series is switched by the correlation judgment circuit.

[0022]

As described above, according to the present invention, only the green component g is interpolated in the horizontal and vertical correlation directions based on the result of the output of the determination circuit. That is, switching is performed to adopt only one of the results of the two interpolations. As a result, a broadband green signal g is secured. The red signal r and the blue signal b are uniquely output based on the values of the peripheral pixels of each color and the green signal. To implement this, the following two processes are performed.

First, a wideband green signal g is secured. FIG. 4 shows the names of the pixels related to the green signal g. The central pixel where the green signal g does not exist is named pixel c. This can be a red cell or a blue cell. Since the green signal g is offset sampled, to interpolate the value of the green signal g at c, a diamond-type two-dimensional filter (or, for short, the average value of the surrounding four pixels, etc.) is usually used. Use. However, in this method, a high-frequency G signal component near the mark x in FIG. 3 cannot be left in principle, as described above.

For this reason, the directionality is detected, and one of the two degenerated components (that is, the horizontal high band or the vertical high band) is determined, and the directional interpolation is performed to perform wideband demodulation. .

However, this processing is not realized as it is, contrary to the two-dimensional sampling theorem well known in communication engineering. Therefore, an assumption (first assumption) that “the values of r, g, and b are substantially equal in each pixel” is set.

The operation will be described below with reference to FIG. The green signal g2 is extracted from the CCD sensor 1, and the red signal r3 and the blue signal b
4 is interpolated by a directional interpolation circuit 5.

Next, the red r and blue b signal components are synthesized by the g-correction interpolation circuits 7 and 8 based on the green signal g6.
Signals 9 and 10 that reproduce the high-frequency components of the red signal r and the blue signal b that have been converted to low frequencies by aliasing
To reduce false colors.

However, also in this case, it is impossible in information theory. Therefore, we make assumptions about the signal. here,
An assumption (second assumption) is made that "the chromaticity is almost constant in the local area". This is not too empirical assumption.

Under the assumption, the g-correction interpolation circuits 7 and 8 interpolate the red signal r as follows, for example. That is, as shown in FIG. 3, the value r ′ to be interpolated is proportional to the value of the surrounding red r sensor pixel and the true value (sensor pixel value; g ₀ ) of the g signal of that pixel. Interpolate as shown in 6. That is, it is obtained from r ′ interpolation value: g ₀ = r true value: g interpolation value. The g interpolation value is as described above.

The reason why the horizontal and vertical high-frequency components of the red signal r and the blue signal b cause false colors is that they become DC and low-frequency components due to aliasing. And its value is determined by the sampling phase. If the above hypothesis holds, assuming that the green signal g is correctly interpolated, this effect is eliminated by performing proportional interpolation, false colors are reduced, and the original signal can be reproduced.

According to this method, multiplication and division are required for signal processing. Therefore, as described later, a convenient method of interpolating with the difference can be considered. This can be approximated sufficiently in the low saturation region.

[0032]

As long as the above two assumptions hold, the original image is correctly reproduced for the horizontal and vertical high frequency components. That is,
Reproduce high-frequency components and reduce false colors. Does not play correctly,
False colors remain when the two assumptions do not hold.

The first assumption does not hold for a high chroma image. What actually matters is the case of high saturation and high frequency components. However, since false colors are masked and inconspicuous, there is no particular problem. In other words, this assumption is extremely violent, but since false color is a major obstacle in the low-saturation area, it is more than expected in that area. If it is not a high-frequency component, there is no problem if the horizontal or vertical interpolation results in the same value.

The second assumption does not hold when there is a sudden change in chromaticity. However, the same phenomenon occurs in conventional interpolation, and does not cause any particular problem.

The oblique high frequency component is not improved even if the two assumptions hold. That is, the false color of the oblique high frequency component is
It cannot be reduced from this principle. Instead, we expect the attenuation due to the aperture effect due to the aperture ratio of each pixel of the CCD sensor, and that there are few such images in nature.

[0036]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A method of obtaining red, green, and blue cell outputs from a Bayer type CCD sensor and a circuit for interpolating the value of a horizontal or vertical peripheral pixel from the value of the peripheral pixel are well known. This is realized even in a normal camera which does not perform the false color reduction processing. In addition, many documents and patents, for example, cited patents, are described in detail from the second half of the second page to the first half of the third page. Therefore, these 3
Color signals are obtained two-dimensionally, and normal interpolation can be easily realized.

First, wideband demodulation of the green signal g by directional interpolation will be described with reference to FIG. Since the green signal g is offset sampling, the pixel c where this color does not exist (FIG. 4,
Interpolate the value of g in the interpolated pixel).

The horizontal high-frequency component and the vertical high-frequency component near the mark x in FIG. 3 degenerate as described above, and cannot be reproduced in principle.

Therefore, as described above, "for each pixel, r,
Considering that “the values of g and b are almost equal” (first assumption), perform directional interpolation demodulation, that is, horizontal interpolation or vertical interpolation.

First, the pixel c for which the green signal g is to be interpolated
Is a pixel of r (cell pixel of r), that is, a case where the pixel position is an even number in both the horizontal direction x and the vertical direction y. The value _r c 11 of a red signal in the cell, directly above the green pixel value _{g U} just below the green average value of the pixel value _{g D (g} U +
g _D) / 2 and a straight left green pixel value _{g L} and the average value of the straight right green pixel value _{g R (g} L +
g _R ) / 2, which horizontal / vertical correlation is higher is identified.

[0041] That is, the averaging circuit 12 immediately below just above the average circuit 13 of the straight left straight right, the output of the absolute value circuits 16 and 17 of the respective difference signal 15 between the _{r c,} the comparator circuit 18 Compare with. Then, the closer one of these (the smaller absolute value) is switched by the switch circuit 19, and
The interpolation value of the green signal g of the pixel is used. That is, vertical interpolation is performed for the former (directly above and below), and horizontal interpolation is performed for the latter (directly left to right). As described above, the green signal g of the pixel c having no green signal value in the sensor is interpolated not vertically and horizontally but vertically or horizontally.

In a region having no high-frequency component, the result is almost the same irrespective of the directional interpolation. Therefore, any interpolation may be appropriately performed without any particular distinction. In the above method, it is interpolated by either. If the difference is small in the above-mentioned "closeness" determination, the above-mentioned soft switching method can be applied.

The same applies when the pixel into which the green signal g is to be interpolated is a blue cell, that is, when both the horizontal x and the vertical y are odd.

Next, in the second stage, the red signal r and the blue signal b
The g-correction interpolation circuits 7 and 8 that perform the interpolation of the above will be described. As a result, high-frequency component reproduction and consequent false color reduction are realized. First, the interpolation of the red signal r is described.

As described above, it is assumed that "the chromaticity is almost constant locally" (second assumption). As shown in FIG. 6, the red signal (r ′: interpolation value) to be interpolated is a peripheral red r sensor pixel and the true value of the g signal of that pixel (sensor pixel value;
g ₀ ). That is, the r ′ interpolation value is obtained as r ′ interpolation value: g ₀ = r true value: g interpolation value. The g interpolation value is as described above.

For example, to obtain an interpolated value of a red r signal in a green g cell pixel sandwiched between two right and left r cell pixels, as shown in FIG. _{': g 0 = (r L} + r R) / 2: assume the _{(g L + g R) /} 2 of such proportional relationship. From this, r ′ = (r _L + r _R ) {g ₀ / (g _L + g _R )} is obtained. This configuration is shown in FIG.

In addition to the horizontal interpolation described above, there are vertical interpolation and interpolation from four corners depending on the pixel position. Regarding the interpolation of the red signal r, as shown in FIG. 7, the pixel position (x,
There are three types according to y). For example, when x = odd and y = even, interpolation is performed from the left and right.

These odd / even states can be determined by the least significant bit (LSB) of the timer 21 in FIG. 1 for driving the sensor pixels. Incidentally, such a determination is extremely common even in a known camera which does not perform the false color reduction processing as in the present invention.

However, when the horizontal / vertical directions are not remarkable, or when g
When the component is small, soft switching may be considered, such as applying a larger load to the normal interpolation value than the suppression interpolation value.

The above is similarly performed for the blue signal b.

[0051]

According to the present invention, in addition to the seemingly violent first assumption, a safe second assumption is set, wide-band demodulation of a green signal g offset-sampled by directional interpolation is performed. We proposed a two-stage method called "g-proportional synthesis" of signals.

As a result of the experiment, wideband demodulation was realized and reduction of false colors was observed even in a general color image where the first assumption was not satisfied.

Although the case of the Bayer primary color type has been described,
If the complementary color type is used, the correlation is further enhanced, and the effect is enhanced. Furthermore, the color difference line-sequential method can be performed more effectively because the first step of this proposal can be omitted.

In the above description, the case of the proportional relationship (the ratio is constant) has been described. However, as mentioned above, it involves multiplication and division and is not desirable in terms of circuitry. Then, as an alternative, assuming that the difference is constant, that is, r ′ − (r ₁ + r ₂ ) / 2 = g ₀ − (g ₁ + g ₂ ) / 2, then r ′ = (r ₁ + r ₂ ) / 2 + Δg ₀ − (g ₁ + g ₂ ) /
The interpolation signal is obtained as in 2 挿, and the operation is simplified.

Here, the description has been made assuming a so-called digital camera. However, an interlaced scanning signal can be obtained by thinning out one scanning line or by adding signals of two scanning lines while changing combinations. .

Although the case where the CCD sensor is of the primary color system of red, green, and blue has been described, an equivalent operation is possible even with a complementary color system if the pixel array is replaced.

Although the case of the CCD image pickup device has been described here, the present invention can be generally applied to a case where an image represented by the same color information arrangement is reproduced. Furthermore, it can also be applied to images that have already been reproduced in color and false colors have occurred.

[0058]

[Brief description of the drawings]

FIG. 1 is a schematic configuration diagram of the present invention.

FIG. 2 is an explanatory view of a Bayer type sensor arrangement which is a main object of the present invention.

FIG. 3 is an explanatory diagram of a frequency spectrum of a video signal to which the present invention is applied;

FIG. 4 is an explanatory diagram of naming of peripheral pixels to be interpolated;

FIG. 5 is a configuration diagram of a g signal interpolation circuit (part of FIG. 1);

FIG. 6 is an explanatory diagram of a method of interpolating an r signal (b signal) by a g-correction circuit.

FIG. 7 is an explanatory diagram of the case of the interpolation method of the r signal (b signal).

FIG. 8 is a circuit diagram of interpolation of an r signal (b signal).

FIG. 9 is an explanatory diagram of a two-dimensional frequency display of a high frequency component.

[Explanation of symbols]

1. . 4. Bayer-type CCD sensor; . Directional interpolation circuit, 7, 8. . g-corrected interpolator, 18. . Horizontal / vertical correlation determination circuit, 21. . (Drives the CCD sensor)
Timer circuit.

Claims (2)

[Claims] 1. A first arrangement arranged in an offset manner (checkered form).
Color pixels (usually referred to as a green signal, hereinafter referred to as a green signal), and second and third color pixels (usually a red signal and a In a device for reproducing image information of a so-called Bayer-type pixel arrangement composed of a blue signal (hereinafter, referred to as a red signal and a blue signal), the green signal value of a pixel corresponding to red or blue is determined in the horizontal and vertical directions of the surrounding green signal. The horizontal or vertical correlation is determined from the relationship between the interpolated value in the direction and the red or blue signal value, and either the horizontal or vertical interpolated value is switched and adopted according to the result. And a green signal obtained by interpolating the green signal obtained from the sensor and the green signal interpolated. Formula. 2. An image reproduction system according to claim 1, wherein color image information in which so-called Bayer-type pixels are arranged is obtained from a photoelectric conversion device.