JPH0795595A - Color image pickup device - Google Patents

Abstract

PURPOSE:To provide a color image pickup device generating a luminance signal and a chrominance signal with high quality. CONSTITUTION:A color signal extract circuit 150 uses corresponding point information obtained by a corresponding point extract circuit 109 to conduct color signal separation processing (generation of a luminance signal and a color difference signal) through the calculation of picture element output signals by CCD image pickup elements 101, 105. Prescribed RGB signals or luminance/ color difference signals are obtained from the luminance signal and the color difference signal via a matrix circuit 151 and a color signal processing circuit 152.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a camera for photographing the same subject by a plurality of image pickup devices, or a camera for photographing a subject at a different time by slightly shifting the position of the same subject. The present invention relates to a video signal processing device for synthesizing and outputting one video signal from signals obtained from.

[0002]

2. Description of the Related Art Lenses, lens drive motors and CCDs
There is a camera which has a plurality of image pickup systems in which image pickup elements such as the above are integrated, and which operates the respective units collectively to photograph the same subject.

FIG. 12 is a block diagram of the camera. This camera includes color CCD elements 101 and 105, color signal extraction circuits 102 and 106, a matrix circuit 103,
107, color signal processing circuits 104 and 108, corresponding point extraction circuit 109, modeling circuit 110, image generation circuit 111, image composition circuit 112, texture mapping circuit 113, switch 114, and 2D display 115. The 3D display 116 and the optical low-pass filters 130 and 131.

First, a processing route when this camera is used as a stereoscopic camera will be described. A subject is photographed by two image pickup systems from slightly different positions or angles, and video signals thereof, for example, RGB are independently generated and displayed on a stereoscopic television. When viewing two images with this parallax on the 3D display 116 capable of stereoscopic viewing,
Humans can feel the subject in three dimensions. The process of generating the luminance signal and the color signal will be described in the next section. In the case of this configuration, the switch 114 is on the A side.

Further, since this apparatus has the circuits 109 to 113, it is possible to obtain the stereoscopic information of the subject. In this case, the corresponding point extraction circuit 109 carries out an operation to find corresponding points of mutual images, and further the modeling circuit 110.
Then, the distance information of the stereoscopic information of the subject is obtained from the positional relationship of the optical system and the focus information and converted into a mathematical model. After that, the image generation circuit 111 generates a three-dimensional image called a wire frame, and the texture mapping circuit 113 stitches the two-dimensional image synthesized by the image synthesis circuit 112. To output this video signal, switch 1
14 is set to the B side. By having such a configuration, it is possible to convert an image from a camera position different from the actually photographed angle. Corresponding point extraction circuit 10
9 is for calculating in which pixel the same subject appears in each of the two image signals for each area within a predetermined range, and the correspondence between the vertical and horizontal directions of the screen is obtained with an accuracy of 1 pixel or less. is there.

As another usage method, one high-quality image can be obtained from the respective image signals obtained from the two image pickup systems. This method is described in JP-A-4-26.
It is described in Japanese Patent No. 2677. This is to calculate the corresponding points of two video signals, correct the trapezoidal distortion caused by parallax, and then overlap the corresponding pixels, reduce the random noise generated in the image sensor, and obtain a high-quality image. Is what you get.

The above-described example has the same configuration and arrangement of the image pickup system, and can be selected by switching the signal processing system and the output form according to the intention of the user.

[0008]

A single-plate color image pickup device in which a color filter is provided in each pixel of a CCD image pickup device is often used for color image pickup in each image pickup system. Although there is another method in which color separation is performed using a prism and a CCD is provided for each color, the former is superior in that it can be made smaller. There are various methods for the characteristics and arrangement of the color filter, but the one called the complementary color mosaic type or the color difference line sequential method, which is easy to obtain the resolution of the luminance signal, is most often used.

When this single plate type color image pickup device is used,
There were the following problems. In a single-plate color imaging device such as the color difference line sequential system, it is impossible to extract a complete luminance signal or RGB signal from one pixel. Therefore, the color signal extraction circuits 102 and 106 obtain data of pixels of different colors in the vicinity, which are obtained at different times, at one timing, and the matrix circuits 103 and 107 combine these signals to obtain RGB signals. White balance processing and, if necessary, gamma correction processing are performed by the color signal processing circuits 104 and 108 in the next stage to create predetermined RGB signals or color difference signals. On the other hand, the luminance signal is generated by the low pass filters 117 and 118.

This image signal processing is performed independently for each of the two image pickup systems, and the image signals obtained from the respective image pickup elements are slightly blurred as compared with a system in which color separation is performed by a prism or a monochrome camera. In addition, when a chromatic fine pattern is photographed, a phenomenon occurs in which a color different from the original color appears. These problems are the same as those of a normal camera that uses a single-plate color imaging method, but these problems are the same as those of a normal camera that uses a single-type color imaging method. Is. When texture mapping processing and high definition processing by superimposing as described above are performed using such deteriorated video signals, only a slight improvement in resolution can be expected, and pseudo color appears. In some cases, it may worsen.

An object of the present invention is to solve the above-mentioned problem of image quality deterioration and to provide a color image pickup apparatus for generating high-quality luminance / color signals.

[0012]

A color image pickup apparatus of the present invention has a single-plate color image pickup apparatus, and in a color apparatus for processing two or more pieces of image information photographed in different photographing position relationships, each image information piece is processed. And means for obtaining information on the positional relationship of corresponding points of the image, and means for generating one prescribed video signal from the information on the positional relationship of corresponding points and each video information.

[0013]

The present invention considers the corresponding points of two images at the time when the color extraction signal is obtained, and performs the optimum luminance signal / color signal processing.

Here, the image pickup device having a color filter generally refers to a color difference line sequential system, a pure color (RGB) stripe system, or a device in which a color filter is provided for each pixel. In addition to CCD solid-state image sensor,
Even in the S-type solid-state image pickup device and the image pickup tube, a color filter is provided on the image pickup surface to pick up a color image with one element. The two or more image pickup device signals refer to image pickup device signals obtained from a plurality of image pickup systems, or image pickup device signals obtained by one image pickup device at different time and shooting positions. Corresponding point information is information that compares the two images for matching and determines the same object, information that automatically vibrates the imaging system and detects the moved command value or movement amount, and spatially beforehand. The displacement information of a plurality of image pickup systems installed by giving a displacement.

[0015]

Embodiments of the present invention will now be described with reference to the drawings.

(First Embodiment) FIG. 1 is a block diagram of a color image pickup apparatus according to the first embodiment.

In this embodiment, the image synthesizing circuit 112 is removed from the conventional example of FIG. 12, and a color signal extracting circuit 150, a matrix circuit 151, a color signal processing circuit 152, a distortion correcting circuit 155 and an enhancing circuit 153 are added. . Here, the corresponding point information obtained by the corresponding point extraction circuit 109 is added to the color signal extraction circuit 150 and the enhancement circuit 153. The output signal of the color signal processing circuit 152 directly passes through the enhancement circuit 153 and the distortion correction circuit 155 and is directly output to the output circuit 2.
It is supplied to the D display 115 or the texture mapping circuit 113. The matrix circuit 151 and the color signal processing circuit 152 operate similarly to the conventional example.

First, the CCD image pickup device 10 in this description
The 1,105 color filter array is shown in FIG. A color filter is attached to each light receiving element, and the types are yellow Ye, magenta Mg, and cyan C.
There are four types, y and green Gr. Here, Ye = R + G Mg = R + B Cy = B + G Gr = GR where R is red, G is green, and B is blue. Is. In each field, the signals of the upper and lower two pixels are added,
It is designed to output every one line, and in the even field and the odd field, the read line is shifted by one line. For example, in the even field, as shown on the left side of the drawing, Ye + Mg, Cy + Gr, ... Are read in the nth line, and Ye + Gr, Cy + are read in the n + 1th line.
It is read as Mg, ....

Here, in the conventional configuration, the color signal extraction circuits 102 and 106 using the delay elements match four neighboring signals of different colors at the same timing. The four kinds of signals are, for example, Ye + Mg Cy + Gr Ye + Gr Cy + Mg. The matrix circuits 103 and 107 perform mutual addition / subtraction processing of these signals to generate RGB signals. Further, the luminance signal is obtained by a moving average of pixels that are adjacent in the horizontal direction. That is, (Ye + Mg) + (Cy + Gr) = 2R + 3G + 2B≈
Calculate the luminance signal. However, the coefficients of the R, G, and B signals in this embodiment may use different values depending on the actual color filter characteristics, but integer values are used for explanation.

Here, in general, when the information of images spatially different in position is added, the same effect as that of executing the spatial digital low-pass filter appears and the pass band of the signal is limited. FIG. 3 shows the frequency characteristics of the video signal. In the above example, since the luminance signal is always obtained by performing the moving average calculation with the adjacent pixel, the luminance signal frequency band is 301. With this characteristic, when the sampling frequency of the image is fs, the signal is completely cut at fs / 2, and the signal passage is limited even at a frequency considerably lower than fs / 2. The color filter is a repeating array of two sets of pixels in the horizontal direction. That is, since the color is sampled at intervals of 2 pixels, the carrier reaches the frequency of fs / 2 for the chromatic color, and a false color is generated. In order to prevent this, the frequency characteristics of the optical low-pass filters 130 and 131 provided in front of the CCD image pickup device have characteristics as shown by 312 in FIG. Due to these factors, the resolution of the video signal is considerably lower than that of a monochrome image pickup device which originally has the same number of pixels.

Therefore, the color signal extraction circuit 150, which is a feature of this apparatus, performs the following operation. First, the corresponding point extraction circuit 109 calculates corresponding points of two images. As for this output value, one point of a certain subject is measured by each CCD image pickup device 10
It has information on which pixel 1, 105 is used for photoelectric conversion. The color signal extraction circuit 150 uses this corresponding point information,
Color signal separation processing is performed by calculation with the pixel output signals of the image pickup elements facing each other.

One example is shown in FIG. Here, the output signal 401 of the CCD image pickup device 101 is the CCD image pickup device 105.
It is assumed that one point of the same subject as the output signal 411 of 1 is obtained. In this case, in this device, the signal 401 and the signal 4
The luminance signal is obtained by obtaining the sum of 11. That is, (Ye + Mg) + (Cy + Gr) = 2R + 3G + 3B (1) Signal 401 Signal 411 Luminance signal 402 412 :: 403 413 404 414 :: On the other hand, the color difference signal is obtained by calculating the difference between the color difference signal and the corresponding CCD image sensor signal. That is, the nth line (Ye + Mg) − (Cy + Gr) = 2R−G (2-1) signal 401 signal 411 R color difference signal 1 (Ye + Mg) − (Cy + Gr) = 2R−G (2-2) signal 412 signal 402 R Color difference signal 2 n + 1th line (Ye + Gr)-(Cy + Mg) = G-2B (3-1) signal 403 signal 413B color difference signal 1 (Ye + Gr)-(Cy + Mg) = G-2B (3-2) signal 414 signal 404 It is obtained from the B color difference signal 2.

This processing is performed every pixel clock. Here, with respect to the frequency band of the luminance signal, since there is no moving average processing when obtaining the luminance signal, half of the pixel sampling frequency fs can be obtained, and the resolution of the image is dramatically improved. The luminance signal frequency band at this time is 302. Further, the sampling frequency of the color difference signal is twice that of the conventional example, that is, each pixel can be obtained. Therefore, the chromatic color is not disturbed by the false color at all.

The pixels are not always completely overlapped as described above. FIG. 5 shows a state in which opposed pixels are displaced by 1 pixel or less in both horizontal and vertical directions. In this case, first, the output signal of the CCD image pickup device 105 is interpolated in advance for each of the four types of color filter signals, and the signal strength at the position of the corresponding pixel signal is predicted. For example, the pixel signal 502 ′ (Ye + Mg) on the CCD 105 side relative to the signal 502 (Cy + Gr)
Is generated, the calculation formula is [signal 502 '] = (1-KH) (1-KV) [signal 513] + KH (1-KV) [signal 512] + (1-KH) KV [signal 511]. + KH KV [Signal 510] (4) KH is corresponding point information (0 ≦ KH ≦ 1) indicating a horizontal distance between 502 and 513. KV is corresponding point information (0 ≦ representing a vertical distance between 502 and 513. It is determined by KV ≤ 1). This equation (4) uses bilinear linear interpolation, and in a device that can use more circuits, a wider range of pixel information may be used, for example, cubic spline interpolation may be performed. By such a calculation, the pixel signal required on the opposite image sensor side is predicted, and then the luminance signal
Generate a color signal. This calculation is based on the above (1) to (3-
2) is used.

In practice, this interpolation processing is performed on both video signals. Each video signal has image distortion caused by lens distortion difference, trapezoidal distortion caused by parallax, and image size difference due to error in focal length setting value of each lens. We have to fix these. Therefore, this pixel interpolation processing and distortion correction processing are performed in the same processing step.

When there is an overlap of 1 pixel or less, the luminance signal band is slightly inferior to 302 and a curve of 303 is obtained. Even so, the bandwidth will be higher than in the conventional example. Further, the color signals are sampled at irregular intervals, but the number of samplings is the same as in the case of FIG.

Further, the pixels of the filters of the same color may overlap depending on the photographing condition. In this case, the spatial sampling interval is not narrowed even if the calculation is performed with the image pickup device facing the image pickup device. However, since a luminance signal and a color difference signal are respectively obtained by both image pickup devices, these signals can be averaged to obtain a signal with a high S / N. Therefore, by performing edge enhancement that raises the high frequency side in the band, the passing frequency is widened and the resolution is improved. Since this signal has a good S / N ratio, noise is not noticeable even when a high frequency operation that is considerably stronger than the conventional edge enhancement is performed. This edge enhancing operation is performed by the enhancing circuit 153.
However, the false color signal is generated as in the conventional example.

In this way, the luminance signal based on the corresponding point information
After the color signal is generated, a predetermined R is passed through the matrix circuit 151 and the color signal processing circuit 152, which operate in the same manner as the conventional one.
Obtain a GB signal or a luminance color difference signal. In this state, the geometrical distortion difference between the two video signals has already been corrected, and therefore the distortional correction circuit 155 processes the geometrical deformation from the desired photographing angle. In the above example, the CCD image pickup device 10
Since the video signal is created based on the pixel signal of 1, geometrical distortion correction is performed between the CCD image pickup device 101 and the shooting direction to be output. The image thus synthesized is finally output to the display 115 or 113.

When the video signal subjected to such processing is viewed on the entire screen, the degree of partial improvement in resolution is different. If this is left as it is, if an object such as a striped object with many repetitions or a tree-lined path with many fine patterns is taken, an unnatural image will result. Further, since the distortion correction circuit 155 performs a kind of interpolation calculation, there is a possibility that the resolution may be uneven. Therefore, in this apparatus, the edge enhancement characteristic of the enhancement circuit 153 is increased or decreased based on the information of the corresponding points to reduce this problem. For example, no enhancement is performed in the state of FIG. 4, slight enhancement is performed in the state of FIG. 5, and strong enhancement is performed when filters of the same color overlap. Then, even in a place where the resolution is lowered by the distortion correction, the edge enhancement is similarly strengthened.

In the processing of the present apparatus, there are few places where the filters of the same color that completely interfere with the false color signal are completely overlapped, and there are only a few points on the screen. Therefore, the characteristics of the optical low-pass filters 130 and 131 can be set to be considerably higher than those of the conventional example. The optical low-pass filter characteristic in this case is shown by 313 in FIG.

By the processing described above, the resolution is improved by at least 50% or more on the entire screen, and a sharper image than that of the conventional example can be obtained. Also, the occurrence of false color does not occur in most places. Such an effect cannot be obtained when the image signals are processed by the image pickup devices and then combined as in the conventional example.

Further, in some cases, the photographing direction of one or both of the image pickup systems may be slightly changed so that one CCD
Different color filters may correspond to the image sensor pixels depending on the time. As a result, the area where the same color filter overlaps always moves, and becomes inconspicuous as a moving image.

Further, when the corresponding point extraction circuit 109 detects a state in which an error occurs or the accuracy is lowered, it is desirable to switch to the conventional video signal processing system without using the pixel signal of the opposing image pickup device. .

(Second Embodiment) The structure of the present invention can be used for other than the color difference line sequential method. The processing contents in the case of the RGB stripe method, which is a general method following the color difference line sequential method, will be described below. FIG. 6 shows a color filter array of this image sensor. In this color filter, filters of R, G, and B colors are arranged in order vertically, and the same color filter is repeated in the horizontal direction at a cycle of three pixels. Therefore, each pixel signal can be treated as an RGB signal of a color signal almost as it is. Here, generally, the luminance signal is generated as follows.

(R + G + B) / 3 This is the sum of the obtained pixel signal, the left pixel, and the signal of the left of the two pixels, and the weighting coefficient is (1,1,1) moving average. Processing. Therefore, the frequency characteristic of the calculation of the luminance signal is as shown by the characteristic 601 in FIG. Since a set of colors is repeated in three pixels, a carrier flutters at a frequency of fs / 3 for chromatic colors. for that reason,
The optical low pass filter must take the characteristics of 611 to cut this carrier. For the above reason, the color filter system can obtain the band of the luminance signal up to 1/3 of the pixel sampling frequency.

In the case of this color filter system, the structure of this apparatus is as shown in FIG. Compared with the configuration shown in FIG. 1, matrix circuits 103 and 10 that generate RGB signals from color difference signals
7 is no longer needed and has been deleted.

The process of luminance signal generation in this apparatus will be described below. FIG. 9 shows two CCD image pickup devices 101 and 10.
5 shows a case where 3n + 1 pixels (n is an integer) are shifted in the horizontal direction. In this case, R G B mth clock [signal 801] + [signal 811] + [signal 812] m + 1 clock [signal 801] + [signal 802] + [signal 812] m + 2 clock [signal 813] + [signal 802] + [signal 812] :::, and so on.
The processing is performed so that the signals of the different color filters closest to are handled. In this case, since only pixel signals in a narrow range are used spatially, the pass band of the luminance signal by moving average extends to 602 in FIG. 7. Further, each of the color signals is processed as R G B m clock signal 801 signal 811, signal 812 m + 1 clock signal 801, signal 802 signal 812 m + 2 clock signal 813 signal 802 signal 812 ::::. That is, the mutual CCD image pickup devices 1
A color signal is generated from the pixels spatially closest to 01 and 105. Therefore, the sampling frequency of the spatial color signal is higher than that of the conventional example, and the interference with the luminance signal by the false color signal is very small.

Other than the example of FIG. 9, like the first embodiment, 1
In the case of a shift of less than a pixel, filters with the same color may overlap, but the processing in each case is similar,
The enhance circuit 153 is controlled based on the corresponding point information,
The processing is performed so that the band of the video signal is broadened as a whole.
As for the optical low-pass filters 130 and 131, similarly to the first embodiment, those having wide frequency characteristics can be used, and a wideband video signal can be obtained as a whole.

(Third Embodiment) The above two embodiments are as follows.
Although the apparatus having two or more CCD image pickup devices has been described, the following configuration can be adopted when the camera can be moved relative to the subject. This movement is, for example, a camera shake of the photographer or an act of intentionally shifting the camera.

FIG. 10 shows a part for generating a two-dimensional video signal in the configuration of the present device. The memory 156, the memory read control circuit 157, and the motion vector detection circuit 15 are shown.
8 and other elements having the same numbers are the same as the constituent elements in FIG. The output of the motion vector detection circuit 158 is input to the memory read control circuit 157 and the color signal extraction circuit 150.

Next, the operation of this apparatus will be described. Memory 1
56 has a capacity for one frame, for example, and is C
The output signals of the CD image sensor 101 are stored one after another. The motion vector detection circuit 158 calculates how much the image has moved in one frame from the output of the CCD image pickup device 101. This calculation requires an image memory for one frame, but the motion vector detection circuit 158 is assumed to include this amount of memory. Also, this image movement amount is
It is calculated for each of the areas into which the screen is finely divided.

Here, the color signal processing circuit 150 is a CCD
A signal from the image sensor 101 and the same signal delayed by one frame are calculated to generate a video signal. At this time, which pixel is to be calculated is determined from the motion vector information. For example, when the motion vector detection circuit 158 detects that the camera or the subject has just moved by one pixel in the horizontal direction in one frame, it can be seen that the pixel correspondence state is the same as in FIG. Therefore, similarly to the configuration of FIG. 1, the luminance signal and the color signal are generated by using the equations (1) to (3-2). Further, when the shift is not an integer number of pixels, the interpolation calculation is performed using the equation (4), and when the color filters of the same pixel overlap, the characteristics of the enhance circuit 153 in the subsequent stage are changed. The configuration is the same. That is, C
Although there is only one CD image pickup device, when the camera moves in time, the memory 156 and the motion vector detection circuit 158 can be used to bring about the same effect as having two CCD image pickup devices, and a wide band. It is possible to obtain various video signals.

(Fourth Embodiment) In the configuration described in the third embodiment, it is necessary for the camera or the subject to make a predetermined movement. With the configuration of this embodiment shown in FIG. 11, a high-definition image can be obtained even if the camera and the subject are not moving.

Therefore, in this embodiment, an optical axis deflector 16 capable of deflecting the optical axis of a variable apex angle prism or the like.
2. The actuator 160, the actual deflection angle sensor 161, and the transmitter 159 are provided.

The transmitter 159 determines the deflection angle of the optical axis deflector 162 according to the lens focal length information and outputs it to the actuator 160. Due to this output command value, the optical axis is deflected at high speed, and the image received by the CCD image pickup element 101 vibrates. This action creates a state similar to the movement of the camera or subject in FIG. In this case, it is not necessary to use the motion vector when determining the pixel to be calculated, and the deflection detection value of the sensor 161 is used. If the response of the optical axis deflector 162 is sufficiently accurate, the command value of the transmitter 159 may be used.

[0046]

As described above, according to the present invention, a predetermined video signal is always generated by using pixels of different color filters from video signals of CCD image pickup devices having different optical positions. There is an effect that it is possible to obtain a high-definition image that is dramatically enlarged and has no false color.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a color image pickup apparatus according to a first embodiment of the present invention.

FIG. 2 is a diagram showing an arrangement of color filters of CCD image pickup devices 101 and 105.

FIG. 3 is a diagram showing frequency characteristics of a video signal.

FIG. 4 is a diagram illustrating a calculation processing example.

FIG. 5 is a diagram illustrating an example of calculation processing.

FIG. 6 is a diagram showing an RGB stripe type filter array.

FIG. 7 is a diagram showing frequency characteristics of a video signal.

FIG. 8 is a configuration diagram of a color image pickup apparatus according to a second embodiment of the present invention.

FIG. 9 is a diagram illustrating an example of calculation processing.

FIG. 10 is a configuration diagram of a color image pickup apparatus according to a third embodiment of the present invention.

FIG. 11 is a configuration diagram of a color image pickup apparatus according to a fourth embodiment of the present invention.

FIG. 12 is a configuration diagram of a conventional example of a color imaging device.

[Explanation of symbols]

 101 CCD image sensor 102 Color signal extraction circuit 103 Matrix circuit 104 Color signal processing circuit 105 CCD image sensor 106 Color signal extraction circuit 107 Matrix circuit 108 Color signal processing circuit 109 Corresponding point extraction circuit 110 Modeling circuit 111 Image generation circuit 113 Texture mapping Circuit 114 Switch 115 2D display 116 3D display 117 Low-pass filter 118 Low-pass filter 130 Optical low-pass filter 131 Optical low-pass filter 150 Color signal extraction circuit 151 Matrix circuit 152 Color signal processing circuit 153 Enhance circuit 155 Distortion correction circuit 156 Memory 157 Memory read control circuit 158 Motion Vector Detection Circuit 159 Oscillator 160 Actuator 161 Deflection Angle Sensor 1 2 optical axis deflector

Claims (1)

[Claims] 1. A color image pickup apparatus having a single-plate color image pickup device for processing two or more pieces of image information photographed at different photographing position relations, and means for obtaining information on the corresponding point positional relation information of images from each image information. And a means for generating one prescribed video signal from the information on the positional relationship of the corresponding points and each video information.