JPH11234690A - Image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a high resolution image with doubled resolutions in the horizontal and vertical directions where false color and false contour are reduced by calculating a signal at a prescribed pixel position with signals having doubled resolutions in the horizontal and vertical directions based on a signal from an edge discrimination means. SOLUTION: A multiplexer 7 multiplexes respective signals to obtain a signal with doubled resolutions in the horizontal and vertical directions based on signals R1, G1, B1 of the number of pixels of an image pickup element decoded by a 1st color decoding means 5 and on signals R2, G2, B2 of the number of pixels of the image pickup element decoded by a 2nd color decoding means 6. An edge discrimination means 8 detects a difference in the horizontal and vertical directions based on pixels at left/right/upper/lower pixels at a pixel position of a G signal, discriminates an edge at the surrounding pixels and outputs a signal 'ed' denoting a discrimination result to a selection means 12. The selection means 12 outputs R, G, B signals at respective pixels at the pixel positions, that is, the R, G, B signals with doubled resolutions in the horizontal and vertical directions of the number of pixels of the image pickup element.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to high image quality and miniaturization of an image pickup apparatus such as an electronic still camera.

[0002]

2. Description of the Related Art Conventionally, in an image pickup apparatus such as an electronic still camera, a single image pickup element (hereinafter, referred to as a single image pickup element) has been used for miniaturization.
(Referred to as a single-chip image sensor). FIG. 18 is a diagram illustrating an example of a color filter array of an imaging device in a conventional imaging device. In the figure, R is an image sensor having a color filter having a spectral characteristic that allows light of R to pass therethrough. Similarly, R and B are image sensors having respective color filters. As shown in FIG. 18, G,
R is arranged every two pixels, and B and G are 2 in the (n + 1) th line.
It is arranged for each pixel. Accordingly, the R and B signals are obtained every four pixels in the upper and lower pixels (shaded portions in the drawing), and the G signal is obtained every two pixels.
Further, when the resolution is converted to twice the number of pixels in each of the horizontal and vertical directions, arithmetic processing such as interpolation is performed from the obtained pixel signals to generate R, G, and B signals of the number of imaging elements. The resolution is increased using each of the R, G, and B signals.

FIG. 19 shows a single-chip image sensor using the primary color filters shown in FIG. 18 that generates R, G, and B signals from signals from the image sensor and further doubles the signals in the horizontal and vertical directions. FIG. 11 is a block diagram illustrating an example of a configuration of a conventional imaging device for obtaining a signal of a resolution. In the figure, 101
Denotes an image sensor; 102, an A / D converter; 103, a frame memory; 104, a signal in the frame memory 103, which is separated into R, G, and B signals; A color restoration unit 105 for restoring a signal is a resolution conversion unit for converting the signal from the color restoration unit 104 into a signal having twice the resolution (an image having four times the number of pixels) in the horizontal and vertical directions, respectively. 1
01 is composed of color filters of pixels R, G, and B as shown in FIG.

Next, the operation will be described. Each pixel signal R, G, B is read from the image sensor 101, and the output is A / D
A / D conversion is performed by the converter 102, and each pixel signal is taken into the frame memory 103. Color restoration means 104
Separates each signal from the signal taken into the frame memory 103 and interpolates and generates a signal of a pixel not obtained in each of R, G, and B signals from a signal of an adjacent pixel,
It calculates and outputs RGB signals for all pixels of the image sensor.

FIG. 20 is a block diagram showing an example of the configuration of the color restoring means 104 in a conventional image pickup apparatus, and its operation will be described. In the figure, reference numeral 111 denotes a separating unit for separating R, G, and B signals from signals in the frame memory 103, and 112 interpolates and generates a pixel signal that is not obtained in each of the separated signals from a signal of an adjacent pixel. This is interpolation means for calculating and outputting RGB signals of all the pixel numbers. The signal in the frame memory 103 is separated into respective R, G, and B signals by a separation unit 111, and each is output to the interpolation unit 112. The R, G, B signals separated by the separation means 111 are shown in FIG.
(A), (b), and (c), the pixels indicated by G, R, and B in the figure are the respective signals obtained from the image sensor 101, and the blank pixels are obtained. This is a pixel signal that has not been set.

[0006] The interpolation method in the interpolation means 112 is G
As for the signal (FIG. 21A), a signal g (hereinafter, pixel position (n, m)) at the vertical n-th line and horizontal m-th pixel position
It is written. ) To interpolate the difference (| G (n−1, m) −G (n + 1, m) |) in the vertical direction and the difference (| G (n, m−1) in the horizontal pixel. ) -G (n,
m + 1) |), and interpolates the pixel signals in the direction in which the difference is small. For example, if the difference between adjacent pixels in the left-right direction is small, g (n, m) = (G (n, m-1) + G
(N, m + 1)) / 2, and when the difference between adjacent pixels in the vertical direction is small, g (n, m) = (G (n−
1, m) + G (n + 1, m)) / 2.

For the R and B signals (FIGS. 21 (b) and (c)), first, horizontal pixel interpolation is performed, and then vertical interpolation is performed. For example, for R in FIG. 21B, interpolation is performed on the vertical n-1 line and the n + 1 line, and the signal at the pixel position (n-1, m) (n + 1, m) is converted into r (n-1, m). ) = (R (n-1, m-1) + R (n-
1, m + 1)) / 2 r (n + 1, m) = (R (n + 1, m-1) + R (n +
1, m + 1)) / 2, and then the pixels of the n-th line are moved up and down (n-1).
And n + 1 lines), the signal of each horizontal pixel position m-1, m, m + 1 is obtained. B can be obtained in a similar manner.

By the above-described interpolation method, it is possible to calculate R, G, and B signals of the number of pixels of all the image pickup devices at the output of the interpolation means 112.

Next, the signal from the single-chip image pickup device is restored to the resolution of the number of pixels of the image pickup device by two signals in the horizontal and vertical directions.
In order to improve the resolution twice, the R, G, and B signals output from the interpolation means 112 are converted to resolution conversion means 105.
Enter to.

In the resolution conversion means 105, the input R,
The G and B signals are each converted into a signal of twice the resolution (an image with four times the number of pixels) in the horizontal and vertical directions. For example,
When the number of pixels of the image sensor is N × M pixels of vertical N lines and horizontal M pixels, the restored R, G, and B signals of N × M pixels are output from the color restoring unit 104, and the resolution converting unit 105 In this case, a signal having a resolution of 2N × 2M pixels of 2N pixels in a vertical 2N line and horizontal is output. In the method of converting the resolution (improving the resolution), the signal of the pixel position (n, m) in the output of the color restoring unit 104 is set to the pixel position (2n, 2m) of the signal of 2N × 2M pixels, and is adjacent to this pixel. (2n, 2m + 1), (2n + 1, 2m), (2n +
The pixels of (1, 2m + 1) have the same value. FIG. 22 shows an example of each signal. In FIG.
, The signal R1 at the pixel position (n, m) is converted into a pixel (2n, 2m) with a signal of twice the resolution in the horizontal and vertical directions,
(2n, 2m + 1), (2n + 1, 2m), (2n +
1, 2m + 1). The same applies to other pixel positions and signals. Therefore, the four pixels in the upper, lower, left and right directions of the 2N × 2M pixel signal are determined by the pixel position (n,
m), R, G, 2N × 2M pixels
B signals can be obtained.

In the above description, the resolution conversion means 10
The method of increasing the resolution at 5 is a case where the same value is inserted into the four pixels in the upper, lower, left and right directions to make the pixels twice as large in the horizontal and vertical directions. It is also conceivable that the image is generated by interpolation using adjacent pixels in the vertical direction.

FIG. 23 shows another example of the structure of the color restoring means in a conventional image pickup apparatus using a single-plate image pickup device of a pixel mixing system disclosed in Japanese Patent Application Laid-Open No. 6-178307. FIG. 7 is a block diagram showing a case where a signal from an image sensor is generated by interpolation from three horizontal scanning lines, similarly to the above-described conventional example. In the figure, 113 is an image sensor, 11
Reference numeral 4 denotes a frame memory, 115 denotes a signal selection circuit, 116 denotes a color interpolation circuit, and 117 denotes an RGB matrix. The signal selection circuit 115 and the RGB matrix 117 constitute a color restoration unit. The image pickup device 113 shown in FIG.
, Four pixels A, B, C, and D (the numbers given to the respective pixel signals indicate pixel positions), and A, B are generated so that color signals can be generated by pixel mixture readout. Of pixels are alternately arranged for each line.

Next, the operation in FIG. 23 will be described. Each pixel signal is read out from the image sensor 113 as it is without performing pixel mixture readout and A / D converted (not shown), and then each pixel signal is taken into the frame memory 114. From the signals captured in the frame memory 114, the signal selection circuit 1
A signal of three adjacent vertical lines is selected by 15 and sent to the color interpolation circuit 116. The color interpolation circuit 116 interpolates and generates the color signals A, B, C, and D from the signals of the three vertical lines, and outputs the signals as RGB signals by the RGB matrix circuit 117.

Here, the color interpolation circuit 116 interpolates and generates each color signal. This interpolation method will be described. For example, in the color signal interpolation generation of the n2 line, the signal selection circuit 115 causes the three vertical lines n1, n2, n3
Are selected and sent to the color interpolation circuit 117. The color signals obtained in the n2 line are C and D pixels, and there are no A and B pixels. Therefore, the pixels A and B are interpolated based on the signals of the n1 and n3 lines in the vertical direction. However, since the positions of the pixels A and B are different in the n1 and n3 lines, it is necessary to change the interpolation coefficient in the horizontal direction. Become. Now, assuming that the color signals A ′, B ′, C ′, and D ′ are generated by interpolation from the five horizontal pixels before interpolation for the third pixel (n2, 3) of the n2 line of the color signal after interpolation. , C ', D' for interpolates generated weighted horizontally only _{center, C 23 '= (C 21} /2 + C 23 + C 25/2) / 2 D 23' = (D 22 + D 24) / 2. On the other hand, A ', B' for _{the, A 23 '= (A 11} /4 + A 13/2 + A 15/4) / 2 +
_{(A 31/4 + A 33} /2 + A 3 5/4) / 2 B 23 '= (B 12/2 + B 14/2) / 2 + (B 32/2 +
From the formula B _34/2 ) / 2, horizontal pixels can be weighted and interpolated.

Next, in the interpolation generation of the color signal of the n3 line, the pixels of A and B are generated by interpolation from the n3 line, and C, D
Pixels are generated by interpolation from lines n2 and n4. That is, in the pixel position (3,3), for _{example, A 33 '= (A 32} + A 34) / 2 B 33' = (B 31/2 + B 33 + B 35/2) / 2 C 33 '= (C 21 / _{4 + C 23/2 + C} 25/4) / 2 + (C
_{41/4 + C 43/2} + C 45/4) / 2 D 33 '= (D 22/2 + D 24/2) / 2 + (D 42/2 + D
_44/2 ) / 2.

In consideration of the fact that the pixel arrangements of A and B are switched for each line, the pixel positions (4,
3), A ₄₃ ′ = (A _31/4 + A _33/2 + A _35/4 ) / 2 + (A
_51/4 + _A53 / 2 + _A55 / 4) / 2 _B43 '= ( _B32 / 2 + _B34 / 2) / 2 + ( _B52 / 2 + B
_{54/2) / 2 C 43} '= (C 41/2 + C 43 + C 45/2) / 2 D 43' = (D 42 + D 44) / 2 In n5 line, to the pixel position (5,3), A ₅₃ '= ( _A52 + _A54 ) / 2 _B53 ' = ( _B52 + _B54 ) / 2 _C53 '= ( _C41 / 4 + _C43 / 2 + _C45 / 4) / 2 + (C
_61/4 + _C62 / 2 + _C65 / 4) / 2 _D53 '= ( _D42 / 2 + _D44 / 2) / 2 + ( _D62 / 2 + D
_64/2 ) / 2. Thereafter, the color signals A ′,
B ', C', and D 'will be generated.

The interpolation-generated A, B, C,
The D signal is output as an RGB signal by the RGB matrix circuit 117, sent to the resolution conversion means shown in FIG. 19, and converted into a signal having twice the resolution in the horizontal and vertical directions. In the above description, the four color signals are A, B, C,
Although described as D, for example, four colors of Mg (magenta), G (green), Cy (cyan), and Ye (yellow) can be considered. May be any coefficient that can generate a color signal by interpolation.

[0018]

In a conventional image pickup device, when a single-chip image pickup device restores a signal of the number of pixels of the image pickup device and increases the resolution to twice the resolution in the horizontal and vertical directions, the image pickup device moves up, down, left and right. The same value is put in four pixels to make a signal of four times the pixel, or a signal of four times the pixel is obtained by interpolation with adjacent pixels, and the resolution is twice as high in the horizontal and vertical directions. Does not take into account changes in the signal in local areas such as edges in the image of
There is a problem that false contours are generated, and there is a problem that improvement in resolution in both horizontal and vertical directions in all signals is not considered.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and has two image sensors which are arranged to be shifted by half a pixel in an oblique direction, and which is read by a first and a second image sensor. A color signal restoring unit for restoring and generating a color signal having a resolution corresponding to the number of pixels of the image sensor for each image sensor from the output color signals; and the first and second outputs, which are outputs from the color signal restoring unit. Means for obtaining a pixel of a signal having a resolution of twice the number of pixels of each image sensor in each of the horizontal and vertical directions from each signal restored from a signal from the image sensor, and a signal having a resolution of twice each of the obtained horizontal and vertical directions. And a first edge determining means for determining an edge component of a peripheral pixel at a predetermined pixel position at the predetermined pixel position. Based on a signal from the first edge determining means, a predetermined signal having a resolution of twice each in the horizontal and vertical directions is provided. Pixel position signal , A high-resolution image with twice the resolution in the horizontal and vertical directions with reduced false colors and false contours can be obtained, and the resolution in the horizontal and vertical directions can be improved. The purpose is to gain.

[0020]

An image pickup apparatus according to the present invention comprises an image pickup apparatus having two image pickup elements which are displaced by a half pixel in an oblique direction, and is read by the first and second image pickup elements. A color signal restoring unit for restoring and generating a color signal having a resolution corresponding to the number of pixels of the image pickup device for each image pickup device from the obtained color signal, and the first and second imagings which are outputs from the color signal restoring unit Means for obtaining a pixel of a signal having a resolution twice as large as the number of pixels of each image sensor in the horizontal and vertical directions from each signal restored from the signal from the element, and a signal having a resolution twice as large as the obtained horizontal and vertical directions. A first edge determination means for determining an edge component of a peripheral pixel at a predetermined pixel position, and a predetermined pixel in a signal having a resolution of twice each in the horizontal and vertical directions based on a signal from the first edge determination means. Position information And has a means for calculating.

Further, in the image pickup apparatus according to the present invention, the first edge determination means detects an edge component in a horizontal direction by calculating an absolute value of a difference between right and left adjacent pixels in a predetermined pixel in a predetermined signal. Means for calculating the absolute value of the difference between the upper and lower pixels to detect a vertical edge component; and means for detecting the horizontal edge component and the means for detecting the vertical edge component. Based on this, the horizontal or vertical edge component of the predetermined pixel is determined.

In the imaging apparatus according to the present invention, the edge component determination in the first edge determination means detects an output from the horizontal edge component detection means or the vertical edge component detection. If the output from the means is larger than a predetermined value, it is determined that an edge component has been detected in a peripheral pixel of the predetermined pixel. If the output of the detecting means is larger than the output of the means for detecting the vertical edge component, it is determined that there is a correlation in the horizontal direction. In addition, the outputs of the means for detecting the horizontal edge component and the means for detecting the vertical edge component are both smaller than a predetermined value. When have are those determined not to detect the edge components.

In the imaging apparatus according to the present invention, the means for calculating a signal at a predetermined pixel position in the signal having a resolution twice as high in the horizontal and vertical directions may include: Vertical direction calculating means for calculating a signal from the average value, horizontal direction calculating means for calculating a signal from the average value of the left and right pixels, and average value calculating means for calculating a signal from the average value of the upper, lower, left and right adjacent pixels are provided. Based on the output of the first edge determination means, select from the output of the horizontal direction calculation means or the output of the vertical direction calculation means, or the output from the average value calculation means, obtain a signal at the predetermined pixel, This is to obtain each signal having a resolution twice as large as the number of pixels in the image sensor in the horizontal and vertical directions.

Further, in the imaging apparatus according to the present invention, the means for calculating the signal at a predetermined pixel position in the signal having a resolution twice as high in the horizontal and vertical directions may include an output of the first edge determination means at the predetermined pixel position. If it is determined that no edge component is detected, the output of the average value calculating means is selected, and if it is determined that there is a correlation in the vertical direction, the output of the vertical direction calculating means is selected, and there is a correlation in the horizontal direction. If the judgment is made, the output of the horizontal direction calculation means is selected.

In the image pickup apparatus according to the present invention, in each of the two image pickup elements which are displaced by a half pixel in the oblique direction, each of the image pickup elements has a vertical two-row horizontal two-column upper and lower four pixels. A first color filter having a spectral sensitivity characteristic for one color signal and a second color filter having a spectral sensitivity characteristic for a second color signal are arranged in the first vertical line, and the second vertical line in the second vertical line. A third color filter having a spectral sensitivity characteristic for the third color signal is arranged in the same column as the pixel position where the first color filter in the first row is arranged, and is arranged in the same column as the second color filter. A first color filter having a spectral sensitivity characteristic with respect to a first color signal is arranged, and the upper and lower four-pixel color filters are sequentially and vertically and horizontally repeatedly arranged. Imaging for color signals The color signal restoring means for restoring and generating a color signal having a resolution corresponding to the number of child pixels is provided based on a peripheral pixel signal of the first color signal at a predetermined pixel position by the first color filter. A second edge determining means for determining a component, and first, second and third color filters read out by the first, second and third color filters based on an output of the second edge determining means. A first calculating means for calculating a signal at the predetermined position in the first color signal based on the color signal; and an output from the first calculating means and a color filter based on an output from the second edge determining means. A second calculating means for calculating the second and third color signals from the second and third color signals to obtain first, second and third color signals of the number of pixels in the image sensor. , The first edge determining means is With respect to the signal output from the means for obtaining the first pixel of the horizontal and vertical directions each 2 times the resolution of the signal by the color signal, and determines the edge component of the surrounding pixels at a given pixel position.

Further, in the image pickup apparatus according to the present invention, the second edge determining means in the color restoring means calculates an absolute value of a difference between left and right adjacent pixels at a predetermined pixel position of the first color signal, and performs horizontal scanning. Horizontal edge detection means for detecting an edge component in the direction, and vertical edge detection means for calculating an absolute value of a difference between upper and lower pixels at a predetermined pixel position of the first color signal to detect a vertical edge component. Determining means for determining a horizontal or vertical edge component of the predetermined pixel based on outputs from the horizontal edge detecting means and the vertical edge detecting means, wherein the determining means comprises the horizontal edge detecting means. If the output from the vertical edge detecting means is larger than a predetermined value, it is determined that an edge component has been detected in the peripheral pixels of the predetermined pixel. Further, when the output from the horizontal edge detecting means is larger than the output from the vertical edge detecting means, there is a correlation in the vertical direction, and the output from the horizontal edge detecting means is smaller than the output from the vertical edge detecting means. In this case, it is determined that there is a correlation in the horizontal direction, and if both the outputs from the horizontal edge detection means and the vertical edge detection means are smaller than a predetermined value, it is determined that no edge component is detected. .

Further, in the image pickup apparatus according to the present invention, the first calculating means for calculating the signal for the predetermined position in the first color signal in the color restoring means includes a predetermined pixel l of the second color signal B. At the position of the row m column B (l, m), the values Ahlpf (l, m), B of the first color signal A and the second color signal B are respectively passed through the horizontal low-pass filter.
hlpf (l, m) is calculated, and Ahlpf (l, m), which is an output signal from the horizontal low-pass filter, and B
hlpf (l, m) and the pixel value B at the above pixel position
By (l, m), the pixel value A (l, m) in the first color signal A in l rows and m columns is represented by A (l, m) = B (l, m) ×
A horizontal signal that is calculated by {Ahlpf (l, m) / Bhlpf (l, m)} and similarly calculates a pixel value of the first color signal A at some other pixel position of the third color signal C Calculating means, and the predetermined pixel l row m column B (l,
m), the first color signal A and the second color signal B
, Avlpf (l, m) and Bvlpf (l, m) are calculated via a vertical low-pass filter, and the output signals from the vertical low-pass filter are Avlpf (l, m) and Bvlpf (l, m). The pixel value A (l, m) in the first color signal A of l rows and m columns is calculated by the ratio of A (l, m) to the pixel value B (l, m) at the pixel position.
(L, m) = B (l, m) × ｛Avlpf (l, m) /
Bvlpf (l, m)} to calculate the third color signal C
A vertical direction signal calculating means for calculating a pixel value of the first color signal A at another pixel position having
Average value for calculating the pixel value A (l, m) in the first color signal A of l rows and m columns from the average value of the upper, lower, left and right adjacent pixels at the position of the predetermined pixel l rows and m columns in the color signal A Calculating means, based on the output of the second edge determining means, selecting from the output of the horizontal direction signal calculating means or the output of the vertical direction signal calculating means, or the output from the average value calculating means, A pixel value A (l, m) of the first color signal A in l rows and m columns is obtained, and a first color signal of the number of pixels in the image sensor is obtained.

Further, in the image pickup apparatus according to the present invention, the first calculating means in the color restoring means may detect that the output of the second edge determining means does not detect an edge component at a position of a predetermined pixel at l rows and m columns. If it is determined, the output of the average value calculation means is selected.If it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculation means is selected.If it is determined that there is a correlation in the horizontal direction, The output of the horizontal direction signal calculation means is selected to obtain a first color signal of the number of pixels in the image sensor.

Further, in the image pickup apparatus according to the present invention, the second calculating means in the color restoring means is arranged so that the second calculating means in the horizontal direction with respect to the output A from the first calculating means at the position of predetermined pixel l row and m column. Value A1hlpf through low-pass filter
(L, m) and the value A1 through the vertical low-pass filter
vlpf (l, m) is calculated, and a value B1hlpf is applied to the second color signal B via a horizontal low-pass filter.
The value C1vlpf (l, m) (or, through a low-pass filter in the vertical direction) for (l, m) and the third color signal C (or
A value B1vlpf (l, m) through a low-pass filter in the vertical direction for the second color signal B and a value C1hlp through a low-pass filter in the horizontal direction for the third color signal C
f (l, m)) and the ratio of A1hlpf (l, m) to B1hlpf (l, m) (or C1hlpf).
(L, m)), A1vlpf (l, m) and C1
vlpf (l, m) (or B1vlpf)
(Ratio to (l, m)) and the pixel value A (l, m) at l rows and m columns of the predetermined pixel in the output A from the first calculating means, the 1st row and m columns of the second color signal The pixel values B (l, m) and C (l, m) of B and the third color signal C are represented by B (l, m)
= A (l, m) × ｛B1hlpf (l, m) / A1hl
pf (l, m)}, C (l, m) = A (l, m) x {C
1vlpf (l, m) / A1vlpf (l, m)｝ (or B (l, m) = A (l, m) × ｛B1vlpf
(L, m) / A1vlpf (l, m)｝, C (l, m)
= A (l, m) × ｛C1hlpf (l, m) / A1hl
pf (l, m)｝), and a predetermined pixel x different from the position of the l row and m column.
At the position of the row y column, the value A2h obtained through the horizontal low-pass filter with respect to the output A from the first calculating means.
Ipf (x, y), a value B2hlpf (x, y) of the second color signal B output from the signal calculating means through a horizontal low-pass filter is calculated, and A2hlpf is calculated.
The ratio of (x, y) to B2hlpf (x, y) and the pixel value A at pixel x row y column at the output A from the first calculating means
According to (x, y), the second color signal B (x, y) at the position of x row and y column is represented by B (x, y) = A (x, y) × ｛B
2hlpf (x, y) / A2hlpf (x, y)}, and the horizontal signal calculating means for calculating the C signal for the third color signal C, and the output A from the first calculating means. A2vlpf (x, y) through a low-pass filter in the vertical direction, and B2vlpf (x, y) through a low-pass filter in the vertical direction with respect to the second color signal B at the output from the signal calculation means. Is calculated, and A2v
By the ratio of lpf (x, y) to B2vlpf (x, y) and the pixel value A (x, y) at the pixel x row y column at the output A from the first calculating means, x row y column The second color signal B (x, y) at the position is represented by B (x, y) = A (x, y) ×
{B2vlpf (x, y) / A2vlpf (x, y)}
Vertical signal calculating means for calculating the C signal in the same manner also for the third color signal C, and a predetermined pixel x in the second and third color signals in the output from the signal calculating means.
Average value calculating means for calculating an average value of obliquely adjacent pixels at the position of the row y column, wherein the horizontal direction signal calculating means and the vertical direction signal calculating means are based on the output of the second edge determining means. , The second and third color signals at the predetermined pixel x row and y column are selected from the respective outputs from the average value calculating means, and the second and third color signals of the number of pixels in the image sensor are obtained. Is what you get.

Further, in the image pickup apparatus according to the present invention, the second calculating means in the color restoring means may detect that the output of the second edge determining means does not detect an edge component at a position of a predetermined pixel x row and y column. If it is determined, the output of the average value calculation means is selected.If it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculation means is selected.If it is determined that there is a correlation in the horizontal direction, The output of the horizontal direction signal calculation means is selected to obtain second and third color signals of the number of pixels in the image sensor.

[0031]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be specifically described with reference to the drawings showing the embodiments. Embodiment 1 FIG. FIG. 1 is a block diagram illustrating an example of a configuration of an imaging device according to an embodiment of the present invention. In the figure, reference numeral 1 denotes an image sensor for capturing signals of two image sensors in which pixels are shifted by half a pixel in an oblique direction, 2 denotes an A / D converter,
Reference numerals 3 and 4 denote frame memories, and reference numerals 5 and 6 denote first and second colors for restoring color signals so as to obtain R, G, and B signals of the total number of pixels in one image sensor in the image sensor 1. The restoration means 7 is a multiplexing means for multiplexing the signals from the first and second color restoration means 5 and 6 to obtain a signal having twice the number of pixels in each of the horizontal and vertical directions. Reference numeral 8 denotes an edge judging means for judging an edge component at a predetermined pixel in the G signal output from the multiplexing means 7; Horizontal direction calculating means for calculating an average value of left and right adjacent pixels at predetermined pixels in each signal from the multiplexing means;
Numeral 11 denotes a vertical direction calculating means for calculating an average value of upper and lower pixels at predetermined pixels in each signal from the multiplexing means 7, and 12 denotes a selecting means. Calculation means 9, horizontal direction calculation means 1
0, the signal is switched and selected from the output of each of the vertical operation means 11 and the multiplexing means 7, and R, which has a resolution of twice the number of pixels in the image sensor in the horizontal and vertical directions, respectively.
G and B signals are output.

Here, in order to increase the resolution, in the image pickup device 1, signals are taken in by arranging two image pickup devices with pixels shifted by half a pixel in an oblique direction. FIG. 2 is a diagram showing a case where pixels according to the first embodiment of the present invention are shifted by half a pixel in an oblique direction.
FIG. 5 is a diagram illustrating an example of the arrangement of the image pickup elements, and includes data D1.
And a signal D2, which is obtained by shifting the data D1 by a half pixel toward the lower left direction. FIG. 3 is a diagram showing an example of an array of color filters of each image sensor according to Embodiment 1 of the present invention, and shows an array of color filters in each image sensor based on the two data D1 and data D2. . The color filter shows an image sensor of a system using primary colors and calling each photoelectric conversion element independently. In FIG.
G is vertical direction 2i (i = 0, 1, 2,...), Horizontal direction 2i
, j (j = 0, 1, 2,...) (hereinafter, referred to as pixel position (2i, 2j)) and pixel position (2i +
1, 2j + 1), a first color filter having a spectral characteristic to pass a G signal, and R is located at a pixel position (2i, 2j +
2) having a spectral characteristic for passing the R signal
Is a third color filter at a pixel position (2i + 1, 2j), which has a spectral characteristic for passing the B signal. As shown in FIG. 3, the R and B signals are obtained every four pixels in the upper and lower portions (hatched portions in the figure), and the G signals are obtained every two pixels. These four upper and lower pixels are repeatedly arranged in the vertical and horizontal directions. ing.

Next, the operation will be described. In the image sensor 1, pixel signals R, G, and B serving as data D1 and data D2 are read from two image sensors in which pixels are shifted by half a pixel in an oblique direction as shown in FIG.
A / D conversion is performed by the D converter 2 and the first and second
Are input to the frame memories 3 and 4. Here, the signals of the image sensors arranged as shown in FIG. 3 based on the data D1 in FIG. 2 are input to the first frame memory 3,
The signals of the imaging elements arranged as shown in FIG. 3 based on the data D2 in FIG. 2 are input to the second frame memory 4. The first color restoration unit 5 restores a pixel signal that is not obtained by separating each of the R, G, and B signals from the signal of the data D1 input to the first frame memory 3 and obtains the number of pixels of the image sensor. , And outputs the R, G, and B signals R1, G1, and B1, respectively.
The pixel signals that are not obtained by separating the R, G, and B signals from the signal of the data D2 input to are output, and the R, G, and B signals R2, G2, and B2 of the number of pixels of the image sensor are output. Then, each output is sent to the multiplexing means 7.

The operation of the first and second color restoration means 5 and 6 will now be described based on the structure of the color restoration means shown in FIG. FIG. 4 is a block diagram showing an example of the configuration of a color restoring unit in the imaging apparatus according to the first embodiment of the present invention. In FIG. 21 is a first for determining an edge component at a predetermined pixel position in the G signal.
The edge determination means 22 of each R, R from the separation means 20,
G component restoration means for restoring the G signal based on the G and B signals and the output from the first edge determination means 21; 23, first RB component restoration means for restoring the R and B signals;
Is a second edge determining means for determining an edge component at a predetermined pixel position in the G signal from the G component restoring means 22;
Reference numeral 5 denotes second RB component restoration means for restoring the R and B signals in accordance with the output of the second edge determination means 24.

The signals from the image sensors of the arrangement shown in FIG. 3 in the frame memory 3 or 4 are separated into R, G and B signals by a separation means 20, respectively. The R and B signals are sent to the G component restoring unit 22 and the first RB
It is sent to the restoration means 23. First edge determination means 21
Then, the edge component at a predetermined pixel position in the G signal is determined, and the determination result is sent to the G component restoring means 22. The pixel signal of the G component is restored so as to obtain the G signal of the total number of pixels in the above. FIG. 5 is a flowchart for explaining the operation of the first edge determining means 21 and the G component restoring means 22 in the color restoring means in the imaging apparatus according to the first embodiment of the present invention. The processing operation of the G component restoring means 22 will be described with reference to FIG.

Now, as shown in FIG. 3, the G signal in the image sensor 1 has pixel positions (2i, 2j) and (2i + 1, 2j).
+1), and in order to obtain a G signal of the number of pixels of the image sensor, pixel positions (2i, 2j + 1) and (2i + 1,
The signal at the pixel 2j) is obtained. Therefore, according to FIG. 5, the pixel position (2i,
The pixels at (2j) and (2i + 1, 2j + 1) output the G signal as they are, and the pixel positions (2i, 2j + 1) and (2i +
In (1, 2j), first, the first edge determination means 21 detects a difference between left and right and upper and lower pixels, that is, an edge component. That is, the absolute value ΔH of the difference between the left and right pixels and the absolute value ΔV of the difference between the upper and lower pixels at the pixel position are calculated. For example, at the pixel position (2i, 2j + 1), the horizontal difference ΔH is ΔH = | G (2i, 2j) −G (2i, 2j + 2) | (1) The vertical difference ΔV is ΔV = | G (2i −1, 2j + 1) −G (2i + 1, 2j + 1) | (2) Hereinafter, the absolute value of the difference between the pixels is referred to as an edge component.

Then, the edge component ΔH in the horizontal direction is obtained.
The change of the signal level in the peripheral pixels in the horizontal and vertical directions is determined based on the vertical edge component ΔV and the signal ed1 indicating the determination result is output from the first edge determination unit 21. When the first edge determination unit 21 determines that both ΔH and ΔV are equal to or smaller than a predetermined value th, it is determined that there is no change in the signal level in the surrounding pixels, and the G component restoration unit 22 determines Does not need to be considered, and the average value of the four pixels at the top, bottom, left and right is calculated and used as the G signal. For example, the G signal at the pixel position (2i, 2j + 1) in this case is g (2i, 2j + 1) = ｛G (2i-1, 2g).
j + 1) + G (2i + 1,2j + 1) + G (2i, 2
j) + G (2i, 2j + 2)｝ / 4.

On the other hand, ΔH or ΔV is a predetermined value th.
If it is larger, it is determined that there is an edge component in that pixel. Further, if ΔH> ΔV, it is determined that the correlation is high in the vertical direction, and if ΔH ≦ ΔV, it is determined that the correlation is high in the horizontal direction. When it is determined that the correlation is high in the vertical direction when ΔH> ΔV, the G component restoring means 22 calculates from the vertical pixel signals of the R, G, and B signals to obtain a G having a vertical correlation. Output a signal. That is,
The processing pixel position (2i,
2j + 1), the value Gvlpf of the G signal through the vertical low-pass filter and the value Rvlpf of the R signal through the vertical low-pass filter are, for example, Gvlpf = ｛G (2i−3, 2j + 1) + G (2i−). 1,2j + 1) + G (2i + 1,2j + 1) + G (2i + 3,2j + 1)｝ / 4 (3) Rvlpf = {R (2i-4,2j + 1) + 2 × R (2i-2,2j + 1) + 2 × R (2i, 2j + 1) ) + 2 × R (2i + 2, 2j + 1) + R (2i + 4, 2j + 1)｝ / 8 (4), and the correlation in the vertical direction is obtained from the ratio of Gvlpf to Rvlpf and the R signal at the pixel position (2i, 2j + 1). The G signal g (2i, 2j + 1) is calculated by the following equation: g (2i, 2j + 1) = R (2i, 2j + 1) × (Gvlpf / Rvlpf) (5) .

Similarly, at the processing pixel position (2i + 1, 2j) where the B signal is obtained, the value Gvlpf through the vertical low-pass filter of the G signal, and the value Gvlpf through the vertical low-pass filter of the B signal. Value Bvlpf
For example, Gvlpf = {G (2i-2, 2j) + G (2i, 2j) + G (2i + 2, 2j) + G (2i + 4, 2j)} / 4 (6) Bvlpf = {B (2i-3, 2j) + 2 × B (2i−1, 2j) + 2 × B (2i + 1, 2j) + 2 × B (2i + 3, 2j) + R (2i + 5, 2j)｝ / 8 (7), and Gvlpf and Bvlpf From the ratio and the B signal at the pixel position (2i + 1, 2j), a G signal g (2i + 1, 2j) having a vertical correlation is expressed by the following equation: g (2i + 1, 2j) = B (2i + 1, 2j) × (Gvlpf / Bvlpf) ) Calculated by (8).

When it is determined that the correlation in the horizontal direction is high when ΔH ≦ ΔV, the G component restoring means 22
A G signal having a correlation in the horizontal direction is output by calculating from the pixel signals in the horizontal direction. At the processing pixel position (2i, 2j + 1), the value Ghlpf of the G signal through the horizontal low-pass filter and the value Rhlpf of the R signal through the horizontal low-pass filter are, for example, Ghlpf = 例 え ば G (2i, 2j). + G (2i, 2j-2) + G (2i, 2j + 2) + G (2i, 2j + 4)｝ / 4 (9) Rhlpf = {R (2i, 2j-3) + 2 × R (2i, 2j-1) + 2 × R (2i, 2j + 1) + 2 × R (2i, 2j + 3) + R (2i, 2j + 5)｝ / 8 (10) , And a G signal g (2i, 2j + 1) having a correlation in the vertical direction is calculated by the following equation: g (2i, 2j + 1) = R (2i, 2j + 1) × (Ghlpf / Rhlpf) (11)

Similarly, at the processing pixel position (2i + 1, 2j) at which the B signal is obtained, the value Ghlpf of the G signal at the processing pixel position (2i + 1, 2j) through the horizontal low-pass filter is obtained. Ghlpf = {G (2i + 1, 2j-1) + G (2i, 2j-3) + G (2i, 2j + 1) + G (2i, 2j + 3)} / 4 (12) Bhlpf = ｛B (2i + 1, 2j-4) + 2 × B (2i + 1, 2j-2) + 2 × B (2i + 1, 2j) + 2 × B (2i + 1, 2j + 2) + B (2i + 1, 2j + 4) From the ratio of Ghlpf to Bhlpf and the B signal at the pixel position (2i + 1, 2j), a G signal g (2i + 1, 2j) having a vertical correlation is calculated from｝ / 8 (13). Equation g (2i + 1,2j) = calculated by B (2i + 1,2j) × (Ghlpf / Bhlpf) (14).

The above equations (5), (8), (11),
The calculation method according to (14) is based on the premise that there is little change in color in the local region. That is, since the ratio of each color signal in the local region is substantially equal, the local method in the horizontal or vertical direction is obtained. The ratio of each color signal in a specific region is given by the ratio of the values of R, G, and B through a horizontal or vertical low-pass filter. Further, the above formulas (5) to
(14) is a calculation example of each horizontal low-pass filter and vertical low-pass filter output. The number of taps and coefficients of the filter are not limited to the above, and other tap numbers and coefficients may be used.

Therefore, the pixel positions (2i, 2j), (2i, 2j + 1), (2i +
A G signal is output at each pixel of (1, 2j) and (2i + 1, 2j + 1), that is, a G signal having a resolution equal to the number of pixels of the image sensor is obtained. Next, the output from the G component restoring unit 22 is sent to the first RB component restoring unit 23 and the second edge determining unit 24.

Next, in the first RB component restoring means 23,
R and B signal pixel positions (2i, 2j), (2i + 1, 2)
j + 1) are generated, and the output is output to the second
To the RB restoration means 25 of the second edge determination means 2
In (4), the pixel positions (2i, 2j + 1) and (2i) are obtained by the G signal output from the G component restoring means 22.
+1 and 2j), the difference between the left and right and upper and lower pixels, that is, the edge component is detected. Second RB component restoration means 2
5, R is determined based on the output of the second edge determination unit 24.
The pixel at the pixel position (2i + 1, 2j) in the signal and the pixel at the pixel position (2i, 2j + 1) in the B signal are restored and generated. FIG. 6 shows the first RB component restoring means 23, the second edge determining means 24 and the second RB component restoring means 25.
6 is a flowchart showing the operation of the first embodiment, and the processing operations of the first and second RB component restoring units 23 and 25 and the second edge determining unit 24 will be described with reference to FIG.

Now, as shown in FIG. 3, the R signal in the image sensor is obtained at the pixel position (2i, 2j + 1) and the B signal is obtained at (2i + 1, 2j). , B signal, the pixel positions (2i, 2j), (2i + 1, 2j) and (2i + 1, 2j) for R are obtained.
The signal at the pixel at (j + 1) is determined, and for B, the signal at the pixel at pixel positions (2i, 2j), (2i, 2j + 1) and (2i + 1, 2j + 1) is determined. FIGS. 7A and 7B are diagrams illustrating R and B of each pixel for explaining calculation of R and B signals, where R and B indicate pixel signals obtained by the image sensor. ing. According to FIG. 6, the first and second RB component restoring means 2
At positions 3 and 25, the pixel position (2i, 2j +
1) The pixel in the pixel signal (2i + 1, 2j) of the B signal outputs a signal as it is. Next, the pixel position (2
In the case of (i, 2j) and (2i + 1, 2j + 1), pixels in either the left or right or up and down directions are obtained for both the R and B signals. I have. Therefore, the first RB component restoration unit 23 receives the R and B signals from the separation unit 20 and the G signal obtained by restoring the signals of all the pixels from the G component restoration unit 22. The operation is performed to obtain the R and B signals of the pixels (2i, 2j) and (2i + 1, 2j + 1). That is, r1 and b1 (pixel positions (2i, 2j)) and r2 and b2 (pixel positions (2i +
The pixel in (1, 2j + 1)) is calculated from pixels in the vertical and horizontal directions of the processing pixel in the G, R, and B signals.

The pixel position (2i, 2i,
j), G, R at the pixel position (2i, 2j)
The value G of the signal through a horizontal low-pass filter
For example, 1hlpf and R1hlpf are expressed as follows: G1hlpf = (g (2i, 2j-1) + g (2i, 2j + 1)) / 2 (15) R1hlpf = (R (2i, 2j-1) + R (2i, 2j + 1)) / 2 ( 16) and the ratio of G1hlpf to R1hlpf and the pixel G (2i, 2j) determine the pixel position (2i, 2j).
Is calculated by the following equation. r (2i, 2j) = G (2i, 2j) × (R1hlpf / G1hlpf) = G (2i, 2j) × (R (2i, 2j−1) + R (2i, 2j + 1)) (g (2i, 2j−)) 1) + g (2i, 2j + 1)) (17)

Similarly, as for the position of b1 in the B signal, a value G1vl obtained through a vertical low-pass filter with respect to the G and B signals at the pixel position (2i, 2j).
pf and B1vlpf are, for example, G1vlpf = (g (2i-1, 2j) + g (2i + 1, 2j)) / 2 (18) B1vlpf = (B (2i-1, 2j) + B (2i + 1, 2j)) / 2 ( 19) and the ratio of G1vlpf to B1vlpf and the pixel G (2i, 2j) are used to calculate the pixel position (2i, 2j).
Is calculated according to the following equation. b (2i, 2j) = G (2i, 2j) × (B1vlpf / G1vlpf) = G (2i, 2j) × (B (2i−1, 2j) + B (2i + 1, 2j)) / (g (2i−1) , 2j) + g (2i + 1, 2j)) (20)

Further, a pixel position (2i + 1, 2
j + 1), the pixel position (2i +
1, 2j + 1), the values G1vlpf, R1vl of the G and R signals through a low-pass filter in the vertical direction.
pf is calculated by, for example, G1vlpf = (g (2i, 2j + 1) + g (2i + 2, 2j + 1)) / 2 (21) R1vlpf = (R (2i, 2j + 1) + R (2i + 2, 2j + 1)) / 2 (22) By the ratio of G1vlpf to R1vlpf and the pixel G (2i + 1, 2j + 1), the pixel position (2i
R signal r (2i + 1, 2j + 1) at (+1, 2j + 1)
Is calculated by the following equation. r (2i + 1, 2j + 1) = G (2i + 1, 2j + 1) × (R1vlppf / G1vlpf) = G (2i + 1, 2j + 1) × (R (2i, 2j + 1) + R (2i + 2, 2j + 1)) / (g (2i, 2j + 1)) + G (2i + 2,2j + 1)) (23)

Similarly, at the pixel position (2i + 1, 2j + 1) indicated by b2 in the B signal, the values G1hlpf, G1hlpf,
B1hlpf is calculated by, for example, G1hlpf = (g (2i + 1, 2j) + g (2i + 1, 2j + 2)) / 2 (24) B1hlpf = (B (2i + 1, 2j) + B (2i + 1, 2j + 2)) / 2 (25) The pixel position (2i
B signal b (2i + 1, 2j + 1) at (+1, 2j + 1)
Is calculated by the following equation. b (2i + 1, 2j + 1) = G (2i + 1, 2j + 1) × (B1hlpf / G1 hlpf) = G (2i + 1, 2j + 1) × (B (2i + 1, 2j) + B (2i + 1, 2j + 2)) / (g (2i + 1, 2j) ) + G (2i + 1, 2j + 2)) (26)

The above equations (17), (20) and (2)
3) and (26) are based on the premise that the change of the color signal in the local region is small similarly to the restoration method in G, that is, the ratio of each signal is almost equal in the local region. G1h in the equations (15) to (26)
lpf, G1vlpf, R1hlpf, R1vlpf,
The formulas for calculating B1hlpf and B1vlpf are calculation examples of low-pass filter outputs in the horizontal and vertical directions, and the number of taps and coefficients of the filter are not limited to the above, and other tap numbers and coefficients may be used.

Next, the pixel position (2
i + 1, 2j) and the pixel position (2i, 2j +
For the signals in 1) (r3 and b3 in FIG. 7),
First, the second edge determination means 24 detects a difference between left and right and upper and lower pixels in the G signal, that is, an edge component. That is, for the G signal from the G component restoring means 22, the absolute value ΔH of the difference between the left and right pixels and the absolute value ΔV of the difference between the upper and lower pixels at the pixel position are calculated. For example, at the pixel position (2i + 1, 2j), the horizontal difference ΔH is ΔH = | G (2i + 1, 2j−1) −G (2i + 1, 2
j + 1) | vertical difference ΔV is as follows: ΔV = | G (2i, 2j) -G (2i + 2, 2j) |

Then, the edge component ΔH in the horizontal direction is obtained.
The change in the signal level in the peripheral pixels in the horizontal and vertical directions is determined based on the vertical edge component ΔV, and a signal indicating the determination result is output to the second RB component restoration unit 25. , R signal (2i
+1 and 2j), the result edr of the edge component at the pixel position (2i + 1, 2j) is sent, and the pixel position (2i, 2j + 1) in the B signal is transmitted at the pixel position (2i, 2j + 1). Result ed of judging edge component
b. If both ΔH and ΔV are equal to or less than the predetermined value th in the determination results edr and edb, it is determined that there is no change in the signal level in the peripheral pixels, and the second
In the RB component restoring means 25, there is no need to consider a change in frequency, and an average value of four pixels adjacent in an oblique direction is calculated to obtain an R or B signal. That is, for the pixel position (2i + 1, 2j) in the R signal (r3 in FIG. 7A), r (2i + 1, 2j) = (R (2i, 2j-1) + R
(2i, 2j + 1) + R (2i + 2, 2j-1) + R
(2i + 2, 2j + 1)｝ / 4 For the pixel position (2i, 2j + 1) in the B signal (b3 in FIG. 7B), b (2i, 2j + 1) = ｛B (2i−1, 2j) + B
(2i-1, 2j + 2) + B (2i + 1, 2j) + B
(2i + 1, 2j + 2)｝ / 4.

On the other hand, ΔH or ΔV is a predetermined value th.
If it is larger, it is determined that there is an edge component in that pixel. Further, if ΔH> ΔV, it is determined that the correlation is high in the vertical direction, and if ΔH ≦ ΔV, it is determined that the correlation is high in the horizontal direction. If it is determined that the correlation is high in the vertical direction when ΔH> ΔV, the second RB component restoration unit 2
In 5, the arithmetic operation is performed from the pixels in the vertical direction in the R, G, and B signals, and a signal having a correlation in the vertical direction is output. At the pixel position (2i + 1,2j) indicated by r3 in the R signal,
The value R2vlpf through a vertical low-pass filter,
G2vlpf is calculated by, for example, R2vlpf = (r (2i, 2j) + r (2i + 2, 2j)) / 2 (27) G2vlpf = (G (2i, 2j) + G (2i + 2, 2j)) / 2 (28) From the ratio of R2vlpf to G2vlpf and pixel g (2i + 1,2j), the pixel value r (2i + 1,2j) of R having a correlation in the vertical direction is calculated by the following equation. r (2i + 1, 2j) = g (2i + 1, 2j) × (R2vlpf / G2vlpf) = g (2i + 1, 2j) × (r (2i, 2j) + r (2i + 2, 2j)) / (G (2i, 2j) + G (2i + 2, 2j)) (29)

Similarly, at the pixel position (2i, 2j + 1) at the pixel position indicated by b3 in the B signal,
The value B2vlpf through a vertical low-pass filter,
Let G3vlpf be, for example, B2vlpf = (b (2i-1, 2j + 1) + b (2i + 1, 2j + 1)) / 2 (30) G3vlpf = (G (2i-1, 2j + 1) + G (2i + 1, 2j + 1)) / 2 (31) Using the ratio of B2vlpf and G3vlpf and the pixel g (2i, 2j + 1), the pixel value b (2i, 2j + 1) having a vertical correlation is calculated by the following equation. b (2i, 2j + 1) = g (2i, 2j + 1) × (B2vlpf / G3vlpf) = g (2i, 2j + 1) × (b (2i−1, 2j + 1) + b (2i + 1, 2j + 1)) / (G (2i− 1, 2j + 1) + G (2i + 1, 2j + 1)) (32)

If it is determined that the correlation is high in the horizontal direction when ΔH ≦ ΔV, the second RB component restoring means 25 calculates from the left-right pixel signals of the R, G, and B signals, Outputs a signal having horizontal correlation. At the pixel position (2i + 1,2j) indicated by r3 in the R signal,
The value R2hlpf through a horizontal low-pass filter,
G2hlpf is, for example, R2hlpf = (r (2i + 1, 2j-1) + r (2i + 1, 2j + 1)) / 2 (33) G2hlpf = (G (2i + 1, 2j-1) + G (2i + 1, 2j + 1)) / 2 (34) From the ratio of R2hlpf to G2hlpf and the pixel g (2i + 1,2j), a pixel value r (2i + 1,2j) of R having a correlation in the horizontal direction is calculated by the following equation. r (2i + 1, 2j) = g (2i + 1, 2j) × (R2hlpf / G2hlpf) = g (2i + 1, 2j) × (r (2i + 1, 2j−1) + r (2i + 1, 2j + 1)) / (G (2i + 1, 2j-1) + G (2i + 1, 2j + 1)) (35)

Similarly, at the pixel position indicated by b3 in the B signal, at the pixel position (2i, 2j + 1), the values B2hlpf, G3h passed through the horizontal low-pass filter.
lpf is calculated by, for example, B2hlpf = (b (2i, 2j) + b (2i, 2j + 2)) / 2 (36) G3hlpf = (G (2i, 2j) + G (2i, 2j + 2)) / 2 (37) From the ratio of B2hlpf to G3hlpf and the pixel g (2i, 2j + 1), a pixel value b (2i, 2j + 1) having a correlation in the horizontal direction is calculated by the following equation. b (2i, 2j + 1) = g (2i, 2j + 1) × (B2hlpf / G3hlpf) = g (2i, 2j + 1) × (b (2i, 2j) + b (2i, 2j + 2)) / (G (2i, 2j) + G (2i, 2j + 2)) (38)

The above equations (29), (32) and (3)
5) and (38) are based on the premise that the change of the color signal in the local region is small as in the above-described restoration method in G, that is, the ratio of each signal is almost equal in the local region. G2h in the equations (27) to (38)
lpf, G2vlpf, G3hlpf, G3vlpf,
R2hlpf, R2vlpf, B2hlpf, B2vl
The formula for calculating pf is an example of calculating the output of the low-pass filter in the horizontal and vertical directions, and the number of taps and coefficients of the filter are not limited to the above, and other tap numbers and coefficients may be used.

From the above, the pixel positions (2i, 2j), (2i, 2j + 1),
The R and B signals at each of the pixels (2i + 1, 2j) and (2i + 1, 2j + 1) are output, that is, R and B signals having a resolution equal to the number of pixels of the image sensor can be obtained. As for the G signal, the G component restoring means 22 has already obtained signals of the resolution corresponding to the number of pixels of the image sensor. Therefore, the edge component in the horizontal and vertical directions is determined using the G component, and the spatial frequency Is determined, and the generation of each color signal is switched based on the result of the determination. In the calculation of the signal when the edge component exceeds a predetermined value, the calculation is performed based on the ratio of the color signal in a local region, and the edge component is calculated. Is less than or equal to a predetermined value, by calculating the average value of peripheral pixels,
At the outputs of the first and second color restoration means 5 and 6, R, G, and B signals having resolutions corresponding to the number of pixels of the image sensor are obtained.

Next, the signals R1 and R2 corresponding to the number of pixels of the image sensor restored by the first color restoring means 5 by the method described above.
G1 and B1 are sent to the multiplexing means 7 and the signals R2,
G2 and B2 are also sent to the multiplexing means 7. The multiplexing means 7 converts the R1, G1, and B1 signals from the first color restoring means 5 and the R2, G2, and B2 signals from the second color restoring means 6 into signals each having twice the resolution in the horizontal and vertical directions. The respective signals are combined and multiplexed so as to achieve. That is, since each signal is a signal restored by the data D1 and D2 shifted by a half pixel in the oblique direction shown in FIG. 2, the above R1, G1, B1 and R
8, G2 and B2 are arranged as shown in FIG. 8 in a signal having a resolution of twice the number of pixels in each image sensor in the horizontal and vertical directions. That is, R1, G1, and B1 are located at pixel positions (2x, 2y + 1) (x and y are integers) in a vertical 2x row and horizontal 2y + 1 column in each signal having twice the resolution in the horizontal and vertical directions.
1 and R2, G at pixel position (2x + 1, 2y)
2 and B2 are located. In FIG. 8, D1 indicates the position of R1 or G1 or B1, and D2 is R2 or G2.
Or the position of B2 is shown. Therefore, the multiplexing means 7 outputs a signal having a resolution of twice the number of pixels arranged in the horizontal and vertical directions as shown in FIGS.

Next, R, G, B, which are the outputs of the multiplexing means 7,
Each signal (FIGS. 9 to 11) is sent to the average value calculation means 9, the horizontal direction calculation means 10, and the vertical direction calculation means 11, and the G signal is also input to the edge determination means 8. Here, FIG.
As shown in FIG. 11, the pixels at the pixel positions (2x, 2y) and (2x + 1, 2y + 1) are not obtained in each signal, and therefore, the signals at these pixels are converted into the average value calculating means 9, The calculation is performed by the horizontal calculation means 10 and the vertical calculation means 11. Then, the average value calculation means 9, the horizontal direction calculation means 10, the vertical direction calculation means 11
Is sent to the selection means 12, and the edge determination means 8 outputs the pixel positions (2x, 2y), (2x
(+1, 2y + 1), the edge components of the peripheral pixels are determined,
The result of the determination is output to the selection means 12.

FIG. 12 is a flowchart for explaining the operation of increasing the resolution in the image pickup apparatus according to the first embodiment of the present invention. Direction calculation means 11
And the operation of the selection means 12. A description will be given with reference to FIG.

The pixel positions (2x, 2y + 1), (2x + 1, 2y) at which the pixel signals shown in FIGS. 9 to 11 are obtained.
As for, the selection means 12 outputs a signal as it is. Pixel positions (2x, 2y), (2x + 1, 2y +
Regarding 1), first, the edge determination means 8 detects differences in the horizontal and vertical directions (that is, edge components) from the left and right and upper and lower pixels at the pixel position of the G signal.
That is, for the G signal from the multiplexing means 7, the absolute value ΔHg of the difference between the left and right pixels at the pixel position and the absolute value ΔVg of the difference between the upper and lower pixels are calculated. For example, the pixel position (2x, 2
In y), the horizontal difference ΔHg is ΔHg = | G (2x, 2y−1) −G (2x, 2y +
1) | The vertical difference ΔVg is ΔVg = | G (2x−1, 2y) −G (2x + 1, 2)
y) |

Based on the horizontal difference ΔHg and the vertical difference ΔVg, an edge of a peripheral pixel in the horizontal and vertical directions is determined, and a signal ed indicating the determination result is output to the selection means 12. In the judgment result ed, ΔHg
If both of .DELTA.Vg and .DELTA.Vg are equal to or less than a predetermined value thg, it is determined that there is no change in the signal level in the peripheral pixels, and the selecting means 1
In step 2, the signal from the average value calculating means 9 is selected. In the average value calculating means 9, the pixel position (2x,
2y) and (2x + 1, 2y + 1), the average value of the four pixels at the top, bottom, left and right is calculated.
In the case of y), the values Ra, G of the R, G, and B signals are given by the following equations.
a and Ba are calculated and output to the selection means 12. Ra (2x, 2y) = ｛R (2x-1, 2y) + R (2
x + 1, 2y) + R (2x, 2y-1) + R (2x, 2
y + 1)｝ / 4 Ga (2x, 2y) = {G (2x−1, 2y) + G (2
x + 1, 2y) + G (2x, 2y-1) + G (2x, 2
y + 1)｝ / 4 Ba (2x, 2y) = {B (2x−1, 2y) + B (2
x + 1,2y) + B (2x, 2y-1) + B (2x, 2
y + 1)｝ / 4

On the other hand, if ΔHg or ΔVg is larger than a predetermined value thg, it is determined that there is an edge component in the pixel. Further, if ΔHg> ΔVg, there is an edge component in the horizontal direction, and there is a correlation in the vertical direction. Is high, and when ΔHg ≦ ΔVg, it is determined that there is an edge component in the vertical direction and the correlation is high in the horizontal direction. And ΔHg
When it is determined that the correlation is high in the vertical direction at> ΔVg,
The signal from the vertical operation means 11 is selected by the selection means 12. The vertical direction calculating means 11 calculates the average value of the pixels in the vertical direction at the pixel positions (2x, 2y) and (2x + 1, 2y + 1) in each signal, and outputs a signal having a correlation in the vertical direction. For example, in the case of the pixel position (2x, 2y), the values Rv, Gv, and B of the R, G, and B signals are expressed by the following equations.
v is calculated and output to the selection means 12. Rv (2x, 2y) = ｛R (2x-1, 2y) + R (2
x + 1, 2y) / 2 Gv (2x, 2y) = {G (2x-1, 2y) + G (2
x + 1,2y)｝ / 2 Bv (2x, 2y) = {B (2x−1,2y) + B (2
x + 1, 2y)｝ / 2

When it is determined that the correlation is high in the horizontal direction when ΔHg ≦ ΔVg, the selection means 12 selects the signal from the horizontal operation means 10. In the horizontal direction calculating means 10, the pixel position (2x, 2y) in each signal,
The average value of the pixels in the horizontal direction at (2x + 1, 2y + 1) is calculated, and a signal having a correlation in the horizontal direction is output. For example, in the case of the pixel position (2x, 2y), R,
The values Rh, Gh, and Bh of the G and B signals are calculated and output to the selection unit 12. Rh (2x, 2y) = ｛R (2x, 2y-1) + R (2
x, 2y + 1)｝ / 2 Gh (2x, 2y) = {G (2x, 2y−1) + G (2
x, 2y + 1)｝ / 2 Bh (2x, 2y) = {B (2x, 2y−1) + B (2
x, 2y + 1)｝ / 2

As described above, the pixel positions (2x, 2y), (2x, 2y + 1), (2x + 1, 2)
y), R at each pixel (2x + 1, 2y + 1),
G and B signals are output, that is, R, G, and B signals having a resolution twice as large as the number of pixels of the image sensor in the horizontal and vertical directions can be obtained. Is determined, the generation of each color signal is switched based on the determination result, and in the calculation of the signal when the edge component exceeds a predetermined value, the calculation is performed from the pixels in the direction where the edge component is small, and the edge component is equal to or less than the predetermined value. By calculating the average value of the peripheral pixels, false colors and false contours in an image with twice the resolution in the horizontal and vertical directions can be reduced.
The resolution of all the signals in both the horizontal and vertical directions is improved.

FIG. 13 is a diagram showing a false color level at a vertical resolution in a simulation image processed by the imaging apparatus according to the first embodiment of the present invention and a conventional apparatus. More specifically, FIG. In the simulation, a false color level (color-color) generated in the vertical direction when the zone plate is processed by the configuration of FIG.
It shows the relationship between the alias level (vertical axis) and the resolution (horizontal axis). The false color level after processing according to the above-described conventional technique is indicated by a broken line, and the false color level after processing according to the first embodiment shown in FIG. 1 is indicated by a solid line. Many false colors are generated in the processing according to the related art, but the processing according to the first embodiment suppresses the generation of false colors. In each process, the level of this false color signal in the vertical direction is integrated and compared.
There is a suppression effect of about 21.5 dB.

In the first embodiment, the configuration of the color restoring means in the first color restoring means 5 and the second color restoring means 6 has been described with reference to the configuration shown in FIG. Even if the restoration method by interpolation is used, R, G, and B signals corresponding to the number of pixels of the image sensor are restored from the signals obtained by the two image sensors that are obliquely shifted by half a pixel, and this pixel shift is performed. If an image having twice the resolution in the horizontal and vertical directions is obtained from the two restored images by the method shown in FIG. 1, the same effect as in the first embodiment can be obtained.

In the first embodiment, the image pickup device 1
The arrangement of the color filters of the two image sensors due to the pixel shift is a primary color filter shown in FIG. 3, and each of the photoelectric conversion elements is described as an independent image sensor. The image sensor of the system may be used, and the spectral characteristics of the color filters are not limited to R, G, and B, and the arrangement of the color filters may be other cases.
An RGB color signal can be reproduced by restoring a signal of the number of pixels of the image sensor from the data obtained by the two image sensors shifted obliquely by half a pixel, and FIG. As long as a signal having twice the resolution in each of the horizontal and vertical directions can be obtained by the method described above, the same effect as described above can be obtained.

For example, as shown in FIG. 14, a pixel position (2i, 2j) and a pixel position (2i + 1, 2j + 1) (i
= 0, 1, 2,..., J = 0, 1, 2,...) At the pixel position (2i, 2j + 1). A second color filter having a spectral characteristic for passing the second color signal B, and a third color filter having a spectral characteristic for passing the third color signal C at the pixel position (2i + 1, 2j). , The respective color signals A, B,
It is only necessary that RGB color signals can be reproduced after C is restored. Further, as shown in FIG. 15, the pixel position (2i, 2j +
1) and pixel positions (2i + 1, 2j) (i = 0, 1, 2,.
, J = 0, 1, 2,...), A first color filter at the pixel position (2i, 2j), and a third color filter at the pixel position (2i + 1, 2j + 1). In this case (the hatched portion in FIG. 15), the same effect can be obtained.

In the first embodiment, the image pickup device 1
FIG. 2 shows the arrangement of two image sensors due to pixel shift in FIG.
However, if the pixel is shifted by a half pixel in the oblique direction, a high-resolution image can be obtained according to the first embodiment even if the pixel is shifted as shown in FIG. Play.

In the first embodiment, the case where the processing of the configuration shown in FIGS. 1 and 4 is performed by hardware has been described, but it goes without saying that the same processing can be performed by software in the imaging apparatus. Thus, the same effect as in the first embodiment can be obtained.

Embodiment 2 In the color restoration means 5 or 6 shown in FIG. 4 of the first embodiment, the pixel position (2i, 2j +
1) and (2i + 1, 2j) to determine the edge components in the upper, lower, left, and right directions.
The pixel position (2i +
1, 2j), the G component for the reconstruction of the B signal
The edge component at the pixel position (2i, 2j + 1) in the signal is configured to be determined.
i, 2j + 1) and (2i + 1, 2j), the edge components of the left and right and upper and lower pixels are detected. Since the upper, lower, left and right pixel signals at these pixel positions are signals obtained from the image sensor, they are the same. Is to be obtained. Therefore, as shown in FIG. 17, the configuration of the color restoration means 5 or 6 is changed by one G component edge determination means to the pixel positions (2i, 2j + 1) and (2i + 1, 2) in the G signal.
It is also possible to adopt a configuration in which the edge component in j) is determined.

In FIG. 17, 20, 22, 23 and 25 are the same as those in the color restoring means 5 and 6 in the image pickup apparatus according to the first embodiment, and 41 is a G component edge judging means.

Next, the operation will be described. A signal from the frame memory 3 or 4 is separated into respective R, G, and B signals by a separating unit 20, and a G component restoring unit 22, a first RB restoring unit 23, and a second RB restoring unit 25 use a pixel of an image sensor. The operation of generating and restoring each signal of the numbers is the same as that in the first embodiment, and therefore the detailed description is omitted.

The G component edge determining means 41 includes a separating means 2
G signal at 0 is input and the pixel position (2i, 2j +
1) and (2i + 1, 2j) (pixel positions indicated by R and B in FIG. 3), that is, differences between left and right and upper and lower pixels, that is, edge components are detected. That is, the absolute value ΔH2 of the difference between the left and right pixels at the pixel position, and the absolute value ΔV2 of the difference between the upper and lower pixels
Is calculated. For example, at the pixel position (2i, 2j + 1), ΔH2 = | G (2i, 2j) −G (2i, 2j + 2) | ΔV2 = | G (2i−1, 2j + 1) −G (2i + 1,
2j + 1) |.

Then, the edge component ΔH in the horizontal direction is obtained.
2 and a vertical edge component ΔV2 to determine a change in signal level in peripheral pixels in the horizontal and vertical directions, and restore a signal ed1 indicating the determination result to the G component according to each pixel position of the input G signal. The edr and edb are output to the second RB component restoring means 25. When both of ΔH2 and ΔV2 are equal to or smaller than a predetermined value, each determination signal determines that there is no change in the signal level in the peripheral pixels.
On the other hand, if ΔH2 or ΔV2 is larger than a predetermined value, it is determined that there is an edge component at that pixel. Further, if ΔH2> ΔV2, it is determined that the correlation is high in the vertical direction. If ΔH2 ≦ ΔV2, It is determined that the correlation is high in the horizontal direction. Here, in the G component restoration means 22,
In order to restore the pixels at the pixel positions (2i, 2j + 1) and (2i + 1, 2j), the edge determination result ed1 is output at both pixel positions.

On the other hand, the second RB component restoring means 25 restores the pixel position (2i + 1, 2j) of the R signal and restores the pixel position (2i, 2j + 1) of the B signal. Therefore, the result of determining the edge component at the pixel position (2i + 1, 2j) is defined as edr, and the second RB
The component restoration unit 25 outputs the signal in synchronization with the processing pixel position in the R signal. In addition, the pixel position (2i,
2j + 1) is the result of the determination of the edge component
As the db, the second RB component restoring unit 25 outputs the signal in synchronization with the processing pixel position in the B signal.

Therefore, in the G component edge determining means 41, the edge component of the G signal corresponding to the pixel position at which the G, R, B signal is restored in the G component restoring means 22 and the second RB component restoring means 25 is obtained. The judgment result is output.

In the second embodiment, the G signal is applied to the pixel positions (2i, 2j) and (2i + 1, 2j + 1).
The R signal is applied to the pixel position (2i, 2j + 1) at the pixel position (2
The case where the B signal is obtained from the image sensor at (i + 1, 2j + 1) has been described. However, as in the first embodiment, the G signal is at pixel positions (2i, 2j + 1) and (2i + 1, 2j) and the pixel position (2i, 2j). ) At the pixel position (2).
i + 1, 2j + 1), a B signal may be arranged,
Further, the color signal is not limited to RGB.

In the second embodiment, as in the first embodiment, the same processing as that of the second embodiment can be performed by the software in the imaging apparatus. A similar effect is achieved.

[0082]

As described above, according to the image pickup apparatus of the present invention, in the image pickup apparatus having two image pickup elements which are arranged to be shifted by a half pixel in the oblique direction, the first and second image pickup elements A color signal restoring unit for restoring and generating a color signal having a resolution corresponding to the number of pixels of the image sensor for each image sensor from the read color signals; and the first and second outputs being the outputs from the color signal restoring unit. Means for obtaining a pixel of a signal having a resolution twice as large as the number of pixels of each image sensor in the horizontal and vertical directions from each signal restored from the signal from the image sensor as described above; A first edge determining means for determining an edge component of a peripheral pixel at a predetermined pixel position in the signal; and a color signal having a resolution twice as high in the horizontal and vertical directions based on a signal from the first edge determining means. Predetermined pixel By having a means for calculating the location of the signal, the horizontal and vertical directions each 2 false color, was reduced false contour
A high-resolution image with double resolution can be obtained, and the resolution in the horizontal and vertical directions can be improved.

According to the imaging apparatus of the present invention,
A first edge determining means for calculating an absolute value of a difference between left and right neighboring pixels in a predetermined pixel in a predetermined color signal to detect a horizontal edge component; and an absolute value of a difference between upper and lower pixels. Calculating a vertical edge component and calculating a horizontal edge component based on an output from the horizontal edge component detecting unit and a vertical edge component detecting unit. By determining the components, it is possible to obtain a high-resolution image with a double resolution in the horizontal and vertical directions in which false colors and false contours are reduced,
In addition, the resolution in the horizontal and vertical directions can be improved.

According to the imaging apparatus of the present invention, the first edge determining means determines the edge component.
If the output from the means for detecting the horizontal edge component or the output from the means for detecting the vertical edge component is larger than a predetermined value, it is assumed that an edge component has been detected in a peripheral pixel of the predetermined pixel. Further, when the output from the means for detecting the edge component in the horizontal direction is larger than the output from the means for detecting the edge component in the vertical direction, there is a correlation in the vertical direction, and the output from the means for detecting the edge component in the horizontal direction is If the output of the means for detecting the vertical edge component is smaller than the output of the means for detecting the vertical edge component, it is determined that there is a correlation in the horizontal direction, and the outputs of the means for detecting the horizontal edge component and the means for detecting the vertical edge component are both predetermined. If the value is smaller than the value, it is determined that no edge component is detected. It can be obtained for high-resolution images of the resolution, and it is possible to improve the resolution horizontal, in the vertical direction.

Further, according to the imaging apparatus of the present invention, the means for calculating the signal at the predetermined pixel position in the signal having the resolution twice as high in the horizontal and vertical directions is provided for each of the color signals at the predetermined pixel position. Vertical calculation means for calculating a signal from the average value of, horizontal calculation means for calculating the signal from the average value of the left and right pixels, and average value calculation means for calculating the signal from the average value of the upper, lower, left and right adjacent pixels Based on the output of the first edge determination means, select from the output of the horizontal direction calculation means or the output of the vertical direction calculation means, or the output from the average value calculation means, obtain a signal at the predetermined pixel, 2 pixels each in the horizontal and vertical directions of the number of pixels in the image sensor
By obtaining each color signal of double resolution, it is possible to obtain a high-resolution image of double resolution in the horizontal and vertical directions with reduced false colors and false contours, and to improve the resolution in the horizontal and vertical directions Can be.

According to the imaging apparatus of the present invention,
If the means for calculating a signal at a predetermined pixel position in the signal having a resolution twice as high in the horizontal and vertical directions determines that the output of the first edge determination means does not detect an edge component at the predetermined pixel position, the average is calculated. Select the output of the value calculation means,
When it is determined that there is a correlation in the vertical direction, the output of the vertical direction calculation means is selected, and when it is determined that there is a correlation in the horizontal direction, the output of the horizontal direction calculation means is selected. It is possible to obtain a high-resolution image with twice the resolution in the horizontal and vertical directions with reduced contours,
The resolution in the vertical direction can be improved.

According to the imaging apparatus of the present invention,
In each of the two image sensors that are displaced by a half pixel in the oblique direction, each of the image sensors has a spectral sensitivity characteristic with respect to a first color signal in four pixels in two rows vertically and two columns vertically. A color filter and a second color filter having a spectral sensitivity characteristic for the second color signal are arranged in a vertical first row, and the first color filter in the vertical first row is arranged in a vertical second row. A third color filter having a spectral sensitivity characteristic for the third color signal is arranged in the same column as the pixel position, and a first color filter having a spectral sensitivity characteristic for the first color signal in the same column as the second color filter. Is an image sensor in which the color filters of the upper and lower four pixels are sequentially arranged in the vertical and horizontal directions sequentially, and the resolution corresponding to the number of pixels of the image sensor with respect to a color signal from each image sensor. Restore the color signal of A second edge determination unit configured to determine an edge component at a predetermined pixel position based on a peripheral pixel signal of the first color signal at a predetermined pixel position by the first color filter; The first, second, and third color signals read out by the first, second, and third color filters based on the output of the second edge determination means at the predetermined position in the first color signal. Based on the output of the first calculating means for calculating the signal and the output of the second edge determining means, the second and third color signals from the color filter and the output of the first calculating means are used. A second calculating means for calculating the color signal of the image sensor, obtaining first, second, and third color signals of the number of pixels in the image sensor, and wherein the first edge determining means calculates the color signal in the horizontal and vertical directions. Obtain pixels of signal with double resolution By judging the edge components of peripheral pixels at predetermined pixel positions with respect to the first color signal output from the stage, false colors and false contours are reduced and a high-resolution image with a double resolution in the horizontal and vertical directions is reduced. And the resolution in the horizontal and vertical directions can be improved.

According to the imaging apparatus of the present invention,
The second edge judging means in the color restoring means is the first
Horizontal edge detection means for calculating the absolute value of the difference between the left and right adjacent pixels at the predetermined pixel position of the color signal to detect a horizontal edge component; A vertical edge detecting means for calculating an absolute value of the difference to detect a vertical edge component, based on an output from the horizontal edge detecting means and the vertical edge detecting means, in a horizontal or vertical direction in the predetermined pixel. A determination unit for determining an edge component;
When the output from the horizontal edge detecting means or the output from the vertical edge detecting means is larger than a predetermined value, the determining means detects an edge component in a peripheral pixel of the predetermined pixel, If the output from the horizontal edge detecting means is larger than the output of the vertical edge detecting means, there is a vertical correlation.If the output from the horizontal edge detecting means is smaller than the output of the vertical edge detecting means, the correlation is horizontal. In addition to determining that there is a correlation depending on the direction, and determining that no edge component is detected when both the outputs from the horizontal edge detecting means and the vertical edge detecting means are smaller than a predetermined value, It is possible to obtain a high-resolution image with twice the resolution in the horizontal and vertical directions with reduced false contours, and in the horizontal and vertical directions. It is possible to improve the definitive resolution.

Further, according to the imaging apparatus of the present invention, the first calculating means for calculating the signal for the predetermined position in the first color signal in the color restoring means includes a predetermined pixel l of the second color signal B. At the position of the row m column B (l, m), the value Ahlpf (l, l) of each of the first color signal A and the second color signal B via a horizontal low-pass filter is given.
m) and Bhlpf (l, m) are calculated, and Ahlpf (l, m) which is an output signal from the horizontal low-pass filter is calculated.
m) and Bhlpf (l, m) and the pixel value B (l, m) at the pixel position, the pixel value A (l, m) of the first color signal A in l rows and m columns is represented by A (L, m) = B (l,
m) × ｛Ahlpf (l, m) / Bhlpf (l,
m), the horizontal direction signal calculating means for calculating the pixel value of the first color signal A at another pixel position of the third color signal C in the same manner, and the predetermined pixel l row m column B At the position (l, m), the values Avlpf (l, m) and Bvlpf (l, l) of the first color signal A and the second color signal B are respectively passed through the low-pass filter in the vertical direction.
m), and output signals from the vertical low-pass filter, Avlpf (l, m) and Bvlpf (l,
m) and the pixel value B (l, m) at the pixel position, the pixel value A (l, l) in the first color signal A in l rows and m columns
m) is given by A (l, m) = B (l, m) × ｛Avlpf
(L, m) / Bvlpf (l, m)}, and the first pixel is similarly set at another pixel position where the third color signal C exists.
A vertical direction signal calculating means for calculating a pixel value in the color signal A of the first color signal A; Average value calculating means for calculating a pixel value A (l, m) of one color signal A, and based on an output of the second edge determining means, an output of the horizontal signal calculating means or a vertical signal calculating means. The pixel value A (l, m) of the first color signal A in the predetermined pixel l row m column is selected from the output of the means or the output from the average value calculation means, and the pixel number A By obtaining a color signal of 1, false colors,
It is possible to obtain a high-resolution image with twice the resolution in the horizontal and vertical directions with reduced false contours, and to improve the resolution in the horizontal and vertical directions.

According to the imaging apparatus of the present invention,
When the first calculating means in the color restoring means determines that the output of the second edge determining means does not detect an edge component at the position of the predetermined pixel l row and m column, the output of the average value calculating means is selected. Then, when it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculation means is selected,
If it is determined that there is a correlation in the horizontal direction, the output of the horizontal direction signal calculation means is selected, and the first color signal of the number of pixels in the image sensor is obtained, whereby the horizontal and vertical directions with reduced false colors and false contours are obtained. A high-resolution image having a resolution twice as high in each direction can be obtained, and the resolution in the horizontal and vertical directions can be improved.

Further, according to the imaging apparatus of the present invention, the second calculating means in the color restoring means determines that the output A from the first calculating means is at the position of the predetermined pixel at l row and m column.
To the value A1hl through a horizontal low-pass filter
pf (l, m) and a value A1vlpf (l, m) through a vertical low-pass filter are calculated, and a value B1hlpf through a horizontal low-pass filter is calculated for the second color signal B.
(L, m) and a value C1vlpf (l, m) through a vertical low-pass filter for the third color signal C (or a value C1vlpf (l, m) through a vertical low-pass filter for the second color signal B) The value B1vlpf (l, m) and the third color signal C
To the value C1h through a horizontal low-pass filter
lpf (l, m)) and A1hlpf (l,
m) and the ratio of B1hlpf (l, m) (or C1hl
pf (l, m)) and the ratio between A1vlpf (l, m) and C1vlpf (l, m) (or B1vlpf).
(Ratio to (l, m)) and the pixel value A (l, m) at l rows and m columns of the predetermined pixel in the output A from the first calculating means, the 1st row and m columns of the second color signal The pixel values B (l, m) and C (l, m) of B and the third color signal C are represented by B (l, m)
= A (l, m) × ｛B1hlpf (l, m) / A1hl
pf (l, m)}, C (l, m) = A (l, m) x {C
1vlpf (l, m) / A1vlpf (l, m)｝ (or B (l, m) = A (l, m) × ｛B1vlpf
(L, m) / A1vlpf (l, m)｝, C (l, m)
= A (l, m) × ｛C1hlpf (l, m) / A1hl
pf (l, m)｝), and a predetermined pixel x different from the position of the l row and m column.
At the position of the row y column, the value A2h obtained through the horizontal low-pass filter with respect to the output A from the first calculating means.
Ipf (x, y), a value B2hlpf (x, y) of the second color signal B output from the signal calculating means through a horizontal low-pass filter is calculated, and A2hlpf is calculated.
The ratio of (x, y) to B2hlpf (x, y) and the pixel value A at pixel x row y column at the output A from the first calculating means
According to (x, y), the second color signal B (x, y) at the position of x row and y column is represented by B (x, y) = A (x, y) × ｛B
2hlpf (x, y) / A2hlpf (x, yy)}, and the horizontal signal calculating means for calculating the C signal for the third color signal C in the same manner, and the output A from the first calculating means. A2vlpf (x, y) through a low-pass filter in the vertical direction, and B2vlpf (x, y) through a low-pass filter in the vertical direction with respect to the second color signal B at the output from the signal calculation means. Is calculated, and A2
By the ratio of vlpf (x, y) to B2vlpf (x, y) and the pixel value A (x, y) at the pixel x row y column at the output A from the first calculating means, x row y column Second in position
B (x, y) = A (x, y)
× ｛B2vlpf (x, y) / A2vlpf (x,
y) The vertical color signal calculating means for calculating the third color signal C in the same manner as in the third color signal C, and the predetermined pixel x in the second and third color signals in the output from the signal calculating means. Average value calculating means for calculating an average value of obliquely adjacent pixels at the position of the row y column, wherein the horizontal direction signal calculating means and the vertical direction signal calculating means are based on the output of the second edge determining means. , The second and third color signals at the predetermined pixel x row and y column are selected from the respective outputs from the average value calculation means, and the second and third color signals of the number of pixels in the image sensor are obtained. As a result, it is possible to obtain a high-resolution image having twice the resolution in the horizontal and vertical directions with reduced false colors and false contours, and to improve the resolution in the horizontal and vertical directions.

Further, according to the imaging apparatus of the present invention,
When the second calculating means in the color restoring means determines that the output of the second edge determining means does not detect an edge component at the position of the predetermined pixel x row and y column, the output of the average value calculating means is selected. Then, when it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculation means is selected,
When it is determined that there is a correlation in the horizontal direction, the output of the horizontal direction signal calculation means is selected and the second and third color signals of the number of pixels in the image sensor are obtained, thereby reducing false colors and false contours. It is possible to obtain a high-resolution image having twice the resolution in each of the horizontal and vertical directions, and to improve the resolution in the horizontal and vertical directions.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an example of a configuration of an imaging device according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of an arrangement of two imaging elements shifted by a half pixel in an oblique direction according to the first embodiment of the present invention;

FIG. 3 is a diagram showing an example of an arrangement of color filters of each image sensor according to Embodiment 1 of the present invention;

FIG. 4 is a block diagram illustrating an example of a configuration of a color restoration unit in the imaging device according to the first embodiment of the present invention;

FIG. 5 is a flowchart illustrating an operation of a first edge determination unit and a G component restoration in the color restoration unit in the imaging device according to the first embodiment of the present invention;

FIG. 6 is a flowchart for explaining an operation of restoring R and B components in a color restoring unit in the imaging device according to the first embodiment of the present invention;

FIG. 7 is a diagram showing pixels of R and B signals for describing an operation of restoring R and B signals in the imaging apparatus according to the first embodiment of the present invention;

FIG. 8 is a diagram illustrating pixels of a signal having twice the resolution in the horizontal and vertical directions for explaining the operation of the multiplexing unit 12 in the imaging apparatus according to the first embodiment of the present invention;

FIG. 9 is a diagram showing pixels of an R signal having twice the resolution in the horizontal and vertical directions in the multiplexing unit 12 in the imaging apparatus according to the first embodiment of the present invention.

FIG. 10 is a diagram showing pixels of a G signal having twice the resolution in the horizontal and vertical directions in the multiplexing unit 12 in the imaging apparatus according to the first embodiment of the present invention.

FIG. 11 is a diagram showing pixels of a B signal having twice the resolution in the horizontal and vertical directions in the multiplexing unit 12 in the imaging apparatus according to the first embodiment of the present invention;

FIG. 12 is a diagram illustrating a flowchart for explaining the operation of increasing the resolution in the imaging device according to the first embodiment of the present invention;

FIG. 13 is a diagram illustrating a false color level at a vertical resolution in a simulation image processed by the imaging device according to the first embodiment of the present invention and a conventional device.

FIG. 14 is a diagram showing an example of another color filter array of the imaging device according to the first embodiment of the present invention.

FIG. 15 is a diagram showing an example of another color filter array of the imaging device according to the first embodiment of the present invention.

FIG. 16 is a diagram showing another example of the arrangement of the two imaging elements shifted by a half pixel in the oblique direction according to the first embodiment of the present invention.

FIG. 17 is a block diagram illustrating an example of a configuration of a color restoring unit in the imaging device according to the second embodiment of the present invention;

FIG. 18 is a diagram illustrating an example of a color filter array of an imaging device in a conventional imaging device.

FIG. 19 is a block diagram illustrating an example of a configuration of a conventional imaging device.

FIG. 20 is a block diagram illustrating an example of a configuration of a color restoration unit in a conventional imaging device.

FIG. 21 is a diagram illustrating pixels of each signal for explaining an operation of increasing the resolution of a conventional imaging device.

FIG. 22 is a diagram illustrating pixels of each signal for describing a method of increasing the resolution of a conventional imaging device.

FIG. 23 is a block diagram illustrating another example of the configuration of the color restoring unit in the conventional imaging apparatus.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Image sensor, 2 A / D converter, 3, 4 frame memory, 5 and 6 color restoration means, 7 multiplexing means, 8 edge determination means, 9 average value calculation means, 10 horizontal direction calculation means, 11 vertical direction calculation means, 12 Selection means, 2
0 separation means, 21 first edge determination means, 22 G
Component restoring means, 23 first RB component restoring means, 24
Second edge determining means, 25 second RB component restoring means.

 ──────────────────────────────────────────────────続 き Continued from the front page (72) Inventor Tetsuya Kuno 2-3-2 Marunouchi, Chiyoda-ku, Tokyo Mitsubishi Electric Corporation

Claims (11)

[Claims] 2. An image display apparatus comprising: two pixels arranged in a diagonal direction and shifted by a half pixel;
A color signal for restoring and generating a color signal having a resolution corresponding to the number of pixels of the image sensor for each image sensor from the color signals read by the first and second image sensors in an image pickup apparatus including a plurality of image sensors; A signal having a resolution twice as large as the number of pixels of each image sensor in the horizontal and vertical directions, based on each signal restored from the signals from the first and second image sensors, which is an output from the color signal restoring means. And a first edge determination means for determining an edge component of a peripheral pixel at a predetermined pixel position in the obtained signal having a resolution twice as high in the horizontal and vertical directions, and the first edge An imaging apparatus comprising: means for calculating a signal at a predetermined pixel position in a signal having twice the resolution in each of the horizontal and vertical directions based on a signal from a determination means. 2. The image processing apparatus according to claim 1, wherein the first edge determination unit calculates an absolute value of a difference between left and right adjacent pixels in a predetermined pixel in a predetermined signal and detects an edge component in a horizontal direction. Means for calculating the absolute value of the difference to detect a vertical edge component; means for detecting the horizontal edge component and means for detecting a vertical edge component; 2. The imaging apparatus according to claim 1, wherein a vertical edge component is determined. 3. The determination of an edge component in the first edge determination means is performed by determining whether an output from the horizontal edge component detection means or an output from the vertical edge component detection means is predetermined. If greater than the value,
Assuming that an edge component is detected in a peripheral pixel of the predetermined pixel,
Further, when the output from the means for detecting the horizontal edge component is larger than the output from the means for detecting the vertical edge component, there is a correlation in the vertical direction, and the output of the means for detecting the horizontal edge component is If the output is smaller than the output of the means for detecting the vertical edge component, it is determined that there is a correlation in the horizontal direction, and the outputs of the means for detecting the horizontal edge component and the means for detecting the vertical edge component are both predetermined values. 3. The imaging apparatus according to claim 1, wherein it is determined that the edge component is not detected when the size is smaller. 4. A means for calculating a signal at a predetermined pixel position in a signal having twice the resolution in each of the horizontal and vertical directions, wherein for each signal at the predetermined pixel position, a vertical signal for calculating a signal from an average value of upper and lower pixels is calculated. The first edge determination means, comprising: a direction calculation means; a horizontal direction calculation means for calculating a signal from the average value of the left and right pixels; and an average value calculation means for calculating a signal from the average values of the upper, lower, left and right adjacent pixels. Based on the output of
Select from the output of the horizontal direction calculation means or the output of the vertical direction calculation means, or the output from the average value calculation means, obtain the signal at the predetermined pixel, and double the number of pixels in the image sensor in each of the horizontal and vertical directions. 2. The image pickup apparatus according to claim 1, wherein each signal of the resolution is obtained. 5. A means for calculating a signal at a predetermined pixel position in a signal having twice the resolution in each of the horizontal and vertical directions determines that the output of the first edge determination means does not detect an edge component at the predetermined pixel position. In this case, the output of the average value calculation means is selected.If it is determined that there is a correlation in the vertical direction, the output of the vertical direction calculation means is selected. The imaging device according to claim 1, wherein an output of the calculation unit is selected. 6. In each of the two image sensors, which are displaced by half a pixel in the oblique direction, each image sensor has a spectral sensitivity characteristic with respect to a first color signal in upper and lower four pixels of two vertical rows and two horizontal columns. And a second color filter having a spectral sensitivity characteristic for a second color signal are arranged in the first vertical line, and the first color in the first vertical line is arranged in the second vertical line. A third color filter having a spectral sensitivity characteristic for the third color signal is arranged in the same column as the pixel position where the filter is arranged, and a spectral sensitivity for the first color signal is arranged in the same column as the second color filter. A first color filter having characteristics is arranged. The image sensor is an image sensor in which the color filters of the upper and lower four pixels are sequentially and repeatedly arranged in the vertical and horizontal directions. Color signals with the resolution of the number of pixels A second edge determining unit configured to determine an edge component at a predetermined pixel position based on a peripheral pixel signal of the first color signal at a predetermined pixel position by the first color filter; The first, second, and third color signals read by the first, second, and third color filters based on the output of the second edge determination means, and A first calculating means for calculating a signal at a position, and a second signal based on the output of the first calculating means and the second and third color signals from the color filter based on the output of the second edge determining means. And a second calculating means for calculating a third color signal, wherein the first, second, and third pixel numbers in the image sensor are determined.
And the first edge determination means determines a predetermined pixel position with respect to a signal output from the means for obtaining a pixel of a signal having a double resolution in each of the horizontal and vertical directions based on the first color signal. The imaging device according to claim 1, wherein an edge component of a peripheral pixel is determined. 7. A horizontal edge detecting means for detecting a horizontal edge component by calculating an absolute value of a difference between left and right adjacent pixels at a predetermined pixel position of the first color signal. A direction edge detection unit, a vertical direction edge detection unit that calculates an absolute value of a difference between upper and lower pixels at a predetermined pixel position of the first color signal to detect a vertical direction edge component, and the horizontal direction edge detection unit. A determination unit configured to determine a horizontal or vertical edge component in the predetermined pixel based on an output from the vertical edge detection unit, wherein the determination unit determines whether an output from the horizontal edge detection unit or the vertical edge If the output from the detecting means is larger than a predetermined value, it is determined that an edge component has been detected in the peripheral pixels of the predetermined pixel. If the output from the stage is larger than the output of the vertical edge detecting means, there is a vertical correlation.If the output from the horizontal edge detecting means is smaller than the output of the vertical edge detecting means, the correlation is more horizontal. 2. The method according to claim 1, further comprising: judging that no edge component is detected when the outputs from the horizontal edge detecting means and the vertical edge detecting means are both smaller than a predetermined value. Item 7. The imaging device according to Item 6. 8. A first calculating means for calculating a signal for a predetermined position in a first color signal in the color restoring means,
At a position of a predetermined pixel l row m column B (l, m) of the second color signal B, the value of each of the first color signal A and the second color signal B via a horizontal low-pass filter Ah
Ipf (l, m) and Bhlpf (l, m) are calculated, and Ah which is an output signal from the horizontal low-pass filter is calculated.
Based on the ratio of lpf (l, m) to Bhlpf (l, m) and the pixel value B (l, m) at the pixel position, the first in l rows and m columns
The pixel value A (l, m) of the color signal A of A is represented by A (l,
m) = B (l, m) × ｛Ahlpf (l, m) / Bhl
pf (l, m)}, and a horizontal direction signal calculating means for calculating a pixel value of the first color signal A at another pixel position of the third color signal C in the same manner; At the position of the row m column B (l, m), the values Avlpf (l, m), Bvlp for the first color signal A and the second color signal B, respectively, are passed through a vertical low-pass filter.
f (l, m) is calculated, and Avlpf (l, m) and Bvlp which are output signals from the vertical low-pass filter are calculated.
f (l, m) and the pixel value B (l,
m), the pixel value A (l, m) in the first color signal A in l rows and m columns is calculated as A (l, m) = B (l, m) × ｛Av
vertical direction signal calculation which calculates the pixel value of the first color signal A at another pixel position where the third color signal C is also calculated by the calculation of lpf (l, m) / Bvlpf (l, m)｝ Means, and the predetermined pixel l in the first color signal A.
An average value calculating means for calculating a pixel value A (l, m) of the first color signal A in l rows and m columns from an average value of the upper, lower, left and right neighboring pixels at the position of the row m column; Based on the output of the edge judging means, the output of the horizontal signal calculating means, the output of the vertical signal calculating means, or the output of the average value calculating means, and the first color in the predetermined pixel l row m column 8. The method according to claim 1, wherein a pixel value A (l, m) of the signal A is obtained to obtain a first color signal of the number of pixels in the image sensor. Imaging device. 9. A method according to claim 1, wherein said first calculating means in said color restoring means determines that an output of said second edge determining means is a predetermined pixel l row m
If it is determined that no edge component is detected at the position of the column, the output of the average value calculating means is selected.If it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculating means is selected. 9. When it is determined that there is a correlation, the output of the horizontal direction signal calculation means is selected to obtain a first color signal of the number of pixels in the image sensor. The imaging device according to any one of the above. 10. The second calculating means in the color restoring means outputs a value A1hlpf (A1hlpf (A) through a horizontal low-pass filter to an output A from the first calculating means at a position of a predetermined pixel at l rows and m columns. l, m) and a value A1vlpf (l, m) through a vertical low-pass filter,
The value B1hlpf (l, m) of the second color signal B through a horizontal low-pass filter and the value C1vlpf of the third color signal C through a vertical low-pass filter
(L, m) (or a value B1vlpf (l, m) through a low-pass filter in the vertical direction for the second color signal B and a value C1hlpf through a low-pass filter in the horizontal direction for the third color signal C (L, m)) and A1hl
A ratio between pf (l, m) and B1hlpf (l, m) (or a ratio between C1hlpf (l, m)) and A1vlpf
The ratio of (l, m) to C1vlpf (l, m) (or
B1vlpf (l, m)) and the pixel value A at the predetermined pixel l row m column in the output A from the first calculating means
From (l, m), pixel values B (l, m) and C (l, m) in the second color signal B and the third color signal C in l rows and m columns are
B (l, m) = A (l, m) × ｛B1hlpf (l,
m) / A1hlpf (l, m)｝, C (l, m) = A
(L, m) × ｛C1vlpf (l, m) / A1vlpf
(L, m)｝ (or B (l, m) = A (l, m) ×
｛B1vlpf (l, m) / A1vlpf (l,
m)｝, C (l, m) = A (l, m) × １C1hlpf
(L, m) / A1hlpf (l, m)｝), and at the position of a predetermined pixel x row y column different from the position of l row m column, the first
A2hlpf (x, y) through a low-pass filter in the horizontal direction with respect to the output A from the calculation means, and through a low-pass filter in the horizontal direction with respect to the second color signal B in the output from the signal calculation means. A value B2hlpf (x, y) is calculated, and A2hlpf (x, y) and B2hlpf (x, y) are calculated.
y) and the pixel x at the output A from the first calculating means
By the pixel value A (x, y) at the row y column, the second color signal B (x, y) at the position at the x row y column is calculated as B (x, y) =
A (x, y) × ｛B2hlpf (x, y) / A2hlp
f (x, y)}, and a horizontal signal calculating means for calculating the C signal in the same manner also for the third color signal C, and a low-pass filter in the vertical direction for the output A from the first calculating means. A2vlpf (x, y) via the low-pass filter in the vertical direction with respect to the second color signal B at the output from the signal calculation means.
y) is calculated, and A2vlpf (x, y) and B2vlpf are calculated.
The second color signal B at the position of x row and y column is obtained from the ratio of (x, y) and the pixel value A (x, y) at the pixel x row and y column at the output A from the first calculating means. (X, y) is converted to B (x,
y) = A (x, y) × ｛B2vlpf (x, y) / A2
vlpf (x, y)}, a vertical signal calculating means for calculating the C signal in the same manner also for the third color signal C, and the second and third color signals in the output from the signal calculating means. Average value calculating means for calculating an average value of pixels obliquely adjacent to each other at the position of the predetermined pixel x row and y column, and the horizontal direction signal calculating means and the vertical direction By selecting from the respective outputs from the direction signal calculating means and the average value calculating means, the second and third color signals at the predetermined pixel x row and y column are obtained, and the second and third pixel numbers in the image sensor are obtained. 9. The image pickup apparatus according to claim 1, wherein three color signals are obtained. 11. When the second calculating means in the color restoring means determines that the output of the second edge determining means does not detect an edge component at a position of a predetermined pixel x row and y column, the average value calculation is performed. The output of the means is selected, and if it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculation means is selected.If it is determined that there is a correlation in the horizontal direction, the output of the horizontal signal calculation means is output. 11. The image pickup apparatus according to claim 1, wherein a second color signal and a third color signal corresponding to the number of pixels in the image pickup element are selected.