JP6095519B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus capable of obtaining a captured image with higher sensitivity in a low-illuminance imaging environment.

  2. Description of the Related Art As a conventional imaging device, a device configured to increase sensitivity and improve S / N by executing a function of adding all digital signals up to N pixels before is known (for example, Patent Document 1). reference).

JP 2000-184274 A (page 4, paragraph 0010)

  Since the conventional image pickup apparatus adds image pickup signals from consecutive adjacent pixels, there is a problem that when a color image pickup element is used, colors are mixed and a color signal cannot be reproduced. In addition, there is a problem that high sensitivity is not sufficient.

  The present invention has been made to solve the above-described problems. Even when the image signal read from the color image sensor is added to the pixel, the color of the image signal is not mixed and the sensitivity of the image signal can be sufficiently increased. The purpose is to.

The imaging apparatus of the present invention
An imaging signal generation unit that performs imaging and generates an imaging signal indicating a pixel value for each pixel of a plurality of pixels that constitute each temporally continuous frame;
Based on the imaging signal generated by the imaging signal generation unit, the pixels in the frame are sequentially designated in each of the successive frames, and the designated pixels are designated in the same frame for the designated pixels. A pixel having a high correlation between the pixel value and the designated pixel is selected from the pixels located around the pixel, and the pixel value of the selected pixel and the pixel value of the designated pixel are added and added. An in-plane pixel adder that outputs the result as an in-plane sensitization signal for the designated pixel;
The consecutive frames are sequentially designated as the attention frame, the pixels in the attention frame are sequentially designated as the attention pixels, and for each attention pixel, from each of one or more frames located in the vicinity of the attention frame, the For each of the pixels located at the same position as the target pixel and its surroundings, the correlation of the in-plane sensitization signal with the target pixel is compared with a correlation presence / absence determination threshold value, and it is determined that there is a correlation by the comparison determination. A pixel having a high correlation is selected from the selected pixels, and the in-plane sensitization signal for the pixel selected from each of the one or more neighboring frames and the in-plane sensitization for the pixel of interest possess a plane between the pixel adding unit for outputting the addition result by adding the signals as a three-dimensional sensitization signal,
The in-plane pixel addition unit
A region composed of pixels having a high correlation with each other is selected from a plurality of predetermined regions each formed by the designated pixel and its surrounding pixels, and the position of the pixel in the selected region And an area selection unit that outputs an addition pixel pattern code indicating the pattern of the selected area,
For the imaging signal of the designated pixel, the addition result obtained by adding the imaging signals of the pixels included in the area selected by the area selection unit is output as an in-plane sensitization signal for the designated pixel. And a selective addition unit to
The inter-plane pixel addition unit
The in-plane sensitization signal and the added pixel pattern code for pixels located in the same position as the target pixel and in the vicinity thereof in each of the frames adjacent to the target frame among the neighboring frames, and Correlation presence / absence determination for determining presence / absence of correlation between the in-plane sensitization signal and the added pixel pattern code for the target pixel and the in-plane sensitization signal for the target pixel and the in-plane sensitization signal other than the target pixel One pixel is selected from the frame based on the comparison result with the threshold .

  According to the present invention, it is possible to sufficiently increase the sensitivity of an image signal read from an image sensor, and to obtain an effect that a subject can be visually recognized even in an extremely dark low-light environment. Even if the image sensor is a color image sensor, colors are not mixed.

It is a block block diagram which shows the imaging device of Embodiment 1 of this invention. It is a figure which shows pixel space arrangement | positioning centering on an attention pixel which has a pixel value output from an image pick-up element. FIG. 3 is a diagram illustrating an arrangement of peripheral pixels when a target pixel is a G pixel in the pixel space arrangement of FIG. 2. FIG. 3 is a diagram illustrating an arrangement of peripheral pixels when a pixel of interest is an R pixel in the pixel space arrangement of FIG. 2. FIG. 3 is a diagram illustrating an arrangement of peripheral pixels when a pixel of interest is a B pixel in the pixel space arrangement of FIG. 2. It is a block block diagram which shows an example of the three-dimensional pixel addition part (6) of FIG. It is a block block diagram which shows an example of the in-plane pixel addition part (20) of FIG. It is a block block diagram which shows an example of the pixel extraction part (21) of FIG. It is a block block diagram which shows an example of the area | region selection part (22) of FIG. (A)-(d) is a figure which shows the pattern of the 1st-4th combination when the attention pixel of a pixel addition part is a G pixel. (A)-(d) is a figure which shows the pattern of the 5th-8th combination when the attention pixel of a pixel addition part is a G pixel. (A)-(d) is a figure which shows the pattern of the 9th to 12th combination when the attention pixel of a pixel addition part is a G pixel. (A)-(d) is a figure which shows the pattern of the 1st-4th combination when the attention pixel of a pixel addition part is R pixel. (A)-(d) is a figure which shows the pattern of the 1st-4th combination when the attention pixel of a pixel addition part is a B pixel. It is a block block diagram which shows an example of the selection addition part (23) of FIG. It is a block block diagram which shows an example of the inter-plane pixel addition part (30) of FIG. It is a block block diagram which shows a part of example of the pixel extraction part (31) of FIG. It is a block block diagram which shows another part of an example of the pixel extraction part (31) of FIG. It is a block block diagram which shows an example of the pixel selection part (351) of FIG. It is a block block diagram which shows the specific example of the pattern discrimination | determination part (361) of FIG. 19 with the correlation discrimination | determination part (381). It is a block block diagram which shows an example of the discrimination | determination part (3611) of FIG. It is a block block diagram which shows the pixel addition part (35b) used in Embodiment 2 of this invention. It is a block block diagram which shows the specific example of the 2nd pattern discrimination | determination part (662) of FIG. 22 with the correlation discrimination | determination part (682). It is a block block diagram which shows an example of the discrimination | determination part (6621) of FIG. It is a block block diagram which shows the specific example of the 1st pattern discrimination | determination part (661) of FIG. 22 with the correlation discrimination | determination part (681). It is a block block diagram which shows an example of the discrimination | determination part (6611) of FIG. It is a block diagram which shows the inter-plane pixel addition part (30b) used in Embodiment 3 of this invention. It is a block block diagram which shows the imaging device of Embodiment 4 of this invention. It is a block block diagram which shows the imaging device of Embodiment 5 of this invention. (A) to (e) are the subject illuminance, the lens aperture, the amplification gain of the programmable gain amplification unit, the sensitization magnification of the pixel addition unit, the exposure time of the imaging unit, and the amplitude of the output signal of the pixel addition unit. It is a figure which shows a relationship. It is a block block diagram which shows the imaging device of Embodiment 6 of this invention.

Embodiment 1 FIG.
FIG. 1 shows an imaging apparatus according to Embodiment 1 of the present invention. The illustrated imaging apparatus captures images in a continuous manner, and sequentially outputs image signals representing temporally continuous frame images. In the following, it is assumed that the image is a color image composed of red, green, and blue pixels.

In FIG. 1, a lens 1 focuses a subject image on an imaging surface of a CCD imaging device 2.
As shown in FIGS. 3 to 5 described later, the CCD image pickup device (CCD) 2 has an R pixel for detecting red light (first color component light) and green light (second color component light). ) For detecting G) and B pixels for detecting blue light (light of the third color component) are arranged in a Bayer array.

The R pixel, the G pixel, and the B pixel are, for example, a photoelectric conversion element that includes a color filter that transmits only red light, a photoelectric conversion element that includes a color filter that transmits only green light, and only blue light. It is composed of a photoelectric conversion element provided with a color filter to be transmitted.
The R pixel, G pixel, and B pixel detect red light, green light, and blue light (first color component light, second color component light, and third color component light). That is, photoelectric conversion is performed, and electric charges generated by the photoelectric conversion are transferred inside the device and output as an electric signal (imaging signal). The imaging signal is output sequentially for each frame.

  The imaging signal includes an R signal (a signal representing the first color component value) that is a pixel signal from the R pixel, a G signal (a signal representing the second color component value) that is a pixel signal from the G pixel, and B A B signal (a signal representing the third color component value) that is a pixel signal from the pixel is included.

From the image pickup signal output from the CCD image pickup device 2, noise and the like are removed by a correlated double sampling processing unit (CDS) 3.
The programmable gain amplifier (PGA) 4 amplifies the output signal of the correlated double sampling processing unit 3 with a gain controlled by the control signal output from the control unit 12 and outputs the amplified signal.
The A / D converter (ADC) 5 converts the output signal of the programmable gain amplifier 4 into a digital signal Pc.

  The imaging device 2, the correlated double sampling processing unit 3, the programmable gain amplification unit 4, and the A / D conversion unit 5 sequentially generate imaging signals for each frame of a plurality of color components obtained as a result of imaging the subject. An imaging signal generation unit 13 is configured. This imaging signal indicates a pixel value for each of a plurality of pixels constituting each temporally continuous frame.

  The three-dimensional pixel addition unit 6 receives the imaging signal Pc output from the A / D conversion unit 5, performs in-plane pixel addition and inter-plane pixel addition, and generates a three-dimensional sensitization signal.

  In the in-plane pixel addition, the pixels in the frame are sequentially specified in each successive frame, and the designated pixels are designated out of the pixels located around the designated pixel in the same frame. A pixel having a high correlation between the pixel value and the selected pixel is selected, and the pixel value of the selected pixel is added to the pixel value of the specified pixel, and the addition result is in-plane sensitized for the specified pixel. Output as a signal.

  In the inter-plane pixel addition, consecutive frames are sequentially designated as a target frame, pixels in the target frame are sequentially specified as a target pixel, and one or more frames positioned in the vicinity of the target frame for each target pixel. For each of the pixels located at the same position and the periphery of the target pixel, the correlation of the in-plane sensitization signal with the target pixel is compared with the correlation presence / absence determination threshold value CRth, and as a result of the comparison determination, In-plane sensitization for a pixel selected from each of one or more neighboring frames by selecting a pixel having a relatively high correlation, for example, the highest correlation among the pixels determined to have correlation. The signal and the in-plane sensitization signal for the target pixel are added, and the addition result is output as a three-dimensional sensitization signal.

When the difference absolute value is used as the correlation index, the smaller the difference absolute value, the higher the correlation. Therefore, the difference absolute value is equal to or less than the threshold value CRth, and the correlation is equal to or greater than the reference value. Means that. On the other hand, when using an index that becomes larger as the correlation is higher, the index being equal to or greater than a threshold means that the correlation is equal to or greater than a reference value.
The correlation presence / absence determination threshold value CRth is given from the control unit 12.

  In the in-plane pixel addition, the in-plane sensitization signal for the sequentially designated pixels is obtained, and in the inter-plane pixel addition, the pixel addition is performed using the in-plane sensitization signal for each pixel of the plurality of frames. The target pixel of the target frame in the inter-plane pixel addition executed at each time point is not the same as the “designated pixel” at the time of in-plane pixel addition. The “selected pixel” is referred to as a target pixel.

  The video signal processing unit 7 performs color synchronization processing, gradation correction processing, noise reduction processing, contour correction processing, white balance adjustment processing, signal amplitude adjustment processing, and color correction processing on the output signal of the three-dimensional pixel addition unit 6. A video signal with the above added is output from the video signal output terminal 8.

  The synchronization signal generation unit 11 generates a vertical synchronization signal VD and a horizontal synchronization signal HD and supplies them to the three-dimensional pixel addition unit 6, the video signal processing unit 7, and the timing generation unit 10.

  The timing generator 10 generates a drive timing signal DRT for the CCD image pickup device 2 and supplies it to the driver 9. The drive unit 9 generates a drive signal DRS for the CCD image pickup device 2 based on the drive timing signal DRT output from the timing generation unit 10. The CCD imaging device 2 performs photoelectric conversion and charge transfer based on the drive signal DRS output from the drive unit 9.

  The control unit 12 controls the aperture of the lens 1, controls the timing of charge readout from the photoelectric conversion element of the CCD image sensor 2 generated by the timing generation unit 10, and the control of the forced discharge timing of the charge (accordingly, the charge accumulation time, ie, the exposure time Control), control of the amplification gain of the programmable gain amplification unit 4, and control of pixel addition processing including setting of the sensitization magnifications La and Lb for the three-dimensional pixel addition unit 6.

  The pixels on the imaging screen of the imaging device 2 are aligned in the horizontal direction (row direction) H and the vertical direction (column direction) V as shown in FIG. 2, and are arranged in a matrix as a whole. The position of each pixel on the screen is represented by the coordinate (h, v) by the horizontal position h and the vertical position v, and the pixel at the coordinate (h, v) is represented by the symbol Phv. The value of h differs by 1 between pixels adjacent in the horizontal direction, and the value of v differs by 1 between pixels adjacent in the vertical direction. That is, the distance (pixel pitch) between adjacent pixels is represented by 1.

  FIG. 2 shows five ranges (5 × 5 pixel range) of 5 horizontal pixels and 5 vertical pixels centered on the pixel of interest P33 (in the in-plane pixel addition), and pixels located in the periphery (horizontal 9 pixels, vertical). The arrangement of a part of pixels within a range of 9 pixels) is shown.

  3, 4 and 5 show the arrangement of R, G and B pixels. In this arrangement, R pixels, G pixels, and B pixels are arranged in a checkered pattern, and an arrangement of four pixels of two horizontal pixels and two vertical pixels is repeated as a basic unit. In this basic unit, two G pixels are arranged on one diagonal line, and R pixels and B pixels are arranged on the other diagonal line. In these drawings, Rhv, Ghv, and Bhv indicate an R pixel, a G pixel, and a B pixel at coordinates (h, v), respectively. In these figures, (h, v) = (3, 3) are the coordinates of the target pixel.

  FIG. 3 shows an arrangement of a 5 × 5 pixel region and its surrounding pixels when the target pixel is a G pixel (a pixel having a G filter) in the pixel space arrangement of FIG. For example, the pixels of each color are arranged so as to repeat a combination of four pixels of R32, G33, B43, and G42 of two horizontal pixels and two vertical pixels as a basic unit.

  FIG. 3 illustrates an example in which the neighboring pixels (pixels before and after one pixel) on the same line of G33 are B pixels, but the neighboring pixels on the same line are R pixels as in G22. There is also an array pattern. In this case, the color arrangement is such that the R pixel and the B pixel are interchanged. However, since the pixel to be added is only the G pixel when the target pixel is the G pixel, only a slight correction is made for one color arrangement. Is true.

  FIG. 4 shows an arrangement of a 5 × 5 pixel region and surrounding pixels when the pixel of interest is an R pixel (a pixel having an R filter) in the pixel space arrangement of FIG. For example, the pixels of each color are arranged so as to repeat a combination of 4 pixels of 2 horizontal pixels and 2 vertical pixels composed of R33, G34, B44, and G43 as a basic unit.

  FIG. 5 shows an arrangement of a 5 × 5 pixel region and its surrounding pixels when the pixel of interest is a B pixel (a pixel having a B filter) in the pixel space arrangement of FIG. For example, the pixels of each color are arranged so as to repeat a combination of four horizontal pixels and two vertical pixels composed of R22, G23, B33, and G32 as a basic unit.

  As illustrated in FIG. 6, the three-dimensional pixel addition unit 6 includes an in-plane pixel addition unit 20 and an inter-plane pixel addition unit 30.

  Imaging signals Pc of R pixels, G pixels, and B pixels arranged in a Bayer array are supplied from the A / D conversion unit 5 to the in-plane pixel addition unit 20 via the input terminal 601.

  The in-plane pixel addition unit 20 sequentially designates the pixels in the frame in each successive frame based on the imaging signal generated by the imaging signal generation unit 13, and within the same frame for the designated pixel. In the above, a pixel having a high correlation with a pixel value of the designated pixel is selected from among pixels located around the designated pixel, and the pixel value of the selected pixel and the designated pixel The pixel values are added, and the addition result is output as an in-plane sensitization signal VAL for the designated pixel.

  The inter-plane pixel addition unit 30 sequentially designates consecutive frames as the attention frame, sequentially designates pixels within the attention frame as the attention pixels, and for each attention pixel, one or two or more positions located in the vicinity of the attention frame. From each of the frames, a pixel having a high correlation of the in-plane sensitization signal VAL with the pixel of interest is selected from the pixels located at the same position as the pixel of interest and the periphery thereof, and the one or more neighboring frames are selected. The in-plane sensitization signal VAL for the pixel selected from each and the in-plane sensitization signal VAL for the pixel of interest are added, and the addition result is output as a three-dimensional sensitization signal Pf.

The in-plane sensitization signal VAL is obtained by adding, for example, four original image pickup signals in the in-plane pixel addition unit 20, and in this case, the in-plane sensitization signal VAL can be sensitized up to four times. The sensitization signal Pf is obtained by adding, for example, five in-plane sensitization signals VAL in the inter-plane pixel addition unit 30. In this case, the sensitization signal Pf can be further sensitized up to 5 times, and therefore, three-dimensional As the sensitized signal Pf, a signal obtained by sensitizing the original image pickup signal up to 20 times can be obtained.
The inter-plane pixel addition unit 30 supplies the three-dimensional sensitization signal Pf from the output terminal 602 to the video signal processing unit 7.

  As illustrated in FIG. 7, the in-plane pixel addition unit 20 includes a pixel extraction unit 21, a region selection unit 22, a selection addition unit 23, and a signal combination unit 24.

The imaging signal Pc output from the A / D conversion unit 5 is input to the input terminal 601 of the three-dimensional pixel addition unit 6, and the above-described in-plane sensitization signal VAL and an addition pixel described later are input from the output terminal 604. A composite signal MIX combined with the pattern code PAT is output.
The region selection unit 22 and the selection addition unit 23 receive the horizontal synchronization signal HD and the vertical synchronization signal VD output from the synchronization signal generation unit 11 of FIG.
Information indicating the sensitization magnification La output from the control unit 12 of FIG. 1 is also input to the selective adding unit 23.

  The imaging signal Pc supplied to the pixel extraction unit 21 via the input terminal 601 represents the pixel values of the R, G, and B pixels that are Bayer-arrayed as described above.

The pixel extraction unit 21 includes the same color filter as that of the pixel of interest, which is positioned in the vicinity of each other in the same frame, by delaying the input imaging signal Pc by different times (detects light of the same color component). Pixel values of a plurality of pixels are extracted simultaneously. In this case, the pixel located at the center of the plurality of pixels is treated as a target pixel, and the other pixels are treated as reference pixels. Therefore, the above extraction process is performed by a plurality of pixels that are located around the pixel of interest in the same frame as the pixel of interest and that have the same color filter as the pixel of interest (detect light of the same color component). It can be said that this is a process of simultaneously extracting pixel values of pixels (reference pixels). Since the extracted pixels are sequentially switched, the above processing also sequentially designates the pixels in the frame, and includes the designated pixels and the surrounding color filter having the same color filter as the designated pixels (the same It can also be said to be a process of simultaneously detecting pixel values of a plurality of pixels.
A plurality of pixels having the same color filter as the pixel of interest (detecting light of the same color component) outputs an imaging signal having the same color component as the pixel of interest.

The region selection unit 22 receives a signal representing the pixel values of the target pixel and the surrounding reference pixels extracted by the pixel extraction unit 21, and includes a combination of the target pixel and one or more pixels located in the vicinity thereof, Accordingly, a plurality of pixel regions constituted by the combination are formed, and a pixel region composed of pixels having high correlation among the plurality of formed pixel regions is selected and output. Since each pixel region includes a pixel of interest, it is possible to select a pixel region composed of pixels having high correlation with the pixel of interest by selecting a pixel region composed of pixels having high correlation with each other. In order to select a pixel region composed of pixels having high correlation between each other, a combination having a minimum difference between the maximum value and the minimum value of the pixels in each pixel region is selected.
Information (addition pixel position information) POS indicating the position of the pixel constituting the selected pixel region is output to the selection / addition unit 23.

  Along with the addition pixel position information POS, information indicating the pattern (type) of the selected pixel region (for example, information indicating which of a plurality of predetermined types belongs) is generated as the addition pixel pattern code PAT. And output to the signal coupling unit 24. The pattern (type) referred to here is defined by, for example, the relative position or orientation of the selected region with respect to the target pixel, and is further defined by the shape of the region.

  For example, a plurality of pixel regions having different patterns, for example, those having different relative positions or orientations with respect to the target pixel are prepared, or a plurality of ones having different shapes along with the relative orientations or orientations are prepared. Among the plurality of pixel regions, the one having the highest correlation is selected, and information indicating the pattern of the selected region is output as the added pixel pattern code PAT.

  The selection / addition unit 23 receives the imaging signal extracted by the pixel extraction unit 21, and designates the imaging signal of each pixel of interest by the pixels included in the pixel region selected by the region selection unit 22, that is, addition pixel position information POS. The imaging signals of the added pixels are added, and the addition result is output as the in-plane sensitization signal VAL. The in-plane sensitization signal VAL output from the selective adding unit 23 is supplied to the signal combining unit 24.

The signal combination unit 24 combines the in-plane sensitization signal VAL output from the selection addition unit 23 and the addition pixel pattern code PAT output from the region selection unit 22 into one signal to generate a composite signal MIX. Output.
For example, when the in-plane sensitization signal VAL is 12 bits and the added pixel pattern code PAT is 4 bits, the composite signal MIX is 16 bits.
The composite signal MIX generated by the signal combining unit 24 is supplied to the inter-plane pixel adding unit 30 via the output terminal 604.

  Hereinafter, the pixel extraction unit 21, the region selection unit 22, the selection addition unit 23, and the signal combination unit 24 will be described in detail.

The pixel extraction unit 21 is configured as shown in FIG. 8, for example.
In FIG. 8, “1L-DL” indicates a 1-line delay unit, “2L-DL” indicates a 2-line delay unit, “1D-DL” indicates a 1-pixel delay unit, and “2D-DL” indicates 2 A pixel delay part is shown. “4D-DL” indicates a 4-pixel delay unit.

  The pixel extraction unit 21 includes two-line delay units 511 and 512, one-line delay units 522 to 525, four-pixel delay units 530 and 531, two-pixel delay units 532 to 537, one-pixel delay units 542 to 545, 547 to 550, 552 to 555, 557 to 560, and 562 to 565 are connected as shown in the figure, and the imaging signals Pc are delayed by different times to output signals of pixels located in the vicinity of each other. To do.

  Of the signals output from the pixel extraction unit 21, the signal output from the delay unit 553 is used as the signal of the target pixel P33 (FIG. 2). If the signal from the delay unit 553 is the signal of the target pixel P33 (FIG. 2), the imaging signal Pc input to the input terminal 601 at this time is the signal of the pixel PRB (FIG. 2), and the target pixel P33 and its surroundings Pixel values (pixel signals representing pixel values) of the pixels P11 to P55, PL3, P3T, P3B, and PR1 to PR5 are simultaneously output.

  Hereinafter, each pixel, the pixel value of the pixel, and the pixel signal are represented by the same symbol. For example, the pixel value and pixel signal of the pixel P33 are also denoted by reference numeral P33.

Hereinafter, processing by each delay unit of the pixel extraction unit 21 will be described in detail.
The pixel signal PRB is sequentially delayed by a two-line delay unit 511, a one-line delay unit 522 to 525, and a two-line delay unit 512, and is two, three, four, five, six, and eight lines with respect to the pixel signal PRB, respectively. Delayed pixel signals PR5, PR4, PR3, PR2, PR1, and PRT are output.
The pixel signal PRB is also delayed by the 4-pixel delay unit 530 and output as the pixel signal P3B.

The pixel signal PR5 output from the two-line delay unit 511 is delayed by two pixels by the two-pixel delay unit 532, and further sequentially delayed by one pixel by the one-pixel delay units 542 to 545. Pixel signals P55, P45, P35, P25, and P15 delayed by 3, 4, 5, and 6 pixels are output.
The pixel signal PR4 output from the one-line delay unit 522 is delayed by two pixels by the two-pixel delay unit 533, and further sequentially delayed by one pixel by the one-pixel delay units 547 to 550, so that the pixel signal PR4 is 2 for each pixel signal PR4. Pixel signals P54, P44, P34, P24, and P14 delayed by 3, 4, 5, and 6 pixels are output.
The pixel signal PR3 output from the 1-line delay unit 523 is delayed by 2 pixels by the 2-pixel delay unit 534, further delayed by 1 pixel by the 1-pixel delay units 552 to 555, and further by 2 by the 2-pixel delay unit 535. Pixel signals P53, P43, P33, P23, P13, and PL3 delayed by 2, 3, 4, 5, 6, and 8 pixels with respect to the pixel signal PR3 are output.

The pixel signal PR2 output from the one-line delay unit 524 is delayed by two pixels by the two-pixel delay unit 536, and further sequentially delayed by one pixel by the one-pixel delay units 557 to 560, so that each pixel signal PR2 is two. Pixel signals P52, P42, P32, P22, and P12 delayed by 3, 4, 5, and 6 pixels are output.
The pixel signal PR1 output from the one-line delay unit 525 is delayed by two pixels by the two-pixel delay unit 537, and further sequentially delayed by one pixel by the one-pixel delay units 562 to 565, so that each pixel signal PR1 is two. Pixel signals P51, P41, P31, P21, and P11 delayed by 3, 4, 5, and 6 pixels are output.
The pixel signal PRT output from the 2-line delay unit 512 is delayed by the 4-pixel delay unit 531 and output as the pixel signal P3T.

  The pixel signals P55 to P11, P3B, PR3, PL3, and P3T represent the pixel values of the pixels P55 to P11, P3B, PR3, PL3, and P3T in FIG. Simultaneously output from the pixel extracting unit 21 and supplied to the region selecting unit 22 and the selective adding unit 23.

  For example, as illustrated in FIG. 9, the region selection unit 22 includes a pixel selection unit 570, change width calculation units 571 to 582, a minimum value calculation unit 585, and a pixel designation unit 587.

  The horizontal synchronization signal HD and the vertical synchronization signal VD output from the synchronization signal generation unit 11 in FIG. 1 are supplied to the pixel selection unit 570, the minimum value calculation unit 585, and the pixel designation unit 587.

The pixel selection unit 570 determines the pixel position of the target pixel P33 based on the horizontal synchronization signal HD and the vertical synchronization signal VD, and specifies the pixel position on the color filter array of the target pixel. The pixel selection unit 570 also specifies whether the target pixel is an R pixel, a G pixel, or a B pixel.
Then, the pixel signals P55 to P11, P3B, PR3, PL3, and P3T supplied from the pixel extraction unit 21 are received, and a plurality of combinations (or pixel regions) including the target pixel and its surrounding pixels are generated. A plurality of such combinations are generated for each pixel of interest.

  If a pixel having a high correlation with the target pixel can be correctly selected as a pixel used for pixel addition in the selection / addition unit 23, resolution degradation of the image after pixel addition can be reduced. Therefore, the region selection unit 22 prepares or forms a plurality of combinations of the pixel of interest and its surrounding pixels as described above, and obtains a high correlation for each combination. Pixel addition is performed using pixels belonging to the combination having the highest correlation.

  For each combination, the level of correlation is evaluated based on the change width of the pixel value given by the difference between the maximum value (maximum pixel value) and the minimum value (minimum pixel value) among the pixel values of the pixels belonging to each combination. Is done. That is, the combination having the smallest change width is selected as the combination having the highest correlation.

  FIG. 10A to FIG. 12D show patterns of combinations of four pixels when performing four-pixel addition when the pixel of interest is a G pixel in the pixel space arrangement of FIG. In the case of four-pixel addition of G pixels, a combination pattern having the highest correlation among twelve combination patterns is obtained.

  FIG. 10A illustrates the target pixel and its surrounding pixels, that is, the target pixel G33, the pixel G31 two lines before the target pixel, the pixel G22 one pixel before the target pixel, and the target pixel. The combination GP1 of the upper block pattern constituted by the pixel G42 after one pixel before one line is shown.

  FIG. 10B shows the target pixel and its peripheral pixels on the right side, that is, the target pixel G33, the pixel G53 two pixels after the target pixel, the pixel G42 one pixel before the target pixel one line, and the target pixel A combination GP2 of the right block pattern constituted by the pixel G44 one pixel after one line is shown.

  FIG. 10C shows the target pixel and its peripheral pixels, that is, the target pixel G33, the pixel G13 two pixels before the target pixel, the pixel G22 one pixel before the target pixel, and the target pixel. A combination GP3 of the left block pattern constituted by the pixel G24 one pixel before one line is shown.

  FIG. 10D shows the target pixel and its lower peripheral pixels, that is, the target pixel G33, the pixel G35 two lines after the target pixel, the pixel G24 one pixel before the target pixel one line, and the target pixel. A combination GP4 of the lower block pattern constituted by the pixel G44 after one pixel after the first line is shown.

  FIG. 11A shows the target pixel and the surrounding pixels above it, that is, the target pixel G33, the pixel G3T four lines before the target pixel, the pixel G31 two lines before the target pixel, and two lines after the target pixel. A combination GP5 of the upper vertical line pattern constituted by the pixels G35 is shown.

  FIG. 11B shows the target pixel and its lower peripheral pixels, that is, the target pixel G33, the pixel G3B four lines after the target pixel, the pixel G35 two lines after the target pixel, and two lines before the target pixel. A combination GP6 of the lower vertical line pattern constituted by the pixels G31 is shown.

  FIG. 11C shows the target pixel and its peripheral pixels on the left side, that is, the target pixel G33, the pixel GL3 four pixels before the target pixel, the pixel G13 two pixels before the target pixel, and two pixels after the target pixel. A left side line pattern combination GP7 composed of pixels G53 is shown.

  FIG. 11D shows the target pixel and its neighboring pixels on the right side, that is, the target pixel G33, the pixel GR3 four pixels after the target pixel, the pixel G53 two pixels after the target pixel, and two pixels before the target pixel. A right side line pattern combination GP8 composed of pixels G13 is shown.

  FIG. 12A shows the target pixel and its upper left peripheral pixel, that is, the target pixel G33, the pixel G11 two pixels before the target pixel two lines before, and the pixel G22 one pixel before the target pixel one line before , And a combination GP9 of the upper left diagonal line pattern constituted by the pixel G44 one pixel after the one line after the target pixel.

  FIG. 12B shows the target pixel and its lower right peripheral pixels, that is, the target pixel G33, the pixel G55 two pixels after the second line of the target pixel, and the pixel one pixel after the first line of the target pixel. A combination GP10 of the lower right diagonal line pattern composed of G44, the pixel one pixel before the target pixel and G22, and G22 is shown.

  FIG. 12C shows the target pixel and its peripheral pixels on the upper right side, that is, the target pixel G33, the pixel G51 two pixels before the target pixel two lines before, and the pixel G42 one pixel after the target pixel one line before , And a combination GP11 of the diagonal line pattern on the upper right side constituted by the pixel G24 one pixel before and one pixel after the target pixel.

  FIG. 12D shows the target pixel and the lower left peripheral pixel, that is, the target pixel G33, the pixel G15 two pixels before the two lines after the target pixel, and the pixel G24 one pixel before the one line after the target pixel. , And a combination GP12 of a diagonal line pattern on the lower left side composed of a pixel G42 one pixel before and one pixel before the target pixel.

  The pixel selection unit 570 sets the combinations GP1 to GP12 as the first to twelfth combinations AP1 to AP12, respectively, and sets the pixel values of the pixels constituting these to the first to twelfth change width calculation units 571 to 582, respectively. Supply.

  FIGS. 13A to 13D show combinations of four pixels when four-pixel addition is performed when the pixel of interest is an R pixel in the pixel space arrangement of FIG. In the case of 4-pixel addition of R pixels, a combination pattern having the highest correlation among the four combination patterns is obtained.

  FIG. 13A shows the target pixel and its upper left peripheral pixel, that is, the target pixel R33, the pixel R31 two lines before the target pixel, the pixel R11 two pixels before the target pixel two lines, and the target pixel. This shows a combination RP1 of the upper left block pattern composed of the pixel R13 two pixels before.

  FIG. 13B shows the target pixel and its peripheral pixels on the upper right side, that is, the target pixel R33, the pixel R31 two lines before the target pixel, the pixel R51 two pixels after the two lines before the target pixel, and the target pixel. The combination RP2 of the upper right side block pattern composed of the pixel R53 after the second pixel is shown.

  FIG. 13C shows the target pixel and its lower left peripheral pixels, that is, the target pixel R33, the pixel R13 two pixels before the target pixel, the pixel R35 two lines after the target pixel, and two lines after the target pixel. A combination RP3 of the lower left block pattern constituted by the pixel R15 two pixels before is shown.

  FIG. 13D shows the target pixel and its lower right peripheral pixels, that is, the target pixel R33, a pixel R53 two pixels after the target pixel, a pixel R35 two lines after the target pixel, and two lines of the target pixel. A combination RP4 of the lower right block pattern constituted by the pixel R55 after the subsequent two pixels is shown.

  The pixel selection unit 570 uses the combinations PR1 to PR4 as the first to fourth combinations AP1 to AP4, respectively, and the pixel values of the pixels constituting these to the first to fourth change width calculation units 571 to 574, respectively. Supply.

  FIGS. 14A to 14D show patterns of combinations of four pixels when four-pixel addition is performed when the pixel of interest is a B pixel in the pixel space arrangement of FIG. In the case of four-pixel addition of B pixels, a combination pattern having the highest correlation among the four combination patterns is obtained.

  FIG. 14A shows the target pixel and its upper left peripheral pixel, that is, the target pixel B33, the pixel B31 two lines before the target pixel, the pixel B11 two pixels before the second line of the target pixel, and the target pixel. A combination BP1 of the upper left block pattern constituted by the pixel B13 two pixels before is shown.

  FIG. 14B shows the target pixel and its peripheral pixels on the upper right side, that is, the target pixel B33, the pixel B31 two lines before the target pixel, the pixel B51 two pixels before the second line of the target pixel, and the target pixel. The combination BP2 of the upper right side block pattern constituted by the pixel B53 after two pixels is shown.

  FIG. 14C shows the target pixel and the lower left peripheral pixel, that is, the target pixel B33, the pixel B13 two pixels before the target pixel, the pixel B35 two lines after the target pixel, and two lines after the target pixel. This shows a combination BP3 of the lower left block pattern composed of the pixel B15 two pixels before.

  FIG. 14D shows the target pixel and its lower right peripheral pixels, that is, the target pixel B33, a pixel B53 two pixels after the target pixel, a pixel B35 two lines after the target pixel, and two lines of the target pixel. A combination BP4 of the lower right block pattern constituted by the pixel B55 after the subsequent two pixels is shown.

  The pixel selection unit 570 sets the combinations PB1 to PB4 as the first to fourth combinations AP1 to AP4, respectively, and sets the pixel values of the pixels constituting these to the first to fourth change width calculation units 571 to 574, respectively. Supply.

  As described above, the pixel combination prepared when the target pixel is a G pixel, and the shape of the region constituting the combination are divided into a block shape (diamond shape) and a line shape (band shape), and the block Is divided according to whether the center of the region is on the upper side, right side, left side, or lower side with respect to the pixel of interest, and for the line shape, the center of the region is the pixel of interest. On the other hand, it is classified according to which of the upper side, the lower side, the left side, the right side, the diagonal left upper side, the diagonal lower right side, the diagonal upper right side, and the diagonal lower left side.

  Therefore, as the added pixel pattern code PAT, the selected combination is any one of FIG. 10A to FIG. 12D, that is, the shape of the region composed of the pixel of interest and the pixels located in the vicinity thereof, , Whether it is block-shaped (diamond-shaped) or line-shaped (band-shaped), and the relative position or orientation of the center of the region with respect to the designated pixel, ie, block-shaped Indicates the side of the top, right, left, or bottom, and for a line type, indicates the top, right, left, bottom, diagonal upper left, diagonal upper right, diagonal lower left, or diagonal upper right side Things are used.

  On the other hand, in the combination of pixels prepared when the target pixel is an R pixel or a B pixel, the shapes of the regions constituting the combination are all block-shaped, and the center of the region is the target pixel. On the other hand, they are classified according to whether they are on the upper left side, the upper right side, the lower left side, or the lower right side.

  Therefore, as the added pixel pattern code PAT, the selected combination is any one of FIGS. 13A to 13D or FIGS. 14A to 14D, that is, the pixel of interest and the periphery thereof. A relative position or azimuth of the center of a region made up of pixels with respect to a designated pixel, that is, one that indicates which side is diagonally upper left, diagonally upper right, diagonally lower left, or diagonally lower right is used.

  However, the present invention is not limited to the above example. That is, the shape of the region composed of the pixel of interest and its surrounding pixels, and the orientation of the center of the region with respect to the pixel of interest may be other than those shown in FIGS. 10 (a) to 14 (d). good. In any case, the added pixel pattern code PAT only needs to indicate the relative position or orientation of the region of the pixel of interest and its surrounding pixels with respect to the pixel of interest.

The first to twelfth change width calculation units 571 to 582 calculate the difference between the maximum pixel value and the minimum pixel value as the change width for the input first to twelfth combinations AP1 to AP12, respectively.
That is, the change width calculation units 571 to 582 compare the input pixel values of the four pixels to obtain the maximum pixel value and the minimum pixel value. Next, the difference between the maximum pixel value and the minimum pixel value is obtained and supplied to the minimum value calculation unit 585 as the change width of the combination (pixel area).

When the target pixel is a G pixel,
The change width calculation unit 571 calculates the change width of the combination of the upper block pattern GP1 input as the first combination AP1, and outputs the change width as the first change width WP1.
The change width calculation unit 572 calculates the change width of the combination of the right block pattern GP2 input as the second combination AP2, and outputs the change width as the second change width WP2.
The change width calculation unit 573 calculates the change width of the combination of the left block pattern GP3 input as the third combination AP3, and outputs the change width as the third change width WP3.
The change width calculation unit 574 calculates the change width of the combination of the lower block pattern GP4 input as the fourth combination AP4, and outputs the change width as the fourth change width WP4.
The change width calculation unit 575 calculates the change width of the combination of the upper vertical line pattern GP5 input as the fifth combination AP5, and outputs the change width as the fifth change width WP5.
The change width calculation unit 576 calculates the change width of the combination of the lower vertical line pattern GP6 input as the sixth combination AP6, and outputs the change width as the sixth change width WP6.
The change width calculation unit 577 calculates the change width of the combination of the left horizontal line pattern GP7 input as the seventh combination AP7, and outputs the change width as the seventh change width WP7.
The change width calculation unit 578 calculates the change width of the combination of the right lateral line pattern GP8 input as the eighth combination AP8, and outputs the change width as the eighth change width WP8.
The change width calculation unit 579 calculates the change width of the combination of the upper left oblique line pattern GP9 input as the ninth combination AP9, and outputs the change width as the ninth change width WP9.
The change width calculation unit 580 calculates the change width of the lower right diagonal line pattern GP10 input as the tenth combination AP10, and outputs the change width as the tenth change width WP10.
The change width calculation unit 581 calculates the change width of the combination of the upper right diagonal line pattern GP11 input as the eleventh combination AP11, and outputs the change width as the eleventh change width WP11.
The change width calculation unit 582 calculates the change width of the combination of the lower left oblique line pattern GP12 input as the twelfth combination AP12, and outputs it as the twelfth change width WP12.

When the target pixel is an R pixel or a B pixel,
The change width calculation unit 571 calculates the change width of the combination of the upper left block pattern RP1 or BP1 input as the first combination AP1, and outputs the change width as the first change width WP1.
The change width calculation unit 572 calculates the change width of the combination of the upper right block pattern RP2 or BP2 input as the second combination AP2, and outputs the change width as the second change width WP2.
The change width calculation unit 573 calculates the change width of the combination of the lower left block pattern RP3 or BP3 input as the third combination AP3, and outputs the change width as the third change width WP3.
The change width calculation unit 574 calculates the change width of the combination of the lower right block pattern RP4 or BP4 input as the fourth combination AP4 and outputs it as the fourth change width WP4.
The change width calculation units 575 to 582 do not perform a change width calculation operation.

An operation when the target pixel of the minimum value calculation unit 585 is a G pixel will be described.
The minimum value calculation unit 585 receives the first to twelfth change widths WP1 to WP12 output from the change width calculation units 571 to 582, obtains the minimum one of them, and determines a combination pattern (addition) with the minimum change width. Information indicating the pixel pattern) (added pixel pattern code) PAT is notified to the pixel specifying unit 587 and the signal combining unit 24.
The pixel designating unit 587 supplies information (added pixel position information) POS indicating the pixel positions of the pixels constituting the combination specified by the added pixel pattern code PAT notified from the minimum value calculating unit 585 to the selective adding unit 23. .

When the change width of the combination of the upper block pattern GP1 is the smallest, the position information POS of the pixels G31, G22, G42, and G33 is supplied to the selection adding unit 23.
When the change width of the combination of the right block pattern GP2 is the smallest, the position information POS of the pixels G42, G33, G53, and G44 is supplied to the selection adding unit 23.
When the change width of the combination of the left block pattern GP3 is the smallest, the position information POS of the pixels G22, G13, G33, and G24 is supplied to the selection adding unit 23.
When the change width of the combination of the lower block pattern GP4 is the smallest, the position information POS of the pixels G33, G24, G44, and G35 is supplied to the selection adding unit 23.

When the change width of the combination of the upper vertical line pattern GP5 is the smallest, the position information POS of the pixels G3T, G31, G33, and G35 is supplied to the selection adding unit 23.
When the change width of the combination of the lower vertical line pattern GP6 is the smallest, the position information POS of the pixels G31, G33, G35, and G3B is supplied to the selection adding unit 23.
When the change width of the combination of the left lateral line pattern GP7 is the smallest, the position information POS of the pixels GL3, G13, G33, and G53 is supplied to the selection adding unit 23.
When the change width of the combination of the right lateral line pattern GP8 is the smallest, the position information POS of the pixels G13, G33, G53, and GR3 is supplied to the selection adding unit 23.

When the change width of the combination of the upper left diagonal line pattern GP9 is the smallest, the position information POS of the pixels G11, G22, G33, and G44 is supplied to the selection adding unit 23.
When the change width of the combination of the lower right diagonal line pattern GP10 is the smallest, the position information POS of the pixels G22, G33, G44, and G55 is supplied to the selection adding unit 23.
When the change width of the combination of the upper right side oblique line pattern GP11 is the smallest, the position information POS of the pixels G51, G42, G33, and G24 is supplied to the selection adding unit 23.
When the change width of the combination of the lower left oblique line pattern GP12 is the smallest, the position information POS of the pixels G42, G33, G24, and G15 is supplied to the selection adding unit 23.

  By such processing, the correlation area detection unit 590 configured by the minimum value calculation unit 585 and the pixel specification unit 587 outputs the first to twelfth change widths WP1 to WP12 output from the change width calculation units 571 to 582. In response, the pixel area composed of the pixels constituting the combination with the smallest change width is determined as the pixel area having the highest correlation, and information indicating the position of the area (the position of the pixels constituting the area) (addition pixel position) Information) POS is supplied to the selective adding unit 23, and information indicating the pattern (type) of the minimum combination (the minimum combination corresponds to any of the patterns in FIGS. 10A to 12D). Is supplied to the signal combining unit 24 as the added pixel pattern code PAT.

An operation when the target pixel of the minimum value calculation unit 585 is an R pixel will be described.
The minimum value calculation unit 585 receives the first to fourth change widths WP1 to WP4 output from the change width calculation units 571 to 574, obtains the minimum one of them, and determines a combination pattern (addition) with the minimum change width. Information indicating the pixel pattern) (added pixel pattern code) PAT is notified to the pixel specifying unit 587 and the signal combining unit 24.
The pixel designating unit 587 supplies information (added pixel position information) POS indicating the pixel positions of the pixels constituting the combination specified by the added pixel pattern code PAT notified from the minimum value calculating unit 585 to the selective adding unit 23. .

When the change width of the combination of the upper left block pattern RP1 is the smallest, the position information POS of the pixels R11, R31, R13, and R33 is supplied to the selection adding unit 23.
When the change width of the combination of the upper right block pattern RP2 is the smallest, the position information POS of the pixels R31, R51, R33, and R53 is supplied to the selection adding unit 23.
When the change width of the combination of the lower left block pattern RP3 is the smallest, the position information POS of the pixels R13, R33, R15, and R35 is supplied to the selection adder 23.
When the change width of the combination of the lower right block pattern RP4 is the smallest, the position information POS of the pixels R33, R53, R35, and R55 is supplied to the selection adding unit 23.

  By such processing, the correlation area detection unit 590 receives the first to fourth change widths WP1 to WP4 output from the change width calculation units 571 to 574, and includes pixels that form a combination with the smallest change width. The pixel area is determined to be the pixel area having the highest correlation, and information (added pixel position information) POS indicating the position of the area (the position of the pixels constituting the area) is supplied to the selective addition unit 23, and the minimum The signal combination unit 24 uses the information indicating the combination pattern (type) (information indicating which of the patterns in FIGS. 13A to 13D is the minimum combination) as the added pixel pattern code PAT. To supply.

An operation when the target pixel of the minimum value calculation unit 585 is a B pixel will be described.
The minimum value calculation unit 585 receives the first to fourth change widths WP1 to WP4 output from the change width calculation units 571 to 574, obtains the minimum one of them, and determines a combination pattern (addition) with the minimum change width. Information indicating the pixel pattern) (added pixel pattern code) PAT is notified to the pixel specifying unit 587 and the signal combining unit 24.
The pixel designating unit 587 supplies information (added pixel position information) POS indicating the pixel positions of the pixels constituting the combination specified by the added pixel pattern code PAT notified from the minimum value calculating unit 585 to the selective adding unit 23. .

When the change width of the combination of the upper left block pattern BP1 is the smallest, the position information POS of the pixels B11, B31, B13, and B33 is supplied to the selection adding unit 23.
When the change width of the combination of the upper right block pattern BP2 is the smallest, the position information POS of the pixels B31, B51, B33, and B53 is supplied to the selection adding unit 23.
When the change width of the combination of the lower left block pattern BP3 is the smallest, the position information POS of the pixels B13, B33, B15, and B35 is supplied to the selection adding unit 23.
When the change width of the combination of the lower right block pattern BP4 is the smallest, the position information POS of the pixels B33, B53, B35, and B55 is supplied to the selection adding unit 23.

  By such processing, the correlation area detection unit 590 receives the first to fourth change widths WP1 to WP4 output from the change width calculation units 571 to 574, and includes pixels that form a combination with the smallest change width. The pixel area is determined to be the pixel area having the highest correlation, and information (added pixel position information) POS indicating the position of the area (the position of the pixels constituting the area) is supplied to the selective addition unit 23, and the minimum The signal combination unit 24 uses information indicating the pattern (type) of the combination (information indicating which of the patterns in FIGS. 14A to 14D is the minimum combination) as an added pixel pattern code PAT. To supply.

  As described above, since a configuration having the highest correlation is obtained from 12 combinations in the case of G pixels, and from 4 combinations in the case of R pixels and B pixels, as pixels used for pixel addition, A pixel having a high correlation with the target pixel can be correctly selected, and resolution degradation of the image after pixel addition can be reduced.

As described above, when the G pixel is the target pixel and the R pixel or the B pixel is the target pixel in the same pixel selection unit 570, change width calculation units 571 to 582, and minimum value calculation unit 585. The same pixel selection unit in which the G pixel is used as the target pixel instead of using a part of the output pixel when the R pixel or the B pixel is used as the target pixel. The change width calculation unit and the minimum value calculation unit may be provided separately from the pixel selection unit, the change width calculation unit, and the minimum value calculation unit when the R pixel or the B pixel is the target pixel.

  Next, the selective addition unit 23 will be described with reference to FIG. As illustrated in FIG. 15, the selection / addition unit 23 includes a pixel selection unit 593 and a pixel addition unit 595.

  Pixel signals P55 to P11, P3B, PR3, PL3, and P3T extracted by the pixel extraction unit 21 are supplied to the pixel selection unit 593.

  The added pixel position information POS notified from the pixel designating unit 587 of the region selecting unit 22 and the horizontal synchronizing signal HD and the vertical synchronizing signal VD output from the synchronizing signal generating unit 11 of FIG. 1 are also supplied to the pixel selecting unit 593. The

The pixel selection unit 593 determines the pixel position of the target pixel P33 based on the horizontal synchronization signal HD and the vertical synchronization signal VD, and specifies the pixel position on the color filter array of the target pixel.
The pixel selection unit 593 also specifies whether the target pixel is an R pixel, a G pixel, or a B pixel. Based on the added pixel position information POS from the pixel specifying unit 587, the pixel positions of the four pixels constituting the selection area are also specified.

  The pixel selection unit 593 supplies the pixel values Ps1 to Ps4 of the four pixels at the pixel position indicated by the addition pixel position information POS to the pixel addition unit 595.

  A signal indicating the sensitization magnification La (La = 1 to 4) output from the control unit 12 is supplied to the pixel addition unit 595.

The pixel addition unit 595 adds the pixel values of the four pixels supplied from the pixel selection unit 593, and combines the value Pe obtained as a result of the addition as an in-plane sensitization signal VAL via the output terminal 606. To the unit 24 . In this addition, an addition coefficient (weighting coefficient) is applied to the pixel value before the addition so that the addition result has a predetermined sensitization factor La.
For example, if the pixel values of four pixels are Ps1, Ps2, Ps3, Ps4 and the sensitization magnification is La,
Pe = (Ps1 + Ps2 + Ps3 + Ps4) × La / 4
The in-plane sensitization signal (value) Pe is obtained by the calculation represented by: In the following description, the in-plane sensitization signal for G pixel is Ge, the in-plane sensitization signal for R pixel is Re, and the in-plane sensitization signal for B pixel is Value) is denoted by Be.

An operation when the pixel of interest is a G pixel will be described.
When the combination of the upper block pattern GP1 as the first combination AP1 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (G31 + G22 + G42 + G33) × La / 4
When the combination of the right block pattern GP2 as the second combination AP2 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (G42 + G33 + G53 + G44) × La / 4
When the combination of the left block pattern GP3 as the third combination AP3 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (G22 + G13 + G33 + G24) × La / 4
When the combination of the lower block pattern GP4 as the fourth combination AP4 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (G33 + G24 + G44 + G35) × La / 4

When the combination of the upper vertical line pattern GP5 as the fifth combination AP5 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (G3T + G31 + G33 + G35) × La / 4
When the combination of the lower vertical line pattern GP6 as the sixth combination AP6 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (G31 + G33 + G35 + G3B) × La / 4
When the combination of the left horizontal line pattern GP7 as the seventh combination AP7 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (GL3 + G13 + G33 + G53) × La / 4
When the combination of the right lateral line pattern GP8 as the eighth combination AP8 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (G13 + G33 + G53 + GR3) × La / 4

When the combination of the upper left diagonal line pattern GP9 as the ninth combination AP9 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (G11 + G22 + G33 + G44) × La / 4
When the combination of the lower right diagonal line pattern GP10 as the tenth combination AP10 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (G22 + G33 + G44 + G55) × La / 4
When the combination of the upper right diagonal line pattern GP11 as the eleventh combination AP11 is selected, the pixel addition unit 595 performs the following calculation.
Ge = (G51 + G42 + G33 + G24) × La / 4
When the combination of the lower left diagonal line pattern GP12 as the twelfth combination AP12 is selected, the pixel addition unit 595 performs the following calculation.
G = (G42 + G33 + G24 + G15) × La / 4

An operation when the target pixel is an R pixel will be described.
When the combination of the upper left block pattern RP1 as the first combination AP1 is selected, the pixel addition unit 595 performs the following calculation.
Re = (R11 + R31 + R13 + R33) × La / 4
When the combination of the upper right block pattern RP2 as the second combination AP2 is selected, the pixel addition unit 595 performs the following calculation.
Re = (R31 + R51 + R33 + R53) × La / 4
When the combination of the lower left block pattern RP3 as the third combination AP3 is selected, the pixel addition unit 595 performs the following calculation.
Re = (R13 + R33 + R15 + R35) × La / 4
When the combination of the lower right block pattern RP4 as the fourth combination AP4 is selected, the pixel addition unit 595 performs the following calculation.
Re = (R33 + R53 + R35 + R55) × La / 4

An operation when the pixel of interest is a B pixel will be described.
When the combination of the upper left block pattern BP1 as the first combination AP1 is selected, the pixel addition unit 595 performs the following calculation.
Be = (B11 + B31 + B13 + B33) × La / 4
When the combination of the upper right side block pattern BP2 as the second combination AP2 is selected, the pixel addition unit 595 performs the following calculation.
Be = (B31 + B51 + B33 + B53) × La / 4
When the combination of the lower left block pattern BP3 as the third combination AP3 is selected, the pixel addition unit 595 performs the following calculation.
Be = (B13 + B33 + B15 + B35) × La / 4
When the combination of the lower right block pattern BP4 as the fourth combination AP4 is selected, the pixel addition unit 595 performs the following calculation.
Be = (B33 + B53 + B35 + B55) × La / 4

  In the above example, for the G pixel, the pixel addition is configured to use a combination pattern assuming an image including a subject with high resolution, such as a vertical, horizontal, diagonal line pattern, and block pattern. They can be used for addition. For this reason, even when a pixel including a scene including a subject including a fine pattern or fine irregularities is added, there is an effect of preventing the high-resolution portion from blurring.

In the above example, only the combination of the block pattern is used for the R pixel and the B pixel. However, as the combination of the R pixel and the B pixel, the combination of the vertical, horizontal, and diagonal line patterns is also used as in the G pixel. You may comprise so that a correlation may be determined about these.
However, in the case of the R pixel and the B pixel, since the distance between the pixels to be added is longer than that of the G pixel, the risk of erroneous determination of the correlation determination is increased, the circuit scale is increased, and the human scale is increased. The combination pattern is determined by comprehensively judging that the sensitivity to the color change is lower than the luminance.

  In the above example, three neighboring pixels are added to the target pixel. However, a combination including more neighboring pixels in the addition target pixel may be used. By doing so, it is possible to further increase the sensitivity.

  Further, for the G pixel, not all of the above 12 combinations of patterns may be used but only a part thereof. For example, only the four patterns shown in FIGS. 10A to 10D may be used, or only the four patterns shown in FIGS. 11A to 11D may be used. Only the four patterns shown in (d) may be used.

  Also, one type of combination pattern for in-plane pixel addition may be used corresponding to the Bayer arrangement of R and B pixels. In this case, in the inter-plane pixel addition described later, the addition target pixel of the neighboring frame is selected only by the correlation determination of the pixel value without determining whether or not the addition pixel pattern codes match.

Next, the operation of the inter-plane pixel addition unit 30 will be described with reference to FIG.
As illustrated in FIG. 16, the inter-plane pixel addition unit 30 includes a pixel extraction unit 31, a signal separation unit 350, an addition pixel selection unit 35, and a pixel addition unit 39. The addition pixel selection unit 35 includes first to fourth pixel selection units 351 to 354.

  The composite signal MIX for each pixel in the continuous frame from the in-plane pixel addition unit 20 is sequentially output from the output terminal 604 of the in-plane pixel addition unit 20, and the pixel extraction unit via the input terminal 605 of the inter-plane pixel addition unit 30. 31 are sequentially input.

  The pixel extraction unit 31 delays the composite signal MIX input to the input terminal 605 for different periods, and simultaneously extracts the composite signal MIX for a plurality of pixels in a plurality of frames located near each other. In this case, the one located in the center of the plurality of frames is treated as the target frame, one pixel in the target frame is extracted as the target pixel, and the same position as the target pixel of the target frame is extracted from each of the other frames. And one or more pixels located in the vicinity thereof are extracted. Therefore, the above extraction processing includes the composite signal MIX for the target pixel in the target frame, and the composite signal MIX for one or more pixels (reference pixels) in the frame (reference frame) located in the vicinity of the target frame. It can be said that it is the process which extracts simultaneously. Since the extracted frames and pixels are sequentially switched, the above processing also sequentially designates consecutive frames as attention frames, sequentially designates pixels in the attention frames as attention pixels, and combines signals for the attention pixels of the attention frames. It can also be said that this is a process of simultaneously extracting MIX and the composite signal MIX for the reference pixels in the frame located in the vicinity of the frame of interest.

Hereinafter, the configuration of the pixel extraction unit 31 will be described, the operation when the G pixel is the target pixel will be described first, and then the operation when the R pixel or the B pixel is the target pixel will be described.
The pixel extraction unit 31 is configured as shown in FIGS. 17 and 18, for example.
17 and 18, “1F-DL” indicates a one-frame delay unit, “1L-DL” indicates a one-line delay unit, “1D-DL” indicates a one-pixel delay unit, and “2D-DL” "Indicates a two-pixel delay unit.

The pixel extraction unit 31 is configured by connecting one frame delay units 311 to 314, one line delay units 3201 to 3218, two pixel delay units 3301 to 3325, and one pixel delay units 3401 to 3408 as illustrated. The composite signal MIX input to the input terminal 605 is delayed for different times, and a composite signal is output for pixels in a frame located near each other.
Of the output signals, the signal output from the delay unit 3313 is used as a composite signal for the target pixel P33 (FIG. 2) of the target frame F2. When the signal from the delay unit 3313 is the signal of the target pixel P33 (FIG. 2) of the target frame F2, the composite signal MIX input to the input terminal 605 at this time is the pixel P55 (FIG. 2) of the frame F0 after two frames. The composite signal MIX for the target pixel P33 of the target frame F2, the same position P33 as the target pixel in the frames F0, F1, F3, and F4 located in the vicinity of the target frame F2, and its The composite signal MIX is simultaneously output for the pixels P31, P22, P42, P13, P53, P24, P44, and P35 located in the periphery.
F1, F3, and F4 respectively represent a frame that is one frame after the frame of interest F2 (back frame), a frame that is one frame before (frame before), and a frame that is two frames before.

  The composite signal MIX for the pixel P55 in the frame F0 may be expressed as MIX (F0, P55) for distinction. The same applies to the composite signal MIX for other pixels. The same applies to the in-plane sensitization signal VAL, the added pixel pattern code PAT, and the three-dimensional sensitization signal Pf. However, when there is no need for distinction, they are simply represented by “MIX”, “VAL”, “PAT”, and “Pf”.

Hereinafter, processing by each delay unit of the pixel extraction unit 31 will be described in detail.
The composite signal MIX (F0, P55) input to the input terminal 605 is sequentially delayed by the 1-frame delay units 311 to 314, and the signal MIX (F1, P55), the signal MIX (F2, P55), and the signal MIX (F3) , P55) and signal MIX (F4, P55).

The signal MIX (F0, P55) input to the input terminal 605 is delayed by the two-pixel delay unit 3301 to become the signal MIX (F0, P35). Further, the signal MIX (F0, P44) is delayed by the one-line delay unit 3201 and the one-pixel delay unit 3401, and further delayed by the two-pixel delay unit 3302 to become the signal MIX (F0, P24).
The output of the 1-line delay unit 3201 is delayed by the 1-line delay unit 3202 to become the signal MIX (F0, P53), delayed by the 2-pixel delay unit 3303 to become the signal MIX (F0, P33), and further 2 pixels The signal is delayed by the delay unit 3304 to become a signal MIX (F0, P13).
The output of the 1-line delay unit 3202 is delayed by the 1-line delay unit 3203 and the 1-pixel delay unit 3402 to become the signal MIX (F0, P42), and further delayed by the 2-pixel delay unit 3305 to be the signal MIX (F0, P22). It becomes.
The output of the 1-line delay unit 3203 is delayed by the 1-line delay unit 3204 and the 2-pixel delay unit 3306 to become a signal MIX (F0, P31).

The signal MIX (F1, P55) output from the one-frame delay unit 311 is delayed by the two-pixel delay unit 3307 to become the signal MIX (F1, P35). Further, the signal MIX (F1, P44) is delayed by the one-line delay unit 3205 and the one-pixel delay unit 3403, and further delayed by the two-pixel delay unit 3308 to become the signal MIX (F1, P24).
The output of the 1-line delay unit 3205 is delayed by the 1-line delay unit 3206 to become a signal MIX (F1, P53), delayed by the 2-pixel delay unit 3309 to be a signal MIX (F1, P33), and further 2 pixels The signal is delayed by the delay unit 3310 to become a signal MIX (F1, P13).
The output of the 1-line delay unit 3206 is delayed by the 1-line delay unit 3207 and the 1-pixel delay unit 3404 to become the signal MIX (F1, P42), and further delayed by the 2-pixel delay unit 3311 to be the signal MIX (F1, P22). It becomes.
The output of the 1-line delay unit 3207 is delayed by the 1-line delay unit 3208 and the 2-pixel delay unit 3312 to become a signal MIX (F1, P31).

  The signal MIX (F2, P55) output from the 1-frame delay unit 312 is delayed by the 1-line delay unit 3209, the 1-line delay unit 3210, and the 2-pixel delay unit 3313 to become the signal MIX (F2, P33).

The signal MIX (F3, P55) output from the one-frame delay unit 313 is delayed by the two-pixel delay unit 3314 to become the signal MIX (F3, P35). Further, the signal MIX (F3, P44) is delayed by the one-line delay unit 3211 and the one-pixel delay unit 3405, and further delayed by the two-pixel delay unit 3315 to become the signal MIX (F3, P24).
The output of the 1-line delay unit 3211 is delayed by the 1-line delay unit 3212 to become the signal MIX (F3, P53), and is delayed by the 2-pixel delay unit 3316 to become the signal MIX (F3, P33). The signal is delayed by the delay unit 3317 to become a signal MIX (F3, P13).
The output of the 1-line delay unit 3212 is delayed by the 1-line delay unit 3213 and the 1-pixel delay unit 3406 to become the signal MIX (F3, P42), and further delayed by the 2-pixel delay unit 3318 to be the signal MIX (F3, P22). It becomes.
The output of the 1-line delay unit 3213 is delayed by the 1-line delay unit 3214 and the 2-pixel delay unit 3319 to become a signal MIX (F3, P31).

The signal MIX (F4, P55 ) output from the one-frame delay unit 314 is delayed by the two-pixel delay unit 3320 to become the signal MIX (F4, P35). Further, the signal MIX (F4, P44) is delayed by the one-line delay unit 3215 and the one-pixel delay unit 3407, and further delayed by the two-pixel delay unit 3321 to become the signal MIX (F4, P24).
The output of the 1-line delay unit 3215 is delayed by the 1-line delay unit 3216 to become the signal MIX (F4, P53), and is delayed by the 2-pixel delay unit 3322 to become the signal MIX (F4, P33). The signal is delayed by the delay unit 3323 to become a signal MIX (F4, P13).
The output of the 1-line delay unit 3216 is delayed by the 1-line delay unit 3217 and the 1-pixel delay unit 3408 to become the signal MIX (F4, P42), and further delayed by the 2-pixel delay unit 3324 to be the signal MIX (F4, P22). It becomes.
The output of the 1-line delay unit 3217 is delayed by the 1-line delay unit 3218 and the 2-pixel delay unit 3325 to become a signal MIX (F4, P31).

  Of the signals MIX (F0, P35) to MIX (F4, P31) generated in this way, the signals MIX (F0, P35) to MIX (F0, P31) are supplied to the first pixel selection unit 351. , Signals MIX (F1, P35) to MIX (F1, P31) are supplied to the second pixel selector 352, and signals MIX (F3, P35) to MIX (F3, P31) are supplied to the third pixel selector. The signals MIX (F4, P35) to MIX (F4, P31) are supplied to the fourth pixel selection unit 354.

  When performing the processing using the R pixel or the B pixel as the target pixel, the pixel extraction unit 31 has, for example, four pixels P44, P24, P42, and P22 of each frame from the configuration illustrated in FIGS. MIX (F0, P44), MIX (F0, P24), MIX (F0, P42), MIX (F0, P22), MIX (F1, P44), MIX (F1, P24), MIX (F1, P42), MIX (F1, P22), MIX (F3, P44), MIX (F3, P24), MIX (F3, P42), MIX (F3, P22), MIX (F4, P44), MIX (F4, P24), MIX ( The parts for outputting F4, P42) and MIX (F4, P22) are not operated, or these outputs are not used.

  As described above, the same pixel extraction unit 31 is caused to perform different operations when the G pixel is the target pixel and when the R pixel or B pixel is the target pixel, or the R pixel or B pixel is the target. When a pixel is used, instead of not using a part of the output pixel, a pixel extraction unit when the G pixel is the target pixel and a pixel extraction when the R pixel or the B pixel is the target pixel The part may be provided separately.

  By determining the pixels to be used when the target pixel is each of the G pixel, R pixel, and B pixel as described above, pixels that are equidistant from the target pixel P33 in all the G pixels, R pixels, and B pixels are to be added. This makes it possible to perform pixel addition while suppressing the occurrence of false colors.

The signal separation unit 350 separates the composite signal MIX (F2, P33) for the target pixel P33 of the target frame F2 into the in-plane sensitization signal VAL (F2, P33) and the added pixel pattern code PAT (F2, P33). To do.
The addition pixel pattern code PAT (F2, P33) and the in-plane sensitization signal VAL (F2, P33) output from the signal separation unit 350 are the first to fourth pixel selection units 351 to 354 of the addition pixel selection unit 35. To be supplied.
The in-plane sensitization signal VAL (F2, P33) is also supplied to the pixel adder 39.

  Hereinafter, the configuration of the pixel selection units 351 to 354 will be described, the operation when the G pixel is the target pixel will be described first, and then the operation when the R pixel or the B pixel is the target pixel will be described. .

As illustrated in FIG. 19, the first pixel selection unit 351 includes a pattern determination unit 361 and a correlation determination unit 381.
Similarly, the second to fourth pixel selection units 352 to 354 include pattern determination units 362 to 364 and correlation determination units 382 to 384, respectively.

  The pattern discriminating units 361 to 364 respectively output the composite signal MIX output from the pixel extraction unit 31, that is, the target pixel position (the same position as the target pixel in the target frame) and the frame F0, F1, F3, and F4, respectively. For each of the composite signals MIX for the pixels at the surrounding positions, it is determined whether or not the added pixel pattern code PAT is the same as the added pixel pattern code PAT of the composite signal MIX for the target pixel of the target frame F2. Together with the determination result (match / mismatch information) PAG, the in-plane sensitization signal VAL of the composite signal MIX is supplied to the corresponding correlation discriminating units 381 to 384, respectively. The in-plane sensitization signal VAL and match / mismatch information PAG for the same pixel are output in association with each other.

Each of the correlation determination units 381 to 384 includes an in-plane sensitization signal VAL for a plurality of pixels (attention pixel position and surrounding pixels) supplied from a corresponding pattern determination unit (any one of 361 to 364), The presence or absence of correlation with the in-plane sensitization signal VAL (F2, P33) for the target pixel P33 of the target frame F2 supplied from the signal separation unit 350 is determined by comparison with the correlation presence / absence determination threshold value CRth.
If there is only one correlated pixel, the in-plane sensitization signal VAL for the one correlated pixel is selected and output.

  On the other hand, when there are a plurality of correlated pixels, one of the in-plane sensitization signals VAL for the correlated pixels is selected based on the match / mismatch information PAG. That is, when the match / mismatch information PAG for any of a plurality of correlated pixels indicates “match”, it is associated with the match / mismatch information PAG indicating “match” (shows “match”). Among the in-plane sensitization signals VAL (for the same pixel as the match / mismatch information PAG), the one having the highest correlation with the in-plane sensitization signal VAL (F2, P33) for the target pixel P33 of the target frame F2 (in-plane The in-plane sensitization signal VAL whose value is closest to the sensitization signal VAL (F2, P33) is selected and output.

  If there is no correlated pixel by comparison with the correlation presence / absence determination threshold value CRth, or if there is no match / mismatch information PAG indicating “match”, all input in-plane sensitization signals VAL are present. Among them, the one having the highest correlation with the in-plane sensitization signal VAL for the target pixel is selected and output.

The correlation between the two in-plane sensitization signals VAL is evaluated by, for example, the absolute value of the difference between the two, and it is determined that the correlation is higher as the absolute value of the difference is smaller. Accordingly, the presence / absence of correlation is determined to have correlation when the absolute value of the difference is equal to or less than the threshold value CRth. As the in-plane sensitization signal having the highest correlation with the in-plane sensitization signal VAL for the target pixel, the in-plane sensitization signal having the smallest absolute value of the difference from the in-plane sensitization signal VAL for the target pixel is used. Is selected.
When the in-plane sensitization signal VAL for a certain pixel is selected, the pixel is selected and becomes an addition target pixel.

  In the above example, if there is no correlated pixel by comparison with the correlation presence / absence determination threshold CRth, or if there is no match / mismatch information PAG indicating “match”, all the input surfaces Among the internal sensitization signals VAL, the signal having the highest correlation with the in-plane sensitization signal VAL for the pixel of interest is selected and output, but instead, the in-plane of the pixel at the same position as the pixel of interest is output. The sensitization signal VAL may be selected and output. When the influence of noise is large, it may be expected that the noise reduction effect of the image with higher sensitivity is expected by selecting the in-plane sensitization signal for the pixel at the same position as the target pixel.

The pattern determination unit 361 of the first pixel selection unit 351 includes first to ninth determination units 3611 to 3619 as shown in FIG.
As illustrated in FIG. 21, the first determination unit 3611 includes a signal separation unit 36111 and a determination unit 36112.

  The signal separation unit 36111 receives the composite signal MIX (F0, P35) for the pixel P35 of the frame F0, and generates the in-plane sensitization signal VAL (F0, P35) and the addition pixel pattern code PAT (F0, P35). To separate.

  The determination unit 36112 compares the addition pixel pattern code PAT (F0, P35) from the signal separation unit 36111 with the addition pixel pattern code PAT (F2, P33) from the signal separation unit 350, and determines whether or not they match. And information (match / mismatch information) PAG (F0, P35) indicating the determination result is output.

  The second to ninth discriminating units 3612 to 3619 are also configured in the same manner as the first pattern discriminating unit 3611, and are combined signals MIX (F0, P44), MIX (F0, P24), MIX (F0, P53), MIX, respectively. (F0, P33), MIX (F0, P13), MIX (F0, P42), MIX (F0, P22), MIX (F0, P31) are subjected to the same processing, and the in-plane sensitization signal VAL ( F0, P44), VAL (F0, P24), VAL (F0, P53), VAL (F0, P33), VAL (F0, P13), VAL (F0, P42), VAL (F0, P22), VAL (F0) , P31), match / mismatch information PAG (F0, P44), PAG (F0, P24), PAG (F0, P53), PAG (F0, P33), PAG (F0, P13), PA (F0, P42), PAG (F0, P22), and outputs the PAG (F0, P31).

The correlation determination unit 381 outputs the in-plane sensitization signal VAL (F0, P35) for the pixels P35, P44, P24, P53, P33, P13, P42, P22, P31 of the frame F0 output from the corresponding pattern determination unit 361. , VAL (F0, P44), VAL (F0, P24), VAL (F0, P53), VAL (F0, P33), VAL (F0, P13), VAL (F0, P42), VAL (F0, P22), VAL (F0, P31)
And the absolute difference value between the in-plane sensitization signal VAL (F2, P33) for the target pixel P33 of the target frame F2.
When there is only one pixel whose calculated difference absolute value is equal to or less than the threshold value CRth, the in-plane sensitization signal VAL for the one pixel is selected and output.

  On the other hand, when there are a plurality of pixels whose calculated difference absolute value is equal to or less than the threshold value CRth, the match / mismatch information PAG among the plurality of pixels indicates that the added pixel pattern code PAT matches. The in-plane sensitization signal VAL having the closest value to the in-plane sensitization signal VAL (F2, P33) for the pixel of interest P33 in the frame of interest F2 is selected from among those selected in the frame F0. An in-plane sensitization signal (also simply referred to as “in-plane sensitization signal selected in frame F0”) VAL (F0) for the pixel to be added is output.

In-plane sensitization signals VAL (F0, P35), VAL (F0, P44), VAL (F0, P24), VAL (F0, P53), VAL (F0, P33), VAL (output from the pattern discrimination unit 361 F0, P13), VAL (F0, P42), VAL (F0, P22), and VAL (F0, P31), and the in-plane sensitization signal VAL (F2, P33) for the target pixel P33 of the target frame F2. When there is no absolute value of the difference of the threshold value CRth or less, or when there is no indication that the added pixel pattern code matches by the match / mismatch information PAG, the correlation determination unit 381 is input All in-plane sensitization signals VAL (F0, P35), VAL (F0, P44), VAL (F0, P24), VAL (F0, P53), VAL (F0, P33) , VAL (F0, P13), VAL (F0, P42), VAL (F0, P22), and VAL (F0, P31), the in-plane sensitization signal VAL (F2, F2) for the target pixel P33 of the target frame F2 The in-plane sensitization signal VAL having a value closest to P33) is selected and output as the in-plane sensitization signal VAL (F0) selected in the frame F0.
The in-plane sensitization signal VAL (F0) output from the correlation determination unit 381 is supplied to the pixel addition unit 39 (FIG. 16) as the output of the first pixel selection unit 351.

The second to fourth pixel selection units 352 to 354 are configured in the same manner as the first pixel selection unit 351, and the pixels P35, P44, P24, P53, P33, P13 of the frames F1, F3, and F4, respectively. The composite signal MIX for P42, P22, and P31 is processed in the same manner as the first pixel selection unit 351,
In-plane sensitization signals (in-plane sensitization signals selected in frames F1, F3, and F4) VAL (F1), VAL (F3), and VAL (F4) for pixels to be added selected in frames F1, F3, and F4 ) Is output.

  The in-plane sensitization signals VAL (F0), VAL (F1), VAL (F3), and VAL (F4) output from the first to fourth pixel selection units 351 to 354 are output from the signal separation unit 350. Together with the in-plane sensitization signal VAL (F2, P33), it is supplied to the pixel adder 39.

As described above, when pixel addition is performed using the G pixel as a target pixel, the pixel selection units 351 to 354 perform processing by using the composite signal MIX for the pixels P35 to P31 of the corresponding frames as inputs, When pixel addition is performed using the B pixel as the target pixel, the pixel selectors 351 to 354 do not use the composite signal MIX for the pixels P44, P24, P42, and P22 of the corresponding frame, but the pixels P35, P53, and P33. , P13 and P31 are processed using the composite signal MIX.
In other words, when performing pixel addition using the G pixel as the target pixel, the pattern determination units 361 to 364 receive the composite signal MIX for the pixels P35 to P31 of the corresponding frames, respectively, and input the in-plane sensitization signal VAL for the same pixel. , And the process of outputting the match / mismatch information PAG, and the correlation determination units 381 to 384 perform processing using the composite signal MIX for the nine pixels output from the pattern determination units 361 to 364, respectively. When pixel addition is performed using the pixel or the B pixel as the pixel of interest, the pattern determination units 361 to 364 do not use the composite signal MIX for the pixels P44, P24, P42, and P22 of the corresponding frame, but use the pixels P35 and P53. , P33, P13, and P31 are used to perform processing, and the correlation discriminating units 381 to 381 84 performs processing by using the composite signal for the five pixels output from each of the pattern determination unit 361 to 364.

  As described above, instead of causing the same pixel selectors 351 to 354 to operate differently when the G pixel is the target pixel and when the R pixel or B pixel is the target pixel, You may provide separately the pixel selection part in the case of making a pixel into an attention pixel, and the pixel selection part in the case of making R pixel or B pixel into an attention pixel.

  The pixel addition unit 39 performs signal separation on the in-plane sensitization signals VAL (F0), VAL (F1), VAL (F3), and VAL (F4) output from the first to fourth pixel selection units 351 to 354. The in-plane sensitization signal VAL (F2, P33) output from the unit 350 is added.

The control unit 12 sets the sensitization magnification Lb (Lb = 1 to 5) in the pixel addition unit 39, and the pixel addition unit 39 performs an addition result on the pixel value before the addition in the above addition. The addition coefficient is multiplied so as to have a predetermined sensitization magnification Lb.
The calculation for obtaining the sensitized signal Pf (F2, P33) is expressed by the following equation.
Pf (F2, P33)
= {VAL (F2, P33) + VAL (F0) + VAL (F1) + VAL (F3)
+ VAL (F4)} × Lb / 5

By this addition, one in-plane sensitization signal VAL (F2, P33) for the target pixel P33 of the target frame F2 is selected from each of the four frames F0, F1, F3, and F4 before and after the target frame F2. In-plane sensitization signals VAL (F0), VAL (F1), VAL (F3), and VAL (F4) for a total of four pixels having a high correlation with the target pixel are added, and the in-plane sensitization signal VAL (F2) is added. , P33) is sensitized with the sensitization magnification Lb, and a three-dimensional sensitized signal Pf (F2, P33) is generated.
The generated three-dimensional sensitized signal Pf is supplied to the video signal processing unit 7 via the output terminal 602.

The product of the sensitization magnification La of the in-plane pixel addition unit 20 and the sensitization magnification Lb of the inter-plane pixel addition unit 30 becomes the sensitization magnification L of the three-dimensional pixel addition unit 6.
The sensitization magnification L is determined, for example, in relation to the subject illuminance.
For example, when the subject illuminance is equal to or higher than a first predetermined value (high illuminance side reference value), the sensitization magnification L is set to 1, and a second predetermined value (low illuminance) lower than the first predetermined value. (Side reference value) or lower, the sensitization magnification is set to 20, and in a range lower than the high illuminance side reference value and higher than the low illuminance side reference (medium illuminance range), the sensitization magnification is gradually increased as the illuminance decreases. .

  In determining the sensitization magnifications La and Lb for setting the sensitization magnification L to a desired value, the ratio between the sensitization magnification La and the sensitization magnification Lb may be kept constant. Instead, for example, when the motion of the image is large and a high resolution is not required so much, the sensitization magnification La is relatively large, the sensitization magnification Lb is relatively small, and conversely, the image motion is small and the resolution When a high image is required, the sensitization magnification Lb may be made relatively large and the sensitization magnification La may be made relatively small.

  In the pixel addition in the in-plane pixel addition unit 20, the pixel value of each pixel may be multiplied by an addition coefficient having a different value to perform weighted addition. For example, in the addition in the pixel addition unit 595, when the sensitization magnification La is 1, the addition coefficient for the target pixel is 1, the addition coefficient for the other pixels is 0, and the sensitization magnification La is the maximum value, for example, 4, When the sensitization magnification La is between 1 and 4, the value when the sensitization magnification La is 1 is changed to the value when the sensitization magnification La is the maximum value. The addition coefficient may be continuously changed.

  Similarly, in the addition of the in-plane sensitization signal VAL in the inter-plane pixel addition unit 30, weighted addition may be performed by multiplying the pixel value of the pixel of each frame by a different addition coefficient. For example, when the sensitization magnification Lb is 1, the addition coefficient for the pixel of interest in the frame of interest is 1, the addition coefficient for the pixels of the other frame is 0, and when the sensitization magnification Lb is the maximum value, for example, 5, When the addition coefficient for all pixels is the same value and the sensitization magnification Lb is between 1 and 5, the addition coefficient is changed from the value when the sensitization magnification L is 1 to the value when the sensitization magnification Lb is the maximum value. You may make it change continuously.

In the above example, the case where the sensitization magnification La up to 4 times is set for the in-plane pixel addition has been described, but it may be set to 4 times or more. When setting the sensitization magnification La of 4 times or more, it is necessary to pay attention to gradation drop.
Similarly, the case where the sensitization magnification Lb up to 5 times is set for the inter-plane pixel addition has been described, but it may be set to 5 times or more. When setting a sensitization magnification Lb of 5 times or more, it is necessary to pay attention to gradation drop.

In the above example, for each target pixel of the target frame, one pixel is selected from four frames near the target frame, and the in-plane sensitization signals for the five pixels in total are added. However, the number of neighboring frames used for selecting the pixel to be added is not limited to four, and may be, for example, more or less, for example, only the frame immediately before the target frame. Alternatively, it may be only a frame after the frame of interest.
When the required sensitization magnification is not high, the necessary frame memory is reduced by reducing the number of neighboring frames used for selecting pixels to be added, thereby reducing the circuit scale and reducing the cost. Can be achieved.
On the other hand, if pixels in a larger number of neighboring frames are included in the addition target pixels, higher sensitivity can be achieved.

  Further, the pixel arrangement of the color filter has been described in the case of the RGB Bayer arrangement. However, if the arrangement of 4 pixels of 2 horizontal pixels and 2 vertical pixels is used as a basic arrangement, an interline arrangement, a stripe arrangement, a Ye pixel, The present invention can be applied to a complementary color array of Mg pixels, G pixels, and Cy pixels, a pixel using W pixels, or any other color filter combination, and the same effect can be obtained.

Although the case where the captured image is a color image has been described above, the present invention can also be applied to a case where the captured image is a monochrome image.
In the case of an image sensor that is not provided with a color filter and can generate a black and white image, it is possible to add pixels with higher correlation that are located closer to each other. By deciding, it is possible to achieve high sensitivity while preventing a decrease in resolution.

  According to the above embodiment, the in-plane pixel addition and the inter-plane pixel addition are performed together, so that the overall sensitization can be performed compared to the case of only the in-plane pixel addition or the case of only the inter-plane pixel addition. The magnification can be increased.

  In addition, in the in-plane pixel addition, pixel addition is performed using a pixel having a high correlation with the target pixel, so that it is possible to improve sensitivity while minimizing degradation of image resolution.

  Furthermore, in the inter-plane pixel addition, the addition target pixel for the inter-plane pixel addition is selected based on the coincidence / mismatch of the addition pixel pattern code with the frame of interest. High sensitivity can be achieved without reducing the resolution.

  In addition, in inter-plane pixel addition, pixel addition is performed using a pixel having a high correlation with the target pixel, so that it is possible to improve sensitivity while minimizing degradation of image resolution.

  Further, in the inter-plane pixel addition, the correlation between the in-plane sensitization signal for the target pixel and the in-plane sensitization signal for the plurality of pixels supplied from the pattern discrimination unit, and the comparison result of the correlation presence / absence determination threshold , The presence / absence of correlation is determined, and the in-plane sensitization signal to be added to the in-plane sensitization signal for the target pixel is selected from the correlated pixels based on the pattern match / mismatch information. Therefore, the addition of the in-plane sensitization signal for the pixel of interest is not suitable for the addition, the addition of the in-plane sensitization signal for a pixel that has no correlation with the pixel of interest but has the same pattern can be prevented, and the periphery with high correlation Pixels can be added, and there is an effect that the sensitivity can be increased without lowering the resolution.

  In the above example, first, in-plane pixel addition is performed to generate a composite signal including the in-plane sensitization signal and the added pixel pattern code, and the inter-plane pixel is generated using the signal generated by delaying the frame signal. Since it is configured to perform addition, three-dimensional sensitization using a pixel having a high correlation with the target pixel of the target frame can be realized with the minimum number of reference pixels, and therefore the circuit scale can be reduced and the cost can be reduced. Can do.

  Furthermore, since pixels of the same filter color are added, a highly sensitive color image can be obtained without mixing colors.

  When a captured image at low illuminance is amplified by an analog amplifier, noise becomes larger than a signal. Further, gradation reduction occurs when amplified by a digital amplifier. As described in the above example, the present invention is located near the pixel of interest spatially or temporally, and high sensitivity is achieved by pixel addition using highly correlated pixels. Get smaller. For example, when two pixels are added, the signal component is doubled, the noise component is square root doubled, and a relatively pure signal component is increased.

  Further, by performing pixel addition immediately after being output from the image sensor (that is, before processing by the video signal processing unit 7), a high-sensitivity signal is generated by pixel addition without being affected by the video signal processing. be able to. When pixel addition is performed at the later stage of video signal processing, color synchronization processing and filtering processing using pixels located in the vicinity are performed, so the decrease in horizontal resolution and vertical resolution is larger than expected. . Further, since the video signal processing is performed with the small amplitude signal, there is a possibility of gradation drop. By performing pixel addition immediately after output from the image sensor (before video signal processing), the signal amplitude can be recovered by pixel addition before image information is lost, improving visibility of image details There is an effect to.

  Further, in the video signal processing by the video signal processing unit 7, nonlinear filter processing and gradation conversion processing are performed. Therefore, when a low amplitude signal is input, the signal amplitude may be lost. For this reason, even if two pixels are added to the output signal of the video signal processing, there is a possibility that the image signal is not doubled. In the example described above, the pixels are added before the video signal processing. Therefore, when two pixels are added, there is an effect that the image signal is doubled.

  Furthermore, since the decrease in frame frequency can be prevented or suppressed, there is no degradation in dynamic resolution, and degradation in horizontal resolution and vertical resolution can be minimized.

Embodiment 2. FIG.
In the first embodiment, when inter-plane pixel addition is performed, the frames F1 and F3 separated by one frame (after one frame period and before one frame period) from the target frame F2 are separated by two frames (two frame periods). and after the two frame periods before) frame F0, consistent with the added pixel pattern code PAT for the pixel of interest also subject frame F2 for any of F4 / mismatch, and plane sensitization signal VAL pixel of interest of the subject frame F2 The pixel to be added is selected based on the correlation with.

  Instead, in the frames F0 and F4 that are two frames away from the target frame F2, not the target pixels of the target frame, but the addition target pixels selected in the adjacent frames (immediately or immediately before) F1 and F3 on the side of the target frame The pixel to be added may be selected based on the match / mismatch of the addition pixel pattern code PAT and the correlation of the in-plane sensitization signal VAL.

Specifically, the addition target pixel is selected in the previous frame F3 with reference to the addition pixel pattern code PAT and the in-plane sensitization signal VAL for the target pixel of the target frame F2, and the addition for the addition target pixel of the previous frame F3 is performed. With reference to the pixel pattern code PAT and the in-plane sensitization signal VAL, an addition target pixel is selected in the further previous frame F4.
Similarly, an addition target pixel is selected in the subsequent frame F1 with reference to the addition pixel pattern code PAT and the in-plane sensitization signal VAL for the attention pixel of the attention frame F2, and the addition pixel pattern for the addition target pixel of the subsequent frame F1. With reference to the code PAT and the in-plane sensitization signal VAL, a pixel to be added is selected in a later frame F0.
The in-plane sensitization signal VAL for the pixels to be added in the preceding and succeeding frames F0, F1, F3, and F4 selected as described above is added to the in-plane sensitization signal VAL of the frame of interest F2.

In order to perform such processing, in this embodiment, an addition pixel selection unit 35b shown in FIG. 22 is used instead of the addition pixel selection unit 35 of FIG.
The addition pixel selection unit 35b of FIG. 22 is generally the same as the addition pixel selection unit 35 of FIG. 19, but includes pixel selection units 651 to 654 instead of the pixel selection units 351 to 354 of FIG.

  Hereinafter, the configuration of the pixel selectors 651 to 654 will be described, the operation when the G pixel is the target pixel will be described first, and then the operation when the R pixel or the B pixel is the target pixel will be described. .

  The pixel selection units 651 to 654 include pattern determination units 661 to 664 and correlation determination units 681 to 684, respectively.

  The pattern discriminating units 662 and 663 are generally the same as the pattern discriminating units 362 and 363 in FIG. 19, but the in-plane sensitization signal VAL and the match / mismatch information PAG for the pixels in the respective neighboring frames F1 and F3 are used. In addition, the addition pixel pattern code PAT is also output.

  Specifically, as shown in FIG. 23, the pattern discriminating unit 662 includes discriminating units 6621 to 6629, and includes in-plane sensitization signals VAL (F1, P35) to VAL (F1, P31) and match / mismatch information. In addition to PAG (F1, P35) to PAG (F1, P31), addition pixel pattern codes PAT (F1, P35) to PAT (F1, P31) are also output.

Therefore, each of the determination units 6621 to 6629 is configured as follows.
For example, the determination unit 6621 includes a signal separation unit 66211 and a determination unit 66212 as illustrated in FIG.

The signal separation unit 66211 is configured in the same manner as the signal separation unit 36111 in FIG. 21, and the composite signal MIX (F1, P35) is converted into the in-plane sensitization signal VAL (F1, P35) and the added pixel pattern code PAT (F1, F1). P35).
The added pixel pattern code PAT (F1, P35) output from the signal separation unit 66211 is not only supplied to the determination unit 66212 but also output to the correlation determination unit 682.

  The determination unit 66212 is configured in the same manner as the determination unit 36112 in FIG. 21, and matches the addition pixel pattern code PAT (F2, P33) and the addition pixel pattern code PAT (F1, P35) supplied from the signal separation unit 66211. / Match determination is performed, and match / mismatch information PAG (F1, P35) is output.

  The other discriminators 6622 to 6629 are also configured in the same manner, and in-plane sensitization signals VAL (F1, P44) to VAL (F1, P31) and match / mismatch information PAG (F1, P44) to PAG (F1, P31) are used. In addition, the addition pixel pattern codes PAT (F1, P44) to PAT (F1, P31) are output to the correlation determination unit 682.

The correlation determination unit 682 supplies the in-plane sensitization signals VAL (F1, P35) to VAL (F1, P31) for the pixels in the frame F1 supplied from the pattern determination unit 662 and the signal separation unit 350. An absolute difference value with respect to the in-plane sensitization signal VAL (F2, P33) for the target pixel P33 of the target frame F2 is calculated.
If only one pixel has a calculated difference absolute value equal to or less than the threshold value CRth, the in-plane sensitization signal VAL for that pixel is selected and output.

  On the other hand, when there are a plurality of pixels having the calculated difference absolute value equal to or less than the threshold value CRth, one of the in-plane sensitization signals VAL for the plurality of pixels is selected. That is, when the match / mismatch information PAG for any of a plurality of correlated pixels indicates “match”, it is associated with the match / mismatch information PAG indicating “match” (shows “match”). Of the in-plane sensitization signals VAL (for the same pixel as the match / mismatch information PAG), the one having the highest correlation with the in-plane sensitization signal VAL (F2, P33) for the target pixel F33 of the target frame F2 (closest) In-plane sensitization signal VAL) having a value is selected and output as an in-plane sensitization signal (in-plane sensitization signal selected in frame F1) VAL (F1) for the addition target pixel selected in frame F1. To do.

  If there is no correlated pixel by comparison with the correlation presence / absence determination threshold value CRth, or if there is no match / mismatch information PAG indicating “match”, all input in-plane sensitization signals VAL are present. Among them, the one having the highest correlation with the in-plane sensitization signal VAL for the target pixel is selected and output.

  Among the in-plane sensitization signals VAL (F1, P35) to VAL (F1, P31) supplied from the pattern discriminating unit 662, the in-plane sensitization signal VAL (F2, P33) for the target pixel P33 of the target frame F2. If there is no one whose absolute difference value is less than or equal to the threshold value CRth, or there is no one indicating that the added pixel pattern codes match by the match / mismatch information PAG, the correlation determination unit 682 inputs Among all the in-plane sensitization signals VAL (F1, P35) to VAL (F1, P31), the value closest to the in-plane sensitization signal VAL (F2, P33) for the target pixel P33 of the target frame F2. Is selected for the pixel to be added selected in the frame F1 (in-plane sensitization signal selected in the frame F1). And outputs as the inner sensitizing signal VAL (F1).

The in-plane sensitization signal VAL (F1) for the pixel selected in the frame F1 output from the correlation determination unit 682 is supplied to the pixel addition unit 39 (FIG. 16) as the output of the pixel selection unit 652, It is also supplied to the correlation determination unit 681.
As described above, the correlation determination unit 682 outputs the in-plane sensitization signal VAL (F1) for the pixel selected in the frame F1, and also adds the pixel pattern code PAT (F1) for the pixel selected in the frame F1. ) Is supplied to the pattern determination unit 661.

  As shown in FIG. 25, the pattern discriminating unit 661 includes discriminating units 6611 to 6619, and instead of the added pixel pattern code PAT (F2, P33) for the pixel of interest, the pattern discriminating unit 661 for the pixel selected in the frame F1. Using the addition pixel pattern code PAT (F1), it is determined whether or not the addition pixel pattern code for the pixel of the frame F0 supplied from the pixel extraction unit 21 matches, and match / mismatch information indicating the determination result PAG (F0, P35) to PAG (F0, P31) are output.

Therefore, each of the determination units 6611 to 6619 is configured as follows.
For example, the determination unit 6611 includes a signal separation unit 66111 and a determination unit 66112 as shown in FIG.

  The signal separation unit 66111 is configured in the same manner as the signal separation unit 36111 of FIG. 21, and the composite signal MIX (F0, P35) is converted into the in-plane sensitization signal VAL (F0, P35) and the added pixel pattern code PAT (F0, P35).

  The determination unit 66112 is configured in the same manner as the determination unit 36112 of FIG. 21, but uses the addition pixel pattern code PAT (F1) instead of the addition pixel pattern code PAT (F2, P33), and uses the signal separation unit 66111. Is matched / mismatched with the added pixel pattern code PAT (F0, P35) supplied from, and the match / mismatch information PAG (F0, P35) is output.

  The other discriminators 6612 to 6619 are configured in the same manner, and receive in-plane sensitization signals VAL (F2, P44) to PAG (F2, P31) and match / mismatch information PAG (F2, P44) to PAG (F2, P31). The result is output to the correlation determining unit 681.

  The correlation determination unit 681 uses the in-plane sensitization signal VAL (F1) instead of the in-plane sensitization signal VAL (F2, P33), and uses the in-plane sensitization signal VAL supplied from the corresponding pattern determination unit 661. A correlation with (F0, P35) to VAL (F0, P31) is calculated, an addition target pixel is selected based on the calculation result, and the in-plane sensitization signal VAL for the selected addition target pixel is determined in the frame. The in-plane sensitization signal VAL (F0) selected in F0 is output.

That is, the correlation determination unit 681 is supplied from the pixel selection unit 652 and the in-plane sensitization signals VAL (F0, P35) to VAL (F0, P31) for the pixels in the frame F0 supplied from the pattern determination unit 661. Also, the absolute value of the difference from the in-plane sensitization signal VAL (F1) selected in the frame F1 is calculated.
If only one pixel has a calculated difference absolute value equal to or less than the threshold value CRth, the in-plane sensitization signal VAL for that pixel is selected and output.

  On the other hand, when there are a plurality of pixels having the calculated difference absolute value equal to or less than the threshold value CRth, one of the in-plane sensitization signals VAL for the plurality of pixels is selected. That is, when the match / mismatch information PAG for any of a plurality of correlated pixels indicates “match”, it is associated with the match / mismatch information PAG indicating “match” (shows “match”). Of the in-plane sensitization signals VAL (for the same pixel as the match / mismatch information PAG), the one having the highest correlation with the in-plane sensitization signal VAL (F1) selected in the frame F1 (the in-plane having the closest value) Sensitization signal VAL) is selected and output as the in-plane sensitization signal VAL (F0) selected in frame F0.

If there is no correlated pixel by comparison with the correlation presence / absence determination threshold value CRth, or if there is no match / mismatch information PAG indicating “match”, all input in-plane sensitization signals VAL are present. Among them, the one having the highest correlation with the in-plane sensitization signal VAL for the target pixel is selected and output.
The in-plane sensitization signal VAL (F0) selected in the frame F0 output from the correlation determination unit 681 is supplied to the pixel addition unit 39 (FIG. 16) as the output of the pixel selection unit 651.

  The pattern discriminating unit 663 and the correlation discriminating unit 683 are configured in the same manner as the pattern discriminating unit 662 and the correlation discriminating unit 682. From the signal separating unit 350, the added pixel pattern code PAT (F2, P33) and the pixel of interest of the frame of interest Upon receiving the in-plane sensitization signal VAL (F2, P33), the same processing is performed on the composite signals MIX (F3, P35) to MIX (F3, P31) for the pixels in the frame F3 from the pixel extraction unit 21. Then, the in-plane sensitization signal (in-plane sensitization signal selected in frame F1) VAL (F3) for the pixel selected in frame F3 is output.

  The pattern discriminating unit 664 and the correlation discriminating unit 684 are configured in the same manner as the pattern discriminating unit 661 and the correlation discriminating unit 681. From the correlation discriminating unit 683, the added pixel pattern code PAT (F3) and the pixel selected in the frame F3 are displayed. Upon receiving the in-plane sensitization signal VAL (F3), the same processing is performed on the composite signals MIX (F4, P35) to MIX (F4, P31) for the pixels in the frame F4 from the pixel extraction unit 21. The in-plane sensitization signal (in-plane sensitization signal selected in frame F4) VAL (F4) for the pixel selected in frame F4 is output.

  As described above, when pixel addition is performed using the G pixel as the target pixel, the pixel selection units 651 to 654 perform processing with the composite signal MIX for the pixels P35 to P31 of the corresponding frames as inputs, When performing pixel addition using the B pixel as the target pixel, the pixel selectors 651 to 654 do not use the composite signals for the pixels P44, P24, P42, and P22 of the corresponding frame, respectively, and the pixels P35, P53, P33, Processing is performed using the composite signal MIX for P13 and P31.

  That is, when pixel addition is performed using the G pixel as the target pixel, the pattern determination units 661 to 664 receive the composite signal MIX for the pixels P35 to P31 of the corresponding frames, respectively, and input the in-plane sensitization signal VAL for the same pixel. , And the process of outputting the match / mismatch information PAG, and the correlation discriminating units 681 to 684 respectively perform the in-plane sensitization signal VAL and the match / mismatch information PAG for the nine pixels output from the pattern discriminating units 661 to 664, respectively. However, when performing pixel addition using the R pixel or B pixel as the pixel of interest, the pattern discriminating units 661 to 664 are combined signals for the pixels P44, P24, P42, and P22 of the corresponding frame, respectively. Without using MIX, the composite signal MIX for the pixels P35, P53, P33, P13, P31 There are performed processing, the correlation determination unit 681 to 684 performs processing by using the five-plane sensitization signal VAL and match / mismatch information PAG for the pixel which is output from each of the pattern determination unit 661-664.

  Note that, as described above, instead of causing the same pixel selectors 651 to 654 to perform different operations when the G pixel is the target pixel and when the R pixel or the B pixel is the target pixel, G You may provide separately the pixel selection part in the case of making a pixel into an attention pixel, and the pixel selection part in the case of making R pixel or B pixel into an attention pixel.

  By performing the processing as described above, it is possible to more appropriately select the pixel to be added in consideration of the motion of the image, and it is possible to suppress a decrease in resolution due to motion blur.

As described above, the case where the frame near the target frame is composed of two frames adjacent to the target frame and two frames that are two frames away from the target frame has been described. However, when a frame that is three frames or more away from the target frame is included The present embodiment can also be applied to.
In this case, for each frame, pixels in the frame are selected using the addition pixel pattern code PAT and the in-plane sensitization signal VAL for other frames adjacent to the frame of interest.

That is, the pattern determining portion is (are m frame (m from e.g. attention frame positive integer) apart frames) each frame addition pixel pattern code PAT for pixels within is adjacent in the frame of interest side with respect to the frame It is determined whether or not the added pixel pattern code PAT for the pixel selected in another frame (a frame that is (m−1) frames away from the frame of interest) matches, and the correlation determination unit detects the frame (frame of interest). Selected in the in-plane sensitization signal VAL for pixels in a frame within a frame (m frames away from) and another frame (a frame that is (m−1) frames away from the frame of interest) adjacent to the frame on the frame of interest Based on the correlation with the in-plane sensitization signal VAL for the selected pixel and the determination result in the pattern determination unit , Within the frame may be decided to select one of the pixel of interest pixel position and surrounding positions.

Embodiment 3 FIG.
In the first embodiment, the in-plane pixel adding unit 20 has the signal combining unit 24 and outputs the composite signal MIX, and the pixel extracting unit 31 of the inter-plane pixel adding unit 30 differs the composite signal MIX from each other. The composite signal MIX for the pixel of interest in the frame of interest and the pixel of interest in the neighboring frame and the surrounding pixels is extracted at the same time with delay, but the in-plane pixel adder 20 has the signal combiner 24. Alternatively, the addition pixel pattern code PAT and the in-plane sensitization signal VAL may be output without being combined (but associated with each other), and the inter-plane pixel addition unit 30b shown in FIG. 27 may be used.

  The inter-plane pixel addition unit 30b in FIG. 27 is generally the same as the inter-plane pixel addition unit 30 in FIG. 16, but instead of the pixel extraction unit 31 in FIG. 16, the addition pixel pattern code PAT is delayed by different times. And a pattern code extraction unit 31p that simultaneously extracts the addition pixel pattern codes PAT (F0, P35) to PAT (F4, P31) for the target pixel of the target frame, the target pixel position of the neighboring frame, and the surrounding pixels. , The in-plane sensitization signals VAL are delayed by different times so that the in-plane sensitization signals VAL (F0, P35) to VAL ( An in-plane sensitization signal extraction unit 31v that simultaneously extracts F4, P31) is provided.

Further, the signal separation unit 350 (FIG. 16) in the inter-plane pixel addition unit 30 is not used, and the addition pixel selection unit 35 c is used instead of the addition pixel selection unit 35.
The addition pixel selection unit 35c is the same as the addition pixel selection unit 35 in FIG. 19, but a signal is sent to each determination unit of the pattern determination units 361 to 364 (corresponding to the determination units 3611 to 3619 in FIG. 20). The separation unit (reference numeral 36111 in FIG. 21) is not provided, and the addition pixel pattern code PAT for each pixel of the neighboring frame supplied from the pattern code extraction unit 31p and the addition pixel pattern code for the target pixel of the target frame Comparison with PAT is performed by a determination unit (corresponding to the determination unit 36112 in FIG. 21), and comparison using a threshold CRth is performed by a correlation determination unit (corresponding to the correlation determination unit 381 in FIG. 20). Output the comparison result.

Embodiment 4 FIG.
In the first embodiment, the configuration using the CCD image sensor 2 as a solid-state image sensor as shown in FIG. 1 has been described. Something like that. Further, not limited to the interline transfer CCD, a frame transfer CCD or a frame interline transfer CCD may be used.

FIG. 28 shows a configuration using the CMOS image sensor 14. The CMOS image sensor may be a device with a single imaging function or a device with integrated peripheral functions.
FIG. 28 shows a case of a CMOS image sensor in which peripheral functions are integrated.
The functions of the CCD imaging device 2, the correlated double sampling processing unit 3, the programmable gain amplification unit 4, the A / D conversion unit 5, and the timing generation unit 10 in FIG. The CMOS image pickup device 14 alone has a function equivalent to that of the image pickup signal generation unit 13 in FIG. 1, that is, an image pickup signal generation unit 13b that generates image pickup signals of a plurality of color components obtained as a result of image pickup by imaging a subject. Has been.

Embodiment 5. FIG.
FIG. 29 shows an imaging apparatus according to Embodiment 5 of the present invention. 29, the detection unit 15 is added, and the control unit 12b is provided instead of the control unit 12 of FIG. 1, except that the control unit 12b is provided. Play.

  The detection unit 15 detects the magnitude of the signal Pf output from the three-dimensional pixel addition unit 6, obtains a signal amplitude level, for example, an average level detection value ASA, and outputs it as illuminance information.

In the above detection, the detection unit 15 divides the sum of the pixel values of all the effective pixels by the total number of effective pixels to obtain the detection value ASA of the average level of the signal amplitude.
Such calculation of the average level is performed, for example, every vertical scanning cycle, and is executed by, for example, integration processing and division processing. The “calculated value” of the average level of the signal amplitude obtained as described above is also referred to as “detected value” of the average level of the signal amplitude.

  Note that the division by the total number of effective pixels in the calculation of the average level of the signal amplitude described above may be realized by a bit shift process of a digital value when the number of pixels is given by 2 to the nth power (n is an integer). . Further, since the total number of effective pixels is a constant in the same system, division of the total number of effective pixels may be omitted.

The control unit 12b is similar to the control unit 12 of the first embodiment, but has an additional function as follows. That is, the control unit 12b controls the aperture of the lens 1 based on the detection value ASA of the average level of the signal amplitude supplied from the detection unit 15, and from the photoelectric conversion element of the CCD image pickup device 2 generated by the timing generation unit 10. Charge readout timing and charge forced discharge timing control (accordingly, charge accumulation time, ie, exposure time control), control of amplification gain of the programmable gain amplification unit 4, and control of pixel addition processing of the three-dimensional pixel addition unit 6. Do.
Further, the video signal processing unit 7 calculates the level of noise included in the output of the three-dimensional pixel addition unit 6 for each vertical scanning period, and supplies it to the control unit 12b.

  The detection unit 15 calculates the average level of the signal amplitude and the noise level every vertical scanning period, instead of performing the signal processing time in the detection unit 15 and the video signal processing unit 7 and the detection unit 15 and the video. Considering the transmission time from the signal processing unit 7 to the control unit 12b, it may be performed only once in several vertical scans.

  Further, the detection unit 15 may perform peak detection of the signal amplitude instead of calculating the average level of the signal amplitude. The output of the detection unit 15 is generated so that the visibility of the subject of interest is high. For example, when it is not desired to saturate a highlight portion to white, peak detection is performed. When a halftone is clearly visible even if the highlight portion is saturated to white, average detection is performed.

  As will be described in detail below, sensitivity control by the three-dimensional pixel addition unit 6 can also be controlled as part of exposure control, so that even when the illuminance environment changes, it is possible to obtain an image in which the subject can always be viewed under optimum conditions. . Further, the signal amplitude can be adjusted by changing the addition coefficient in the three-dimensional pixel addition unit 6.

  The controller 12b performs automatic exposure control so that the detected value ASA of the average level of the signal amplitude obtained from the detector 15 is constant. When the signal amplitude is large in imaging in a bright environment, the control unit 12b controls the aperture of the lens 1 to reduce the amount of light incident on the CCD image sensor 2, or the forced charge discharge timing by the timing generation unit 10 In this adjustment, the exposure time is reduced by controlling to forcibly discharge the charge accumulated in the photoelectric conversion element of the CCD image pickup device 2.

  When the signal amplitude becomes small due to imaging in a dark environment, the control unit 12b amplifies the imaging signal by controlling to increase the amplification gain of the programmable gain amplification unit 4. However, if the amplification gain is too large, noise becomes conspicuous and image quality deteriorates. As another method, the control unit 12b can extend the exposure time by controlling to read out charges from the photoelectric conversion element of the CCD image pickup device 2 in units of vertical scanning periods. However, if the exposure time is too long, the moving subject becomes an afterimage and the image quality deteriorates. Furthermore, a section for performing interpolation (frame interpolation) of missing images in units of vertical scanning periods is required.

Similar to the first embodiment, the control unit 12b of the present embodiment can change the sensitization magnification L to the three-dimensional pixel addition unit 6, and sets it to 1 to 20 times, for example. The sensitization magnification L is set (adjusted) based on the illuminance information from the detection unit 15 and the exposure parameters. Based on the set sensitization magnification L, the sensitization magnification La in the in-plane pixel addition and the sensitization magnification Lb in the inter-plane pixel addition are determined.
As described in the first embodiment, the addition coefficient used for pixel addition in the pixel addition unit 595 of the in-plane pixel addition unit 20 is adjusted according to the set sensitization magnification La, and the set sensitization magnification is set. The addition coefficient used for pixel addition in the pixel addition unit 39 of the inter-plane pixel addition unit 30 is adjusted according to Lb. Therefore, the addition coefficient in the in-plane pixel addition and the addition coefficient in the inter-plane pixel addition are adjusted based on the illuminance information.

Hereinafter, an example of a procedure for sensitivity adjustment when the subject illuminance changes will be described. First, description will be made assuming that the exposure time is maintained at a constant value (standard exposure time) Tr.
When the subject illuminance gradually decreases and the detection value ASA of the average level of the signal amplitude decreases (FIG. 30E), the aperture of the lens 1 is controlled in the open direction (range Sa in FIG. 30A). ), Maintaining the average level of signal amplitude. Here, “maintaining the average level of the signal amplitude” means maintaining the average level of the signal amplitude at the output of the three-dimensional pixel adder 6, and thus the average level of the signal amplitude expressed by the output of the detector 15. It means to do.

  After the aperture of the lens 1 is opened (fully opened), control is performed to increase the amplification gain of the programmable gain amplifier 4 (range Sb in FIG. 30B), and the average level of the signal amplitude is maintained. After the amplification gain of the programmable gain amplification unit 4 increases and becomes larger than the predetermined upper limit value UGL of the amplification gain, the sensitization magnification L of the three-dimensional pixel addition unit 6 is controlled to increase (FIG. 30). (C) Range Sc), maintaining the average level of the signal amplitude.

  The average level ASA can be maintained by controlling the sensitization magnification L until the sensitization magnification L reaches the maximum value (L = 20). When the subject illuminance further decreases, the average level ASA starts to decrease.

  In the standard exposure time Tr, the illuminance HL at which the output of the three-dimensional pixel adding unit 6 is at a predetermined level when the lens aperture is opened, the amplification gain is maximized, and the sensitization magnification L is 1, is set to the high illuminance side reference value. When the illuminance is 1/20 of the high illuminance reference value HL, that is, at the standard exposure time Tr, the lens aperture is opened, the amplification gain is maximized, and the sensitization magnification L is 20, the three-dimensional pixel adding unit 6 Is the low illuminance side reference value.

  When the illuminance of the subject gradually becomes brighter and exceeds the low illuminance side reference value LL and the detection value ASA of the average level of the signal amplitude is about to increase, control is performed to reduce the sensitization magnification L of the three-dimensional pixel addition unit 6. (Range Sc in FIG. 30C), the average level ASA of the signal amplitude is maintained. When the sensitization magnification L is reduced to 1 time, the control is performed so as to reduce the amplification gain of the programmable gain amplification unit 4 to maintain the average level ASA of the signal amplitude. After the amplification gain of the programmable gain amplifier 4 decreases (range Sb in FIG. 30B) and becomes smaller than the predetermined lower limit value LGL, the aperture of the lens 1 is controlled in the light shielding direction (FIG. 30 ( The average level of the signal amplitude is maintained in the range Sa) of a) (FIG. 30 (e)). As the illuminance further increases, the average level ASA increases.

  As a result of the above control, the average level ASA of the signal amplitude can be kept constant in the range from the lower limit LL to the upper limit UL as shown by the solid line in FIG.

Although the exposure time is constant, the exposure time may be controlled according to the illuminance of the subject. For example, even when the illuminance decreases and the sensitization magnification L reaches the maximum value, the exposure time may be increased when the signal amplitude does not reach a sufficient value (range in FIG. 30D). Sd). Conversely, if the illuminance increases and the diaphragm is maximized (the F value is maximized), or if the signal amplitude is too large, the exposure time may be shortened (see FIG. 30D). Range Se).
When such exposure time control is applied, the average level ASA of the signal amplitude can be kept constant in the range from the lower limit LLe to the upper limit ULe, as shown by the dotted line in FIG.

  The predetermined upper limit value UGL of the amplification gain is determined based on the noise level detection value ANL detected by the video signal processing unit 7. (Considering that it is necessary to increase the amplification gain when the illuminance of the subject decreases and the S / N of the output of the image sensor 2 decreases accordingly) The detected value ANL of the noise level is the average level of the signal amplitude. The amplification gain of the programmable gain amplifier 4 when the predetermined noise ratio (first predetermined noise ratio, that is, the allowable upper limit value) NPR1 with respect to the detection value ASA is set as the predetermined upper limit value UGL. The first predetermined noise ratio NPR1 is set to 1/50, for example.

The noise level detection value ANL is obtained by extracting noise components by noise reduction processing and dividing the sum of absolute values of noise components in the entire effective pixel range by the total number of effective pixels. By the noise reduction process, a noise reduction signal NRS in which the noise of the input signal is reduced is obtained. A noise component can be extracted by subtracting the noise reduction signal NRS from the input signal (a signal before the noise reduction process is performed in the video signal processing unit 7). The “calculated value” of the noise level obtained as described above may be referred to as “detected value”.

Since the allowable noise level for visual recognition of the subject varies depending on the application, the first predetermined noise ratio NPR1 varies depending on the application of the imaging apparatus, such as whether S / N is important or image resolution is important. The control unit 12b dynamically observes the gain set in the programmable gain amplification unit 4 and the detection value ANL of the noise level supplied from the video signal processing unit 7 to the control unit 12b, and dynamically increases a predetermined upper limit of the amplification gain. The programmable gain amplification unit 4 and the three-dimensional pixel addition unit 6 may be controlled by determining the value UGL, and the noise level detection value ANL is the average signal amplitude detection value ASA before the imaging device is shipped from the factory. A storage unit (nonvolatile memory or battery) that measures the amplification gain reaching the first predetermined noise ratio NPR1 and retains the stored content as the predetermined upper limit value UGL even when the power of the imaging device is turned off. The control unit 12b refers to the predetermined upper limit value UGL of the amplification gain, and can be programmed. Amplifying unit 4 and may be controlled three-dimensional pixel addition section 6.

  The predetermined lower limit value LGL of the amplification gain is determined based on a noise level detection value ANL supplied from the video signal processing unit 7 to the control unit 12b. The amplification gain of the programmable gain amplifying unit 4 when the noise level detection value ANL falls below a predetermined noise ratio (second predetermined noise ratio) NPR2 with respect to the detection value ASA of the average level of the signal amplitude is the predetermined gain. The lower limit is LGL. The second predetermined noise ratio is determined based on the first predetermined noise ratio NPR1 and the sensitivity magnification (20 times) by the three-dimensional pixel addition unit 6. For example, the second predetermined noise ratio NPR2 is defined as 1/1000 (= (1/50) × (1/20)).

Since the allowable noise level for visual recognition of the subject differs depending on the application, the second predetermined noise ratio NPR2 varies depending on the application of the imaging device, such as whether S / N is important or image resolution is important. The control unit 12b dynamically observes the gain set in the programmable gain amplification unit 4 and the detection value ANL of the noise level supplied from the video signal processing unit 7 to the control unit 12b, and dynamically lowers the predetermined lower limit of the amplification gain. The programmable gain amplification unit 4 and the three-dimensional pixel addition unit 6 may be controlled by determining the value LGL, and the noise level detection value ANL becomes the average signal amplitude detection value ASA before the image pickup apparatus is shipped from the factory. , The amplification gain reaching the second predetermined noise ratio NPR2 is measured, and the predetermined lower limit value LGL is written in the storage unit 16 that can retain the stored contents even when the imaging apparatus is turned off. The programmable gain amplifier 4 and the three-dimensional pixel adder 6 may be controlled with reference to the predetermined lower limit value LGL of the amplification gain.

  The control unit 12b controls the signal amplitude adjustment function by controlling the aperture of the lens 1, the exposure time of the CCD image pickup device 2, the amplification gain of the programmable gain amplification unit 4, and the pixel addition in the three-dimensional pixel addition unit 6. (The average level of the signal amplitude of the output of the three-dimensional pixel adder 6) is maintained.

  Since the control as described above is performed, the conversion value obtained by performing the reverse calculation based on the exposure control parameter with respect to the output of the detection unit 15 is a value corresponding to the illuminance of the subject. That is, it is determined whether the illuminance obtained by such back calculation is higher than the high illuminance side reference value, lower illuminance side luminance value, or in the range between these reference values, and pixel addition is performed according to the determination result. Control (adjustment of sensitization sensitivity) can be performed.

  Since it is configured as described above, it is possible to output an image with optimal brightness with good visibility by sequentially switching aperture control, amplification gain control, pixel addition control, and exposure time control during exposure control. There is an effect that can be done.

  Further, the pixel addition unit is configured so that the sensitization magnification can be set not by the number of pixels to be added but by the addition coefficient, and the sensitization magnification L can be set by a value including not only an integer but also a decimal. Among them, the pixel addition control is also seamlessly switched using the addition coefficient with a value after the decimal point, and there is an effect that an easy-to-view image can be output without a sudden change in the brightness of the image in the process of illuminance change.

Embodiment 6 FIG.
FIG. 31 shows an imaging apparatus according to Embodiment 6 of the present invention. In FIG. 31, except for the point that the photometric unit 17 is added and the point that the control unit 12c is provided instead of the control unit 12, the same effect as the first embodiment is obtained.

  The photometry unit 17 measures the illuminance of the subject in the direction of incidence of light on the lens 1. Mounting and positioning of an illuminance sensor (not shown) of the photometry unit 17 is determined based on the optical axis of the lens, and the illuminance of the subject imaged by the lens 1 is measured.

  The control unit 12c is similar to the control unit 12 of the first embodiment, but has an additional function as follows. That is, the control unit 12 c controls the aperture of the lens 1 based on the illuminance value supplied from the photometry unit 17, the charge readout timing from the photoelectric conversion element of the CCD image sensor 2 generated by the timing generation unit 10, and the charge forcing. Control of the discharge timing (accordingly, control of charge accumulation time, that is, exposure time), control of the amplification gain of the programmable gain amplifier 4 and control of the pixel addition processing of the three-dimensional pixel adder 6 are performed.

In accordance with the set value table stored in the storage unit 16, the control unit 12 c performs the aperture of the lens 1, the exposure time of the CCD image pickup device 2, the amplification gain of the programmable gain amplification unit 4, and the sensitization of the three-dimensional pixel addition unit 6. Set the magnification.
In the set value table, the aperture of the lens 1, the exposure time of the CCD image pickup device 2, the amplification gain of the programmable gain amplification unit 4, and the sensitization magnification of the three-dimensional pixel addition unit 6 are registered for each illuminance value.

  When the illuminance is bright, the exposure time of the image sensor 2 is the standard exposure time Tr based on the frame rate, the amplification gain of the programmable gain amplification unit 4 is 1 time, and the sensitization magnification L of the 3D pixel addition unit 6 is 1 time. And the aperture of the lens 1 is reduced (range Sa in FIG. 30A). When the lens 1 reaches the maximum aperture and the illuminance further increases, the exposure time of the image sensor 2 is controlled to be shorter than the standard exposure time Tr (range Se in FIG. 30D).

  When the illuminance becomes dark, the exposure time of the image sensor 2 is the standard exposure time Tr based on the frame rate, the amplification gain of the programmable gain amplification unit 4 is 1 time, and the sensitization magnification L of the 3D pixel addition unit 6 is 1 time. After setting, the aperture of the lens 1 is opened (range Sa in FIG. 30A). When the lens 1 is opened and the illuminance is further darkened, the amplification gain of the programmable gain amplifier 4 is increased from 1 (range Sb in FIG. 30B). The amplification gain of the programmable gain amplifier 4 is the above upper limit value (the value of the amplification gain when the level of noise included in the output of the three-dimensional pixel adder 6 reaches the first predetermined ratio, that is, When the noise level becomes darker when the noise level becomes a maximum gain value that satisfies the condition that the first predetermined ratio (the upper limit value of the allowable range) is not exceeded, the sensitization magnification of the three-dimensional pixel addition unit 6 is increased. It is increased from 1 time (range Sc in FIG. 30C). When it becomes darker, the exposure time is lengthened (range Sd in FIG. 30D).

  Since it is configured as described above, an aperture image, amplification gain control, pixel addition control, and exposure time control can be sequentially switched during exposure control to output an image with optimal brightness with good visibility. There is an effect that can be done.

  Further, the pixel addition unit is configured so that the sensitization magnification can be set not by the number of pixels to be added but by the addition coefficient, and the sensitization magnification L can be set by a value including not only an integer but also a decimal. Among them, the pixel addition control is also seamlessly switched using the addition coefficient with a value after the decimal point, and there is an effect that an easy-to-view image can be output without a sudden change in the brightness of the image in the process of illuminance change.

  In the fifth and sixth embodiments, it has been described that the exposure time is lengthened when the signal amplitude is not sufficient even when the sensitization magnification is maximized. This is a result of giving priority to keeping the frame rate unchanged. It is. When the resolution is more important than the frame rate, the control to increase the exposure time is performed first, and the sensitization magnification is increased when the signal amplitude is not sufficient even if the exposure time is increased (for example, up to a predetermined value). Alternatively, the control for increasing the sensitization magnification and the control for extending the exposure time may be performed in parallel.

    DESCRIPTION OF SYMBOLS 1 Lens, 2 CCD image pick-up element, 3 Correlation double sampling process part, 4 Programmable gain amplification part, 5 A / D conversion part, 6 Three-dimensional pixel addition part, 7 Video signal processing part, 8 Video signal output terminal, 9 Drive Unit, 10 timing generation unit, 11 synchronization signal generation unit, 12, 12b, 12c control unit, 13, 13b imaging signal generation unit, 14 CMOS image sensor, 15 detection unit, 16 storage unit, 17 photometry unit, 20 in-plane pixel Addition unit, 30 inter-pixel addition unit, 21 pixel extraction unit, 22 region selection unit, 23 selection addition unit, 24 signal combination unit, 31 pixel extraction unit, 35 addition pixel selection unit, 39 pixel addition unit, 350 signal separation unit 351-354, 651-654 pixel selection unit, 361-364, 661-664 pattern discrimination unit, 81-384, 681-684 Correlation determination unit, 570 pixel selection unit, 571-582 change width calculation unit, 585 minimum value calculation unit, 587 pixel designation unit, 590 correlation region detection unit, 593 pixel selection unit, 595 pixel addition unit 3611 to 3619, 6611 to 6619 discrimination unit, 36111, 66111, 66211 signal separation unit, 36112, 66112, 66212 judgment unit.

Claims (12)

An imaging signal generation unit that performs imaging and generates an imaging signal indicating a pixel value for each pixel of a plurality of pixels that constitute each temporally continuous frame;
Based on the imaging signal generated by the imaging signal generation unit, the pixels in the frame are sequentially designated in each of the successive frames, and the designated pixels are designated in the same frame for the designated pixels. A pixel having a high correlation between the pixel value and the designated pixel is selected from the pixels located around the pixel, and the pixel value of the selected pixel and the pixel value of the designated pixel are added and added. An in-plane pixel adder that outputs the result as an in-plane sensitization signal for the designated pixel;
The consecutive frames are sequentially designated as the attention frame, the pixels in the attention frame are sequentially designated as the attention pixels, and for each attention pixel, from each of one or more frames located in the vicinity of the attention frame, the For each of the pixels located at the same position as the target pixel and its surroundings, the correlation of the in-plane sensitization signal with the target pixel is compared with a correlation presence / absence determination threshold value, and it is determined that there is a correlation by the comparison determination. A pixel having a high correlation is selected from the selected pixels, and the in-plane sensitization signal for the pixel selected from each of the one or more neighboring frames and the in-plane sensitization for the pixel of interest possess a plane between the pixel adding unit for outputting the addition result by adding the signals as a three-dimensional sensitization signal,
The in-plane pixel addition unit
A region composed of pixels having a high correlation with each other is selected from a plurality of predetermined regions each formed by the designated pixel and its surrounding pixels, and the position of the pixel in the selected region And an area selection unit that outputs an addition pixel pattern code indicating the pattern of the selected area,
For the imaging signal of the designated pixel, the addition result obtained by adding the imaging signals of the pixels included in the area selected by the area selection unit is output as an in-plane sensitization signal for the designated pixel. And a selective addition unit to
The inter-plane pixel addition unit
The in-plane sensitization signal and the added pixel pattern code for pixels located in the same position as the target pixel and in the vicinity thereof in each of the frames adjacent to the target frame among the neighboring frames, and Correlation presence / absence determination for determining presence / absence of correlation between the in-plane sensitization signal and the added pixel pattern code for the target pixel and the in-plane sensitization signal for the target pixel and the in-plane sensitization signal other than the target pixel An imaging device that selects one pixel from the frame based on a result of comparison with a threshold . The imaging apparatus according to claim 1 , wherein the pattern is defined by a relative position or orientation of an area including the designated pixel and surrounding pixels with respect to the designated pixel. The imaging apparatus according to claim 2 , wherein the pattern is also defined by a shape of an area composed of the designated pixel and surrounding pixels. The inter-plane pixel addition unit
At least in each of the frames adjacent to the target frame, among the pixels located at the same position as the target pixel and its surroundings, there is a correlation with the target pixel based on the comparison result with the correlation presence / absence determination threshold value. When the determined pixel is present and there is a pixel having the same added pixel pattern code as that of the target pixel, the target pixel and the surface among the pixels having the same added pixel pattern code Select the pixel with the highest correlation of the internal sensitization signal,
At least in each of the frames adjacent to the target frame, it is determined that there is a correlation with the target pixel based on the comparison result with the correlation presence / absence determination threshold among the pixels located at the same position as the target pixel and its surroundings. If there is no pixel that has been changed, or if there is no pixel that has the same pixel code addition as the pixel of interest, among the pixels located at the same position as the pixel of interest and its surroundings, The imaging apparatus according to any one of claims 1 to 3 , wherein a pixel having the highest correlation between the target pixel and the in-plane sensitization signal is selected. The in-plane pixel addition unit
A pixel extraction unit that delays the imaging signals generated by the imaging signal generation unit by different times and simultaneously extracts signals representing pixel values of the designated pixel and surrounding pixels;
The region selection unit
Of the pixels extracted by the pixel extraction unit, a plurality of combinations of the designated pixel and one or more pixels located in the vicinity thereof are formed, and the combination of the plurality of combinations claim difference between the maximum value and the minimum value of the pixel values of the pixels constituting the combination combinations is the minimum, and selecting as constituting an area composed of pixels of high correlation between the mutual 1 5. The imaging device according to any one of items 1 to 4 . The inter-plane pixel addition unit
The addition pixel pattern code output from the in-plane pixel addition unit is delayed by different times, and the addition pixel pattern code for the target pixel of the target frame is positioned in the neighboring frame, and the target pixel A pattern code extraction unit that simultaneously extracts the added pixel pattern code for pixels located at the same position as the pixel and its surroundings;
At least in each of the frames adjacent to the target frame, the added pixel pattern code for each of the pixels located at and around the same position as the target pixel is the added pixel pattern code for the target pixel of the target frame. further comprising a pattern determination unit which performs the same whether the determined imaging apparatus according to any one of claims 1 to 5, characterized in. The inter-plane pixel addition unit
The in-plane sensitization signal output from the in-plane pixel addition unit is delayed by different times, and is located in the in-plane sensitization signal for the pixel of interest in the frame of interest and in the neighboring frame, An in-plane sensitization signal extracting unit that simultaneously extracts the in-plane sensitization signals for pixels located at the same position as the target pixel and its surroundings;
At least in each of the frames adjacent to the target frame, the in-plane sensitization signal for each of the pixels located at and around the same position as the target pixel and the in-plane increase for the target pixel of the target frame. A correlation discriminating unit that selects one of the pixels located at the same position as the target pixel and its surroundings in the frame based on a comparison result between the sense signal and the correlation presence / absence determination threshold value. The imaging device according to claim 6 . The in-plane pixel addition unit
A signal combining unit that combines the in-plane sensitization signal and the addition pixel pattern code to generate and output a composite signal;
The inter-plane pixel addition unit
The composite signal output from the signal combining unit of the in-plane pixel addition unit is delayed by different times and located in the composite signal for the pixel of interest of the frame of interest and the neighboring frame, An extraction unit that simultaneously extracts the composite signal for pixels located at the same position as the target pixel and its surroundings;
At least in each of the frames adjacent to the target frame, the added pixel pattern code included in the composite signal for each of the pixels located at and around the same position as the target pixel is the target pixel of the target frame. A pattern discriminating unit for determining whether or not it is the same as the added pixel pattern code included in the composite signal;
At least in each frame adjacent to the target frame, the in-plane sensitization signal and the target pixel of the target frame included in the composite signal for each of the pixels located at and around the same position as the target pixel The same position as the target pixel in the frame based on the comparison result between the in-plane sensitization signal included in the composite signal and the correlation presence / absence determination threshold value and the determination result in the pattern determination unit the imaging apparatus according to any one of claims 1 to 5, characterized in that it comprises a correlation determination unit which selects one of the pixels located in the neighborhood. The neighboring frames include not only a frame adjacent to the frame of interest but also a frame that is two frames or more away from the frame of interest,
The pattern discrimination unit
The addition pixel pattern code for pixels in a frame that is two frames or more away from the frame of interest is the addition pixel pattern code for pixels selected in another frame adjacent to the frame on the frame of interest side Determine whether they match,
The correlation determination unit includes the in-plane sensitization signal for a pixel in a frame that is two frames or more away from the frame of interest, and a pixel selected in another frame adjacent to the frame on the frame of interest. On the basis of the comparison result between the in-plane sensitization signal and the correlation presence / absence determination threshold value and the determination result in the pattern determination unit, the pixel at the same position as the target pixel and its peripheral position in the frame The image pickup apparatus according to claim 7 or 8 , wherein one of them is selected. The neighboring frame is
The imaging apparatus according to any one of claims 1 9, characterized in that it comprises at least one frame after the frame and one frame period of one frame period prior to the frame of interest. The neighboring frame is
A frame one frame period before the frame of interest;
A frame two frames before the frame of interest;
A frame after one frame period with respect to the frame of interest;
The imaging apparatus according to any one of claims 1 to 10 , further comprising a frame after two frame periods with respect to the frame of interest. At least one of the addition of the pixel value in the in-plane pixel addition unit and the addition of the in-plane sensitization signal in the inter-plane pixel addition unit is performed by weighted addition using an addition coefficient determined based on the sensitization magnification. Done,
An illuminance information generator for generating illuminance information representing the illuminance of the subject;
The imaging apparatus according to any one of claims 1 to 11, further comprising a controller for determining the sensitizing factor on the basis of the illuminance information.

Images