JP2004304380A - Photographing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a photographing apparatus capable of simply correcting the image shake in real time. <P>SOLUTION: The photographing apparatus is provided with: a first imaging element 10 for carrying out ordinary photographing a second imaging element 20 for imaging an extracted luminance-changing part; and an image processing circuit 40 for using an imaging signal from the second imaging element 20 to apply processing to eliminate the image shake (image shake caused by camera-shake or the like) of the image taken by the first imaging element 10. When a part arises where luminance is changed by the mobile body existing in the photographing area seen from the second imaging element, the second imaging element 20 outputs an imaging signal of a level different from the level of an imaging signal denoting the other parts whose luminance does not change in the photographing area. The image processing circuit 40 carries out image shake elimination processing including e.g., processing of subtracting the imaging signal outputted from the element 20 from the imaging signal outputted from the element 10. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an imaging apparatus capable of removing image blurring during imaging of a shooting target area.
[0002]
[Conventional technology and its problems]
When an image of a subject is picked up by an image pickup device, if a hand or the like of a person who supports the image pickup device shakes, image shake may occur as if the image flows in an obtained image. Conventionally, electronic correction and optical correction have been known as techniques for correcting such camera shake (image shake). For example, Patent Literature 1 is known as electronic correction, and Patent Literature 2 is known as optical correction.
[0003]
Electronic correction cannot correct camera shake in real time within one imaging frame. Optical correction can be performed in real time irrespective of the frame, but requires a detector for detecting camera shake and a device for controlling the optical system.
[0004]
[Patent Document 1] JP-A-4-255179
[Patent Document 2] JP-A-3-200299
[0005]
[Problems to be solved by the invention]
Therefore, an object of the present invention is to provide an imaging apparatus that can easily correct image shake in real time.
[0006]
[Means for Solving the Problems]
One of the present inventors has previously developed an imaging device capable of extracting and imaging a moving object simply and quickly without requiring complicated image processing (Japanese Patent Application No. 2002-375029). This image sensor, when viewed from the image sensor, has a part whose luminance changes due to a moving object in a subject, and indicates the presence of a moving object at a level different from that of an image signal representing another part of the subject whose luminance does not change. It can output an imaging signal. For example, it can generate an imaging signal that can display at least a part of a moving object with an image density different from that of a still background portion.
[0007]
According to this imaging device, it is possible to detect a change in luminance of a subject extending between frames, and to obtain information on a portion captured as a shake of the subject in a normal solid-state imaging device.
[0008]
The present inventor has paid attention to this point in the course of conducting application research on such an imaging element capable of moving body extraction imaging, combining this imaging element with an imaging element that performs normal imaging, and by using these, the same imaging area is formed at the same time. It has been found that by processing an imaging signal obtained by an ordinary imaging element using an imaging signal obtained by an imaging element capable of capturing and extracting a moving object, it is possible to easily and quickly remove an image shake caused by a camera shake or the like.
[0009]
Based on such knowledge, the present invention provides an imaging device that solves the above-described problems.
A first image sensor that normally captures an image of the imaging target area;
A second image pickup device capable of extracting and imaging the portion where the luminance changes in the imaging target region when the portion is changed;
An image processing unit that removes image shake of an image captured by the first image sensor using an image signal from the second image sensor;
An imaging device provided with:
It is preferable that the imaging region of the second imaging device is the same as the imaging region of the first imaging device. However, as long as the two regions are not different regions having no overlap at all, the two regions are not completely the same region. In both cases, the present invention is established. However, when the entire imaging region of the first imaging device is set as the target region of the image blur removal processing, the imaging region of the second imaging device needs to cover the entire imaging region of the first imaging device.
[0010]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described.
The imaging apparatus according to the embodiment includes a first imaging element that normally captures an imaging target area, a second imaging element that can extract and image a luminance change portion due to a moving object or the like in the imaging target area, An element controller for controlling the elements and an image processing unit are included. The image processing unit uses an image pickup signal (image pickup data) from the second image pickup device to remove image shake of an image picked up by the first image pickup device.
[0011]
When there is a portion in which the luminance changes due to a moving object or the like in the imaging area (imaging target area) under the control of the element controller, the second imaging element includes another part in the imaging area where the luminance does not change. Is output at a level different from that of the image signal representing the image signal.
[0012]
According to this imaging device, the second imaging device and the first imaging device that performs normal imaging simultaneously image the same subject (the same imaging region when viewed from the imaging device), and the image processing unit uses the second imaging device By processing an imaging signal (imaging data) obtained by the normal imaging device using the obtained imaging signal (imaging data), it is possible to easily remove image shake in a captured image of the normal imaging device due to camera shake or the like in real time. be able to.
[0013]
This imaging apparatus can remove not only image shake caused by a shake of a hand or the like of a person supporting an image sensor, but also image shake caused by a moving object moving at high speed in an image pickup area. Even if the moving body that moves at high speed is two or more objects that move in different directions, the image blur caused by the moving body can be removed.
[0014]
The processing performed by the image processing unit to remove the image blur of the image captured by the first image sensor using the imaging signal from the second image sensor includes:
(1) a process including a process of subtracting an imaging signal (imaging data) output from the second imaging device from an imaging signal (imaging data) output from the first imaging device for normal imaging;
(2) A process including a process of adding an imaging signal (imaging data) output from the second imaging device to an imaging signal (imaging data) output from the first imaging device can be exemplified.
[0015]
In the case of performing the subtraction processing, for example, assuming that the moving object moves from the position p1 to the position p2 in one frame, the image data after the image shake removal when the moving object is at the position p2 is obtained, and the addition processing is performed. Can obtain image data after image blur removal when the moving object is at the position p1.
[0016]
A switching instruction unit that instructs switching of the image blur removal processing in the image processing unit, wherein the image processing unit subtracts an imaging signal output from the second imaging element from an imaging signal output from the first imaging element; or An image blur removal process including a process of adding an image signal output from the second image sensor to an image signal output from the first image sensor is performed, and the subtraction process is performed on the image processing unit by the switching instruction unit. Alternatively, an instruction to switch the addition process may be made possible.
This is advantageous for operations such as checking a still image without shake while moving forward and backward by frames.
[0017]
In any case, it is preferable that the image processing unit performs an appropriate offset process along with the subtraction, the addition, and the like in order to obtain a good image.
In order to facilitate arithmetic processing in the image processing unit, it is preferable that the number of imaging pixels and the pixel array pitch be the same between the first image sensor and the second image sensor.
[0018]
The following are typical examples of the second image sensor. That is,
Each is a photoelectric conversion element, a logarithmic conversion unit including a logarithmic conversion transistor for converting the output of the photoelectric conversion element into an electric signal proportional to the logarithmic value of the amount of light incident on the element, and the output from the logarithmic conversion unit. A plurality of pixels having an integrating circuit for storing;
An output circuit for receiving and outputting a signal from each of the pixels,
Under the control of the element controller, a first signal output from the photoelectric conversion element via the logarithmic conversion unit is applied to the integration circuit while a first voltage for imaging is applied to the logarithmic conversion transistor during imaging. When the logarithmic conversion transistor is reset, a second voltage is applied to the logarithmic conversion transistor to reset the transistor, and a second signal obtained by the reset is stored in the integration circuit. When there is a portion in which the luminance changes due to a moving object or the like in the imaging target region by making the voltage different from the voltage when the normal imaging state is obtained, the portion includes the first signal stored in the integration circuit and the first signal stored in the integration circuit. A difference-changed part extraction imaging state is realized in which the difference from the second signal is different from that of the other part of the object area where the luminance does not change. ,
The output circuit is an image sensor that outputs a signal corresponding to a difference between the first signal and the second signal.
[0019]
By the way, in the case where the moving body moves in the imaging area, for example, focusing on the edge of the moving body, the luminance of the stationary background portion with respect to the edge changes when the moving body edge moves, as viewed from the imaging device. Therefore, it can be said that a portion in the imaging region where the luminance changes depending on the moving object indicates at least a part of the moving object.
[0020]
Therefore, in the second imaging device just described,
"When there is a portion where the luminance changes in the shooting target area, the difference between the first signal and the second signal stored in the integration circuit is the same for the portion where the brightness in the shooting target area does not change. To realize a brightness change portion extraction imaging state that is different from that for the portion of
"When a moving object is included in the imaging region (imaging target region), for at least a part of the moving object, the difference between the first signal and the second signal stored in the integration circuit is included in the imaging region. A moving object extraction imaging state that is different from that of the still background portion to be obtained, "or
"When a moving object is included in an imaging region (imaging target region), a moving object extraction imaging state is generated in which an imaging signal that can display at least a part of the moving object at a different density from a still background portion included in the imaging region is realized. Then, it can be rephrased.
[0021]
In any case, as the above-mentioned “making the second voltage different from the voltage when the normal imaging state is obtained”, for example, “the absolute value of the difference between the second voltage and the first voltage is the normal imaging state” Is set to be smaller than the value in the case of obtaining (for example, to be substantially half the value in the case where the normal imaging state is realized) "or" without changing the magnitude of the second voltage. And setting the application period (pulse width of the second voltage) of the second voltage to be shorter than when the normal imaging state is obtained. In short, as will be understood from the description below, the voltage may be different from the voltage in the case where the normal imaging state is obtained so that the reset remains in the logarithmic conversion transistor.
[0022]
Further, a switch for electrically connecting and disconnecting the photoelectric conversion element and the logarithmic conversion transistor may be further provided. In this case, the image sensor controller may be configured to reset the transistor by applying the second voltage to the logarithmic conversion transistor in a state where the switch is turned off.
According to the second image pickup device, an image pickup signal capable of displaying a still background portion with a uniform image density and at least a part of a moving object with a "bright" or "dark" image density than the still background portion can be obtained. It is possible to get. More broadly, it is possible to obtain an image signal that can be displayed with a “bright” or “dark” image density in a portion where the luminance changes in the imaging region than in a portion where the luminance does not change.
[0024]
Therefore, when employing the second image sensor, the image processing unit processes the image signal obtained by the second image sensor, of which the “bright” signal is brighter and the “dark” signal is darker, Thereafter, it may be used for image blur removal processing. This makes it possible to correct image shake more accurately.
[0025]
Next, some specific examples of the imaging device will be described with reference to the drawings.
<Imaging device example 1>
(1) Overall configuration of imaging device example 1
This imaging device (camera) has the configuration shown in FIG.
The imaging apparatus shown in FIG. 1 includes a solid-state imaging device 10 and another solid-state imaging device 20 that perform normal imaging of a subject (an imaging region viewed from the imaging device), and further controls the operation of these imaging devices. It includes an element controller 30 and an image processing circuit 40. The solid-state imaging device 20 is configured to extract and image a luminance change portion due to a moving object or the like in an imaging region, that is, to perform luminance change portion extraction imaging (hereinafter, referred to as “moving object extraction imaging”).
[0026]
The incident light (image light) from the subject is guided to the half mirror M via the optical system 50, passes through the half mirror M, and forms an image on the first image sensor 10, while being reflected by the half mirror M and the second image. An image is formed on the image sensor 20.
[0027]
The imaging signals output from the imaging devices 10 and 20 are digitized by an analog / digital converter A / D and input to the image processing circuit 40. The operation of each of the imaging devices 10 and 20 is controlled by the device controller 30.
[0028]
(2) First imaging element 10 for normal imaging
Next, the first imaging device 10 for normal imaging will be described in detail with reference to FIGS. FIG. 2 is a block diagram illustrating the overall configuration of the image sensor 10. FIG. 2 also shows the element controller 30 described above. FIG. 3 is a diagram showing a configuration of a pixel portion or the like in which pixels are arranged, FIG. 4 is a diagram showing a configuration of one pixel, and FIG. 5 is a timing chart showing driving voltage application when realizing a normal imaging state.
[0029]
(2-1) Overall configuration and operation of the image sensor 10
As shown in FIG. 2, the image sensor 10 includes a pixel unit G, a vertical scanning circuit 1 and an output circuit 8 connected to the pixel unit G, and the output circuit 8 is connected to the horizontal scanning circuit 2 and an output amplifier. Am is connected. An imaging signal is output from the output amplifier Am to the image processing circuit 40 via the converter A / D.
[0030]
The imaging element 10 also includes a voltage regulator Rg for supplying various predetermined voltage signals and the like to the pixel unit G and the like under the instruction of the element controller 30. Further, a timing generator TG for operating each unit at a predetermined timing under the instruction of the element controller 20 is also provided.
[0031]
As shown in FIG. 3, pixels G11 to Gmn for imaging are arranged in a matrix (matrix arrangement) in the pixel unit G. The vertical scanning circuit 1 sequentially scans row (row) lines 31, 32,..., 3n for applying a scanning signal φV to each pixel, and outputs each pixel via lines 41, 42,. Is supplied to a capacitor C described later.
[0032]
The horizontal scanning circuit 2 sequentially reads out the photoelectric conversion signals derived from each pixel to the output signal lines 61, 62,... 6m in the horizontal direction (row direction) for each pixel. In FIG. 3, reference numeral 5 denotes a power supply line.
In FIG. 4, which will be described later, the row (row) line is indicated by 3, the φVD supply line is indicated by 4, and the output signal line is indicated by 6.
[0033]
3n, 41, 42,... 4n, output signal lines 61, 62,... 6m, power supply line 5, and other lines ( For example, a clock line and a bias supply line are also connected, but these are not shown in FIG.
[0034]
The output circuit 8 shown in FIG. 2 includes the constant current power supplies 71, 72,... 7m, the selection circuits 81, 82,.
[0035]
7m are connected to output signal lines 61, 62, ... 6m, respectively. The selection circuits 81, 82,..., 8m are circuits that sample and hold video signals and noise signals given from the pixels G11 to Gmn via the signal lines 61 to 6m. When the video signal and the noise signal are sequentially transmitted from the selection circuits 81, 82,... 8m, the correction circuit 9 performs a correction process, that is, a process of subtracting the difference between the video signal and the noise signal from the video signal. The video signal from which noise has been removed is output to the outside for image display. Note that a DC voltage VPS is applied to one end of each of the constant current sources 71 to 7m.
[0036]
In the solid-state imaging device 10, a video signal and a noise signal output from the pixel Gab (a: a natural number of 1 ≦ a ≦ m, b: 1 ≦ b ≦ n) are output to the output signal line 6a, respectively. And is amplified by a constant current power supply 7a connected to the output signal line 6a. The video signal and the noise signal output from the pixel Gab are sequentially sent to the selection circuit 8a, and the video signal and the noise signal are sampled and held in the selection circuit 8a. After the held video signal is transmitted to the correction circuit 9, the sampled and held noise signal is also transmitted to the correction circuit 9.
[0037]
The correction circuit 9 corrects the video signal supplied from the selection circuit 8a based on the noise signal also supplied from the selection circuit 8a, amplifies the noise-removed video signal via the output amplifier Am, and outputs the amplified signal. Output to As a configuration example of each of the selection circuits 81 to 8m and the correction circuit 9, there is a configuration presented by the present applicant in Japanese Patent Application Laid-Open No. 2001-223948. .. 8m may include a correction circuit.
[0038]
(2-2) Pixel Configuration of Image Sensor 10
Next, one example of each of the pixels G11 to Gmn will be described with reference to FIG.
The pixel shown in FIG. 4 is a photodiode PD, which is an example of a photoelectric conversion element, and a logarithmic conversion MOS transistor T2 for converting the output of the photodiode PD into an electric signal proportional to the logarithmic value of the amount of incident light. And an integration circuit IT including a capacitor C for storing the output of the logarithmic conversion unit L.
[0039]
More specifically, in each pixel, the drain of the switching MOS transistor T1 is connected to the cathode of the photodiode PD whose anode potential is VPD (or may be the ground potential), and the source of the transistor T1 is a MOS transistor for logarithmic conversion. The gate and drain of T2 are connected to the gate of MOS transistor T3. The transistor T3 flows a current corresponding to the logarithmically converted signal.
[0040]
The source of the MOS transistor T3 is connected to the gate of the source follower amplification MOS transistor T5 and the drain of the capacitor reset MOS transistor T4, and the source of the MOS transistor T5 is connected to the switching (signal reading) MOS transistor T6. The drain is connected. The source of the MOS transistor T6 is connected to the output signal line 6 (corresponding to the output signal lines 61 to 6m in FIG. 3). Each of the MOS transistors T1 to T6 is a P-channel transistor.
[0041]
The signal φVPS is input to the source of the logarithmic conversion MOS transistor T2. The drain potentials of the MOS transistors T3 and T5 are set to VPD. Note that the drain may be grounded. A capacitor C is connected to the source of the MOS transistor T3, and a reference voltage (signal φVD) for integrating the electric signal from the photodiode PD with the capacitor C is input to the capacitor C.
[0042]
The DC voltage RSB is input to the source of the MOS transistor T4, and the signal φRST is input to the gate of the transistor T4. Further, a signal φS for turning on and off the transistor T1 is input to the gate of the MOS transistor T1, and a signal φV for turning on and off the transistor T6 is input to the gate of the MOS transistor T6.
[0043]
In the pixel thus configured, the constant current power supply 7 (corresponding to the constant current power supplies 71 to 7m in FIG. 3) to which the DC voltage VPS is applied to the minus terminal via the MOS transistor T6 and the output signal line 6 is provided. , Connected to the source of the MOS transistor T5.
Therefore, when the MOS transistor T6 is on, the MOS transistor T5 operates as a source follower MOS transistor and outputs a voltage signal amplified by the constant current power supply 7 to the output signal line 6.
[0044]
By configuring the source follower circuit in this way, an amplifier circuit that outputs a large signal is configured. Therefore, a signal which is sufficiently amplified by this amplifier circuit is obtained, so that processing in a subsequent signal processing circuit (not shown) is facilitated. Further, the constant current power supplies 71 to 7m constituting the load resistance portion of the amplifier circuit are not provided in the pixels but are provided for each of the output signal lines 61 to 6m to which a plurality of pixels arranged in the column direction are connected. Thus, the number of load resistors or constant current power supplies can be reduced, and the area occupied by the amplifier circuit on the semiconductor chip can be reduced.
[0045]
(2-3) Normal imaging and sensitivity variation detection by the imaging device 10
Next, first, a normal imaging operation by the above-described imaging device and a sensitivity variation detection operation of each pixel will be described.
[0046]
As the signal φVPS supplied to the source of the logarithmic conversion MOS transistor T2, a high (High) and low (Low) voltage signal is used to obtain a normal imaging state. That is, the voltage for operating the transistor T2 in the sub-threshold region is set to low when reading out the video signal and the noise signal due to the sensitivity variation, and when the transistor T2 is reset, the voltage applied to the transistor T2 is higher than the voltage. A high (Hi) voltage that allows a larger current to flow than when the signal φVPS is applied is adopted.
As will be described later, when the capacitor C is reset after reading the noise signal, a voltage having a value between about high and low is applied to the source of the transistor T2 in order to remove an afterimage signal from the photoelectric conversion element.
[0047]
As the reference voltage φVD applied to the capacitor C, a ternary voltage signal is employed. That is, the voltage value when the capacitor C is integrated is set to the highest Vh, the voltage value for reading the video signal is set to Vm lower than Vh, and the voltage value for reading the noise signal is set to Vl lower than Vm.
[0048]
In the following description, application of a voltage signal or the like to a pixel or the like is performed from a voltage regulator Rg under timing control by a timing generator TG.
[0049]
(2-3-1) Video signal (image signal) output
In the following description, the signal φS for turning on and off the MOS transistor T1 is always low during the imaging operation, and the transistor T1 is on. Further, the signal φRST applied to the capacitor resetting transistor T4 is set high (Hi) to turn off the transistor T4. Then, the signal φVPS applied to the source of the transistor T2 is set low and the voltage value of the signal φVD applied to the capacitor C is set to Vh so that the integration operation by the capacitor C is enabled so that the MOS transistor T2 operates in the sub-threshold region. .
[0050]
When light from the imaging area enters the photodiode PD in such a state, a photocurrent is generated, and a photocurrent is applied to the gates of the transistors T2 and T3 due to the sub-threshold characteristic of the transistor T2. As a result, a voltage corresponding to a value converted so as to change in a natural logarithm is generated.
[0051]
Based on the voltage which changes in a natural logarithm with respect to the incident light amount, the drain current amplified by the transistor T3 flows from the capacitor C, and the capacitor C discharges. Therefore, the gate voltage of the MOS transistor T5 becomes a voltage proportional to the natural logarithm of the integrated value of the amount of incident light. Then, in order to read a video signal obtained by performing the integration operation by the capacitor C, the voltage value of the signal φVD is set to Vm, and a low pulse signal φV is given to the MOS transistor T6. As a result, a source current corresponding to the gate voltage of the MOS transistor T5 flows to the output signal line 6 via the MOS transistor T6.
[0052]
At this time, since the transistor T5 operates as a source follower type MOS transistor, a video signal appears on the output signal line 6 as a voltage signal. Thereafter, the signal φV is set high to turn off the transistor T6, and the voltage value of the signal φVD is set to Vh. As described above, the video signal output via the transistors T5 and T6 has a value proportional to the gate voltage of the transistor T5, and is therefore a signal proportional to the natural logarithm of the integrated value of the amount of light incident on the photodiode PD.
[0053]
(2-3-2) Sensitivity variation detection (noise signal output)
As shown in FIG. 5, when a pulse signal φVD having a voltage value Vm and a low pulse signal φV are supplied and a video signal is output, the signal φVD is set to Vh, and then the signal φS is set high to turn off the transistor T1. To start the reset operation. At this time, positive charges flow from the source side of the transistor T2, and the negative charges accumulated in the gate and drain of the transistor T2 and the gate of the transistor T3 are recombined. Goes up.
[0054]
However, when the potential of the gate and drain of the transistor T2 rises to a certain value, the reset speed becomes slow. In particular, this tendency becomes remarkable when a bright imaging region suddenly becomes dark. Therefore, next, the signal φVPS applied to the source of the transistor T2 is set to high. As described above, by increasing the source voltage of the transistor T2, the amount of positive charges flowing from the source side of the transistor T2 increases, and the negative and accumulated charges at the gate and drain of the transistor T2 and the gate of the transistor T3. The charges recombine quickly. At this time, the signal φRST is set to low to turn on the transistor T4, thereby initializing the voltage of the connection node between the capacitor C and the gate of the transistor T5.
[0055]
When the potential of the gate and the drain of the transistor T2 is further increased by raising the signal φVPS to high, the signal φVPS supplied to the source of the transistor T2 is lowered to return the potential state of the transistor T2 to the original state. As described above, when the potential state of the transistor T2 is reset to the original state, the signal φRST is set high and the transistor T4 is turned off.
[0056]
Then, the capacitor C performs an integration operation, and the voltage at the connection node between the capacitor C and the gate of the transistor T5 becomes in accordance with the reset gate voltage of the transistor T2. Therefore, a pulse signal φV is supplied to the gate of the transistor T6 to turn on the transistor T6 and set the voltage value of the signal φVD to Vl. As a result, an output current representing a variation in sensitivity of each pixel due to a variation in the characteristics of the transistors T2 and T3 flows through the output signal line 6.
[0057]
At this time, since the transistor T5 operates as a source follower type MOS transistor, a noise signal appears on the output signal line 6 as a voltage signal. After that, φS is set to low, the transistor T1 is turned on, the voltage applied to the source of the transistor T2 is set to a voltage of an intermediate value between high and low, and the pulse signal φRST is applied to the transistor T4 to connect the capacitor C with the gate of the transistor T5. Of the connection node is reset so that the imaging operation can be performed.
[0058]
In the above description, the voltage φVD applied to the capacitor C for integrating the electric signal obtained by the photoelectric conversion is set to the three values of Vh, Vm, and Vl, but the voltage applied to the capacitor C for integrating the electric signal is described. φVD may be constant. However, by adopting the three values as described above, it is possible to reduce the offset in the video signal from which noise has been removed, thereby reducing the operating range of an analog / digital converter or the like connected downstream of the image sensor. Can be used effectively. Note that the voltage value of the signal φVD applied to the capacitor C at the time of reading the video signal may be higher than the voltage value applied at the time of integration.
[0059]
In the above-described imaging device, each pixel is configured using a P-channel MOS transistor. However, the pixel may be configured using an N-channel MOS transistor. At this time, the polarity of each element is reversed. The polarity of the constant current power supplies 71 to 7m provided in the solid-state imaging device is opposite to that of FIG. Except for the above, the imaging device is substantially the same as the already described imaging device.
[0060]
(3) Second imaging element 20 for moving body extraction imaging
(3-1) Configuration of the second image sensor 20
The second imaging device 20 for moving body extraction imaging is driven as shown in FIG. 6 instead of the driving shown in FIG. 5 in the imaging device 10 shown in FIGS. 2 to 4, thereby enabling the moving body extraction imaging. Is the same as that of the first image sensor 10. The number of imaging pixels and the pixel arrangement pitch in the second imaging device 20 are the same as those in the first imaging device 10.
[0061]
(3-2) Operation of second image sensor 20
The operation for outputting the video signal is the same as that of the first imaging device 10. However, in the noise signal output operation, as shown in FIG. 6, instead of the high signal φVPS in the case of the noise signal output in the normal imaging state, the signal φVPS employed in the reset processing of the MOS transistor T2 is changed to a high level as shown in FIG. And a voltage signal having a value approximately between the low and high values. Thereby, moving object extraction (extraction of a luminance change portion) can be performed. Otherwise, the process is the same as the noise output process in the first element 10. Note that, instead of the low signal φVPS in the case of outputting a noise signal in the first image sensor 10, a voltage signal having a value intermediate between high and low may be adopted.
[0062]
Also in the element 20, similarly to the element 10, the video signal output from each pixel when imaging is performed with φVPS low and the MOS transistor T2 reset by setting φVPS to an intermediate value between high and low. Then, the difference from the noise signal (noise signal) output from each pixel is subtracted from the video signal by the difference processing by the correction circuit 9 and output.
[0063]
However, instead of the high signal φVPS in the case of the noise signal output in the case of the first element 10, a voltage signal having an intermediate value between high and low is employed as the signal φVPS employed in the reset processing of the MOS transistor T2. As a result, a reset remains in the MOS transistor T2. In addition, the reset residue is larger for a pixel having a larger incident light amount and smaller for a pixel having a smaller incident light amount due to the characteristics of the MOS transistor T2. Therefore, the video signal after the difference processing has the same value for each pixel that captures an image of a subject portion whose luminance does not change. Thus, a signal capable of gray display is output substantially uniformly for the still background portion of the subject. By using this, the still background portion can be displayed almost uniformly in gray.
[0064]
On the other hand, for moving objects, the brightness is “dark” → “bright”, “dark” → “bright” → “dark”, “bright” → “dark”, “bright” → “dark” → “bright”, etc. It changes during the integration period of the video signal.
[0065]
If the brightness changes from “dark” to “bright” and remains in the “bright” state until the start of the reset period, the video signal will be higher than in the “bright” state during the integration period. Become. Since the noise signal has a value depending only on the voltage of the integration circuit input section at the start of reset, the noise signal is in the “bright” state at the start of reset, and thus the noise signal is always the same as in the “bright” state. Then, the output of the video signal after the difference processing decreases, and the portion where the light incident on the element 20 changes from “dark” to “bright” can be displayed dark. In addition, when compared with the case where the video signal is always in the “dark” state, the video signal is lower and the noise signal is lower, but the amount of the video signal lower than the amount of the noise signal is smaller, and the video after the difference processing is performed. The signal output decreases.
[0066]
If the brightness changes from “dark” to “bright” to “dark” and remains in the “dark” state until the start of the reset period, the video signal will always be in the “dark” state during the integration period. It is lower than the comparison. Since the noise signal has a value dependent only on the voltage of the integration circuit input portion at the start of reset, the noise signal is in the “dark” state at the start of reset, and thus the noise signal is always the same as in the “dark” state. Then, the output of the video signal after the difference processing increases, and the portion where the light incident on the element 20 changes from “dark” to “bright” to “dark” can be displayed brightly.
[0067]
When the brightness changes from “bright” to “dark”, the video signal output after the difference processing increases for the same reason, and the brightness change portion can be displayed brightly.
When the brightness changes from “bright” to “dark” to “bright”, the output of the video signal after the difference processing decreases, and the brightness change portion can be displayed dark.
Due to such a phenomenon, it is possible to extract a portion of the subject in which the incident light has changed, thereby detecting the presence of a moving object.
[0068]
If an image of a subject including a moving object is captured by the imaging element 10 for normal imaging and an image for one frame is displayed on a display, for example, an image as shown in FIG. If the same subject is imaged at the same time and an image for one frame is displayed on the display, an image as shown in FIG. 7B is obtained.
[0069]
7A and 7B, the moving object is a bright sphere, and moves at high speed from right to left in the figure in front of a uniformly dark background. In normal imaging by the imaging device 10, as shown in FIG. 7A, a portion corresponding to subject shake is displayed at an intermediate density between “bright” and “dark”. In the moving body extraction imaging by the imaging device 20, as shown in FIG. 7B, a portion in the subject where there is no change in luminance is displayed at an intermediate uniform density, and the sphere moves from “dark” to “bright”. The part that changes is dark, and the part that changes from “bright” to “dark” is displayed bright.
[0070]
(4) Image blur removal processing in imaging apparatus example 1
The imaging signal output from the first imaging element 10 for normal imaging and the imaging signal output from the second imaging element 20 for moving object extraction described above are digitized by an A / D converter, respectively, and are processed by an image processing circuit. Input to 40. The image processing circuit 40 combines the image data from the two devices 10 and 20 by subtracting the image signal (image data) of the image sensor 20 from the image signal (image data) of the image sensor 10 and further adjusting the offset. Then, the image blur of the normal captured image by the image sensor 10 is removed, and the processed image data is output and provided for display on a display, recording on a recording medium, and the like.
[0071]
More specifically, according to the study of the present inventor, the imaging signal of the previous n-th frame is subtracted from the imaging signal of the (n + 1) -th frame obtained by imaging the moving object with the normal imaging device, and the image signal is appropriately offset. When the adjustment is performed, image data similar to image data obtained by the moving object extraction type image sensor is obtained.
[0072]
This means that the imaging data of the moving object extraction type imaging device can be regarded as reflecting the change of the moving object between frames in the normal imaging device.
On the other hand, the moving object extraction type imaging device exhibits the moving object extracting function even when the subject luminance changes within one frame. The change in luminance within one frame corresponds to a shake of a subject or a shake of a camera supporting means (such as a shake of a camera) in an ordinary imaging device.
[0073]
As described above, since the imaging data of the moving object extraction type imaging device reflects the change in luminance due to the moving object, the imaging of one frame of the moving object extraction type imaging device at the same time from the imaging data of one frame of the normal imaging device at the same time By subtracting the data and performing an appropriate offset adjustment, it is possible to cancel image shake caused by subject shake, hand shake, and the like.
[0074]
FIG. 7C subtracts the image data shown in FIG. 7B from the image data shown in FIG. 7A so that the density of the background part approaches the density of the background part of FIG. Is an image subjected to offset processing. As a result, correction is performed so that a portion corresponding to subject shake (image shake portion) is removed from the image of FIG. 7A obtained by normal imaging.
[0075]
In this way, the image processing circuit 40 converts the imaging signal L (n) of the n-th frame captured by the normal imaging device 10 into the imaging signal D (n) of the n-th frame captured by the moving object extraction type imaging device 20. ) Is subtracted and the offset adjustment α is performed to synthesize data from the two elements 10 and 20. That is,
L1 (n) = L (n) -D (n) + α
L1 (n) is generated by calculation, and this is output as one piece of image data.
Thus, it is possible to obtain a video signal in which the image blur caused by the high-speed movement of the moving object, the camera shake, or the like is canceled in real time by simple signal processing.
[0076]
<Imaging device example 2>
This imaging apparatus is the same as the imaging apparatus shown in FIG. 1 and the like except that the image processing circuit 40 in the imaging apparatus shown in FIG. 1 and the like is changed to operate as follows.
[0077]
The image processing circuit in the image pickup apparatus processes the image pickup signal D (n) from the image pickup device 20, converts the “bright” signal of the image pickup signal to be brighter, and converts the “dark” signal to be darker. Then, by subtracting the processed image signal from the image signal L (n) of the image sensor 10 and performing an offset process, the image data L2 (n) on which the image blur correction has been performed more accurately is output.
Specifically, in this imaging apparatus, the arithmetic processing performed by the image processing circuit in the imaging apparatus example 1 is
L2 (n) = L (n) -f (D (n)) + α
This is changed to an arithmetic process for generating L2 (n).
f (D (n)) means an operation for performing conversion corresponding to γ adjustment of the variable D (n).
[0078]
<Imaging device example 3>
This imaging apparatus is the same as the imaging apparatus shown in FIG. 1 and the like except that the image processing circuit 40 in the imaging apparatus shown in FIG. 1 and the like is changed to operate as follows.
[0079]
The image processing circuit in the imaging apparatus includes an imaging signal L (n + 1) of the (n + 1) th frame captured by the imaging element 10 and an imaging signal D (n) of the (n + 1) th frame captured by the moving object extraction type imaging element 20. (n + 1) is added to perform offset adjustment.
As a result, image data in which the image blur has been removed from the data L (n) of the n-th frame immediately before the normal imaging element 10 is generated and output.
[0080]
<Imaging device example 4>
This imaging device is an imaging device in which the image processing circuit in the imaging device of Example 3 is changed to one that performs the following processing.
[0081]
That is, the image processing circuit in this image pickup device processes the image pickup signal D (n + 1) from the image pickup device 20 to make the “bright” signal brighter and “dark” as in the case of the image pickup device of Example 2 above. The image is subjected to conversion processing to make the signal darker, and thereafter, the image pickup signal after the processing is added to the image pickup signal L (n + 1) of the image pickup device 10 and the offset processing is executed, so that the image in which the image shake correction is more accurately performed. Output data.
[0082]
<Imaging device example 5>
This imaging device (camera) has the configuration shown in FIG.
The imaging device shown in FIG. 8 includes a solid-state imaging device 10 for performing normal imaging of a subject and a solid-state imaging device 20 for performing moving body extraction imaging, and further includes an element controller 30 and an image processing circuit that control the operation of these imaging devices. 400, a controller 60 for controlling the circuit 400, and an operation panel PA connected to the controller 60. Although not shown, an optical system similar to that shown in FIG. 1 and a half mirror for guiding image light from a subject to the image pickup devices 10 and 20 are also provided.
[0083]
The imaging devices 10 and 20 and the device controller 30 are the same as the imaging devices 10 and 20 and the controller 30 in the imaging device example 1.
The imaging signals output from the imaging devices 10 and 20 are digitized by an analog / digital converter A / D and input to the image processing circuit 400. The operation of each of the imaging devices 10 and 20 is controlled by the device controller 30.
[0084]
In this imaging apparatus, the image processing circuit 400 is a circuit capable of performing both image blur removal processing including subtraction processing in the imaging apparatus example 1 or 2, and image blur removal processing including addition processing in the imaging apparatus example 3 or 4. Switching between the image shake removal processing including the subtraction processing and the image shake removal processing including the addition processing can be performed by an operator's instruction from the operation panel PA via the controller 60.
[0085]
According to this imaging apparatus, there is an advantage that a work such as checking a still image without shake while moving forward and backward by frames can be performed.
[0086]
<Imaging device example 6>
This imaging device (camera) has the configuration shown in FIG.
The imaging apparatus shown in FIG. 9 includes a solid-state imaging device 10, a solid-state imaging device 20, an element controller 30 that controls the operation of these imaging devices, and a imaging signal of the imaging device 10 that is input via an analog-to-digital converter A / D. One image processing circuit 701, a second image processing circuit 702 to which an image pickup signal of the image pickup device 20 is input via an analog / digital converter A / D, and a storage device 80 for storing image data output from these circuits 701 and 702. Have.
[0087]
The imaging apparatus further issues an instruction to the third image processing circuit 700 that reads out the image data stored in the storage device 80 under the instruction of the controller 90, performs predetermined processing, and outputs the processed image data, and the controller 90. An operation panel PA 'is also provided.
[0088]
The imaging devices 10 and 20 and the device controller 30 are the same as the imaging devices 10 and 20 and the controller 30 in the imaging device example 1. Although not shown, an optical system similar to that shown in FIG. 1 and a half mirror for guiding image light from a subject to the imaging devices 10 and 20 are also provided.
[0089]
The first image processing circuit 701 performs predetermined processing such as gain adjustment and offset adjustment, and the second image processing circuit 702 also performs predetermined processing such as gain adjustment and offset adjustment. The third image processing circuit 700 can perform any of a plurality of predetermined processes such as an image blur removal process including a subtraction process similar to the imaging device example 1 and an image blur removal process including an addition process similar to the imaging device example 3. Circuit. Switching of the plurality of types of processing can be performed by an operator's instruction from the operation panel PA 'via the controller 90.
[0090]
According to this imaging apparatus, output signals from each of the imaging elements 10 and 20 can be individually stored, and when the stored data is reproduced, a synthesis process can be performed to remove image blur. Rather than immediately synthesizing the output signals from the respective image sensors by the image processing circuit, the synthesizing process is performed when the output signals of the image sensors 10 and 20 are temporarily stored as individual data. Therefore, the conditions of the synthesis processing can be changed arbitrarily and the processing can be executed any number of times.
[0091]
The second image processing circuit 702 performs the conversion process such that the “bright” signal is brighter and the “dark” signal is darker in the imaging signals from the imaging device 20 as in the imaging device examples 2 and 4. Anything that can be done. In that case, the image blur removal processing in the third image processing circuit 700 can be performed using the signal after the conversion processing.
[0092]
<Example of Still Another Imaging Device>
In the imaging apparatus examples 1 to 6 described above, the image data after the analog-to-digital conversion processing from the two imaging elements 10 and 20 are synthesized, but a differential amplifier is provided before the analog-to-digital converter A / D. Then, it is also possible to combine the signals from the respective image sensors in the form of analog signals. By doing so, the load on the image processing circuit that performs the image blur removal processing is reduced accordingly.
[0093]
【The invention's effect】
As described above, according to the present invention, it is possible to provide an imaging apparatus that can easily correct image shake in real time.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating an overall configuration of an imaging device according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a configuration of a solid-state imaging device for normal imaging and moving body extraction imaging.
FIG. 3 is a diagram illustrating a configuration of a pixel unit and the like in which pixels are arranged.
FIG. 4 is a diagram showing a configuration of one pixel.
FIG. 5 is a timing chart illustrating an example of application of an element driving voltage in an imaging element for normal imaging (first imaging element).
FIG. 6 is a timing chart showing an example of application of an element drive voltage in an imaging element (second imaging element) for moving body extraction imaging.
7A illustrates an example of imaging of a subject including a moving object in normal imaging by a first imaging device, and FIG. 7B illustrates the same moving object in moving object extraction imaging by a second imaging device. FIG. 7C shows an example of an image of the subject included at the same time, and FIG. 7C shows an example of the image after the image blur removal processing.
FIG. 8 is a block diagram illustrating a configuration of an imaging device according to another embodiment of the present invention.
FIG. 9 is a block diagram illustrating a configuration of an imaging apparatus according to yet another embodiment of the present invention.
[Explanation of symbols]
10 First imaging device for normal imaging
20 Second imaging device for moving body extraction imaging
30 Image sensor controller
40, 400, 700, 701, 702 Image processing circuit
50 Optical system
60, 90 Controller for image processing circuit
80 storage device
PA, PA 'operation panel
A / D analog / digital converter
G pixel section 81-8m selection circuit
1 vertical scanning circuit 9 correction circuit
2 horizontal scanning circuit 10 voltage controller
3, 31-3n Row (Row) Line Am Output Amplifier
4, 41-4n line Rg voltage regulator
5 Power line TG Timing generator
6, 61 to 6 m Output signal line G11 to Gmn Pixel
71-7m constant current power supply PD photodiode
8 Output circuit
T1 switching MOS transistor
T2 MOS transistor for logarithmic conversion
T3 MOS transistor
T4 MOS transistor for capacitor reset
T5 MOS transistor for source follower amplification
T6 MOS transistor for signal reading
C capacitor (capacitor for integration)

Claims (5)

A first image sensor that normally captures an image of the imaging target area;
A second image pickup device capable of extracting and imaging the portion where the luminance changes in the imaging target region when the portion is changed;
An image processing unit that removes image shake of an image captured by the first image sensor using an image signal from the second image sensor;
An imaging device comprising: The second image sensor outputs an image signal of a different level from an image signal representing another portion of the portion to be photographed where the luminance does not change in the portion. The image pickup apparatus according to claim 1, wherein the image pickup apparatus performs an image blur removal process including a process of subtracting an image signal output from the second image sensor from the output image signal. The second image sensor outputs an image signal of a different level from an image signal representing another portion of the portion to be imaged that does not change in luminance, with respect to the portion,
A switching instruction unit for instructing switching of image blur removal processing in the image processing unit is provided,
The image processing unit is configured to perform a process of subtracting an imaging signal output from the second imaging device from an imaging signal output from the first imaging device or the second imaging to an imaging signal output from the first imaging device. 2. The image pickup apparatus according to claim 1, wherein the image shaking removal processing includes processing for adding an image pickup signal output from the element, and the switching instruction unit can instruct the image processing unit to switch the image shake removing processing. apparatus. The imaging device according to claim 1, wherein the first imaging device and the second imaging device have the same number of imaging pixels and the same pixel arrangement pitch. The second image sensor includes:
Each is a photoelectric conversion element, a logarithmic conversion unit including a logarithmic conversion transistor for converting the output of the photoelectric conversion element into an electric signal proportional to the logarithmic value of the amount of light incident on the element, and the output from the logarithmic conversion unit. A plurality of pixels having an integrating circuit for storing;
An output circuit for receiving and outputting a signal from each of the pixels,
In a state where a first voltage for imaging is applied to the logarithmic conversion transistor during imaging, a first signal output from the photoelectric conversion element via the logarithmic conversion unit is accumulated in the integration circuit, and the logarithmic conversion is performed. At the time of resetting the transistor, a second voltage is applied to the logarithmic conversion transistor to reset the transistor, a second signal obtained from the logarithmic conversion unit from the logarithmic conversion unit is accumulated in the integration circuit, and the second voltage is When there is a portion where the luminance changes in the imaging target region by making the voltage different from the voltage when the normal imaging state is obtained, the first signal and the second signal respectively stored in the integration circuit are included in the portion where the luminance changes. The difference of the brightness change portion extraction imaging state that is different from that of the other portion in which the brightness in the shooting target area does not change is realized,
The imaging device according to claim 1, wherein the output circuit outputs a signal corresponding to a difference between the first signal and the second signal.

Images