JP6512291B2 - Signal processing method and driving method of solid-state imaging device - Google Patents

Description

  The present invention relates to a signal processing method and a driving method of a solid-state imaging device. In particular, the present invention relates to a signal processing method and a driving method of a solid-state imaging device suitably used to capture a phenomenon that occurs at high speed in a short time.

  BACKGROUND ART High-speed imaging devices (high-speed video cameras) for continuously capturing, at high speed, phenomena that occur at high speed in a short time, such as explosion, destruction, combustion, collision, or discharge, are conventionally known (for example, non-patent documents 1). Such an apparatus uses a solid-state imaging device having a special structure different from an imaging device used for a general video camera, digital camera, and the like because continuous imaging is performed at a speed of about one million frames per second or more. Be

  The present inventors have proposed one such solid-state imaging device in Patent Document 1. The solid-state imaging device includes N (for example, 100,000) pixels and M (for example, 128) storage elements corresponding to the N pixels. The N pixels are aggregated into a pixel area, and a two-dimensional array (for example, a 250 × 400 grid array) is formed in the pixel area. Further, N × M storage elements are disposed in a storage area spatially separated from the pixel area. A pixel output line extends from each pixel in the pixel area, and is connected to M storage elements corresponding to the pixel via a switching transistor.

  In the solid-state imaging device, an operation of accumulating signals in each pixel for a predetermined sampling time and subsequently holding an output signal in one of M storage elements corresponding to the pixels is repeated. Then, when the output signal is held in all the M storage elements, the processing returns to the first storage element, and the pixel signals are sequentially overwritten from the storage element. The accumulation of the signal in each pixel and the output to the storage element are simultaneously performed in all the pixels. After the end of imaging, the signals held in the respective storage elements are read out of the solid-state imaging element, and M (for example, 128 frames) images are created.

Patent No. 4931160 gazette

Kondo et al. 5: "Development of High Speed Video Camera HyperVision HPV-1", Shimadzu Review, Shimadzu Review and Editing Department, published on September 30, 2005, vol. 62, Vol. 1, No. 2, p. 79-86

  In the above-described solid-state imaging device, generally, two or more digits of storage elements are provided in comparison with the number of pixels, and in the case of the above-described example, for example, the number of storage elements is ten million or more. Usually, solid-state imaging devices are manufactured in a clean room to prevent the entry of dust, which is a factor causing defects in the elements, but it is still difficult to completely prevent the entry of dust. In addition, a defect of the element may occur due to a factor other than the mixing of dust, such as a defective formation of a wiring pattern. Therefore, as described above, in the process of manufacturing a solid-state imaging device having a very large number of storage devices, it is difficult to avoid the occurrence of defects in the storage devices at a certain rate. Loss of data caused by the failure of one storage element is only for the signal of one pixel in one frame and does not have a fatal effect on the entire imaging data, but if there is an area without a signal in the image, it looks Had the problem of making it feel unnatural.

  The problem to be solved by the present invention is that a solid-state imaging device having a configuration in which a plurality of storage elements for holding an output signal from one pixel is associated with one pixel is a part of the storage element. A signal processing method and a driving method of a solid-state imaging device capable of acquiring imaging data without omission even when a failure occurs.

The first aspect of the present invention made to solve the above-mentioned problems is
A plurality of pixels disposed adjacent to each other and outputting signals according to the intensity of light received by each;
A plurality of predetermined storage elements provided for each of the plurality of pixels, wherein the plurality of storage elements sequentially hold signals output from each of the plurality of pixels; A method of processing a signal read out from part or all of the plurality of storage elements respectively corresponding to
Identifying a nonconforming signal that does not meet predetermined conditions from among the signals read from the storage element;
And estimates a normal signal on the basis of the signals acquired at a time adjacent to the non-conforming signal in a pixel corresponding to the storage element before Symbol incompatible signal is read.

The arrangement of the plurality of pixels adjacent to each other is typically a two-dimensional arrangement (for example, a lattice or honeycomb arrangement), but is a one-dimensional arrangement such as linear, curvilinear, or looped. May be
The nonconforming signal which does not conform to the predetermined condition is, for example, read out from a memory element or the like (including a signal line connected to the memory element or the like) which is known to have a defect in the manufacturing process. Signal, or a signal whose difference in magnitude between adjacent pixels is larger than a predetermined threshold, or a signal whose signal value is zero although the image is photographed with a certain brightness or more (the signal itself is a storage element (Including the case of not being held by

  In the solid-state imaging device, a signal corresponding to the intensity of light received at the same timing in a plurality of pixels is read out from each pixel at a predetermined sampling cycle, and held in order in a plurality of storage elements corresponding to the pixels. In the case where signal output lines connecting each pixel and a plurality of storage elements corresponding to the pixel are provided independently for each pixel, output of signals from each pixel to the storage element is simultaneously performed on all the pixels. On the other hand, in the case of a configuration in which a plurality of pixels share one signal output line, the plurality of pixels are sequentially switched to output a signal to a storage element corresponding to the pixel. In the configuration in which the signal output line is provided independently for each pixel, the sampling period can be shortened because the time required for outputting the signal to the storage element is short. Further, in the configuration in which a plurality of pixels share one signal output line, the ratio of the light receiving area in the pixel area can be increased by reducing the number of signal output lines.

  For example, in the case of a solid-state imaging device in which a plurality of pixels are two-dimensionally arranged in a grid, four or eight pixels (peripheral adjacent to a pixel (target pixel) corresponding to a storage element from which a nonconforming signal is read In the pixel), the values of the signals acquired at the same timing as the nonconforming signal are averaged to estimate a normal signal (peripheral correction). Alternatively, the value of one or more signals acquired before and / or after the nonconforming signal in the target pixel is averaged to estimate a normal signal (back and forth correction). Alternatively, a normal signal is estimated by both peripheral correction and longitudinal correction. As described above, when the signal processing method according to the first aspect of the present invention is used, even if there is a problem in a part of the signal read from the storage element of the solid-state imaging device, it should be read from the storage element A normal signal can be estimated to obtain imaging data without omission.

  When estimating the above-mentioned normal signal, whether the peripheral correction, the front / rear correction or both of them should be performed may be determined by the user according to the characteristics of the object to be imaged. However, in the case of an unskilled user, it is difficult to make such judgment properly.

Therefore, in the first aspect, further,
A signal acquired at the same time as the nonconforming signal, and a signal acquired at the time adjacent to the nonconforming signal, for the pixel disposed adjacent to the pixel corresponding to the storage element from which the nonconforming signal is read out Find the degree of correlation between
When the degree of correlation is smaller than a predetermined value, a signal acquired at a time corresponding to the nonconforming signal in a pixel arranged adjacent to a pixel corresponding to the storage element from which the nonconforming signal is read it is preferable that to estimate the normal signal on the basis of.

  In this aspect, the degree of correlation between the signals acquired at the same time as the nonconforming signal in the peripheral pixels and the signals acquired before and after the nonconforming signal is determined. The degree of correlation of the signal is determined from the nonconforming signal and the magnitude of the change in the value of the signal acquired before and after it. For example, when the calculated degree of correlation is a value equal to or greater than a predetermined threshold value, the signal is small (or stationary) and the calculated degree of correlation is less than a predetermined threshold value. , It can be considered that the signal is intense. Therefore, a normal signal can be estimated by the back-and-forth correction in the former case and by the peripheral correction in the latter case.

  Further, correction processing combining the front and back correction and the peripheral correction may be performed. For example, according to the value of the degree of correlation, each of the front and back correction and the peripheral correction may be weighted, and both may be used to estimate a normal signal.

In addition, the second aspect of the present invention made to solve the above-mentioned problems is
One or more pixels each outputting a signal according to the intensity of the light received;
A driving method of a solid-state imaging device including: a plurality of predetermined storage devices provided for each of the one or the plurality of pixels;
Identifying a nonconforming storage element which does not conform to a predetermined condition among the predetermined plurality of storage elements corresponding to each of the one or more pixels;
For each of the one or more pixels, a normal storage element obtained by removing the nonconforming storage element from a plurality of predetermined storage elements associated with the pixel is used to sequentially hold the signal output from the pixel It is characterized by

  The non-compliant storage element is, for example, a storage element for which a malfunction has been confirmed in a predetermined inspection, a storage element for which a normal signal could not be read out in the previous imaging, or provided for writing or reading out a signal. Storage element having a defect in the signal output line being

  The second aspect of the present invention does not use storage elements that do not conform to the predetermined conditions (a failure has occurred), but uses only normal storage elements to hold signals for a plurality of frames. If read out in order, normal imaging data without omission can be obtained.

  In a solid-state imaging device, a storage element holding an output signal from a pixel is often designated by a shift register. The shift register selectively connects the pixel to one of the plurality of storage elements corresponding to the pixel, and switches the storage element connected to the pixel in order each time one pulse is received. Therefore, after holding the output signal from the pixel in the memory element one before the non-conforming memory element, the specified memory element is skipped by continuously transmitting two pulses to the shift register, and the normal memory Only elements can be used to hold the output signals from the pixels in order.

  By using the signal processing method for a solid-state imaging device according to the first aspect of the present invention, even if there is a problem in a part of the signal read from the storage device of the solid-state imaging device, A normal signal to be read out can be estimated to obtain missing imaging data. In addition, by using the method for driving a solid-state imaging device according to the second aspect of the present invention, it is possible to acquire imaging data without omission even when there is a defect in a part of the storage device of the solid-state imaging device.

The structural example of the solid-state image sensor which implements the signal processing method and drive method of the solid-state image sensor which concerns on this invention, and a control * signal processing part. The circuit block diagram of the pixel in a present Example. FIG. 2 is a diagram showing connection of pixels and a storage unit in the present embodiment, and a circuit configuration in the storage unit. The example of the drive pulse signal transmitted to a shift register. 7 shows an example of a transmission pattern of drive pulses in Embodiment 1. 6 shows an example of signal processing in Embodiment 1. 6 shows another example of signal processing in Embodiment 1. 15 shows still another example of signal processing in the first embodiment. FIG. 6 is a configuration diagram of a solid-state imaging device and a control / signal processing unit according to a second embodiment. 6 is a flowchart of a signal processing method according to a second embodiment. 7 shows a calculation example using the signal processing method of the second embodiment. 16 shows another calculation example using the signal processing method of the second embodiment. 21 shows still another calculation example using the signal processing method of the second embodiment. 11 shows an example of a transmission pattern of drive pulse signals in the method of driving a solid-state imaging device in Embodiment 3.

  One embodiment of a signal processing method and a driving method of a solid-state imaging device according to the present invention will be described below with reference to the drawings. FIG. 1 is a schematic plan view showing the layout on the semiconductor chip of the solid-state imaging device 1 in this embodiment, and a schematic configuration diagram of a control / signal processing unit 50 that controls the imaging operation of the solid-state imaging device and processes signals. It is.

First, a schematic configuration of the solid-state imaging device 1 in the present embodiment will be described.
As shown in FIG. 1, the solid-state imaging device 1 has a rectangular pixel area 2 in which a plurality of pixels are arranged, and a rectangular shape which is spatially separated from the pixel area 2 and in which a plurality of storage elements are arranged. A storage area 3 is provided. In the pixel area 2, 100,000 pixels 10 are arranged in a 250 × 400 two-dimensional array. Also in the storage area 3, storage units 30 of the same number (ie 100,000) as the pixels 10 are arranged in a 250 × 400 two-dimensional array. Further, 128 storage elements are arranged in each storage unit 30, and the pixel 10 and the storage element in the storage unit 30 corresponding to the pixel 10 described later are connected by the pixel output line 40. The signals read out from the pixel 10 are sequentially held in the storage element in the storage unit 30 corresponding to the pixel 10.

  In the vicinity of the storage area 3, a vertical scanning circuit area 4 and a horizontal scanning circuit area 5 are provided. In the vertical scanning circuit area 4 and the horizontal scanning circuit area 5, circuits such as shift registers for controlling readout of signals from the storage elements included in the storage unit 30 in the storage area 3 are disposed.

  One circuit configuration of the pixel 10 arranged in the pixel area 2 is shown in FIG. The pixel 10 includes a photodiode 11, a transfer transistor 12, a reset transistor 13, a floating diffusion 14, a buffer transistor 15, a first current blocking transistor 16, a first bias transistor 17, a first capacitor 18, and a first sampling. Transistor 19, a second sampling transistor 20, a second capacitor 21, a source follower amplification transistor 22, a second current blocking transistor 23, a load current source transistor 24, and an output control transistor 25. There is.

  The gate terminals of the transfer transistor 12, the reset transistor 13, the first current blocking transistor 16, the first sampling transistor 19, and the second sampling transistor 20 are driven to have φT, φR, φX1, φNS, and φSS, respectively. A drive line for supplying a pulse signal (control signal) is connected. A drive line for supplying a drive pulse signal (control signal) of φX2 is connected to the gate terminals of the second current blocking transistor 23 and the output control transistor 25.

  FIG. 3 is a diagram showing the connection of the pixel 10 and the storage unit 30 corresponding to the pixel 10, and the circuit configuration in the storage unit 30. As shown in FIG. Although only one pixel 10 and the storage unit 30 corresponding to the pixel 10 are described in FIG. 3, as described above, both the pixel 10 and the storage unit 30 are arranged in a 250 × 400 two-dimensional array It is done.

  The pixel output line 40 extended from the pixel 10 is connected to the storage element in the storage unit 30 via the writing transistor 31. The storage unit 30 is provided with sampling transistors 36 and capacitors 37 (storage elements) for the number L of accumulation frames (in this example, L = 128). A drive line for supplying a common drive pulse signal φWS to all the storage units 30 is connected to the gate terminal of the write transistor 31. At the gate terminal of the sampling transistor 36, a drive pulse signal φVCLK1_O <? For specifying L sampling transistors 36 included in one storage unit 30 in order. The shift register VSR1 that supplies> (? Is 0 to 127) is connected. Shift register VSR1 operates in response to start signal φVS1 and clock signal φVCLK1. As shown in FIG. 4, the start signal φVS1 is transmitted to the shift register VSR1 first, and then the clock signal φVCLK1 is transmitted at a predetermined interval. The L sampling transistors 36 are sequentially switched on / off each time one clock signal φVCLK1 is received.

  Between the write transistor 31 and the storage unit 30, a vertical read transistor 38 and a horizontal read transistor 39 for reading a signal from the storage unit 30 are connected. Connected to the vertical read transistor 38 is a shift register VSR0 for supplying a drive pulse signal of φVCLK0_O <*> (* is 0 to 249) specifying the vertical position of the two-dimensionally arranged storage units 30. . Shift register VSR0 operates in response to start signal φVS0 and clock signal φVCLK0. Further, to the horizontal read transistor 39, a shift register HSR is connected which supplies a drive pulse signal of φHCLK_O <#> (# is 0 to 399) for specifying the horizontal position of the storage unit 30. The shift register HSR operates in response to the start signal φHS and the clock signal φHCLK.

Next, the imaging operation of the solid-state imaging device 1 will be described with reference to the transmission pattern of the drive pulse signal of FIG.
First, the transfer transistor 12 is turned off (.phi.T: on.fwdarw.off) to make the photodiode 11 in a floating state, and accumulation (exposure) of charges generated by photoelectric conversion is started. At this time, the floating diffusion 14 is reset to the reset voltage VR. After a short delay, the reset transistor 13 is turned off (φR: on → off) to set the floating diffusion 14 in the floating state. At this time, the voltage of the floating diffusion 14 is reset voltage VR + reset noise voltage. Subsequently, the first sampling transistor 19 and the second sampling transistor 20 are both turned on (φNS, φSS: off → on), and the terminal voltage of the first capacitor 18 on the second sampling transistor 20 side and the second capacitor All the terminal voltages on the side of the second sampling transistor 20 of 21 are reset to the reset voltage VR.

  Furthermore, only the first sampling transistor 19 is turned off (φ NS: on → off), and the terminal of the first capacitor 18 on the second sampling transistor 20 side and the terminal of the second capacitor 21 on the second sampling transistor 20 side are Both are in a floating state. At this time, if the voltage applied to the first capacitor 18 through the buffer transistor 15 is the reset voltage + the reset noise voltage, the voltage applied to the gate terminal of the source follower amplification transistor 22 becomes the reset voltage VR. Noise is removed. Thereafter, the transfer transistor 12 is turned on (φ T: off → on) to transfer the signal charge from the photodiode 11 to the floating diffusion 14. As a result, the voltage applied to the gate terminal of the source follower amplification transistor 22 decreases from the reset voltage VR by [net signal voltage] × [gain of CDS (Correlated Double Sampling) circuit].

  Furthermore, when the second sampling transistor 20 is also turned off (φ SS: on → off), the voltage applied to the gate terminal of the source follower amplification transistor 22 is determined. The voltage at this time is [net signal voltage] × [gain of CDS circuit] from which the reset noise voltage is removed. Thereafter, the output control transistor 25 is turned on (φX2: off → on), and a signal voltage not including reset noise in the floating diffusion 14 is output to the pixel output line 40 via the source follower amplification transistor 22.

  As described above, while the signal is output from the pixel 10 to the pixel output line 40, the storage unit 30 transmits the start signal φVS1 and the first clock signal φVCLK to the shift register VSR1, and corresponds to the pixel 10 by the shift register VSR1. The sampling transistor 36 of the first storage element to be turned on is turned on (φVCLK1_O <0>: off → on). Then, the writing transistor 31 is turned on (φ WS: off → on) to hold the output signal from the pixel 10 in the first memory element. This completes one cycle.

  In the second to 128th cycles, the control pulse signal is transmitted in the same manner as described above except for the start signal φ VS1 to turn on / off each transistor. As shown in FIG. 4, each time one clock signal φVCLK1 is transmitted, the storage element for turning on the sampling transistor 36 is sequentially switched, and the output signal from the pixel 10 is held. When the output signal is held in (the capacitors 37 of) the L storage elements, the above-described series of operations are repeated again, and the pixel signal is overwritten in order from the (first capacitor 37) of the first storage element.

  After the imaging operation, the storage unit 30 is sequentially designated by the vertical readout transistor 38 and the horizontal readout transistor 39, and the storage elements in the storage unit 30 are sequentially designated (the sampling transistor 36 is turned on). The signal held in (the capacitor 37 of) the storage element is read out of the solid-state imaging element 1.

Hereinafter, a specific example of the signal processing method according to the present invention will be described.
The configuration of the control / signal processing unit 50 will be described with reference to FIG. 1 again. The control / signal processing unit 50 includes an imaging control unit 52, a nonconforming signal identification unit 53, a signal estimation unit 54, and an image generation unit 55 as functional blocks in addition to the storage unit 51. The substance of the control / signal processing unit 50 is a personal computer, and the input unit 60 and the display unit 70 are connected. Also, each functional block described above is embodied by executing the control / signal processing program stored in the storage unit 51. The storage unit 51 stores the defect information of the storage element found in the manufacturing process (for example, there is a defect in the storage element in which the signal of the Kth frame of the Xth pixel is held). In addition, even if the user performs preliminary imaging under a certain lightness, and stores the information of storage elements whose values of the signals read from the respective storage elements do not match the lightness at the time of preliminary imaging in the storage unit 51. Good.

  When imaging is instructed by the user, the imaging control unit 52 transmits a drive pulse signal to the solid-state imaging device 1 described above based on the instruction content (imaging time, sampling interval and the like) to operate each unit. When the imaging is completed, the signal held in the storage element is read in the above-described procedure and stored in the storage unit 51.

  When all the signals are stored in the storage unit 51, the nonconforming signal identification unit 53 reads out the defect information of the storage element stored in the storage unit 51, and identifies the corresponding nonconforming signal. Further, the stored signals are compared with each other to confirm the presence or absence of nonconforming signals other than the signal corresponding to the defect information of the storage element. Specifically, the signal value is zero although the signal difference between the adjacent pixels is greater than a predetermined threshold value designated by the user in advance or the image is captured with a certain brightness or more. And the like (including the case where the signal itself is not held in the storage element) and the like are additionally specified as the nonconforming signal.

  When the nonconforming signal is identified, the signal estimation unit 54 estimates a normal signal to replace the nonconforming signal. In the present embodiment, two estimation methods (front and back correction, peripheral correction) described below are used.

In the first method (back and forth correction), as shown in FIG. 6, the signal of the same pixel 10 in the previous frame (K-1 frame) of the dropped frame (K frame) and the signal in the subsequent frame (K + 1 frame) The signal of the same pixel 10 is averaged.
In the second method (peripheral correction), as shown in FIG. 7, it is a signal in a missing frame (K frame) and is adjacent to the pixel (target pixel) corresponding to the nonconforming signal in the top, bottom, left, and right Average the signals of the pixels (peripheral pixels). Alternatively, a total of eight signals may be averaged, including the signals of four pixels diagonally adjacent to the pixel corresponding to the nonconforming signal. In this case, in addition to simple averaging of eight signals, weighting may be applied to signals of pixels adjacent vertically and horizontally and pixels adjacent diagonally and averaged.
Moreover, these two methods can also use either one or both in combination.

  When a normal signal is estimated by the signal estimation unit 54, the image generation unit 55 creates image data using the estimated signal, and displays the image data on the display unit 70. As described above, when the signal processing method of the present embodiment is used, a defect occurs in the storage element in the manufacturing process of the solid-state imaging device 1, or a defect occurs in writing of a signal to the storage element or reading from the storage element. Even in the case where a problem occurs, it is possible to estimate a normal signal to be originally read from the storage element and obtain imaging data without a dropout.

  It is also possible to estimate the normal signal using the method described below. In the second method, the signal read from one storage element is used as it is as one signal constituting imaging data, but in this modification, the signals at the same point in four adjacent pixels are averaged. The average value is used as one signal in imaging data. That is, from the pixel arrangement surrounded by a solid line shown in FIG. 8, a signal surrounded by a broken line is generated to generate imaging data. Then, when there is a nonconforming signal, the three signals excluding the signal are averaged to obtain one signal in the imaging data. Also by this method, it is possible to obtain imaging data without omission as described above.

  In the first embodiment, the user determines either signal processing using back-and-front correction, peripheral correction, or both to estimate a normal signal. However, in the case of an unskilled user, it is difficult to make such judgment properly. In the second embodiment, a configuration in which even an unskilled user can estimate a normal signal will be described.

  FIG. 9 is a configuration of the solid-state imaging device 1 and the control / signal processing unit 150 in the second embodiment, and FIG. 10 is a flowchart of signal processing in the second embodiment. The configuration and operation of the solid-state imaging device 1 are as described above, and thus the description thereof is omitted. Further, among the components of the control / signal processing unit 150, the storage unit 151, the imaging control unit 152, the nonconforming signal identification unit 153, the signal estimation unit 154, and the image generation unit 155 are also the same as in the first embodiment. Omit. In the second embodiment, the control / signal processing unit 150 further includes a correlation degree calculation unit 156 and a correction mode determination unit 157 as functional blocks.

  Hereinafter, a specific signal processing method will be described by taking, as an example, a case where there is a defect in the storage element in which the signal of the Kth frame of the Xth pixel is held. Hereinafter, the X-th pixel (target pixel) is represented by the origin of the xy coordinates, and eight pixels (peripheral pixels) adjacent to the target pixel are represented by coordinate positions. That is, the coordinates of eight peripheral pixels are (x, y) = (-1, 1), (-1, 0), (-1, 1), (0, -1), (0, 1), It becomes (1, -1), (1, 0), (1, 1).

  When imaging is instructed by the user through the input unit 160, the imaging operation by the solid-state imaging device 1 is executed under the control of the imaging control unit 152. Then, when the signal held in each storage element is read out, the nonconforming signal identification unit 153 identifies the nonconforming signal.

上式(1)において、A _{Ｋ−１（y, x）}は、Ｋ−１フレームにおける座標(x, y)の画素の信号値、A _{Ｋ（y, x）}は、Ｋフレームにおける座標(x, y)の画素の信号値である。また、A _Ｋ （数式では上線）はＫフレームにおける周辺画素の信号値の平均値であり、次式(2)から計算される。

Subsequently, the correlation degree calculation unit 156 uses the signal values of eight peripheral pixels in the K-1 frame and the signal values of eight peripheral pixels in the K frame to calculate the correlation degree C-1 (forward The correlation between the frame and the frame is calculated (step S1). In Equation (1) used here and Equation (3) described later, the degree of correlation is calculated to be higher as the signal has less motion.

In the above equation (1), _{AK-1 (y, x)} is the signal value of the pixel at coordinates (x, y) in the K-1 frame, and _{AK (y, x)} is the coordinates (x ₎ in the K frame. , y) are the signal values of the pixels. Further, A _K (upper line in the formula) is an average value of signal values of peripheral pixels in the K frame, and is calculated from the following formula (2).

上式(3)において、A _Ｋ＋１ （y, x）は、Ｋ＋１フレームにおける座標(x, y)の画素の信号値である。
Using the signal values of the eight peripheral pixels in the K frame and the signal values of the eight peripheral pixels in the K + 1 frame, the correlation degree calculation unit 156 further calculates the correlation degree C ₊₁ (rear frame and the corresponding frame) from (The degree of correlation) is calculated (step S2).

In the above equation (3), A _{K + 1} (y, x) is a signal value of the pixel at coordinates (x, y) in the K + 1 frame.

When the correlation degree calculation unit 156 calculates the correlation degrees C ₋₁ and C ₊₁ from the above equations (1) and (3), the correction mode determination unit 157 subsequently determines the correction mode from these values (see FIG. Step S3). Specifically, when the correlation degrees C ₋₁ and C ₊₁ are 1, it is determined that the signal has a small amount of movement (or a stationary signal), and the correction mode is determined to be the front-back correction. On the other hand, when the correlation degrees C ₋₁ and C ₊₁ are 0, it is determined that the signal is a signal with strong motion, and the correction mode is determined to be the peripheral correction. When the correlation degrees C ₋₁ and C ₊₁ are intermediate values between 0 and 1, it is determined that the signal is a moving image signal (for example, a signal obtained by photographing the movement of an object). The correlation _C -1 of adjacent _frames, and sets the _C weighting factor directly before and after the frame the value of _{+1 W K-1, W K} + 1, from 2 values of correlation _{C -1, C +1} before and after the frame The subtracted value is set to the weighting coefficient W _{K of the} frame (step S4).

When the correction mode determination unit 157 determines the preceding and subsequent frames and the weighting factor of the frame, the signal estimation unit 154 estimates the value of a normal signal from the following equation (4) (step S5). Then, the image generation unit 155 generates an image using the estimated signal value, and displays the image on the screen of the display unit 170.

  Here, specific calculation examples will be described for three types of signals (imaging signal of a stationary body, imaging signal of a moving body, and imaging signal of a light emitter).

FIG. 11 shows an example of signal values of a stationary signal (a signal having a large correlation with the preceding and following frames). 11 (a) shows the signal value of the K-1 frame (eight pixels adjacent to the X-th pixel and the pixel (hereinafter, referred to as "surrounding pixels".), Wherein only the signal value in), FIG. 11 (b 11) is the signal value of K frame (the signal value for which the shaded signal value “10” is to be originally obtained), and FIG. 11 (c) is the signal value of K + 1 frame.

In the example illustrated in FIG. 11 , the correlation degree C ₋₁ (= weighting coefficient W _K−1 ) of the previous frame is 1.00, and the correlation degree C ₊₁ ( = weighting coefficient W _{K + 1} ) of the back frame is 1.00. Therefore, the correction mode is determined by the correction mode determination unit 157 as back and forth correction, and the signal estimation unit 154 obtains the estimated value 10 of the normal signal from the average of the signal values of the Xth pixels in the previous and subsequent frames. This value corresponds to the value that should be originally obtained at the target pixel. If the signal value is estimated to be 381 by the peripheral average in this example, the value deviates from the signal value to be originally obtained, but the correct value can be estimated by using the method of the second embodiment.

FIG. 12 shows an example of signal values of an imaging signal of a moving object (a signal in which an object moving from upper right to lower left is photographed and the correlation with the preceding and following frames is intermediate). 12 (a) shows the signal value of the K-1 frame, FIG. 12 (b) shows the signal value of the K frame (the signal value for which the shaded signal value "1000" should be originally obtained), and FIG. 12 (c) shows K + 1 It is a signal value of a frame. In this example, the degree of correlation C ₋₁ (= weighting coefficient W _K−1 ) = 0.94 of the previous frame, and the degree of correlation C ₊₁ (= weighting coefficient W _{K + 1} ) = 0.76 of the subsequent frame. A correction mode using both correction and marginal correction is determined, and the weighting factor W _K = 0.30 of the frame is calculated. Then, the signal estimation unit 154 obtains an estimated value 331 of a normal signal. Assuming that the signal value is estimated by the peripheral average in this example, it is 243. Therefore, it can be seen that using the method of Example 2 is closer to the signal value to be originally obtained than using the marginal average.

FIG. 13 shows an imaging signal of a light emitter (an example in which an object with strong motion is photographed and there is no correlation with the preceding and following frames). 13 (a) shows the signal value of the K-1 frame, FIG. 13 (b) shows the signal value of the K frame (the signal value for which the shaded signal value "1000" should be originally obtained), and FIG. 13 (c) shows K + 1 It is a signal value of a frame. In this example, the correlation degree C ₋₁ (= weighting coefficient W _K−1 ) = 0.00 of the previous frame and the correlation degree C ₊₁ (= weighting coefficient W _{K + 1} ) = 0.00 of the later frame. Therefore, the correction mode is determined to be peripheral correction by the correction mode determination unit 157, and the estimated value 875 of a normal signal is determined by the signal estimation unit 154.

  As can be seen from the above-described three calculation examples, in the method of the second embodiment, the correction mode is determined based on the degree of correlation between the preceding and succeeding frames and the relevant frame, and a normal signal value is estimated. Therefore, even an unskilled user can correct the nonconforming signal to an appropriate value.

In the above example, for ease of description, correlation C _-1, C ₊₁ is determined before and after correction of the correction mode if it is 1, correlation C _-1, if C ₊₁ is 0 Although the signal is determined to be a signal with strong motion, and the correction mode is determined to be the peripheral correction, the value of the degree of correlation with which the correction mode is determined may be set appropriately. For example, it is possible to use front-back correction when the degree of correlation is 0.9 or more, and use peripheral correction when it is 0.1 or less . Further, in the above example (when the reference value for determining the correction mode is 1, 0), determination of the correction mode (step S3) by the correction mode determination unit 157 is skipped, and a weighting factor is calculated to be a normal signal. The same results can be obtained by estimating the values.

  In addition, the formula for calculating the degree of correlation described above is also an example, and in consideration of the characteristics of the object to be photographed (such as the characteristics of the movement and the degree of coloration) and the photographing conditions (such as the length of the sampling interval) The degree of correlation may be calculated. The following is an example of the calculation formula of the degree of correlation.

式(5)は各画素の前後のフレームの信号値の差の二乗の値を用いるもの、式(6)は各画素の前後のフレームの信号値の差の絶対値を用いるもの、式(7)は正規化相互相関を用いるものである。上述した式(1), (3)では動きが少ないほど相関度の値が小さくなったが、式(5), (6)では動きが少ないほど相関度の値が小さくなる。なお、上式(5)〜(7)を用いる場合には、それぞれ重み付け係数を正規化するための値（上記例における「２」に相当）を適宜に設定して正常な値を推定する。

Equation (5) uses the square of the difference between the signal values of the frames before and after each pixel, and equation (6) uses the absolute value of the difference between the signal values of the frames before and after each pixel, ) Uses normalized cross correlation. In the equations (1) and (3) described above, the value of the degree of correlation decreases as the motion decreases, but in the equations (5) and (6), the value of the degree of correlation decreases as the motion decreases. In addition, when using the said Formula (5)-(7), the value (equivalent to "2" in the said example) for normalizing a weighting coefficient is each set suitably, and a normal value is estimated.

Furthermore, in the above-described example, the correlations C _-1 and C ₊₁ of the preceding and following frames are used as weighting coefficients of the preceding and following frames as they are, but other weighting factors are obtained by squaring the correlations C _-1 and C ₊₁ of the preceding and following frames. Various modifications are possible, such as.

  Next, an embodiment of a method of driving a solid-state imaging device according to the present invention will be described. The configuration of the solid-state imaging device 1 is the same as that described with reference to FIGS. The driving method of this embodiment is characterized by the transmission form of the driving pulse signal.

  In the present embodiment, when the user instructs to start imaging, the imaging control unit 52 reads the defect information of the storage element stored in the storage unit 51 in the storage unit 51 and specifies the nonconforming storage element. Here, the case where the third storage element of the L storage elements corresponding to a certain pixel 10 is a nonconforming storage element will be described as an example. When the defect information of the storage element is read out, the imaging control unit 52 creates a transmission pattern of the drive pulse signal shown in FIG. That is, in the driving method of the present embodiment, by adding the skip signal to the clock signal φVCLK1 to the shift register VSR1, the storage element which has been found to have a defect in advance is skipped and the pixel 10 Hold the output signal. Also, even after the end of imaging, storage elements skipped at the time of imaging are skipped and signals are sequentially read out from the other storage elements.

  In the driving method of this embodiment, a storage element known to have a defect in the manufacturing process and a storage element from which a normal signal was not read at the previous imaging time are regarded as incompatible elements before the start of imaging. A signal output from the pixel 10 is held using only normal storage elements specified and excluding the storage elements. Therefore, it is possible to obtain imaging data which is not affected by a defect of the memory element.

  The above embodiments are all examples and can be appropriately modified in accordance with the spirit of the present invention. The number and arrangement of pixels and the number and arrangement of storage elements are merely examples, and the same signal processing method and driving method as described above can be used for solid-state imaging devices having other configurations. For example, although the pixel output line 40 is provided independently for each pixel 10 in the above embodiment, a plurality of pixels 10 share one pixel output line, and the pixels 10 to be sequentially connected are switched by a shift register or the like. Also good.

  In the above embodiment, the solid-state imaging device in which the pixels are two-dimensionally arranged in a grid is exemplified. However, the above signal processing method and driving method can be used for a solid-state imaging device in which the pixels are arranged in a honeycomb. Further, among the signal processing methods of the above-described embodiment, a signal of the adjacent frame is averaged to estimate a normal signal, or the driving method of the above-described embodiment is a solid-state imaging device in which pixels are arranged one-dimensionally. It can also be used for a curvilinear shape or a loop shape. Moreover, it is also possible to combine and use said Examples 1-3. For example, imaging is performed by skipping storage elements whose defects have been confirmed before the start of imaging (corresponding to the third embodiment), and a signal that is read out after the imaging is not compatible (that is, reading process from the storage element) In the case where a signal in which a failure has occurred is included, it is possible to estimate a normal signal (corresponding to Examples 1 and 2) to replace the nonconforming signal.

DESCRIPTION OF SYMBOLS 1 solid image pickup element 2 pixel area 3 memory area 4 vertical scanning circuit area 5 horizontal scanning circuit area 10 pixel 11 photodiode 12 transfer transistor 13 reset transistor 14 floating diffusion 15 for buffer Transistor 16 first current blocking transistor 17 first biasing transistor 18 first capacitor 19 first sampling transistor 20 second sampling transistor 21 second capacitor 22 source follower amplifying transistor 23 second 2 Current interrupting transistor 24 Load current source transistor 25 Output control transistor 30 Memory unit 31 Writing transistor 36 Sampling transistor 37 Capacitor 38 Vertical reading transistor 39 Water Readout transistor 40: Pixel output line 50, 150: Control and signal processing unit 51, 151: Storage unit 52, 152: Imaging control unit 53, 153: Nonconforming signal identification unit 54, 154: Signal estimation unit 55, 155: Image Generation unit 156: Correlation degree calculation unit 157: Correction mode determination unit 60, 160: Input unit 70, 170: Display unit

Claims (7)

A plurality of pixels disposed adjacent to each other and outputting signals according to the intensity of light received by each;
A plurality of predetermined storage elements provided for each of the plurality of pixels, wherein the plurality of storage elements sequentially hold signals output from each of the plurality of pixels; A method of processing a signal read out from part or all of the plurality of storage elements respectively corresponding to
Identifying a nonconforming signal that does not meet predetermined conditions from among the signals read from the storage element;
Signal processing method of the solid-state imaging device and estimates a normal signal based on the previous SL is a signal obtained in the time incompatible signal adjacent to the non-conforming signal at the corresponding pixel storage element that is read. further,
A signal acquired at the same time as the nonconforming signal, and a signal acquired at the time adjacent to the nonconforming signal, for the pixel disposed adjacent to the pixel corresponding to the storage element from which the nonconforming signal is read out Find the degree of correlation between
When the degree of correlation is smaller than a predetermined value, a signal acquired at a time corresponding to the nonconforming signal in a pixel arranged adjacent to a pixel corresponding to the storage element from which the nonconforming signal is read signal processing method for a solid state image pickup device according to claim 1, characterized in that you estimate the normal signal on the basis of. Processing using the signal acquired at a time corresponding to the nonconforming signal in a pixel disposed adjacent to the pixel corresponding to the storage element from which the nonconforming signal is read using the degree of correlation; and the nonconforming signal Weighting is applied to processing using a signal acquired at a time adjacent to the nonconforming signal in a pixel corresponding to the storage element from which the data is read out, and both processing is performed to estimate a normal signal. The signal processing method of the solid-state imaging device according to claim 2. A plurality of pixels disposed adjacent to each other and outputting signals according to the intensity of light received by each;
A plurality of predetermined storage elements provided for each of the plurality of pixels, wherein the plurality of storage elements sequentially hold signals output from each of the plurality of pixels; A device for processing a signal read from part or all of the plurality of storage elements respectively corresponding to the pixels of
A nonconforming signal identification unit that identifies nonconforming signals that do not conform to predetermined conditions from among the signals read from the storage element;
A signal estimation unit for estimating a normal signal based on the previous SL is a signal obtained in the time incompatible signal adjacent to the non-conforming signal in a pixel corresponding to the storage element that is read,
A signal processing apparatus comprising: further,
A signal acquired at the same time as the nonconforming signal, and a signal acquired at the time adjacent to the nonconforming signal, for the pixel disposed adjacent to the pixel corresponding to the storage element from which the nonconforming signal is read out A correlation degree calculation unit for determining the correlation degree between
The signal estimation unit corresponds to the nonconforming signal in a pixel disposed adjacent to a pixel corresponding to the storage element from which the nonconforming signal is read, when the degree of correlation is smaller than a predetermined value. signal processing device for a solid state image pickup device according to claim 4, characterized in that you estimate the normal signal on the basis of the signal acquisition time. Processing using the signal acquired at a time corresponding to the nonconforming signal in a pixel disposed adjacent to the pixel corresponding to the storage element from which the nonconforming signal is read using the degree of correlation; and the nonconforming signal Weighting is applied to processing using a signal acquired at a time adjacent to the nonconforming signal in a pixel corresponding to the storage element from which the data is read out, and both processing is performed to estimate a normal signal. The signal processing device according to claim 5. One or more pixels each outputting a signal according to the intensity of the light received;
A driving method of a solid-state imaging device including: a plurality of predetermined storage devices provided for each of the one or the plurality of pixels;
Identifying a nonconforming storage element which does not conform to a predetermined condition among the predetermined plurality of storage elements corresponding to each of the one or more pixels;
For each of the one or more pixels, a normal storage element obtained by removing the nonconforming storage element from a plurality of predetermined storage elements associated with the pixel is used to sequentially hold the signal output from the pixel A method of driving a solid-state imaging device characterized by:

Images