JP2018186460A - Imaging apparatus and processing method of imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of reducing the influence of coloring due to the leakage of an electric charge from a photoelectric conversion part to a holding part and to provide the processing method of the imaging apparatus.SOLUTION: The imaging apparatus includes a plurality of pixels (102) which generate a video signal by photoelectric conversion, a detection part (202) which detects external flash light on the basis of the video signal, and a processing part (204) which performs processing for reducing the influence of the external flash light with respect to the video signal of a frame previous to a frame in which the external flash light is detected, when the external flash light is detected by the detection part. Each of the plurality of pixels includes the photoelectric conversion part which converts light into the electric charge and stores it, the holding part which holds the electric charge, an amplification part which amplifies a signal based on the electric charge, a first transfer switch which transfers the electric charge of the photoelectric conversion part to the holding part, and a second switch which transfers the electric charge of the holding part to the amplification part.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging device and a processing method of the imaging device.

  A CMOS type image sensor (hereinafter referred to as a CMOS sensor) is used as an imaging element of an imaging apparatus typified by a video camera. A CMOS sensor mainly adopts a rolling shutter system that sequentially exposes horizontal lines and sequentially reads out video signals for each line to generate one frame. In recent years, an increasing number of CMOS sensors have adopted a global shutter system that reads out an image simultaneously with exposure timing of horizontal lines in all rows. In general, in the rolling shutter system, rolling shutter distortion occurs due to a shift in exposure time in each row. On the other hand, the global shutter method is effective when shooting a moving object without causing rolling shutter distortion by performing exposure simultaneously in all rows.

  In order to realize the global shutter system, the CMOS sensor has a holding unit that holds charges in addition to the photoelectric conversion unit. However, the global shutter method has a holding unit as compared with the rolling shutter method, so that the area of the photoelectric conversion unit is reduced, the amount of saturated charge that can be processed by one pixel is small, and the saturated charge flows into the holding unit. . In the global shutter method, the external flash light such as strobe light causes a charge saturated in a pixel having a high sensitivity of the photoelectric conversion unit to flow into the holding unit, and is read as a pseudo signal colored in a band shape. is there.

  In Patent Document 1, a line where the influence of an external flash such as a strobe light starts and ends is detected from an increase in the luminance level of an image, and a luminance gain is set for each of an affected line and an unaffected line. It describes how to make adjustments.

JP 2011-193194 A

  However, in the global shutter system, the photoelectric conversion unit may read the N−1th frame charge from the holding unit while accumulating the Nth frame charge. For this reason, in the global shutter system, when an external flash is incident while the Nth frame charge is being accumulated, a pseudo signal colored in a band shape is generated for the (N-1) th frame, not the Nth frame. There is. Also, in the global shutter method, the horizontal line exposure timing is read out simultaneously for all rows, so that an increase in luminance level due to external flash occurs for all lines in the Nth frame. The line where the influence starts and ends cannot be detected.

  An object of the present invention is to provide an imaging device and a processing method of the imaging device that can reduce the influence of coloring due to charge leakage from the photoelectric conversion unit to the holding unit.

  The imaging device of the present invention includes a plurality of pixels that generate a video signal by photoelectric conversion, a detection unit that detects an external flash based on the video signal, and when the external flash is detected by the detection unit, A processing unit that performs processing to reduce the influence of the external flash on the video signal of the frame one frame before the frame in which the external flash is detected, and each of the plurality of pixels converts light into electric charge. A photoelectric conversion unit that converts and accumulates; a holding unit that holds charge; an amplification unit that amplifies a signal based on the charge; a first transfer switch that transfers the charge of the photoelectric conversion unit to the holding unit; And a second transfer switch for transferring the charge of the holding unit to the amplification unit.

  The influence of coloring due to charge leakage from the photoelectric conversion unit to the holding unit can be reduced.

It is a block diagram showing an example of composition of an imaging device concerning a 1st embodiment. It is a figure which shows the structural example of the pixel part which concerns on 1st Embodiment. 3 is a timing chart illustrating the operation of the image sensor according to the first embodiment. It is a figure which shows the image output from the external output circuit which concerns on 1st Embodiment. 3 is a flowchart showing the operation of the LSI according to the first embodiment. 3 is a timing chart showing the operation of the LSI according to the first embodiment. It is a block diagram which shows the structural example of the imaging device which concerns on 2nd Embodiment. It is a timing chart which shows operation of an image sensor concerning a 2nd embodiment. 10 is a timing chart showing the operation of the LSI according to the second embodiment. It is a block diagram which shows the structural example of the imaging device which concerns on 3rd Embodiment. 10 is a flowchart showing the operation of an LSI according to a third embodiment. 10 is a timing chart illustrating an operation of an LSI according to a third embodiment. It is a block diagram which shows the structural example of the imaging device which concerns on 4th Embodiment. 10 is a timing chart illustrating an operation of an LSI according to a fourth embodiment.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus according to the first embodiment of the present invention. The imaging apparatus has an imaging element 100 and an LSI 200 and takes a moving image. The imaging device can be applied to a smartphone, a tablet, an industrial camera, a medical camera, and the like in addition to a digital camera and a video camera. Hereinafter, a processing method of the imaging apparatus will be described. The image sensor 100 is, for example, a CMOS sensor, and includes a timing control unit 101, a pixel unit 102, a vertical scanning circuit 103, a column circuit 104, a horizontal transfer circuit 105, a signal processing circuit 106, and an external output circuit 107. And a controller circuit 108. The LSI 200 is, for example, an FPGA or an ASIC, and includes an external input circuit 201, a flash detection circuit 202, a frame memory 203, a color suppression circuit 204, a camera signal processing unit 205, a video output unit 206, and a CPU 207. .

  First, the image sensor 100 will be described. The controller circuit 108 is an interface unit that performs serial communication between the image sensor 100 and the LSI 200, and receives a control signal from the CPU 207 of the LSI 200 to the image sensor 100. The timing control unit 101 supplies an operation clock signal and a timing signal to each block of the image sensor 100 to control the operation.

  The pixel unit 102 has a plurality of pixels that are arranged in a two-dimensional matrix and generate a video signal by photoelectric conversion. Each pixel has a photoelectric conversion unit that performs photoelectric conversion according to the amount of incident light and outputs a voltage. The photoelectric conversion unit is, for example, a photodiode. A color filter and a microlens are mounted on the surface of the photoelectric conversion unit of each pixel, and so-called color filters of R (red), G (green), and B (blue) are used. It takes a periodic structure of Bayer array using RGB primary color filters. The arrangement is not limited to the Bayer arrangement, and other arrangements may be used. The pixel unit 102 implements a global shutter method. Note that the configuration of the pixel portion 102 will be described later with reference to FIG.

  The vertical scanning circuit 103 performs timing control for sequentially reading out pixel signal voltages of a plurality of pixels arranged in a matrix in the pixel unit 102 in one frame. The signal lines 117 in each column are connected to the pixels in each column in the pixel unit 102, respectively. The vertical scanning circuit 103 sequentially reads out pixel signals to the signal lines 117 of each column from the upper row to the lower row in one frame, in units of rows. The column circuit 104 includes an amplifier for electrically amplifying the pixel signal of the signal line 117 of each column, and an analog-digital conversion circuit that converts the pixel signal from analog to digital.

  The horizontal transfer circuit 105 outputs pixel signals in units of rows output from the column circuit 104 to the signal processing circuit 106 in order for each column. The signal processing circuit 106 performs digital signal processing on the pixel signal. Specifically, the signal processing circuit 106 simply performs gain calculation by adding a certain amount of offset value to the pixel signal by digital processing and performing shift calculation and multiplication. The pixel portion 102 has a light-shielded pixel region, and the signal processing circuit 106 performs a digital black level clamping operation of a normal pixel signal using the light-shielded pixel signal.

  The signal processing circuit 106 outputs the digitally processed signal to the external output circuit 107. The external output circuit 107 is a serializer, converts the multi-bit parallel signal output from the signal processing circuit 106 into a serial signal, converts the serial signal into, for example, an LVDS signal, and outputs it to the LSI 200 as a video signal. The image sensor 100 outputs video signals of a plurality of frames at predetermined time intervals to the LSI 200.

  Next, the LSI 200 will be described. The external input circuit 201 is a deserializer, and converts the serial signal input from the external output circuit 107 in the image sensor 100 into a multi-bit parallel signal for each pixel. The external input circuit 201 outputs the converted multi-bit parallel signal for each pixel to the flash detection circuit 202 and the frame memory 203 as a video signal.

  The flash detection circuit 202 extracts the luminance signal of the video signal input from the external input circuit 201, and calculates the average value of the luminance signal within one frame period. The flash detection circuit 202 holds the average luminance signal value of the previous frame and calculates difference data from the average luminance signal value of the current frame. If the difference data is larger than the threshold value, the flash detection circuit 202 determines that an external flash has entered and outputs flash detection information to the color suppression circuit 204. The flash detection circuit 202 is a detection unit. When the difference data is larger than the threshold, the flash detection circuit 202 determines that an external flash has been detected for the current frame. When the difference data is smaller than the threshold, It is determined that no external flash is detected for the frame.

  The frame memory 203 has a capacity capable of holding a video signal for two frame periods. The frame memory 203 delays the video signal input from the external input circuit 201 by 2 frames and outputs the delayed video signal to the color suppression circuit 204.

  The color suppression circuit 204 receives the flash detection information from the flash detection circuit 202 and the video signal from the frame memory 203. The color suppression circuit 204 changes a correction value for performing color suppression processing on the video signal according to the flash detection information, performs color suppression processing according to the correction value, and performs camera signal processing on the video signal after the color suppression processing. The data is output to the unit 205.

  The camera signal processing unit 205 includes a gamma processing circuit, a luminance adjustment circuit, and the like, performs signal processing on the video signal, and outputs the video signal after the signal processing to the video output unit 206. The video output unit 206 converts the video signal into a format for output to the outside of the imaging apparatus. For example, the video output unit 206 converts the video signal into a serial digital interface (hereinafter referred to as SDI) format. The CPU 207 inputs setting values for determining the time of one frame period and the accumulation time of the image sensor 100 from the user interface, and outputs them to the controller circuit 108.

  FIG. 2 is a diagram illustrating a configuration example of one pixel among a plurality of pixels arranged in the pixel unit 102 of FIG. The plurality of pixels have the same configuration. The pixel includes a photoelectric conversion unit 110, a transfer transistor TX1, a holding unit 112, a transfer transistor TX2, a holding unit 114, a reset transistor RES, an amplification transistor 115, and a selection transistor SEL. The signal lines 117 in each column are connected to the pixels in each column in the pixel unit 102, respectively.

  The photoelectric conversion unit 110 is, for example, a photodiode, and converts incident light into electric charge and accumulates it. The transfer transistor TX1 is a transfer switch, and transfers the charge accumulated in the photoelectric conversion unit 110 to the holding unit 112 according to the gate voltage. The transfer transistor TX2 is a transfer switch, and transfers the electric charge held in the holding unit 112 to the holding unit 114 and the amplification transistor 115 according to the gate voltage. The reset transistor RES resets the voltage of the holding unit 114 to the power supply voltage VDD according to the gate voltage. The amplification transistor 115 is an amplification unit, amplifies a signal based on the electric charge of the holding unit 114, and outputs the amplified signal to the signal line 117 via the selection transistor SEL. The selection transistor SEL outputs the output voltage of the amplification transistor 115 to the signal line 117 according to the gate voltage. The transistor OFD is a reset switch, and resets the electric charge of the photoelectric conversion unit 110 according to the gate voltage.

  By providing the holding unit 112, the holding unit 112 can accumulate the electric charge of the photoelectric conversion unit 110 until the voltage corresponding to the electric charge of the holding unit 114 is output to the signal line 117. The unit 110 can accumulate charges based on photoelectric conversion. Therefore, the pixel unit 102 can perform a global shutter operation in which exposure is performed on all pixels simultaneously.

  FIG. 3 is a timing chart showing the operation of the image sensor 100. Hereinafter, the operation frame rate of the image sensor 100, the operation of the photoelectric conversion unit 110, the operation of the transistor OFD, the operation of the transfer transistor TX1, the operation of the holding unit 112, the operation of the transfer transistor TX2 of each line, and the external The output video signal of the output circuit 107 will be described. At time tf, an external flash is incident on the pixel unit 102.

  Times t <b> 0 to t <b> 3 are the operation period of the (N−1) th frame of the image sensor 100. Times t3 to t6 are operation periods of the Nth frame of the image sensor 100. Times t6 to t9 are an operation period of the (N + 1) th frame of the image sensor 100. Times t9 to t12 are the operation period of the (N + 2) th frame of the image sensor 100.

  At times t <b> 0 to t <b> 1, the transistors OFD of all the pixels are turned on and reset the charges generated in the photoelectric conversion units 110 of all the pixels, respectively. Similarly, at times t3 to t4, t6 to t7, and t9 to t10, the transistors OFD of all the pixels are turned on and reset the charges generated in the photoelectric conversion units 110 of all the pixels, respectively.

  At times t1 to t3, the photoelectric conversion units 110 of all the pixels accumulate the Nth frame charges generated by the photoelectric conversion, respectively. From time t4 to t6, the photoelectric conversion units 110 of all the pixels accumulate the N + 1th frame charge generated by the photoelectric conversion, respectively. From time t7 to t9, the photoelectric conversion units 110 of all the pixels accumulate the charges of the (N + 2) th frame generated by the photoelectric conversion, respectively. At times t10 to t12, the photoelectric conversion units 110 of all the pixels accumulate the charges of the (N + 3) th frame generated by the photoelectric conversion.

  Immediately before time t0, the transfer transistors TX1 of all the pixels are turned on and transfer the N−1th frame charge accumulated in the photoelectric conversion units 110 of all the pixels to the holding units 112 of all the pixels, respectively. Immediately before time t3, the transfer transistors TX1 of all the pixels are turned on, and transfer the N-th frame charges accumulated in the photoelectric conversion units 110 of all the pixels to the holding units 112 of all the pixels, respectively. Immediately before time t6, the transfer transistors TX1 of all the pixels become conductive, and transfer the N + 1th frame charge accumulated in the photoelectric conversion units 110 of all the pixels to the holding units 112 of all the pixels, respectively. Immediately before time t9, the transfer transistors TX1 of all the pixels are turned on, and transfer the charges of the (N + 2) th frame accumulated in the photoelectric conversion units 110 of all the pixels to the holding units 112 of all the pixels, respectively. The holding unit 112 for all pixels holds the charge of the (N + 2) th frame.

  At times t <b> 0 to t <b> 3, the holding unit 112 for all pixels holds the charge of the (N−1) th frame. From time t3 to t6, the holding unit 112 for all pixels holds the charge of the Nth frame. From time t6 to t9, the holding unit 112 for all pixels holds the charge of the (N + 1) th frame. From time t9 to t12, the holding unit 112 of all pixels holds the charge of the (N + 2) th frame.

  At time t0 to t2, the transfer transistor TX2 of the L-th line from the first line to the last line is turned on in a row sequence, and outputs a signal of the (N-1) th frame to the signal line 117 in a row sequence. From time t3 to t5, the transfer transistor TX2 in the L-th line from the first line to the last line is turned on in the row order, and outputs the N-th frame signal to the signal line 117 in the row order. From time t6 to t8, the transfer transistor TX2 in the L-th line from the first line to the last line is turned on in the row order, and outputs the signal of the (N + 1) th frame to the signal line 117 in the row order. From time t9 to t11, the transfer transistor TX2 in the L-th line from the first line to the last line is turned on in the row order, and outputs the signal in the N + 2 frame to the signal line 117 in the row order.

  At time t0 to t2, the external output circuit 107 outputs a signal of the (N−1) th frame. From time t3 to t5, the external output circuit 107 outputs an Nth frame signal. From time t6 to t8, the external output circuit 107 outputs the signal of the (N + 1) th frame. From time t9 to t11, the external output circuit 107 outputs a signal of the (N + 2) th frame.

  At time tf in FIG. 3, external flashlight enters the pixel unit 102. Then, the charge of the photoelectric conversion unit 110 that is accumulating the charge of the (N + 1) th frame is saturated, and the holding unit 112 that holds the charge of the Nth frame from the photoelectric conversion unit 110 even when the transfer transistor TX1 is not conductive. Electric charge leaks out.

  At the time tf, the transfer transistors TX2 of the first line to the (M-1) th line have already transferred the charges of the first line to the (M-1) th line of the N frame from the holding unit 112 to the holding unit 114. On the other hand, the transfer transistors TX2 for the Mth to Lth lines do not transfer the charges for the Mth to Lth lines of the Nth frame from the holding unit 112 to the holding unit 114. In the M-th to L-th line holding units 112, the charges on the M-th to L-th lines in the Nth frame are not yet transferred, and thus are affected by the charges leaked from the photoelectric conversion unit 110. In general, among the R (red), G (green), and B (blue) pixels, the G (green) pixel has the highest sensitivity and is easily saturated. The eye pixel signal appears to be colored green.

  FIG. 4 corresponds to the timing chart of FIG. 3 and is a diagram in which video signals output from the external output circuit 107 to the LSI 200 are arranged for each frame. The video signal of the (N-1) th frame is output as a normal image because no external flash is incident. As described above, the video signal of the Nth frame is output as a normal image in the first line to the (M-1) th line, and is output as a green colored image in the Mth line to the Lth line. The video signal of the (N + 1) th frame is output as a high-brightness flash image on the first line to the L-th line because external flash light is incident when the photoelectric conversion unit 110 accumulates charges. The video signal of the (N + 2) th frame is output as a normal image.

  FIG. 5 is a flowchart showing the operation of the LSI 200. Hereinafter, a method in which the LSI 200 corrects the green colored image of the Nth frame will be described with reference to FIG. In step S <b> 100, the external input circuit 201 inputs a serial video signal from the image sensor 100, converts the input serial signal into a multi-bit parallel signal for each pixel, and outputs it to the flash detection circuit 202 and the frame memory 203. To do. The frame memory 203 holds the video signal. Next, in step S101, the flash detection circuit 202 calculates and holds the average luminance value of the video signal of the current frame.

  Next, in step S102, the flash detection circuit 202 calculates a difference between the average luminance value of the video signal of the current frame and the average luminance value of the video signal of the previous frame, and determines whether the difference is equal to or less than a predetermined value. judge. If the flash detection circuit 202 determines that the difference is equal to or smaller than a predetermined value, the flash detection circuit 202 outputs flash detection information indicating that a flash image (external flash) has not been detected to the color suppression circuit 204. The process proceeds to step S103. If the flash detection circuit 202 determines that the difference is greater than a predetermined value, the flash detection circuit 202 outputs flash detection information indicating that a flash image has been detected to the color suppression circuit 204, and the process proceeds to step S104.

  In step S103, the color suppression circuit 204 sets a color suppression correction value Cw so that R (red), G (green), and B (blue) are uniform because no flash image is detected, and step S105. Proceed with the process.

  In step S104, since the flash image is detected, the color suppression circuit 204 detects a color suppression correction value Cg that reduces the gain of G (green) among R (red), G (green), and B (blue). Is set, and the process proceeds to step S105.

  In step S <b> 105, the color suppression circuit 204 reads from the frame memory 203 the video signal of the frame immediately before the video signal of the frame held in step S <b> 100 by the frame memory 203. In step S <b> 106, the color suppression circuit 204 performs color suppression processing on the video signal read from the frame memory 203.

  Next, in step S107, the LSI 200 determines whether or not to end the operation. When it is determined that the operation is to be ended, the processing is ended. When it is determined that the operation is not to be ended, the processing is performed to step S108. Proceed. In step S108, the LSI 200 stands by until the video signal of the next frame is input. When the video signal of the next frame is input, the process returns to step S100 and the processing of the next frame is repeated.

  FIG. 6 is a timing chart showing the operation of the LSI 200. Using FIG. 6, the horizontal axis represents time, the input signal to the flash detection circuit 202, the detection result of the flash detection circuit 202, the color suppression correction value applied to the color suppression circuit 204, and the frame memory 203. The operation timing of the read video signal will be described.

  The flash detection circuit 202 receives the video signal of the (N-1) th frame at times t20 to t21, the video signal of the Nth frame at times t21 to t22, the video signal of the (N + 1) th frame at times t22 to t23, and N + 2 at the times t23 to t24. Input the video signal of the frame.

  The flash detection circuit 202 outputs, to the color suppression circuit 204, flash detection information indicating that the external flash is not detected for the video signal of the (N−2) th frame from time t20 to t21. In addition, the flash detection circuit 202 outputs flash detection information indicating that the external flash is not detected from the video signal of the (N−1) th frame to the color suppression circuit 204 at times t21 to t22. In addition, the flash detection circuit 202 outputs flash detection information indicating a result of not detecting external flash to the video signal of the Nth frame to the color suppression circuit 204 from time t22 to t23. Further, the flash detection circuit 202 outputs flash detection information indicating the result of detecting the external flash to the video signal of the (N + 1) th frame to the color suppression circuit 204 from time t23 to t24.

  The color suppression circuit 204 sets the color suppression correction value to Cw from time t20 to t23 when no external flash is detected, and sets the color suppression correction value to Cg from time t23 to t24 when external flash is detected.

  The frame memory 203 reads a frame delayed by 2 frames from the input video signal. The frame memory 203 has an N-3th frame video signal at times t20 to t21, an N-2th frame video signal at times t21 to t22, an N-1th frame video signal at times t22 to t23, and a time t23 to t22. At t24, the video signal of the Nth frame is read out.

  The color suppression circuit 204 performs color suppression processing on the video signal of the N-3th frame with the color suppression correction value Cw. In addition, the color suppression circuit 204 performs color suppression processing on the video signal of the (N−2) th frame with the color suppression correction value Cw. The color suppression circuit 204 performs color suppression processing on the video signal of the (N−1) th frame with the color suppression correction value Cw. The color suppression circuit 204 performs color suppression processing on the video signal of the Nth frame with the color suppression correction value Cg.

  When an external flash is detected, the color suppression circuit 204 reduces the influence of the external flash on the video signal of the N frame one frame before the N + 1 frame in which the external flash is detected. Color suppression processing is performed with the suppression correction value Cg. Specifically, when no external flash is detected, the color suppression circuit 204 uses the correction value Cw for the video signal of the N−1th frame before the Nth frame where no external flash is detected. Perform suppression processing. In addition, when an external flash is detected, the color suppression circuit 204 performs a color suppression process with a correction value Cg on the video signal of the N frame one frame before the N + 1 frame in which the external flash is detected. Do.

  Through the above operation, the LSI 200 applies G of R (red), G (green), and B (blue) to the video signal of the Nth frame colored green due to the influence of the charge leakage of the photoelectric conversion unit 110. The color suppression process can be executed with the color suppression correction value Cg that decreases the gain of (green). The LSI 200 detects the flash image and changes the color suppression correction value according to the detection result, thereby reducing the influence of coloring due to charge leakage to the holding unit 112 in the global shutter sensor.

  In this embodiment, the delay amount of the frame memory 203 has been described as two frames. However, the flash image is detected not in all pixels in one frame period but in the first few lines of the frame period, so that one frame and several frames are detected. A delay amount for the line may be used.

(Second Embodiment)
FIG. 7 is a block diagram illustrating a configuration example of an imaging apparatus according to the second embodiment of the present invention. Hereinafter, the points of the present embodiment different from the first embodiment will be described. In the first embodiment, the LSI 200 detects a flash image and executes a color suppression process with the color suppression correction value Cg on all frames that are affected by charge leakage. On the other hand, in the second embodiment, the LSI 200 executes the color suppression process with the color suppression correction value Cg only for the lines where charge leakage may occur.

  The imaging apparatus in FIG. 7 is different from the imaging apparatus in FIG. 1 in that an LUT 308 and a color suppression circuit 304 are provided instead of the color suppression circuit 204. The LUT 308 is a look-up table that associates a line in which the transfer transistor TX2 is turned on simultaneously with the start of charge accumulation in the photoelectric conversion unit 110 from the time of one frame period and the charge accumulation time. The LUT 308 receives information on the time and charge accumulation time for one frame period from the CPU 207, and outputs information on the line in which the transfer transistor TX 2 is turned on to the color suppression circuit 304 simultaneously with the start of charge accumulation in the photoelectric conversion unit 110. Hereinafter, the line in which the transfer transistor TX2 is turned on simultaneously with the start of charge accumulation in the photoelectric conversion unit 110 is referred to as a P-th line. The color suppression circuit 304 changes the color suppression correction value for color suppression processing of the video signal from the Pth line onward according to the flash detection information from the flash detection circuit 202 and the information from the LUT 308.

  FIG. 8 is a timing chart showing the operation of the image sensor 100. Hereinafter, the operation frame rate of the image sensor 100, the operation of the photoelectric conversion unit 110, the operation of the transistor OFD, the operation of the transfer transistor TX1, the operation of the holding unit 112, the operation of the transfer transistor TX2 of each line, and the external The output video signal of the output circuit 107 will be described. At time tf, an external flash is incident on the pixel unit 102.

  FIG. 8 differs from FIG. 3 in that the charge accumulation start timing t1 of the photoelectric conversion unit 110 and the timing t1 at which the transfer transistor TX2 of the P-th line is turned on are the same. Similarly, in FIG. 8, the charge accumulation start timings t4, t7, and t10 of the photoelectric conversion unit 110 are the same as the timings t4, t7, and t10 at which the transfer transistor TX2 of the P-th line is turned on.

  When the external flash light enters the pixel portion 102 during the period when the transistor OFD in each frame period is not conductive, the charge of the photoelectric conversion unit 110 is saturated and the charge leaks from the photoelectric conversion unit 110 to the holding unit 112. However, even if an external flash is incident on the pixel unit 102 during the period when the transistor OFD at time t0 to t1 is conducting, the charge of the photoelectric conversion unit 110 is reset by the transistor OFD. No charge leaks to 112. That is, the phenomenon that the charges on the first line to the P-1 line read from the holding unit 112 during the time t0 to t1 are colored green due to leakage does not occur. Similarly, at times t3 to t4, t6 to t7, and t9 to t10, the charge on the first line to the P-1 line is not colored green due to leakage.

  FIG. 9 is a timing chart showing the operation of the LSI 200. FIG. 9 shows an input video signal to the flash detection circuit 202, a detection result of the flash detection circuit 202, a color suppression correction value applied to the color suppression circuit 304, and a video signal read from the frame memory 203. In the first embodiment (FIG. 6), from time t23 to t24, the color suppression circuit 204 sets the color suppression correction value Cg for all lines in the Nth frame.

  On the other hand, in the present embodiment (FIG. 9), the color suppression circuit 304 at time t23 to t30 causes the first line of the Nth frame to P− where there is no possibility that green coloring is generated due to charge leakage. A color suppression correction value Cw is set for the signal on the first line. Then, at times t30 to t24, the color suppression circuit 304 performs color suppression correction values for the signals on the Pth to Lth lines of the Nth frame where green coloring may occur due to charge leakage. Set Cg.

  As described above, when no external flash is detected, the color suppression circuit 304 performs color correction with the correction value Cw on the video signal of the (N−1) th frame before the Nth frame where no external flash is detected. Perform suppression processing. In addition, when an external flash is detected, the color suppression circuit 304 converts the video signals of the first line to the (P−1) -th line in the N frame one frame before the N + 1 frame where the external flash is detected. On the other hand, color suppression processing is performed with the correction value Cw. The video signals of the first to P-1 lines are video signals of lines in which the transfer transistor TX2 transfers the charge of the holding unit 112 to the amplification transistor 115 while the transistor OFD resets the charge of the photoelectric conversion unit 110. It is. In addition, when an external flash is detected, the color suppression circuit 304 applies to the video signals of the P-th line to the L-th line in the Nth frame one frame before the N + 1th frame in which the external flash is detected. Then, color suppression processing is performed with the correction value Cg. The video signals of the P-th line to the L-th line are video signals of lines in which the transfer transistor TX2 transfers the charge of the holding unit 112 to the amplification transistor 115 while the transistor OFD does not reset the charge of the photoelectric conversion unit 110. .

  Note that the color suppression circuit 304 sets the color suppression correction value to Cg during the period from time t30 to t24, but the charge is obtained by setting the color suppression correction value to Cg during the period from time t23 to t24. The effect of coloring due to leakage may be reduced.

  As described above, the color suppression circuit 304 sets the color suppression correction value of the signal of the line that may be affected by coloring due to charge leakage to Cg from the frame rate and the accumulation time. Thereby, the influence of coloring due to the charge leakage to the holding portion 112 in the global shutter sensor can be reduced.

(Third embodiment)
FIG. 10 is a block diagram illustrating a configuration example of an imaging apparatus according to the third embodiment of the present invention. Hereinafter, the points of the present embodiment different from the first embodiment will be described. In the first embodiment, the LSI 200 detects a flash image, sets a color suppression correction value to Cg for a frame that is affected by charge leakage according to the detection result, and executes color suppression processing. In the third embodiment, the LSI 200 does not read a frame that has an influence of charge leakage, but instead reads another frame that has no influence.

  The imaging apparatus in FIG. 10 is different from the imaging apparatus in FIG. 1 in that a frame memory 403 is provided instead of the frame memory 203 and the color suppression circuit 204. The frame memory 403 receives the flash detection information from the flash detection circuit 202 and changes the read address according to the flash detection information.

  FIG. 11 is a flowchart showing the operation of the LSI 200. Hereinafter, a method in which the LSI 200 corrects the green colored image of the Nth frame will be described with reference to FIG. In the flowchart of FIG. 11, steps S203 and S204 are provided instead of steps S103 and S104, and step S106 is deleted from the flowchart of FIG. Hereinafter, the points of the flowchart of FIG. 11 different from the flowchart of FIG. 5 will be described.

  In step S102, when the flash detection circuit 202 determines that the difference between the average luminance values of the video signals of the current frame and the previous frame is equal to or smaller than a predetermined value, the process proceeds to step S203, and is larger than the predetermined value. If it is determined, the process proceeds to step S204.

  In step S203, the frame memory 403 sets the value of the read address to (write address-2), and the process proceeds to step S105. In step S204, the frame memory 403 sets the value of the read address to (write address-3), and proceeds to step S105.

  In step S105, the frame memory 403 reads the video signal of the frame indicated by the read address set in step S203 or S204, and advances the process to step S107. When the difference between the average luminance values of the video signals of the current frame and the previous frame is equal to or smaller than a predetermined value and no flash image is detected, the frame memory 403 is the previous video signal of the frame held in step S100. Read the video signal of the frame. When the difference between the average luminance values of the video signals of the current frame and the previous frame is larger than a predetermined value and a flash image is detected, the frame memory 403 stores the three video signals of the frames held in step S100. Read the video signal of the previous frame.

  FIG. 12 is a timing chart showing the operation of the LSI 200. 12 shows an input video signal to the flash detection circuit 202, a write address of the frame memory 403, a detection result of the flash detection circuit 202, a read address of the frame memory 403, and a video signal read from the frame memory 403. . In FIG. 6, the color suppression circuit 204 sets the color suppression correction value to Cg during the period from time t23 to t24 when the flash image is detected. On the other hand, in FIG. 12, in the period from time t23 to t24 when the flash image is detected, the frame memory 403 sets the read address to (write address-3).

  First, the write address of the frame memory 403 will be described. The frame memory 403 writes the signal of the (N-1) th frame to the write address ADD-1 from time t20 to t21, and writes the signal of the Nth frame to the write address ADD from time t21 to t22. The frame memory 403 writes the signal of the (N + 1) th frame to the write address ADD + 1 from time t22 to t23, and writes the signal of the (N + 2) th frame to the write address ADD + 2 from time t23 to t24.

  Next, the read address of the frame memory 403 will be described. From time t20 to t21, since no flash image is detected, the frame memory 403 sets (write address-2) as the read address ADD-3, and reads the signal of the N-3th frame of the read address ADD-3. Next, at time t21 to t22, since no flash image is detected, the frame memory 403 sets (write address-2) as the read address ADD-2, and the N-2th frame signal of the read address ADD-2. Is read. Next, at time t22 to t23, since no flash image is detected, the frame memory 403 sets (write address-2) as the read address ADD-1, and the signal of the (N-1) th frame of the read address ADD-1 Is read. Next, at time t23 to t24, since the flash image is detected, the frame memory 403 sets (write address-3) as the read address ADD-1, and the N-1th frame of the read address ADD-1 Read the signal.

  When a flash image is detected, the frame memory 403 does not read the signal of the Nth frame in which coloring due to charge leakage has occurred, but instead of the signal of the (N−1) th frame that is not affected by the charge leakage. read out. When a flash image is detected, the frame memory 403 can reduce the influence of coloring due to charge leakage to the holding unit 112 in the global shutter sensor by changing the read address.

  In the present embodiment, the delay amount of the frame memory 403 has been described as two frames. Therefore, the read address set in step S203 in FIG. 11 is set to (write address-2), and the read address set in step S204 is set to (write address− 3). The setting of the read address may be appropriately changed according to the delay amount in the frame memory 403.

  The frame memory 403 stores the video signal generated by the image sensor 100. If no external flash is detected, the frame memory 403 reads the video signal of the (N-1) th frame before the Nth frame where no external flash is detected. Further, when an external flash is detected, the frame memory 403 reads the video signal of the (N−1) th frame that is two frames before the N + 1th frame in which the external flash is detected.

(Fourth embodiment)
FIG. 13 is a block diagram illustrating a configuration example of an imaging apparatus according to the fourth embodiment of the present invention. Hereinafter, differences of the present embodiment from the third embodiment will be described. In the third embodiment, the LSI 200 detects a flash image, and replaces all lines of a frame having an influence of charge leakage with all frames of a frame having no influence of charge leakage according to the detection result. In the present embodiment, the LSI 200 replaces only a frame line in which charge leakage may occur with a frame line that is not affected by charge leakage.

  The imaging apparatus in FIG. 13 is different from the imaging apparatus in FIG. 10 in that an LUT 308 and a frame memory 503 are provided instead of the frame memory 403. The LUT 308 is the same as that described in the second embodiment (FIG. 7). The LUT 308 outputs, to the frame memory 503, information on a line through which the transfer transistor TX2 is turned on simultaneously with the start of charge accumulation in the photoelectric conversion unit 110. In the description of this embodiment, the line on which the transfer transistor TX2 is turned on simultaneously with the start of charge accumulation in the photoelectric conversion unit 110 is the P-th line. The frame memory 503 changes the read address after the P line of the frame of the flash image to the read address after the P line of the previous frame in accordance with the flash detection information from the flash detection circuit 202 and the information from the LUT 308. To do.

  FIG. 14 is a timing chart showing the operation of the LSI 200. FIG. 14 shows an input video signal to the flash detection circuit 202, a write address of the frame memory 503, a detection result of the flash detection circuit 202, a read address of the frame memory 503, and a video signal read from the frame memory 503. . In the third embodiment (FIG. 12), the frame memory 403 replaces all the lines of the Nth frame with all the lines of the (N−1) th frame during the period of time t23 to t24.

  Next, the read address of the frame memory 403 of this embodiment (FIG. 14) will be described. At times t23 to t30, the frame memory 503 reads (write address-2) as a read address for the signals of the first line to the P-1 line where there is no possibility that the green coloring is generated due to the charge leakage. Set as ADD. Then, the frame memory 503 reads signals on the first line to the (P-1) th line of the Nth frame indicated by the read address ADD.

  At times t30 to t24, the frame memory 503 reads (write address-3) as a read address with respect to signals on the (P-1) -th line to the L-th line that may be colored green due to charge leakage. Set as ADD-1. Then, the frame memory 503 reads the signals of the (P-1) -th line to the L-th line of the (N-1) th frame indicated by the read address ADD-1.

  As described above, in the frame memory 503, there is a possibility that the influence of coloring due to the charge leakage may occur in the read address of the line in the range where the influence of coloring due to the charge leakage may occur from the frame rate and the accumulation time. Change the read address to a line with no range. Thereby, the influence of coloring due to the charge leakage to the holding portion 112 in the global shutter sensor can be reduced.

  The frame memory 503 stores the video signal generated by the image sensor 100. When no external flash is detected, the frame memory 503 reads the video signal of the (N-1) th frame before the Nth frame where no external flash is detected. Further, when an external flash is detected, the frame memory 503 reads the video signals of the first line to the (P-1) -th line in the N frame one frame before the N + 1 frame in which the external flash is detected. The video signals of the first to P-1 lines are video signals of lines in which the transfer transistor TX2 transfers the charge of the holding unit 112 to the amplification transistor 115 while the transistor OFD resets the charge of the photoelectric conversion unit 110. It is. Further, when an external flash is detected, the frame memory 503 reads the video signals of the P-th line to the L-line in the N−1 frame two frames before the N + 1 frame where the external flash is detected. The video signals of the P-th line to the L-th line are video signals of lines in which the transfer transistor TX2 transfers the charge of the holding unit 112 to the amplification transistor 115 while the transistor OFD does not reset the charge of the photoelectric conversion unit 110. .

  According to the first to fourth embodiments, in the global shutter type sensor, when an external flash is incident while the photoelectric conversion unit 110 is accumulating the charge of the (N + 1) th frame, the photoelectric conversion unit 110 to the holding unit 112. Electric charge leaks into the box. The holding unit 112 stores the Nth frame charge. Therefore, the video signal of the Nth frame is colored green due to the influence of charge leakage due to external flash. The color suppression circuit or the frame memory is a processing unit and, when an external flash is detected, reduces the influence of the external flash on the video signal of the frame one frame before the frame where the external flash is detected. Process. Thereby, the influence of coloring due to charge leakage from the photoelectric conversion unit 110 to the holding unit 112 in the global shutter sensor can be reduced.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

102 pixel unit, 110 photoelectric conversion unit, TX1, TX2 transfer transistor, 112 holding unit, 115 amplification transistor, OFD transistor, 202 flash detection circuit, 203, 403, 503 frame memory, 204, 304 color suppression circuit

Claims (9)

A plurality of pixels that generate video signals by photoelectric conversion;
A detection unit for detecting an external flash based on the video signal;
A processing unit that performs a process of reducing the influence of the external flash on a video signal of a frame one frame before the frame in which the external flash is detected when an external flash is detected by the detection unit; Have
Each of the plurality of pixels is
A photoelectric conversion unit that converts light into electric charge and stores it;
A holding unit for holding electric charge;
An amplifier for amplifying a signal based on the charge;
A first transfer switch for transferring the charge of the photoelectric conversion unit to the holding unit;
An image pickup apparatus comprising: a second transfer switch configured to transfer the charge of the holding unit to the amplification unit.   When the external flash is not detected, the processing unit performs a color suppression process with a first correction value on a video signal of a frame one frame before the frame where the external flash is not detected, and the external flash The color suppression processing is performed with a second correction value on a video signal of a frame one frame before the frame in which the external flash is detected. Imaging device. Each of the plurality of pixels has a reset switch for resetting the electric charge of the photoelectric conversion unit,
The processor is
When the external flash is not detected, color suppression processing is performed with a first correction value on the video signal of the frame one frame before the frame where the external flash is not detected,
When the external flash is detected, the second transfer switch is in a period in which the reset switch resets the electric charge of the photoelectric conversion unit in a frame one frame before the frame where the external flash is detected. Performs a color suppression process with a first correction value on the video signal of the line in which the charge of the holding unit is transferred to the amplification unit, and the reset switch does not reset the charge of the photoelectric conversion unit. 2. The imaging according to claim 1, wherein the second transfer switch performs color suppression processing with a second correction value on a video signal of a line in which the charge of the holding unit is transferred to the amplification unit. apparatus.   When the external flash is not detected, the processing unit performs processing such that red, green, and blue are uniform with respect to the video signal of the frame one frame before the frame where the external flash is not detected, When the external flash is detected, a process is performed in which the gain of red, green, and blue is reduced with respect to the video signal of the frame one frame before the frame where the external flash is detected. The imaging device according to claim 1, wherein the imaging device is performed. Each of the plurality of pixels has a reset switch for resetting the electric charge of the photoelectric conversion unit,
The processor is
If the external flash is not detected, the video signal of the frame one frame before the frame where the external flash is not detected is processed so that red, green, and blue are uniform.
When the external flash is detected, the second transfer switch is in a period in which the reset switch is resetting the charge of the photoelectric conversion unit in a frame one frame before the frame where the external flash is detected. Performs a process for making red, green and blue uniform on the video signal of the line in which the charge of the holding unit is transferred to the amplification unit, and the reset switch resets the charge of the photoelectric conversion unit. Performing processing such that the green gain of red, green, and blue decreases for the video signal of the line in which the second transfer switch transfers the charge of the holding unit to the amplification unit during a period of no The imaging apparatus according to claim 1. The processing unit is a frame memory that stores a video signal generated by the plurality of pixels,
When the external flash is not detected, the frame memory reads a video signal of a frame one frame before the frame where the external flash is not detected. When the external flash is detected, the external flash is detected. The image pickup apparatus according to claim 1, wherein a video signal of a frame two frames before the set frame is read out. Each of the plurality of pixels has a reset switch for resetting the electric charge of the photoelectric conversion unit,
The processing unit is a frame memory that stores a video signal generated by the plurality of pixels,
The frame memory is
When the external flash is not detected, the video signal of the frame one frame before the frame where the external flash is not detected is read.
When the external flash is detected, the second transfer switch is in a period in which the reset switch resets the electric charge of the photoelectric conversion unit in a frame one frame before the frame where the external flash is detected. Reads the video signal of the line in which the charge of the holding unit is transferred to the amplification unit, and the reset switch resets the charge of the photoelectric conversion unit in the frame two frames before the frame where the external flash was detected. 2. The imaging apparatus according to claim 1, wherein the second transfer switch reads out a video signal of a line in which the charge of the holding unit is transferred to the amplifying unit during a period when there is no period. Each of the plurality of pixels has a reset switch for resetting the electric charge of the photoelectric conversion unit,
When the external flash is detected, the processing unit is configured so that the reset switch does not reset the charge of the photoelectric conversion unit in a frame one frame before the frame where the external flash is detected. The imaging apparatus according to claim 1, wherein the second transfer switch performs a process of reducing the influence of the external flash on the video signal of the line in which the charge of the holding unit is transferred to the amplification unit. . Each of the plurality of pixels
A photoelectric conversion unit that converts light into electric charge and stores it;
A holding unit for holding electric charge;
An amplifier for amplifying a signal based on the charge;
A first transfer switch for transferring the charge of the photoelectric conversion unit to the holding unit;
A processing method of an imaging apparatus having a second transfer switch for transferring the charge of the holding unit to the amplification unit,
Generating a video signal by photoelectric conversion with the plurality of pixels;
A step of detecting an external flash based on the video signal by a detection unit;
When the processing unit detects the external flash, performing a process of reducing the influence of the external flash on the video signal of the frame one frame before the frame where the external flash is detected. A processing method for an imaging apparatus, comprising:

Images