JP2020031257A - Imaging apparatus, camera, and imaging method - Google Patents

Abstract

To provide an imaging apparatus capable of increasing higher image processing speed than the prior art.SOLUTION: An imaging apparatus 10 includes: a solid-state image pickup device 100 having a plurality of pixels 210 arranged in a matrix and capable of nondestructive reading; and an image processing part 320 that generates a captured image P13 by correcting a first-pixel signal obtained from the solid-state image pickup device 100 after exposure is completed. The image processing part 320 performs the correction on the basis of a second-pixel signal obtained from the solid-state image pickup device 100 by nondestructive reading during exposure.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to an imaging device, a camera including the imaging device, and an imaging method thereof.

  2. Description of the Related Art Conventionally, an imaging device that captures an image using an image sensor is known (for example, see Patent Document 1).

JP 2008-042180 A

  Various kinds of image processing are performed in the imaging apparatus, and it is desired that the imaging apparatus has a high speed of the image processing.

  Therefore, the present disclosure provides an imaging device, a camera, and an imaging method in which the speed of image processing is higher than in the related art.

  In order to achieve the above object, an imaging device according to an embodiment of the present disclosure is arranged in a matrix, a solid-state imaging device having a plurality of pixels capable of nondestructive readout, and from the solid-state imaging device after the end of exposure. An image processing unit that generates a captured image by performing correction on the obtained first pixel signal, wherein the image processing unit obtains a second image obtained from the solid-state imaging device by the nondestructive readout during the exposure. The above correction is performed based on the pixel signal.

  Further, a camera according to an aspect of the present disclosure includes the imaging device described above.

  In addition, the imaging method according to an aspect of the present disclosure includes acquiring a first pixel signal from a solid-state imaging device having a plurality of pixels arranged in a matrix and capable of nondestructive readout after the end of exposure, A captured image is generated by performing correction on the first pixel signal based on the second pixel signal acquired from the solid-state imaging device by non-destructive readout.

  According to the imaging device, the camera, and the imaging method according to the present disclosure, the speed of image processing is increased.

FIG. 1 is a block diagram illustrating an overall configuration of an imaging device according to an embodiment. FIG. 2 is a diagram illustrating an example of a circuit configuration of a pixel according to an embodiment. FIG. 2 is a functional block diagram of a camera including the imaging device according to the embodiment; 5 is a flowchart illustrating an operation of the imaging device according to the embodiment. 9 is a flowchart illustrating an operation of the imaging apparatus according to the conventional example. FIG. 4 is a diagram for describing a flow of image processing according to the embodiment. FIG. 4 is a diagram illustrating an image generated by the image processing according to the embodiment. FIG. 1 is an external view of a camera including an imaging device according to an embodiment.

  Hereinafter, an imaging device, a camera, and an imaging method according to the present disclosure will be described in detail with reference to the drawings. Each of the embodiments described below shows a preferred specific example of the present disclosure. Therefore, the numerical values, shapes, materials, constituent elements, arrangement and connection forms of the constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples, and do not limit the present disclosure. Therefore, among the components in the following embodiments, components that are not described in the independent claims that indicate the highest concept of the present disclosure are described as arbitrary components.

  It is noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the claimed subject matter. In addition, each drawing is a schematic diagram, and is not necessarily strictly illustrated.

(Embodiment)
An embodiment will be described below with reference to FIGS.

[1. Overall configuration of imaging apparatus]
First, the overall configuration of the imaging device according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram illustrating an overall configuration of an imaging device 10 according to the present embodiment. The imaging device 10 illustrated in FIG. 1 includes a solid-state imaging device 100 and a signal processing unit 300. Further, the solid-state imaging device 100 includes a pixel array unit 110, a column AD conversion unit 120, a row scanning unit 130, a column scanning unit 140, and a drive control unit 150. In the pixel array section 110 and its peripheral area, a column signal line 160 is arranged for each pixel column, and a scanning line 170 is arranged for each pixel row.

  The pixel array unit 110 is an imaging unit in which a plurality of pixels 210 are arranged in a matrix.

  The column AD conversion (analog / digital converter) unit 120 obtains, holds, and outputs a digital value corresponding to the amount of light received by the pixel 210 by digitally converting a signal (analog signal) input from each column signal line 160. This is a conversion unit.

  The row scanning unit 130 has a function of controlling the reset operation, the charge accumulation operation, and the read operation of the pixel 210 on a row-by-row basis.

  The column scanning unit 140 causes the signal processing unit 300 to output the digital values for one row held in the column AD conversion unit 120 sequentially to the row signal line 180.

  The drive control unit 150 controls each unit by supplying various control signals to the row scanning unit 130 and the column scanning unit 140. The drive control unit 150 supplies various control signals to the row scanning unit 130 and the column scanning unit 140 based on a control signal from the signal processing unit 300, for example.

[2. Configuration of solid-state imaging device]
Next, the configuration of the solid-state imaging device 100 will be described in detail with reference to FIG.

[2-1. Pixel]
First, the pixel 210 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a circuit configuration of the pixel 210 according to the present embodiment.

  The pixel 210 includes a photoelectric conversion element 211, a reset transistor 212, an amplification transistor 213, a selection transistor 214, and a charge storage unit 215.

  The photoelectric conversion element 211 is a photoelectric conversion unit that performs photoelectric conversion of received light into signal charges (pixel charges). Specifically, the photoelectric conversion element 211 includes an upper electrode 211a, a lower electrode 211b, and a photoelectric conversion film 211c sandwiched between both electrodes. The photoelectric conversion film 211c is a film formed of a photoelectric conversion material that generates electric charge in accordance with received light, and in this embodiment, is formed of an organic photoelectric conversion film including an organic molecule having a high light absorption function. ing. In other words, the photoelectric conversion element 211 according to the present embodiment is an organic photoelectric conversion element having an organic photoelectric conversion film, and the solid-state imaging device 100 is an organic sensor using the organic photoelectric conversion element. Note that the organic photoelectric conversion film is formed over a plurality of pixels 210. Each of the plurality of pixels 210 has an organic photoelectric conversion film.

  The thickness of the photoelectric conversion film 211c is, for example, about 500 nm. The photoelectric conversion film 211c is formed by using, for example, a vacuum evaporation method. The organic molecule has a high light absorption function over the entire range of visible light having a wavelength of about 400 nm to about 700 nm.

  Note that the photoelectric conversion element 211 included in the pixel 210 according to the present embodiment is not limited to the above-described organic photoelectric conversion film, and may be, for example, a photodiode including an inorganic material. .

  The upper electrode 211a is an electrode facing the lower electrode 211b, and is formed on the photoelectric conversion film 211c so as to cover the photoelectric conversion film 211c. That is, the upper electrode 211a is formed over the plurality of pixels 210. The upper electrode 211a is made of a transparent conductive material (for example, ITO: indium / titanium / tin) to make light incident on the photoelectric conversion film 211c.

  The lower electrode 211b is an electrode for extracting electrons or holes generated in the photoelectric conversion film 211c between the lower electrode 211b and the upper electrode 211a. The lower electrode 211b is formed for each pixel 210. The lower electrode 211b is made of, for example, Ti, TiN, Ta, Mo, or the like.

  The charge storage unit 215 is connected to the photoelectric conversion element 211, and stores the signal charge extracted via the lower electrode 211b.

The reset transistor 212 has a drain supplied with the reset voltage V _RST , a source connected to the charge storage unit 215, and resets (initializes) the potential of the charge storage unit 215. Specifically, when a predetermined voltage is supplied (turned on) from the row scanning unit 130 to the gate of the reset transistor 212 via the reset scanning line 170A, the reset transistor 212 causes the potential of the charge storage unit 215 to change. Reset. By stopping the supply of the predetermined voltage, signal charges are stored in the charge storage unit 215 (exposure is started).

The amplification transistor 213 has a gate connected to the charge storage unit 215, a power supply voltage _VDD supplied to the drain, and outputs a pixel signal corresponding to the amount of signal charge stored in the charge storage unit 215.

  The selection transistor 214 has a drain connected to the source of the amplification transistor 213, a source connected to the column signal line 160, and determines a timing at which a pixel signal is output from the amplification transistor 213. Specifically, when a predetermined voltage is supplied to the gate of the selection transistor 214 from the row scanning unit 130 via the selection scanning line 170B, a pixel signal is output from the amplification transistor 213.

  The pixel 210 having the above structure can perform nondestructive readout. Here, the non-destructive readout means reading out a pixel signal corresponding to the amount of charge (signal charge) stored in the charge storage unit 215 during exposure without destroying the charge (signal charge). The term “during exposure” is used to mean an arbitrary timing within the exposure time.

[2-2. Other Configurations]
The column AD converter is configured by AD converters 121 provided for each column signal line 160. The AD converter 121 is, for example, a 14-bit AD converter. The AD converter 121 outputs a digital value corresponding to the amount of light received by the pixel 210 by, for example, converting the analog pixel signal output from the pixel 210 into a digital signal by a ramp method. The AD converter 121 has a comparator and an up / down counter (not shown).

  Here, the A / D conversion by the ramp method is an A / D conversion using a ramp wave. When an analog input signal is input, a ramp wave whose voltage rises at a constant slope is started, and from the time of the start. This method measures the time until the voltages of both signals (input signal and ramp wave) match, and outputs the measured time as a digital value. The comparator compares the voltage of the column signal with the voltage of the reference signal input as a ramp wave, and outputs a signal indicating the timing at which the voltage of the reference signal matches the voltage of the column signal.

  The up-down counter counts down (or up-counts) during the period from when the reference signal is input to the comparator until the reference signal reaches the voltage of the column signal indicating the reference component, and then the reference voltage is input to the comparator. Up-counting (or down-counting) in the period from when the reference signal reaches the voltage of the column signal indicating the signal component, to obtain a digital signal corresponding to the difference obtained by subtracting the reference component from the signal component of the column signal. Lastly retain the value.

  The digital value held in each up / down counter is sequentially output to the row signal line 180 and output to the signal processing unit 300 via an output circuit (not shown, such as an output buffer).

  The drive control unit 150 controls the row scanning unit 130 and the column scanning unit 140 to perform a reset operation, a charge accumulation operation, and a read operation in the pixel 210, or a digital signal from the AD converter 121 to the signal processing unit 300. Control the output operation.

  For example, upon receiving a read instruction from the signal processing unit 300, the drive control unit 150 controls the row scanning unit 130 to sequentially apply a predetermined voltage to the selection scanning line 170B and output a pixel signal (analog). . Further, the drive control unit 150 controls the column scanning unit 140 to sequentially output the pixel signals (digital values) held in the AD converter 121 to the signal processing unit 300.

[3. Configuration of Signal Processing Unit]
Next, the signal processing unit 300 will be described with reference to FIG. FIG. 3 is a functional block diagram of the camera 1 including the imaging device 10 according to the present embodiment. The camera 1 shown in FIG. 1 includes a solid-state imaging device 100, a signal processing unit 300, a lens 400, a display unit 500, and an operation unit 600. The signal processing unit 300 includes a control unit 310, an image processing unit 320, and a memory 330.

  The light that has passed through the lens 400 enters the solid-state imaging device 100. The signal processing unit 300 drives the solid-state imaging device 100 and captures a pixel signal (digital value) from the solid-state imaging device 100. For example, when the control unit 310 controls the drive control unit 150, the image processing unit 320 captures a pixel signal from the solid-state imaging device 100. The image processing unit 320 performs a predetermined signal processing on the pixel signal acquired from the solid-state imaging device 100 to generate a captured image. The generated captured image is stored in the memory 330. The generated captured image is output to the display unit 500. Note that the pixel signal (digital value) is an example of a first pixel signal. Here, the captured image is, for example, an image displayed or recorded by a user in the case of a camera or the like.

  The control unit 310 reads a program from the memory 330, for example, and executes the read program. In the above description, the control unit 310 controls the drive control unit 150. However, the control unit 310 may control other units. For example, when the operation unit 600 receives an input from a user, the control unit 310 may perform control according to the input. When the operation unit 600 receives an instruction to capture an image from a user, the control unit 310 may control the lens 400 (specifically, a motor that controls the position of the lens 400) to adjust the focus of the subject and the like.

  The image processing section 320 generates a captured image by performing a predetermined correction on the first pixel signal acquired from the solid-state imaging device 100 after the end of the exposure. Specifically, a captured image is generated by performing a predetermined correction on the first pixel signal based on the second pixel signal acquired from the solid-state imaging device 100 by non-destructive reading during exposure. The predetermined correction is to obtain information for correction (for example, a correction value to be applied to the first pixel signal) unless image data for one frame (second pixel signal for one frame) is used. For example, correction of gradation, correction of camera shake, or correction of white balance is performed. The present embodiment is characterized in that the above-described correction is performed on the first pixel signal based on the analysis result of the second pixel signal.

  In the gradation correction, for example, the first pixel signal is corrected so that overexposure and underexposure of black are unlikely to occur in imaging of a scene with a large difference in brightness such as backlight. Specifically, the overexposed pixels are corrected to be dark or the underexposed pixels are corrected to be bright. In order to perform such a correction, the image processing unit 320 uses a second pixel signal (image data generated by the second pixel signal) acquired by nondestructive reading. Specifically, the first pixel signal is corrected by generating a blurred image from the second pixel signal for one frame and calculating the generated blurred image and the first pixel signal. The correction of the first pixel signal is performed for each pixel 210. The correction value of the first pixel signal is obtained by calculating the brightness of the first pixel signal of one pixel 210 and the brightness of the pixel of the blurred image corresponding to the first pixel signal. By correcting the first pixel signal with the correction value for each pixel 210, a captured image is generated. Note that the blurred image is an example of a correction image.

  Similarly, in a scene having a small difference in brightness, a captured image is generated by calculating a blurred image and a first pixel signal and correcting the first pixel signal from the calculation result. In the camera shake correction, an image generated by the second pixel signal acquired by the non-destructive readout (an example of an image for correction) and, for example, an image generated by the first or second pixel signal in the immediately preceding frame are used. , And corrects the first pixel signal according to the obtained motion vector. The signal processing unit 300 calculates one motion vector for one frame image. For example, an image of one frame is divided into a plurality of windows (small areas smaller than the image of one frame), a motion vector is obtained in units of windows, and a motion vector in an image of one frame is obtained from the motion vector in each window. . Obtaining a motion vector using the correction image is an example of image analysis.

  In the white balance correction as well, the white balance of the first pixel signal is corrected from the analysis result of the second pixel signal in the same manner as described above. For example, when white balance is corrected after performing offset correction (correction of an OB step when converting an analog signal into a digital signal), the second pixel acquired by non-destructive reading is also used in offset correction. A signal may be used. It should be noted that camera shake correction, white balance correction, and the like do not need to use a blurred image, and can be performed using an image for one frame acquired by nondestructive reading.

  Note that generation of a correction image or image analysis using the generated correction image is an example of image processing.

  The image data (image) generated by the second pixel signal acquired by the nondestructive readout during the exposure is an image that is overall darker than the captured image generated by the first pixel signal acquired after the exposure is completed. Becomes Therefore, the brightness of the image data is adjusted by amplifying the second pixel signal (correcting the gain with respect to the brightness) before generating the correction image using the second pixel signal. Good. Specifically, adjustment for increasing the brightness may be performed.

  The memory 330 functions as a work memory of the image processing unit 320. The memory 330 temporarily stores image data processed by the image processing unit 320, image data (first pixel signal) input from the solid-state imaging device 100 before being processed by the image processing unit 320, and the like. .

  The memory 330 can be realized by, for example, a dynamic random access memory (DRAM) or a ferroelectric memory. Note that the memory 330 only needs to be included in the imaging device 10 and does not need to be included in the signal processing unit 300.

  The display unit 500 is a display device that displays a captured image generated by the signal processing unit 300, and is, for example, a liquid crystal monitor. The display unit 500 can display various setting information. For example, the display unit 500 can display shooting conditions (aperture, ISO sensitivity, and the like) at the time of shooting.

  The operation unit 600 is an input unit that receives an input from a user, and is, for example, a release button or a touch panel. For example, a touch panel is adhered to a liquid crystal monitor. It accepts imaging instructions from the user, changes in imaging conditions, and the like.

  Note that the imaging device 10 may include an interface (not shown) for performing communication between an external circuit and the solid-state imaging device 100 or the signal processing unit 300. The interface is, for example, a communication port including a semiconductor integrated circuit.

[4. Processing of imaging device]
Next, the processing of the imaging device 10 will be described.

[4-1. Processing flow of imaging apparatus]
The processing flow of the imaging device 10 will be described with reference to FIGS. 4A and 4B in comparison with a conventional example. FIG. 4A is a flowchart illustrating the operation of the imaging device 10 according to the present embodiment. FIG. 4B is a flowchart showing the operation of the imaging apparatus according to the conventional example. The conventional example shows an example in which correction is performed using a pixel signal (first pixel signal) acquired after the end of exposure. That is, in the conventional example, the correction is performed without performing the nondestructive reading. FIGS. 4A and 4B are examples of a flowchart showing the operation when performing gradation correction.

  First, the solid-state imaging device 100 starts exposure under the control of the control unit 310, for example (S1). As a result, charges corresponding to the amount of received light are accumulated in each of the plurality of pixels 210. Specifically, a charge corresponding to the amount of light received by the photoelectric conversion element 211 is stored in the charge storage unit 215.

  Then, the control unit 310 performs a nondestructive readout control for reading out the charges accumulated in the charge accumulation units 215 of the respective pixels 210 during the exposure without destroying the charges. More specifically, the control unit 310 controls the drive control unit 150 and performs AD conversion in the column AD conversion unit 120 for each pixel row, thereby converting the accumulated charge (pixel signal) into a digital value (pixel signal) corresponding to the accumulated charge. The second digital signal is converted into a second pixel signal, and the converted digital value is sequentially output to the signal processing unit 300 by the column scanning unit 140. That is, the signal processing unit 300 acquires the second pixel signal by non-destructive reading (S2). Note that after performing the nondestructive read, the reset of the potential of the charge accumulation unit 215 by the reset transistor 212 is not performed.

  The image processing unit 320 generates a blurred image (an example of a correction image) from the acquired second pixel signal (S3). For example, the image processing unit 320 may generate an image generated from the second pixel signal or a blurred image having a lower resolution than the captured image. For example, when the resolution of the image or the captured image generated from the second pixel signal is 5000 × 4000, the resolution of the blurred image is 50 × 40 or the like. The blurred image is generated by accumulating and reducing the second pixel signal for one frame and further enlarging the image. Thus, the capacity of the memory 330 for temporarily holding the blurred image can be reduced. In performing the gradation correction, the correction can be performed even if the resolution of the blurred image is about 50 × 40. The generated blurred image may be held by the image processing unit 320 or may be stored in the memory 330, for example. Note that the blurred image may be generated by performing a low-pass filter process on an image generated by the acquired second pixel signal.

  Here, the exposure in the solid-state imaging device 100 ends (S4). That is, in the imaging device 10 according to the present embodiment, the generation of the blurred image by the image processing unit 320 is completed before the exposure of the solid-state imaging device 100 ends. In other words, the control unit 310 executes non-destructive reading from the exposure time and the time (first time) required from the acquisition of the second pixel signal to the completion of generation of the blurred image (S2 and S3). May be controlled. For example, the control unit 310 may cause the solid-state imaging device 100 to execute non-destructive reading at a time that is a first time earlier than the time at which the exposure ends. This makes it possible to further increase the brightness of the second pixel signal acquired by the nondestructive readout and generate a blurred image by the end of the exposure. Accordingly, it is possible to perform correction on the first pixel signals sequentially acquired for each pixel 210 after the end of the exposure at the time when the first pixel signals are acquired. Further, since the brightness of the second pixel signal (information on the brightness indicated by the second pixel signal) is bright, the influence of noise when applying a gain to the second pixel signal can be reduced. Note that the first time can be specified before the exposure is started, based on the processing capability of the signal processing unit 300, the number of pixels of the pixel array unit 110, and the like.

  If the correction is a camera shake correction, in step S3, generation of a correction image and image analysis of the correction image are performed. In this case, the control unit 310 determines the non-destructive time from the exposure time and the time (second time) required from the acquisition of the second pixel signal to the completion of generation and analysis of the correction image (S2 and S3). The timing at which reading is performed may be controlled. For example, the control unit 310 may cause the solid-state imaging device 100 to execute the nondestructive readout at a time that is a second time earlier than the time at which the exposure ends.

  Next, the control unit 310 controls the drive control unit 150 after the end of the exposure, so that the column AD conversion unit 120 performs AD conversion for each pixel column, and converts the accumulated charge into a digital value corresponding to the accumulated charge. . The converted digital values are sequentially output to the signal processing unit 300 by the column scanning unit 140. That is, the signal processing unit 300 acquires the first pixel signal (S5). More specifically, the signal processing unit 300 sequentially acquires the first pixel signals for each of the plurality of pixels 210. Note that reading of the first pixel signal is performed, for example, by normal reading. The normal readout is a destructive readout in which the stored charge is destroyed after the pixel signal is read (the potential of the charge storage unit 215 is reset by turning on the reset transistor 212).

  Using the blurred image stored in the memory 330, the image processing unit 320 corrects the first pixel signal from the blurred image and the first pixel signal to generate a captured image (S6). Note that when correcting the first pixel signal, it is possible to perform correction for noise (such as a streak as an image) generated in the solid-state imaging device 100, light amount adjustment for the pixels 210 around the lens 400, and the like. Good. As a result, the time required for the correction can be reduced as compared with the case where each correction is performed separately.

  The image processing unit 320 displays the captured image on the display unit 500 included in the camera 1 or captures the image on an external storage medium via a storage unit (not illustrated) or an interface (not illustrated) included in the camera 1, for example. Store the image.

  Subsequently, an operation of the imaging apparatus according to the conventional example will be described with reference to FIG. 4B. Note that the same operations as those of the imaging device 10 according to the present embodiment are denoted by the same reference numerals as in FIG. 4A, and description thereof may be omitted.

  First, the solid-state imaging device starts exposure (S1). The signal processing unit does not perform image processing (does not perform reading or the like) during the exposure, and the exposure ends (S4). At this point, no blurred image has been generated.

  The signal processing unit acquires the first pixel signal after the end of the exposure (S5). The signal processing unit generates an intermediate image by performing, on the first pixel signal, correction for noise generated in the solid-state imaging device, light amount adjustment for pixels around the lens, and the like (S11). These processes are performed before storing in the memory. Then, the intermediate image is stored in the memory (S12). That is, the conventional method requires time for storing the intermediate image in the memory. Further, a memory for storing the intermediate image is required, and therefore, the capacity of the memory needs to be increased. For example, in order to store an intermediate image, an extra memory capable of storing one frame of image data is required.

  Next, a blurred image is generated using the intermediate image (S13). An intermediate image is used as an image for generating a blurred image. The method of generating a blurred image is the same as the method of generating a blurred image according to the present embodiment (see S3). Note that, for example, gain correction is not performed on the first pixel signal. Since step S13 is performed after the end of the exposure, the conventional method requires time for the process after the end of the exposure.

  Further, an intermediate image is read from the memory (S14), and a corrected image is generated using the blurred image generated in step S13 to generate a captured image (S15). That is, the conventional method requires time for reading the intermediate image from the memory.

  As described above, according to the conventional method, since the image processing is started after the exposure is completed, the time required for the image processing is long. Specifically, compared to the imaging device 10 according to the present embodiment, image processing after the end of exposure requires more time than the imaging device 10 according to the present embodiment for the time required for the processing in steps S11 to S14. On the other hand, the imaging device 10 according to the present embodiment can perform image processing using an image acquired by non-destructive reading during exposure, so that the time required for image processing can be reduced as compared with a conventional imaging device. Can be.

[4-2. Comparison of signal processing unit processing]
Next, the image processing performed by the signal processing unit of the imaging device will be described in more detail with reference to FIGS.

  First, the image processing performed by the signal processing unit 300 will be described with reference to FIG. FIG. 5 is a diagram for comparing the flow of image processing. FIG. 5A is a diagram for explaining the flow of image processing of the imaging apparatus according to the conventional example. FIG. 5B is a diagram for explaining the flow of image processing of the imaging device 10 according to the present embodiment.

  First, the processing of the signal processing unit from the start of exposure (S1) to the end of exposure (S4) will be described.

  As shown in FIG. 5A, in the conventional method, the signal processing unit does not perform image processing from the start of exposure to the end of exposure.

  As shown in FIG. 5B, in the present embodiment, the signal processing unit 300 acquires a second pixel signal by performing non-destructive reading during exposure (S2). Then, a blurred image is generated using the second pixel signal (S3). FIG. 5B shows an example in which the signal processing unit 300 performs non-destructive reading and image processing so that the generation of the blurred image is completed by the end of the exposure (S4). By performing the nondestructive reading, the signal processing unit 300 can complete the image processing in step S13 in the conventional method during the exposure.

  Even if the exposure time is short and the processing in step S3 is not completed before the end of the exposure, part of the processing in step S3 can be performed during the exposure. It is possible to shorten the time from the end to the end of the image processing. In other words, the non-destructive readout may be performed before the end of the exposure.

  Next, processing after the end of exposure will be described.

  In the conventional method, image processing is started after the end of exposure. First, an intermediate image is generated from the first pixel signal acquired after the end of the exposure (S11). This is a process performed before storing the data in the memory, and includes, as described above, correction for noise and light amount adjustment for pixels around the lens. Note that the processing performed before storing in the memory may be other than the above.

  The generated intermediate image is stored in the memory (S12). The intermediate image is an image for one frame. The image processing unit generates a blurred image using the intermediate image (S13). That is, the image processing unit generates a blur image for one frame.

  Then, the image processing unit generates a captured image from the intermediate image stored in the memory and the blurred image generated in step S13 (S15).

  In the conventional method, the signal processing unit starts the image processing after the end of the exposure, so that it takes time from the end of the exposure to the end of the image processing. Specifically, it takes time to perform steps S11 to S15. Therefore, the time required from the start of exposure to the end of image processing (imaging time Ta) also requires time. Further, a memory for storing the intermediate image is required.

  Subsequently, image processing of the signal processing unit 300 according to the present embodiment will be described.

  As described above, in the present embodiment, the generation of the blurred image is completed by the end of the exposure. Therefore, the image processing performed after the end of the exposure is generation of a captured image. Specifically, the image processing unit 320 generates a captured image from the first pixel signal obtained after the end of the exposure and the blurred image generated in step S3 (S6).

  Thereby, as shown in FIG. 5, the time from the end of exposure to the end of image processing can be reduced. That is, the imaging time Tb of the imaging device 10 according to the present embodiment can also be reduced (Tb <Ta). Specifically, the imaging time can be reduced by the shortened time T (Tb-Ta) shown in the figure.

  The image processing unit 320 sequentially obtains the first pixel signal for each pixel 210 after the end of the exposure. For example, a first pixel signal for each pixel 210 sequentially acquired and a pixel of a blurred image corresponding to the pixel 210 are calculated, and the first pixel signal is corrected according to the calculation result. That is, it is possible to sequentially acquire the first pixel signal for each pixel 210 and perform the sequential correction. The sequentially corrected first pixel signals are stored in, for example, the memory 330 sequentially. As a result, the time for storing the first pixel signal before correction in the memory 330 or the time for reading out the first pixel signal stored for correction can be omitted, so that the imaging time Tb is further reduced. can do. Further, since the capacity for storing the first pixel signal is unnecessary, the capacity of the memory 330 can be reduced.

  Further, for example, when the reading method is a rolling shutter, the first pixel signal obtained by the conventional method and the first pixel signal obtained by the method of the present embodiment are the same signal. Note that the reading method is not limited to the rolling shutter. For example, a global shutter may be used. When the solid-state imaging device 100 has an organic photoelectric conversion film, a shutter function can be realized by adjusting a voltage applied to the organic photoelectric conversion film, so that a global shutter can be realized without adding an element such as a memory. can do. In addition, by allowing all the organic photoelectric conversion films to function as shutters during a period in which reading is performed by the global shutter, accumulation of charges in the charge accumulation unit 215 during that period can be suppressed. That is, by using the organic photoelectric conversion film, rolling distortion generated in the global shutter can be reduced without adding an element. Note that the non-destructive read and the read after the end of the exposure are performed by the same method.

  Next, the generated image will be described with reference to FIG. FIG. 6 is a diagram for comparing images generated by the image processing. FIG. 6A is a diagram for explaining an image generated by image processing of an imaging device according to a conventional example. FIG. 6B is a diagram for explaining an image generated by the image processing of the imaging device 10 according to the present embodiment. FIG. 6B shows an example in which the generation of the blurred image is completed during the exposure.

  As shown in FIG. 6A, in the conventional method, since no image processing is performed during exposure, no image is generated. After the end of the exposure (S4), the signal processing unit generates the corrected intermediate image P1 from the first pixel signal. The intermediate image P1 is an image for one frame. The intermediate image P1 shows a case where an image that is entirely dark (dark due to an actual subject) is captured, and is indicated by dot-shaped hatching.

  Next, a blurred image P2 is generated from the generated intermediate image P1. The blurred image is also an image for one frame.

  Subsequently, a captured image P3 is generated from the intermediate image P1 and the blurred image P2. That is, in the conventional method, it was necessary to generate three images. FIG. 6A shows an example in which the captured image P3 is corrected to slightly brighten the intermediate image P1. Note that the correction is not limited to this.

  As shown in FIG. 6B, in the present embodiment, image processing is performed during exposure. Specifically, the signal processing unit 300 generates the blurred image P12 by non-destructive reading during exposure (S2). Since non-destructive readout is performed during exposure, the blurred image P12 generated using the second pixel signal obtained therefrom becomes a dark image as a whole. Thus, the image processing unit 320 may adjust the gain of the second pixel signal, and generate the blurred image P12 using the adjusted second pixel signal. As a result, it is possible to obtain a blurred image P12 substantially the same as the blurred image P2 in the conventional example.

  Next, a captured image P13 is generated from the blurred image P12 generated by adjusting the gain and the first pixel signal acquired after the end of the exposure. Thereby, the captured image P13 generated and the captured image P3 generated by the conventional method are substantially the same image. By adjusting the gain of the second pixel signal, the accuracy of the captured image P13 can be improved.

[5. camera]
Examples of the camera 1 in which the above-described imaging device 10 is built include, for example, a digital still camera 1A shown in FIG. 7A and a digital video camera 1B shown in FIG. 7B. For example, by incorporating the imaging device 10 according to the present embodiment in a camera such as that shown in FIG. 7A or FIG. 7B, the user can take an image via the operation unit 600 as described above. Is issued (after the shutter is released) and the time until the captured image is displayed on the display unit 500 can be reduced. In addition, when imaging continuously, it is possible to reduce the time from when the shutter is released to when the next shutter is released.

[6. Effects]
As described above, the imaging device 10 according to the present embodiment is obtained from the solid-state imaging device 100 having a plurality of pixels arranged in a matrix and capable of nondestructive readout and from the solid-state imaging device 100 after the end of the exposure. An image processing unit 320 that generates a captured image P13 by performing correction on the first pixel signal. The image processing unit 320 performs correction based on the second pixel signal acquired from the solid-state imaging device 100 by non-destructive reading during exposure.

  In the conventional method, the correction is performed based on the first pixel signal acquired after the end of the exposure. Therefore, the image processing cannot be started until the exposure is completed.

  The imaging device 10 according to the present embodiment acquires a second pixel signal during exposure by nondestructive readout. Therefore, processing for correcting the first pixel signal using the second pixel signal during exposure (for example, generation of a correction image for correcting the first pixel signal or analysis of the correction image) ) Can be done. As a result, the time from the end of exposure to the end of image processing can be reduced as compared with the conventional method. In other words, the time from the start of exposure to the end of image processing (imaging time Tb) can be reduced. That is, the speed of image processing is increased.

  Further, a control unit 310 is provided. The image processing unit 320 generates an image for correction to be used for correction from the second pixel signal, or generates an image for correction, and performs image processing of performing image analysis using the generated image for correction. Then, the control unit 310 controls the timing at which the solid-state imaging device 100 performs non-destructive reading so that the image processing in the image processing unit 320 ends before the exposure ends.

  Accordingly, for example, when the correction is a gradation correction, a blurred image (an example of a correction image) P12 is generated by using the second pixel signal acquired by the nondestructive readout before the exposure is completed. Can be. That is, the signal of the pixel 210 can be sequentially corrected when the signal for each pixel 210 is obtained for the first pixel signal obtained after the end of the exposure. If the correction is a correction of camera shake or a correction of white balance, a correction image is generated by using the second pixel signal acquired by the nondestructive readout before the exposure is completed, and the correction image is generated. Image analysis can be performed. That is, since the result of the image analysis can be obtained before the end of the exposure, when the signal for each pixel 210 is obtained with respect to the first pixel signal obtained after the end of the exposure, Corrections can be made to the signal. Therefore, the imaging time Tb can be further reduced.

  Further, a memory 330 is provided. Then, the image processing unit 320 performs a correction corresponding to the pixel 210 on the first pixel signal sequentially obtained for each pixel 210, and stores the corrected first pixel signal in the memory 330.

  Thus, since the correction can be performed without temporarily storing the first pixel signal in the memory 330, the time for storing and reading out the first pixel signal before correction in the memory 330 can be omitted. Further, since a memory for storing the first pixel signal before correction is not required, the capacity of the memory 330 can be reduced.

  The correction image is a blurred image P12 having a lower resolution than the captured image P13. Then, the image processing unit 320 generates the blurred image P12 from the second pixel signal to which the gain correction for the brightness has been added.

  By using the blurred image P12, the processing amount in the signal processing unit 300 can be reduced, so that the processing time in the signal processing unit 300 can be reduced. Further, the blurred image P12 generated by adding the gain correction to the second pixel signal is an image having substantially the same brightness as the blurred image P2 generated in the conventional example. As a result, the first pixel signal can be corrected more accurately.

  The correction is correction of gradation, correction of camera shake, or correction of white balance.

  This makes it possible to reduce the imaging time Tb when performing gradation correction, camera shake correction, or white balance correction.

  Further, each of the plurality of pixels 210 has an organic photoelectric conversion film.

  Thus, the shutter function can be realized by adjusting the voltage applied to the organic photoelectric conversion film, so that a global shutter can be realized without adding an element such as a memory. Therefore, even when the subject is moving, image data with less distortion can be obtained.

  Further, the camera 1 according to the present embodiment includes the imaging device 10 described above.

  Accordingly, it is possible to reduce the time from when the user gives an imaging instruction via the operation unit 600 (after the shutter is released) to when the captured image is displayed on the display unit 500. In addition, when imaging continuously, it is possible to reduce the time from when the shutter is released to when the next shutter is released.

  Further, the imaging method according to the present embodiment obtains the first pixel signal from the solid-state imaging device 100 having the plurality of pixels 210 capable of nondestructive readout after the end of the exposure, and performs the nondestructive readout during the exposure. The captured image P13 is generated by correcting the first pixel signal based on the second pixel signal acquired from the solid-state imaging device 100.

  Thus, the second pixel signal can be obtained during the exposure by the non-destructive readout, so that the blurred image can be generated using the second pixel signal during the exposure. As a result, the time from the end of exposure to the end of image processing can be reduced as compared with the conventional method. In other words, the time (imaging time) from the start of exposure to the end of image processing can be reduced.

(Other embodiments)
As described above, the embodiments have been described as examples of the technology in the present disclosure. For that purpose, the accompanying drawings and the detailed description have been provided.

  Therefore, among the components described in the accompanying drawings and the detailed description, not only components that are essential for solving the problem, but also components that are not essential for solving the problem, in order to illustrate the above technology. May also be included. Therefore, it should not be immediately recognized that these non-essential components are essential based on the fact that the non-essential components are described in the accompanying drawings and the detailed description.

  Further, since the above-described embodiments are for exemplifying the technology according to the present disclosure, various changes, replacements, additions, omissions, and the like can be made within the scope of the claims or the equivalents thereof.

  For example, in the above embodiment, an example in which the image processing unit 320 generates the blurred image P12 has been described, but the present invention is not limited to this. For example, when the solid-state imaging device 100 that can perform pixel mixing is used, the image processing unit 320 may use a pixel-mixed signal acquired from the solid-state imaging device 100 as a blurred image.

  Thereby, the same effect as that of the embodiment can be obtained.

  In the above description, the example in which the image processing unit 320 corrects the gradation using the blurred image P12 has been described, but the present invention is not limited to this. For example, the image processing unit 320 may obtain a histogram for the brightness of the image from the second pixel signal, and perform gradation correction on the first pixel signal using the result as a correction signal. Note that performing gradation correction on the first pixel signal in accordance with the histogram for brightness generated from the second pixel signal acquired from the solid-state imaging device 100 is an example of image processing performed by the signal processing unit 320. It is.

  Thereby, the same effect as in the case where the gradation is corrected using the blurred image P12 is obtained.

  Further, in the above description, an example has been described in which the first pixel signal is not stored in the memory 330 but is corrected by the image processing unit 320 to generate a captured image. However, the present invention is not limited to this. For example, the first pixel signal for one frame is temporarily stored in the memory 330, the image processing unit 320 reads the first pixel signal from the memory 330 for each pixel 210, and outputs the read first pixel signal and the blurred image. Thus, the captured image may be generated by performing correction on the first pixel signal.

  As a result, compared to the conventional method, the speed of image processing can be increased because the blurred image is generated by non-destructive reading.

  In the above description, an example in which nondestructive reading is performed once during exposure has been described, but the number of times of performing nondestructive reading is not limited to this. Non-destructive readout may be performed a plurality of times during exposure. For example, when performing camera shake correction, non-destructive readout may be performed twice during exposure, a motion vector may be obtained from image data based on each pixel signal, and correction may be performed on the first pixel signal. Further, in order to reduce the influence of the subject blur and the flash light during the exposure period, for example, the image processing unit 320 reduces the influence of the subject blur and the flash light from a plurality of images acquired by the non-destructive readout. A small number of images may be selected as correction images.

  Further, in the above, the camera 1 in which the imaging device 10 according to the present disclosure is built has been described, but the application is not limited thereto. Various electronic devices incorporating the imaging device 10 according to the present disclosure are also included in the present disclosure.

  In addition, each component (functional block) in the imaging device 10 may be individually formed into one chip by a semiconductor device such as an IC (Integrated Circuit) and an LSI (Large Scale Integration), or may include a part or the whole. May be integrated into one chip. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. After manufacturing the LSI, an FPGA (Field Programmable Gate Array) that can be programmed or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used. Furthermore, if an integrated circuit technology that replaces the LSI appears due to the progress of the semiconductor technology or another derived technology, the functional blocks may be integrated using the technology. The application of biotechnology and the like is possible.

  In addition, all or a part of the above various processes may be realized by hardware such as an electronic circuit, or may be realized by using software. Note that the processing by software is realized by a processor included in the imaging device 10 executing a program stored in a memory. Further, the program may be recorded on a recording medium and distributed or distributed. For example, by installing a distributed program on a device having another processor and causing the processor to execute the program, the device can perform the above-described processing.

  Further, a mode realized by arbitrarily combining the components and functions described in the above-described embodiments is also included in the scope of the present disclosure.

  INDUSTRIAL APPLICATION This indication is widely applicable to the imaging device which images an image.

Reference Signs List 1 camera 1A digital still camera 1B digital video camera 10 imaging device 100 solid-state imaging device 110 pixel array unit 120 column AD conversion unit 121 AD converter 130 row scanning unit 140 column scanning unit 150 drive control unit 160 column signal line 170 scanning line 170A Reset scanning line 170B Selection scanning line 180 Row signal line 210 Pixel 211 Photoelectric conversion element 211a Upper electrode 211b Lower electrode 211c Photoelectric conversion film 212 Reset transistor 213 Amplification transistor 214 Selection transistor 215 Charge storage unit 300 Signal processing unit 310 Control unit 320 Image processing unit 330 Memory 400 Lens 500 Display unit 600 Operation unit P1 Intermediate image P2, P12 Blurred image P3, P13 Captured image Ta, Tb Captured time

Claims (8)

A solid-state imaging device having a plurality of pixels arranged in a matrix and capable of nondestructive readout,
An image processing unit that generates a captured image by performing correction on the first pixel signal acquired from the solid-state imaging device after the end of the exposure,
The image processing unit,
An imaging device that performs the correction based on a second pixel signal acquired from the solid-state imaging device by the nondestructive readout during the exposure. Furthermore, a control unit is provided,
The image processing unit generates a correction image to be used for the correction from the second pixel signal, or generates the correction image, performs image processing to perform image analysis using the generated correction image,
The imaging device according to claim 1, wherein the control unit controls the timing at which the solid-state imaging device performs the nondestructive readout such that the image processing in the image processing unit ends before the exposure ends. apparatus. In addition, it has memory,
The image processing unit performs the correction corresponding to the pixel on the first pixel signal sequentially acquired for each pixel, and stores the corrected first pixel signal in the memory. 3. The imaging device according to 2. The correction image is a blurred image having a lower resolution than the captured image,
The imaging device according to claim 2, wherein the image processing unit generates the blurred image from the second pixel signal to which a gain correction for brightness has been added. The imaging device according to any one of claims 1 to 4, wherein the correction is correction of gradation, correction of camera shake, or correction of white balance. The imaging device according to claim 1, wherein each of the plurality of pixels has an organic photoelectric conversion film. A camera comprising the imaging device according to claim 1. From a solid-state imaging device having a plurality of pixels arranged in a matrix and capable of non-destructive readout, obtain a first pixel signal after the end of exposure,
An imaging method for generating a captured image by correcting the first pixel signal based on a second pixel signal obtained from the solid-state imaging device by non-destructive readout during the exposure.