JPWO2018168458A1 - Imaging device - Google Patents

Abstract

The present invention provides an imaging apparatus capable of improving the correction accuracy based on the black level when different imaging conditions are used. The imaging device 10 includes an effective pixel unit 111 for imaging that outputs first data according to imaging conditions, and an OB pixel unit 112 for black level detection that outputs second data according to imaging conditions. The DSP / sequencer unit 223 periodically switches the imaging conditions. The black level detection system 230 includes a DSP BL detection unit 1 (221) and a DSP BL detection unit 2 (222). The DSP BL detection unit 1 (221) and the DSP BL detection unit 2 (222) are provided for each imaging condition, and detect the black level for each imaging condition based on the second data for each imaging condition. The BL correction unit 114 corrects the first data for each imaging condition based on the black level for each imaging condition.

Description

  The present invention relates to an imaging apparatus.

  There has been known an imaging apparatus capable of eliminating a black level offset between fields and obtaining a beautiful image (see, for example, Patent Document 1).

  This publication states that “the imaging sensor has an imaging area, a front OB portion having a photoelectric conversion element structure, a rear OB portion not having a photoelectric conversion element structure, and a dummy pixel portion formed in a horizontal transfer portion. The black level clamping is performed based on the output signal of the rear OB unit, and black level signals are taken out from the front OB unit, the rear OB unit, and the dummy pixel unit, respectively. When the temperature becomes high, the difference between the output signal of the rear OB part and the output signal of the front OB part is added to the effective pixel output signal to perform black level correction. " Note that OB is an abbreviation for optical black.

JP 2007-27845 A

  In order to adjust the black level, it is necessary to detect the black level. In the black level detection method, it is conceivable to detect black levels that differ depending on position information such as screen areas and fields, or conditions such as gain level and exposure time, and individually correct them.

  However, Patent Document 1 discloses a Wide image capturing method in which an individual exposure time is set for each field (even row / odd row) or frame in order to capture a subject with a wide dynamic range without saturation or blackout. There is no description about Dynamic Range (WDR) imaging.

  In order to improve black level detection accuracy, pixel values of optical black pixels and dummy pixels may be integrated over a plurality of fields or frames. In that case, when combined with WDR imaging, black levels of fields or frames with different imaging conditions are detected together, and the accuracy may rather deteriorate. In this case, the black level is not normally corrected, and the relationship between the pixel signal output value with respect to the luminance of the subject is deviated, and there is a problem that black floating and black crushing occur.

  An object of the present invention is to provide an imaging apparatus capable of improving the correction accuracy based on the black level when different imaging conditions are used.

  In order to achieve the above object, the present invention provides a first pixel unit for imaging that outputs first data according to imaging conditions, and a black level detection that outputs second data according to the imaging conditions. An imaging device having a second pixel unit, an imaging condition control unit that periodically switches the imaging conditions, and provided for each imaging condition, and for each imaging condition based on the second data for each imaging condition A black level detection system including a black level detection unit that detects the black level; and a correction unit that corrects the first data for each imaging condition based on the black level for each imaging condition.

  According to the present invention, it is possible to improve the correction accuracy based on the black level when different imaging conditions are used. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is a figure which shows the structure of the imaging device as a comparative example. 1 is a diagram illustrating a configuration of an imaging apparatus according to Embodiment 1. FIG. It is a figure which shows the structure of the imaging device of Embodiment 2. FIG. It is a figure which shows the structure of the imaging device of a modification.

  Hereinafter, the configuration and operation of the imaging apparatus according to the first and second embodiments of the present invention will be described with reference to the drawings.

(Comparative example)
FIG. 1 shows a configuration diagram of an imaging apparatus 100 as a comparative example.

  The imaging device 100 includes a sensor unit 110 and a DSP unit 120 (DSP: Digital Signal Processor).

  The sensor unit 110 includes an effective pixel unit 111, an OB pixel unit 112 (OB: Optical Black), an AFE unit 113 (AFE: Analog Front End), a BL correction unit 114 (BL: Black Level), a noise correction unit 115, and a pedestal unit. 116 and a sensor / sequencer unit 117.

  The imaging apparatus 100 is, for example, an in-vehicle camera or the like, and is characterized in that it captures and processes a WDR video by multiple exposure in order to acquire a video with a wide dynamic range. In addition, the in-vehicle camera is characterized in that a wide temperature guaranteed range is required.

  The sensor unit 110 is an image sensor that captures an image with a configuration such as a complementary MOS (CMOS) or a charge coupled device (CCD).

  In the effective pixel unit 111, pixels that apply photoelectric conversion and voltage conversion to luminance coming from the direction of a subject (for example, a road condition for an in-vehicle sensor) are arranged two-dimensionally in a matrix shape or a honeycomb shape. It has been done.

  The OB pixel unit 112 is an array of pixels that are shielded so that dark current can be detected, or dummy pixels that do not include dark current and are used to detect the characteristics of signals in the readout system.

  The AFE unit 113 converts the voltage value into digital image data by A / D conversion.

  The sensor BL detection unit 118 outputs the average value, median value, median value, etc. in the spatial direction in the digital image data output from the OB pixel unit 112 and converted by the AFE unit, or in the time direction in a plurality of frames or field periods. Apply a process to output a numerical value called "black level". This black level is obtained by detecting the level of the pixel value when light does not enter the digital image data obtained by digitally converting the video output from the effective pixel unit 111 by the AFE unit 113 due to light shielding or the like.

  The BL correction unit 114 receives the black level output from the sensor BL detection unit 118 via the sensor / sequencer unit 117, and outputs image data obtained by subtracting and correcting the black level from the effective image data. By subtracting the black level, the pixel value level at the time of shading is corrected to a reference value such as zero.

  The noise correction unit 115 corrects noise by performing image filter processing on the image data output from the BL correction unit 114 in the spatial direction or the time direction.

  The pedestal unit 116 adds a predetermined pixel value to the image data output from the noise correction unit 115 and outputs the result.

  The sensor / sequencer unit 117 receives imaging conditions such as analog gain and exposure time from the DSP / sequencer unit 124, which will be described later, and determines the operation content and operation timing of each unit in the sensor unit so that the specified imaging conditions can be realized. Control. Further, in order to realize WDR imaging by multiple exposure, imaging can be performed by switching imaging conditions for each field or each frame.

  The DSP unit 120 includes a pre-processing unit 121, a WDR synthesis unit 122, a post-processing unit 123, and a DSP / sequencer unit 124. The DSP unit inputs image data captured by the sensor unit 110.

  The preprocessing unit 121 applies image processing such as noise removal, pedestal addition, and digital gain processing to the image data output from the sensor unit 110. For example, the digital image processing for the same purpose may be duplicated in the sensor unit 110 depending on circumstances at the time of designing, such as designing a DSP unit in-house using an off-the-shelf product as the sensor unit 110.

  The WDR synthesis unit 122 receives the image data output from the preprocessing unit 121 as an input, synthesizes a plurality of images captured under individual imaging conditions, and creates and outputs one image having a wide dynamic range.

  The post-processing unit 123 receives image data output from the WDR synthesis unit 122 as input. For example, in the in-vehicle camera, the post-processing unit 123 detects, for example, a vehicle, a pedestrian, an obstacle, a sign, a sign, a white line, and the like, and outputs the detected location and contents.

  As another configuration of the DSP unit 120, there is a case where the WDR synthesis unit 122 does not perform synthesis and the post-processing unit 123 performs recognition processing individually for each imaging condition. Generally, when combining, the processing amount of the post-processing unit 123 can be reduced, and when not combining, the detection accuracy of the post-processing unit 123 can be improved.

  The DSP / sequencer unit 124 controls the operation of the entire imaging apparatus including designation of imaging conditions for multiple exposure.

  Next, the problem of the imaging device 100 of the comparative example shown in FIG. 1 will be described.

As for the region connected to the pixel in the effective pixel portion 111 and the OB pixel portion 112, the black level increases as the temperature rises as a general characteristic of the CMOS image sensor.
Further, generally, when one or more of three elements of high temperature, high magnification analog gain, and long exposure time are provided, the amount of increase in the black level becomes large.

  In general, when there is a lot of noise in the sensor unit 110 (in particular, noise detected in a light-shielded state called dark noise), the output signal value of the OB pixel unit is averaged over a plurality of frames. Therefore, the black level may be accurately measured with less influence of noise. For example, in general, the black level of each frame is detected, and further averaging processing is performed by a time (that is, frame or field direction) FIR filter (FIR: Finite Impulse Response) or IIR filter (IIR: Infinite Impulse Response). In some cases, the black level is detected with high accuracy by reducing noise as compared with the case of detecting in one frame.

  However, in the case of realizing WDR imaging by multiple exposure in the imaging apparatus 100, when the black level detection method using a plurality of frames or fields as described above is used, the multiple exposure condition (for example, In the exposure condition 1 based on the combination of the high magnification gain and the long exposure and the exposure condition 2 based on the combination of the low magnification gain and the short exposure, the original black level differs depending on the exposure condition. The black level of the frame is detected in a mixed manner, and the accuracy is rather deteriorated. In this case, the black level is not normally corrected, and the relationship between the pixel signal output value with respect to the luminance of the subject is deviated, and there is a problem that black floating and black crushing occur.

(Embodiment 1)
FIG. 2 shows an example of a configuration diagram of the imaging apparatus 200 of the present embodiment.

  The imaging apparatus 200 includes a sensor unit 110 and a DSP unit 220.

  The sensor unit 110 has the same configuration as the sensor unit 110 of FIG. 1 as a comparative example. In other words, the imaging device 10 has an effective pixel unit 111 (first pixel unit) for imaging that outputs first data according to imaging conditions, and a black level detection that outputs second data according to imaging conditions. An OB pixel portion 112 (second pixel portion). Note that the imaging condition may include at least one of exposure time and gain. Thereby, an imaging parameter correlated with the black level can be used as an imaging condition.

  The DSP unit 220 includes a pre-processing unit 121, a WDR synthesis unit 122, a post-processing unit 123, a DSP BL detection unit 1 (221), a DSP BL detection unit 2 (222), and a DSP / sequencer unit 223.

  The pre-processing unit 121, the WDR synthesis unit 122, and the post-processing unit 123 have the same configuration as that of the comparative example in FIG.

  The DSP BL detection unit 1 (221) detects the black level from the image data output from the sensor unit 110. Similarly, the DSP BL detection unit 2 (222) detects the black level from the image data output from the sensor unit 110. These detection units are characterized by being prepared individually for each imaging condition. In this embodiment, the case where there are two imaging conditions is illustrated.

  In other words, the black level detection system 230 includes a DSP BL detection unit 1 (221) and a DSP BL detection unit 2 (222) as black level detection units. The DSP BL detection unit 1 (221) and the DSP BL detection unit 2 (222) are provided for each imaging condition, and are based on the second data output from the OB pixel unit 112 (second pixel unit) for each imaging condition. To detect the black level for each imaging condition.

  The DSP / sequencer unit 223 performs the same processing as the DSP / sequencer unit 124 of FIG. 1 of the comparative example, and controls which detection unit is operated according to the imaging conditions. Also, the DSP / sequencer unit 223 is a sensor / sequencer unit that corrects the effective image data captured under the same imaging condition for the black level detected by the DSP BL detection unit 1 (221) or 2 (222). For 117, a different black level is output at each timing.

  As with the DSP / sequencer unit 124 of FIG. 1, the DSP / sequencer unit 223 (imaging condition control unit) periodically switches the imaging conditions. The BL correction unit 114 (correction unit) corrects the first data output from the effective pixel unit 111 for each imaging condition based on the black level for each imaging condition. The WDR synthesis unit 122 (synthesis unit) synthesizes the first data corrected for each imaging condition. Thereby, black floating and black crushing are suppressed in the image indicated by the synthesized first data.

  Alternatively, as another configuration, black level correction may be performed by the preprocessing unit 121 instead of the BL correction unit 114. For example, when the BL correction unit 114 has a specification of the sensor unit 110 that cannot change the correction amount in accordance with switching of the imaging condition, the pre-processing unit 121 performs the processing.

(Operation in Embodiment 1)
Next, the operation of the imaging apparatus 200 having the configuration of this embodiment will be described. As an example of multiple exposure, particularly in this embodiment, exposure condition 1 (imaging condition 1) based on a combination of high magnification gain and long time exposure, and exposure condition 2 (imaging condition 2) based on a combination of low magnification gain and short time exposure, The double exposure configuration is as follows. For example, assume that the DSP / sequencer unit 223 specifies the operation timing at which the exposure conditions 1 and 2 are switched alternately for each frame.

  At this time, the DSP BL detection unit 1 (221) detects the black level by averaging the pixel values output from the OB pixel unit 112 in the period of the exposure condition 1 in the spatial direction, and detects the black level using the FIR or IIR filter. Thus, the black level in the case of the exposure condition 1 is detected by averaging in the time direction.

  Similarly, the DSP BL detection unit 2 (222) detects the black level by averaging the pixel values output from the OB pixel unit 112 in the period of the exposure condition 2 in the spatial direction, and detects the black level using the FIR or IIR filter. Thus, the black level in the case of the exposure condition 2 is detected by averaging in the time direction.

  In other words, each black level detection unit averages the black level for each imaging condition. Thereby, the black level detected for every imaging condition does not interfere with each other in averaging.

  The black level in each of the exposure conditions 1 and 2 is output to the BL correction unit 114 through the DSP / sequencer unit 223 and the sensor / sequencer unit 117. The BL correction unit 114 subtracts and corrects the black level in the case of the exposure condition 1 at the timing when the image data captured under the exposure condition 1 in the effective pixel unit 111 is output from the AFE unit 113. Similarly, the BL correction unit 114 subtracts and corrects the black level in the case of the exposure condition 2 at the timing when the image data captured under the exposure condition 2 in the effective pixel unit 111 is output from the AFE unit 113.

  As described above, according to the first embodiment, the black level can be corrected by detecting an appropriate black level for each exposure condition of multiple exposure. For this reason, there is an effect that the relationship between the pixel signal output value with respect to the luminance of the subject does not collapse even at a high temperature, and it is possible to prevent the occurrence of black float or black collapse.

  That is, the correction accuracy based on the black level can be improved when different imaging conditions are used.

(Embodiment 2)
FIG. 3 shows an example of a configuration diagram of the imaging apparatus 300 (stereo camera) of the present embodiment.

  In the present embodiment, there are two (plural) image sensors 10 and a black level detection system 230 is provided for each image sensor 10.

  According to the present embodiment, an appropriate black level can be detected for each exposure condition (imaging condition) for multiple exposure and for each image sensor 10 to correct the black level.

(Modification)
FIG. 4 shows an example of a configuration diagram of an imaging apparatus 300A (stereo camera) according to a modification.

  In this modification, there are two image sensors 10 and the black level detection system 230 is provided only for one image sensor 10 (first image sensor). Based on the black level for each imaging condition detected by the black level detection system 230, the BL correction unit 114 (correction unit) performs first imaging element 10 (first imaging element and second imaging element) for each imaging condition. The first data output from the pixel unit 112 is corrected. Thereby, correction of the 1st data output from each image sensor 10 can be prepared.

  When there are three or more image sensors 10, the first output from the first pixel units 112 of all the image sensors 10 based on the black level for each imaging condition detected by one black level detection system 230. Data is corrected.

  In addition, this invention is not limited to above-described embodiment, Various modifications are included. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to the one having all the configurations described. Further, a part of the configuration of an embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of an embodiment. In addition, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  The imaging apparatus 200 according to the first embodiment is a monocular image processing camera including one sensor unit (camera unit). The imaging apparatus 300 according to the second embodiment and the imaging apparatus 300A according to the modified example include two sensor units. The number of sensor units is arbitrary.

  In the above embodiment, the sensor unit 110 (imaging unit) includes the imaging element 10 and captures an image. The DSP unit 220 (image processing unit) processes an image. Here, the black level detection system 230 may be provided in the sensor unit 110. On the other hand, the BL correction unit 114 (correction unit) may be provided in the DSP unit 220. For example, as described above, the preprocessing unit 121 may perform the function of the BL correction unit 114. Thereby, for example, the arrangement location of the black level detection system 230 and the BL correction unit 114 can be changed according to the design.

  The imaging condition may include a temperature around the imaging device. The temperature around the imaging device is measured by, for example, a temperature sensor.

  Further, each of the above-described configurations, functions, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files for realizing each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

  In addition, the following aspects may be sufficient as embodiment of this invention.

  (1) An image sensor having an effective pixel used for photographing and a black level detection pixel used for detecting a black level, and detecting a black level based on a signal from the black level detection pixel. A black level detection unit; a correction unit that corrects a signal from the effective pixel based on the black level; and an imaging parameter control unit that controls an imaging parameter of the imaging element. An imaging apparatus having a plurality of black level detection units provided for different imaging parameters.

  (2) The imaging device according to (1), wherein the correction unit corrects a signal from the effective pixel based on a plurality of black levels detected by the plurality of black level detection units.

  (3) The imaging device according to (1), wherein the imaging parameter is at least one of an exposure time and a gain.

  (4) The imaging apparatus according to (1), wherein the imaging device includes a plurality of imaging devices, and the black level detection unit is provided for each of the plurality of imaging devices.

  (5) The imaging apparatus according to (4), wherein the black level detected by the black level detection unit of one image sensor is used for correction by the correction unit of the other image sensor.

  (6) An imaging unit having the imaging element and an image processing unit that performs image processing based on an image acquired from the imaging unit, wherein the black level detection unit is provided in the imaging unit or the image processing unit. The imaging apparatus according to (1).

  According to the above (1) to (6), even if the pixel values of the optical black pixels and the dummy pixels are integrated over a plurality of fields or frames in order to improve the black level detection accuracy, The black level can be detected individually for each. Since the correct black level can be detected and corrected for each imaging condition, there is an effect that black floating and black crushing do not occur and the relationship of the pixel signal output value with respect to the luminance of the subject does not shift.

DESCRIPTION OF SYMBOLS 10 ... Imaging device, 100 ... Imaging device (comparative example), 110 ... Sensor part (comparative example), 111 ... Effective pixel part, 112 ... Optical black (OB) pixel part, 113 ... Analog front end (AFE) part, 114 ... Black level (BL) correction unit, 115 ... Noise correction unit, 116 ... Pedestal unit, 117 ... Sensor sequencer unit, 118 ... Sensor BL detection unit, 120 ... Digital signal processor (DSP) unit (comparative example), 121 ... Pre-processing unit, 122 ... Wide Dynamic Range (WDR) synthesis unit, 123 ... Post-processing unit, 124 ... DSP / sequencer unit, 200 ... Imaging device (Embodiment 1), 220 ... DSP unit, 221 ... DSP BL detection unit, 1222 ... DSP BL detector, 2223 ... DSP / sequencer 230, black level detection system, 300, imaging device (second embodiment), 300A, imaging device (modification).

Claims (8)

An imaging device having a first pixel unit for imaging that outputs first data according to imaging conditions, and a second pixel unit for black level detection that outputs second data according to the imaging conditions;
An imaging condition controller that periodically switches the imaging conditions;
A black level detection system that is provided for each imaging condition and includes a black level detection unit that detects the black level for each imaging condition based on the second data for each imaging condition;
A correction unit that corrects the first data for each imaging condition based on the black level for each imaging condition;
An imaging apparatus comprising: The imaging apparatus according to claim 1,
The imaging conditions are:
An imaging apparatus comprising at least one of exposure time and gain. The imaging apparatus according to claim 1,
The image sensor is
There are multiple
The black level detection system includes:
It is provided with respect to each said image pick-up element. The imaging device characterized by the above-mentioned. The imaging apparatus according to claim 1,
The image sensor is
There are at least two,
The black level detection system includes:
Provided only for the first image sensor;
The correction unit is
Based on the black level for each imaging condition detected by the black level detection system, the first data output from the first pixel unit of the first imaging element and the second imaging element for each imaging condition An imaging device characterized by correcting. The imaging apparatus according to claim 1,
An image pickup unit having the image pickup device and picking up an image;
An image processing unit for processing the image,
The black level detection system includes:
An image pickup apparatus provided in the image pickup unit or the image processing unit. The imaging apparatus according to claim 5,
The correction unit is
An image pickup apparatus provided in the image pickup unit or the image processing unit. The imaging apparatus according to claim 1,
Each of the black level detection units
The black level is averaged for each of the imaging conditions. The imaging apparatus according to claim 1,
An image pickup apparatus comprising: a combining unit that combines the first data corrected for each image pickup condition.

Images