JP6705054B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging device.

2. Description of the Related Art There is known an image pickup apparatus capable of eliminating a black level offset between fields to obtain a beautiful image (see, for example, Patent Document 1).

In this publication, "the image sensor includes an image pickup area, a front OB section having a photoelectric conversion element structure, a rear OB section having no photoelectric conversion element structure, and a dummy pixel section formed in a horizontal transfer section. The black level is clamped on the basis of the output signal of the rear OB section, and the black level signals are extracted from the front OB section, the rear OB section, and the dummy pixel section, for example, the temperature of the image sensor is detected. When the temperature becomes high, the black level correction is performed by adding the difference between the output signal of the rear OB section and the output signal of the front OB section to the effective pixel output signal." OB is an abbreviation for optical black.

JP, 2007-27845, A

To adjust the black level, it is necessary to detect the black level. In the black level detection method, it is conceivable to detect different black levels depending on position information such as screen areas and fields, or conditions such as gain level and exposure time, and individually correct the black levels.

However, in Patent Document 1, in order to capture an image of a subject having a wide dynamic range without saturation or underexposure, an individual exposure time is set for each field (even number row/odd number row) or frame to capture a plurality of images. There is no description about Dynamic Range (WDR) imaging.

In order to increase the black level detection accuracy, pixel values of optical black pixels and dummy pixels may be integrated over a plurality of fields or frames. In that case, when combined with WDR imaging, black levels of fields or frames with different imaging conditions are mixed and detected, which may rather deteriorate the accuracy. In this case, the black level is not normally corrected, and the relationship of the pixel signal output value with respect to the brightness of the subject is deviated, and there is a problem that black floating or black crushing occurs.

An object of the present invention is to provide an image pickup apparatus capable of improving the correction accuracy based on the black level when different image pickup conditions are used.

In order to achieve the above object, the present invention provides a first pixel portion for image pickup that outputs first data according to an image pickup condition and a black pixel detection portion that outputs second data according to the image pickup condition. An imaging device having a second pixel unit, an imaging condition control unit that periodically switches the imaging conditions, an imaging condition control unit that is provided for each imaging condition, and for each imaging condition based on the second data for each imaging condition. A black level detection system including a black level detection unit that detects the black level, and a correction unit that corrects the first data for each imaging condition based on the black level for each imaging condition.

According to the present invention, the correction accuracy based on the black level can be improved when different imaging conditions are used. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

It is a figure which shows the structure of the imaging device as a comparative example. 1 is a diagram showing a configuration of an image pickup apparatus according to a first embodiment. FIG. 9 is a diagram showing a configuration of an image pickup apparatus according to a second embodiment. It is a figure which shows the structure of the imaging device of a modification.

Hereinafter, the configuration and operation of the imaging device according to the first and second embodiments of the present invention will be described with reference to the drawings.

(Comparative example)
FIG. 1 shows a block diagram of an image pickup apparatus 100 as a comparative example.

The image pickup apparatus 100 includes a sensor unit 110 and a DSP unit 120 (DSP: Digital Signal Processor).

The sensor unit 110 includes an effective pixel unit 111, an OB pixel unit 112 (OB: Optical Black), an AFE unit 113 (AFE: Analog Front End), a BL correction unit 114 (BL: Black Level), a noise correction unit 115, and a pedestal unit. 116 and a sensor/sequencer unit 117.

The image pickup apparatus 100 is, for example, a vehicle-mounted camera, and is characterized by picking up and processing a WDR image by multiple exposure in order to obtain an image with a wide dynamic range. Further, the on-vehicle camera is characterized in that a wide temperature guarantee range is required.

The sensor unit 110 is an image sensor that takes an image with a configuration such as Complementary MOS (CMOS) or Charge Coupled Device (CCD).

In the effective pixel portion 111, pixels that apply photoelectric conversion and voltage conversion to the brightness coming from the direction of the subject (for example, in the case of a vehicle-mounted sensor, etc.) are arranged two-dimensionally in a matrix or honeycomb arrangement. It was done.

The OB pixel unit 112 is an array of pixels that are shielded so that a dark current can be detected, or dummy pixels that do not contain a dark current and that detect the characteristics of a signal of a readout system.

The AFE unit 113 converts the voltage value into digital image data by A/D conversion.

The sensor BL detection unit 118 outputs an average value, a median value, a median value, etc. in the spatial direction in the digital image data output from the OB pixel unit 112 and converted by the AFE unit, or in the time direction in a plurality of frames or field periods. Apply processing to output a number called the "black level". The black level is obtained by digitally converting the image output from the effective pixel unit 111 by the AFE unit 113, and detecting the level of the pixel value when light does not enter the digital image data due to light shielding or the like.

The BL correction unit 114 receives the black level output from the sensor BL detection unit 118 via the sensor sequencer unit 117, and outputs image data obtained by subtracting and correcting the black level from the effective image data. By subtracting the black level, the level of the pixel value when light is shielded is corrected to a reference value such as zero.

The noise correction unit 115 performs image filter processing on the image data output by the BL correction unit 114 in the spatial direction or the time direction to correct noise.

The pedestal unit 116 adds a predetermined pixel value to the image data output by the noise correction unit 115 and outputs the image data.

The sensor/sequencer unit 117 receives the imaging conditions such as the analog gain and the exposure time from the DSP/sequencer unit 124, which will be described later, and sets the operation content and operation timing of each unit in the sensor unit so that the specified imaging conditions can be realized. Control. Further, in order to realize WDR imaging by multiple exposure, imaging conditions can be switched for each field or each frame for imaging.

The DSP unit 120 includes a pre-processing unit 121, a WDR combining unit 122, a post-processing unit 123, and a DSP/sequencer unit 124. The DSP unit receives the image data picked up by the sensor unit 110.

The preprocessing unit 121 applies image processing such as noise removal, pedestal addition, and digital gain processing to the image data output from the sensor unit 110. For example, there is a case in which digital image processing for the same purpose is duplicated in the sensor unit 110 due to designing circumstances, such as using an off-the-shelf product as the sensor unit 110 and designing the DSP unit in-house.

The WDR synthesizing unit 122 receives the image data output from the pre-processing unit 121 as an input, synthesizes a plurality of images captured under individual image capturing conditions, and creates and outputs one image having a wide dynamic range.

The post-processing unit 123 receives the image data output by the WDR synthesis unit 122 as an input. For example, in the vehicle-mounted camera, the post-processing unit 123 detects a vehicle, a pedestrian, an obstacle, a sign, a sign, a white line, and the like, and outputs the detected location and content.

As another configuration of the DSP unit 120, the WDR synthesizing unit 122 may not perform synthesis, and the post-processing unit 123 may perform recognition processing individually for each imaging condition. Generally, when combining, the processing amount of the post-processing unit 123 can be reduced, and when not combining, the detection accuracy of the post-processing unit 123 can be improved.

The DSP/sequencer unit 124 controls the operation of the entire image pickup apparatus including the designation of the image pickup conditions for multiple exposure.

Next, problems of the image pickup apparatus 100 of the comparative example shown in FIG. 1 will be described.

The black level of the effective pixel portion 111 and the OB pixel portion 112, which are connected to the pixels, increases as the temperature rises as a general characteristic of the CMOS image sensor.
Further, generally, when one or more of the three elements of high temperature, high-magnification analog gain, and long exposure time are prepared, the black level increase amount becomes large.

In general, when the sensor unit 110 has a lot of noise (particularly, dark noise, which is detected in a light-shielded state), the output signal value of the OB pixel unit is averaged over a plurality of frames. The black level may be accurately measured by reducing the influence of noise. For example, generally, the black level of each frame is detected, and further averaged by a time (that is, frame or field direction) FIR filter (FIR: Finite Impulse Response) or IIR filter (IIR: Infinite Impulse Response). By implementing the above method, a method of reducing the noise and detecting the black level more accurately than in the case of detecting in one frame may be taken.

However, in the case of implementing WDR imaging by multiple exposure in the image pickup apparatus 100, when the black level detection method using a plurality of frames or fields is adopted as described above, the exposure condition of multiple exposure (for example, In the exposure condition 1 by the combination of the high magnification gain and the long time exposure, and the exposure condition 2 by the combination of the low magnification gain and the short time exposure, the original black level is different for each exposure condition. The black levels of the frame are mixed and detected, and the accuracy deteriorates. In this case, the black level is not corrected normally, and the relationship of the pixel signal output value with respect to the brightness of the subject is deviated, resulting in black floating or black crushing.

(Embodiment 1)
FIG. 2 shows an example of a configuration diagram of the image pickup apparatus 200 of the present embodiment.

The imaging device 200 includes a sensor unit 110 and a DSP unit 220.

The sensor unit 110 has the same configuration as the sensor unit 110 of FIG. 1 which is a comparative example. That is, the image pickup device 10 includes an effective pixel portion 111 (first pixel portion) for image pickup that outputs first data according to the image pickup condition, and a black level detection portion that outputs second data according to the image pickup condition. And an OB pixel portion 112 (second pixel portion). The imaging condition may include at least one of the exposure time and the gain. Thereby, the imaging parameter correlated with the black level can be used as the imaging condition.

The DSP unit 220 includes a pre-processing unit 121, a WDR synthesis unit 122, a post-processing unit 123, a DSP BL detection unit 1 (221), a DSP BL detection unit 2 (222), and a DSP/sequencer unit 223.

The pre-processing unit 121, the WDR synthesis unit 122, and the post-processing unit 123 have the same configuration as that of the comparative example in FIG.

The DSP BL detection unit 1 (221) detects the black level from the image data output by the sensor unit 110. Similarly, the DSP BL detection unit 2 (222) detects the black level from the image data output from the sensor unit 110. These detection units are characterized in that they are individually prepared for each imaging condition. In the present embodiment, the case where there are two imaging conditions is illustrated.

In other words, the black level detection system 230 is composed of the DSP BL detection unit 1 (221) and the DSP BL detection unit 2 (222) as the black level detection unit. The DSP BL detection unit 1 (221) and the DSP BL detection unit 2 (222) are provided for each imaging condition, and based on the second data output from the OB pixel unit 112 (second pixel unit) for each imaging condition. To detect the black level for each imaging condition.

The DSP/sequencer unit 223 performs the same processing as the DSP/sequencer unit 124 of FIG. 1 of the comparative example, and controls which detection unit is operated according to the imaging conditions. Further, the DSP/sequencer unit 223 corrects the effective image data imaged under the same imaging condition for the black level detected by the DSP BL detection unit 1 (221) or 2 (222). A different black level is output to 117 at each timing.

Note that, similarly to the DSP/sequencer unit 124 of FIG. 1, the DSP/sequencer unit 223 (imaging condition control unit) periodically switches the imaging conditions. The BL correction unit 114 (correction unit) corrects the first data output from the effective pixel unit 111 for each imaging condition based on the black level for each imaging condition. The WDR synthesis unit 122 (synthesis unit) synthesizes the first data corrected for each imaging condition. As a result, black floating and black crushing are suppressed in the image represented by the combined first data.

Alternatively, as another configuration, the black level correction may be performed by the preprocessing unit 121 instead of the BL correction unit 114. For example, when the BL correction unit 114 has the specifications of the sensor unit 110 that cannot switch the correction amount according to the switching of the imaging conditions, the pre-processing unit 121 performs the correction.

(Operation in Embodiment 1)
Next, the operation of the image pickup apparatus 200 having the configuration of this embodiment will be described. Multiple exposure, particularly in the present embodiment, as an example, exposure condition 1 (imaging condition 1) based on a combination of high-magnification gain and long-time exposure, and exposure condition 2 (imaging condition 2) based on a combination of low-magnification gain and short-time exposure , And double exposure. For example, it is assumed that the DSP/sequencer unit 223 specifies the operation timing at which the exposure conditions 1 and 2 are alternately switched for each frame.

At this time, the DSP BL detection unit 1 (221) detects the black level by averaging the pixel values output from the OB pixel unit 112 in the spatial direction during the exposure condition 1 and detects the black level. Then, the black level under the exposure condition 1 is detected by averaging in the time direction.

Similarly, the DSP BL detection unit 2 (222) detects the black level by averaging the pixel values output from the OB pixel unit 112 in the spatial direction during the exposure condition 2 and detects the black level, and detects the black level. Then, the black level under the exposure condition 2 is detected by averaging in the time direction.

In other words, each black level detection unit averages the black levels for each imaging condition. This prevents the black levels detected for each imaging condition from interfering with each other in averaging.

The black level under each of the exposure conditions 1 and 2 is output to the BL correction unit 114 via the DSP/sequencer unit 223 and the sensor/sequencer unit 117. The BL correction unit 114 corrects by subtracting the black level under the exposure condition 1 at the timing when the image data captured under the exposure condition 1 in the effective pixel unit 111 is output from the AFE unit 113. Similarly, the BL correction unit 114 subtracts and corrects the black level under the exposure condition 2 at the timing when the image data captured under the exposure condition 2 in the effective pixel unit 111 is output from the AFE unit 113.

As described above, according to the first embodiment, it is possible to detect an appropriate black level for each exposure condition of multiple exposure and correct the black level. Therefore, there is an effect that the relationship of the pixel signal output value with respect to the brightness of the subject is not broken even at high temperature, and it is possible to prevent the occurrence of black floating or black crushing.

That is, the correction accuracy based on the black level can be improved when different imaging conditions are used.

(Embodiment 2)
FIG. 3 shows an example of a configuration diagram of the image pickup apparatus 300 (stereo camera) of the present embodiment.

In the present embodiment, there are two (a plurality of) image pickup devices 10, and the black level detection system 230 is provided for each image pickup device 10.

According to the present embodiment, it is possible to detect an appropriate black level for each exposure condition (imaging condition) of multiple exposure and for each image sensor 10 and perform the correction of the black level.

(Modification)
FIG. 4 shows an example of a configuration diagram of an image pickup apparatus 300A (stereo camera) of a modified example.

In this modification, there are two image pickup devices 10, and the black level detection system 230 is provided only for one image pickup device 10 (first image pickup device). The BL correction unit 114 (correction unit), based on the black level for each imaging condition detected by the black level detection system 230, the first of the imaging device 10 (first imaging device and second imaging device) for each imaging condition. The first data output from the pixel unit 112 is corrected. Thereby, the correction of the first data output from each of the image pickup devices 10 can be made uniform.

In addition, when there are three or more image pickup devices 10, the first pixel units 112 of all the image pickup devices 10 output based on the black level for each image pickup condition detected by one black level detection system 230. The data is corrected.

It should be noted that the present invention is not limited to the above-described embodiment, and various modifications are included. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those including all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add/delete/replace other configurations with respect to a part of the configurations of the respective embodiments.

The imaging apparatus 200 of the first embodiment shows a monocular image processing camera having one sensor section (camera section), and the imaging apparatus 300 of the second embodiment and the modified imaging apparatus 300A have two sensor sections. However, the number of sensor units is arbitrary.

In the above-described embodiment, the sensor unit 110 (imaging unit) includes the image sensor 10 and captures an image. Further, the DSP unit 220 (image processing unit) processes an image. Here, the black level detection system 230 may be provided in the sensor unit 110. On the other hand, the BL correction unit 114 (correction unit) may be provided in the DSP unit 220. For example, as described above, the preprocessing unit 121 may perform the function of the BL correction unit 114. Thereby, for example, the arrangement positions of the black level detection system 230 and the BL correction unit 114 can be changed according to the design.

The imaging condition may include the temperature around the imaging device. The temperature around the imaging device is measured by, for example, a temperature sensor.

Further, the above-described respective configurations, functions and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Further, each of the above-described configurations, functions, and the like may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as a program, a table, and a file that realizes each function can be placed in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

The embodiment of the present invention may have the following aspects.

(1) An image sensor having an effective pixel used for photographing and a black level detecting pixel used for detecting a black level, and a black level is detected based on a signal from the black level detecting pixel. A black level detection unit, a correction unit that corrects a signal from the effective pixel based on the black level, and an imaging parameter control unit that controls an imaging parameter of the image sensor, the black level detection unit, An imaging apparatus having a plurality of black level detection units provided for different imaging parameters.

(2) The imaging device according to (1), wherein the correction unit corrects the signal from the effective pixel based on the plurality of black levels detected by the plurality of black level detection units.

(3) The imaging device according to (1), wherein the imaging parameter is at least one of exposure time and gain.

(4) The image pickup device according to (1), wherein the image pickup device has a plurality of image pickup devices, and the black level detection unit is provided for each of the plurality of image pickup devices.

(5) The image pickup device according to (4), wherein the black level detected by the black level detection unit of one image pickup device is used for correction by the correction unit of the other image pickup device.

(6) An image pickup unit including the image pickup device, and an image processing unit that performs image processing based on an image acquired from the image pickup unit, wherein the black level detection unit is the image pickup unit or the image processing unit. The imaging device of (1) which has.

According to the above (1) to (6), in order to improve the detection accuracy of the black level, even if the pixel values of the optical black pixels and the dummy pixels are integrated over a plurality of fields or frames, the imaging condition is The black level can be detected individually for each. Since the correct black level can be detected and corrected for each imaging condition, black floating or black crushing does not occur, and there is an effect that the relationship of the pixel signal output value with respect to the brightness of the subject does not shift.

DESCRIPTION OF SYMBOLS 10... Imaging element, 100... Imaging device (comparative example), 110... Sensor part (comparative example), 111... Effective pixel part, 112... Optical black (OB) pixel part, 113... Analog Front End (AFE) part, 114 Black level (BL) correction unit, 115... Noise correction unit, 116... Pedestal unit, 117... Sensor/sequencer unit, 118... Sensor BL detection unit, 120... Digital Signal Processor (DSP) unit (comparative example), 121... Pre-processing unit, 122... Wide Dynamic Range (WDR) combining unit, 123... Post-processing unit, 124... DSP/sequencer unit, 200... Imaging device (Embodiment 1), 220... DSP unit, 221... DSP BL detection unit, 1222... DSP BL detection unit, 2223... DSP/sequencer unit, 230... Black level detection system, 300... Imaging device (Embodiment 2), 300A... Imaging device (modification).

Claims (8)

An image pickup device having a first pixel portion for image pickup that outputs first data according to the image pickup condition, and a second pixel portion for detecting black level that outputs second data according to the image pickup condition,
An imaging condition control unit that periodically switches the imaging condition,
A black level detection system which is provided for each of the imaging conditions, and includes a black level detection unit which detects the black level for each of the imaging conditions based on the second data for each of the imaging conditions;
A correction unit that corrects the first data for each of the imaging conditions based on the black level for each of the imaging conditions;
An imaging device comprising: The imaging device according to claim 1, wherein
The imaging conditions are
An imaging device comprising at least one of an exposure time and a gain. The imaging device according to claim 1, wherein
The image sensor is
There are multiple,
The black level detection system,
An imaging device provided for each of the imaging devices. The imaging device according to claim 1, wherein
The image sensor is
There are at least two,
The black level detection system,
Provided only for the first image sensor,
The correction unit is
Based on the black level for each of the image capturing conditions detected by the black level detection system, the first data output from the first pixel section of the first image sensor and the second image sensor for each of the image capturing conditions is output. An imaging device characterized by correction. The imaging device according to claim 1, wherein
An image pickup section having the image pickup element, for picking up an image;
An image processing unit that processes the image,
The black level detection system,
An imaging device provided in the imaging unit or the image processing unit. The imaging device according to claim 5, wherein
The correction unit is
An imaging device provided in the imaging unit or the image processing unit. The imaging device according to claim 1, wherein
Each of the black level detectors,
An image pickup apparatus, wherein the black level is averaged for each of the image pickup conditions. The imaging device according to claim 1, wherein
An imaging apparatus comprising: a combining unit that combines the first data corrected for each of the imaging conditions.

Images