JP5427744B2 - Image processing device - Google Patents

Description

  The present invention relates to an exposure control unit of an image processing apparatus that detects an object using an image captured by an image sensor.

  2. Description of the Related Art Conventionally, in a camera mounted on a vehicle, expansion of a dynamic range with respect to light input has been a problem so that an object with various luminances can be photographed depending on time and place.

  However, it is practically difficult to have a wide dynamic range that can be visually recognized from sunlight to under the stars. Therefore, many in-vehicle cameras perform exposure control that controls exposure conditions such as aperture, exposure time, and gain according to the signal output level of the image sensor, even if the dynamic range of the image sensor is somewhat narrow. The target is efficiently photographed in the luminance range. Therefore, exposure control is an indispensable function for in-vehicle cameras, and various problems and solutions for exposure control are provided. In particular, since automobiles are high-speed moving bodies, there is a problem in speeding up and optimizing exposure control so that it can cope with instantaneous brightness changes in tunnels, etc. The following are provided as solutions. Yes.

  In Patent Document 1, a plurality of types of exposure control maps (exposure showing a luminance-image density conversion relationship with respect to exposure time / gain settings) are used as a conversion map from the luminance of an imaging target to an image density that is a signal output level of a captured image. Control output characteristics), the brightness of the object to be imaged is obtained based on the current image density and the number of the exposure control map currently in use, and the exposure control map is such that the brightness is the image density of the control target level. By obtaining this, the exposure control is optimized.

  Details of the processing will be described with reference to the example of FIG. FIG. 2 is a diagram for explaining conventional exposure control using an exposure control map, and 16 exposure control maps of n = 1 to 16 are prepared.

First, in FIG. 2A, the n _t (= 7) -th exposure control map currently used at the current time t is read, and the current value y _t of the target image density is set to n _t (= 7). th image density at an exposure control map - to luminance conversion, obtains the current value x _t of the target luminance. Next, in FIG. 2 (b), the most appropriate exposure control map number n _t such that the output image density becomes the control target value y _ref with respect to the luminance x _t obtained in FIG. 2 (a). The optimum map for _{obtaining +1} is selected. In FIG. 2B, the 10th exposure control map is most suitable, and 10 is the next value n _{t + 1 of} the exposure control map number, and the exposure time and gain corresponding to the exposure control map number are set.

  In Patent Document 2, a conversion table of image density and road surface brightness at camera aperture amount and shutter speed is prepared in advance, and the road surface brightness obtained from the aperture amount, shutter speed and image density of the camera currently in use is prepared. Based on the difference between the image density and the target, a conversion table for determining the camera aperture and shutter speed is set in advance, and this table is used to determine the next aperture of the camera. The exposure control is optimized by obtaining the amount and the shutter speed.

JP 2009-157087 A JP-A-11-205663

  However, since the exposure control means of Patent Documents 1 and 2 is intended to set the exposure conditions so that the image density of the control target is set to a single target such as a vehicle or a road surface, a plurality of types are different. When there are detection targets (hereinafter referred to as multiple targets) at the same time, it is not possible to set the exposure condition as a control target for any target in one image, and it is necessary depending on the number of detection targets. The number of images and cameras to be increased increases, and the cost and processing time increase.

  An object of the present invention is to provide an image processing apparatus capable of preventing an increase in cost due to an increase in cameras and an increase in processing time due to an increase in the number of images even when a plurality of objects exist simultaneously.

In order to achieve the above object, in the image processing apparatus of the present invention, linear or non-linear exposure showing the conversion relationship between the luminance of the object imaged by the vehicle-mounted image sensor and the image density detected from the imaged image. A plurality of storage means stored in association with a plurality of types of control factors, a current value of each control factor determined in advance, and a plurality of objects detected from an image captured by an in-vehicle image sensor Image density / luminance conversion means for outputting current values of brightness of a plurality of objects based on the current value of the image density and the plurality of exposure control output characteristics stored in the storage means, and image density / luminance conversion Based on the current values of the brightness of the plurality of objects output from the means, the target values of the image densities of the plurality of objects determined in advance, and the plurality of exposure control output characteristics stored in the storage means Is it an image captured by the image sensor? Image density of the plurality of detected objects, and a control factor output means for outputting a plurality of types of control elements so as to coincide with the target value of image density, a configuration having to.

  The present invention can provide an image processing apparatus capable of preventing an increase in cost due to an increase in cameras and an increase in processing time due to an increase in the number of images even when a plurality of objects exist simultaneously.

It is a figure which shows one Embodiment of the processing flow of exposure control of the image processing apparatus which concerns on this invention. It is a figure for demonstrating the exposure control of the prior art example using an exposure control map. It is a figure for demonstrating the exposure control using the exposure control map of this invention. It is a figure for demonstrating the value of each exposure factor with respect to the exposure control map number of this invention. It is a figure showing the outline of the whole vehicle to which the image processing device concerning the present invention is applied. It is a figure which shows the detailed structure of the image processing apparatus which concerns on this invention. It is a figure which shows the detailed structure of the camera of the image processing apparatus which concerns on this invention. It is a figure for demonstrating the change of the exposure time and gain of this invention. It is a figure for demonstrating knee correction in the HDR image generation of this invention. It is a figure for demonstrating the gamma correction of this invention.

  Embodiments to which the present invention is applied will be described below with reference to the drawings.

  First, the exposure control means of the image processing apparatus of the present invention will be described. The means for acquiring the image density to be detected has been described in the background art, and will be omitted.

  FIG. 1 shows a processing flow of the present invention. The exposure control map is an exposure control output characteristic indicating a conversion relationship between luminance and image density with respect to setting of control factors such as exposure time, gain, knee point, and gamma coefficient. Details of the knee point and gamma coefficient settings will be described later. By storing in advance a plurality of types of exposure control maps that are exposure control output characteristics of the image density with respect to the target luminance and the inverse characteristics thereof in association with combinations of a plurality of types of values in these control factors. As shown in FIG. 1, there are two types of operations: current map selection 1 to image density-luminance conversion 2 operation and luminance-image density conversion 3 to optimum map selection 4 operation.

  The features of the present invention are stored in a storage means, current values of predetermined control factors, current values of image densities of a plurality of objects detected from images captured by a camera that is an in-vehicle image sensor, and Based on a plurality of exposure control output characteristics, image density luminance conversion means for outputting current values of the luminance values of the plurality of objects, and current values of luminance values of the plurality of objects outputted from the image density luminance conversion means And the image density of the plurality of objects detected from the image picked up by the vehicle-mounted image sensor based on the predetermined target value of the image density of the plurality of objects and the exposure control output characteristic, Control factor output means for outputting the plurality of types of control factors that coincide with the target value. Furthermore, it has control factor update means for updating the control factor.

  Since the exposure control map number is an index for unifying and managing the values of the control factors, there are as many combinations of the values of the control factors.

  FIG. 4 shows the value of each exposure factor for the exposure control map number.

  In this embodiment, using means for synthesizing three image signals to be described later to generate one HDR image signal, exposure time and gain are combined in 16 steps, gamma correction is performed in 13 steps, and knee correction is performed in total of 20 steps. The stage can be changed. Therefore, there are 163 = 4096 kinds of exposure time / gain types of three images transmitted as exposure control signals to the image sensor, and types of knee points and gamma coefficients transmitted as image correction signals to the DSP. There are 13 × 20 = 260 ways. Therefore, the maximum value of the exposure control map number to be prepared is the total number 163 × 20 × 13 = 1064960 when all the values of a plurality of types of control factors are combined. Regarding the order of the values of the respective control factors with respect to the exposure control map number, the values of the respective control factors are set so that the image density of the output image increases with respect to the luminance of the subject as the exposure control map number increases as shown in FIG. Arrange in ascending order.

  This makes it possible to set not only a linear but also a more complicated non-linear exposure control map. A detailed method for setting the exposure control map will be described later.

  FIG. 3 shows details of the operation from the current map selection 1 to the image density-luminance conversion 2 and the operation from the luminance-image density conversion 3 to the optimum map selection 4 in FIG. Note that the current map is the current exposure control map, and the optimal map is the optimal exposure control map.

The operation of the image density / luminance conversion 2 from the current map selection 1 in FIG. 3 (a) aims at deriving the luminance X of a plurality of objects from the current image. The value of the control factor are set to the image obtained at the current time t corresponds to the current exposure control map number n _t. The current map is selected from the current exposure control map number n _t by the current map selection 1 and the current image density {y _{t, 1} , yt _{, 2} , yt _{, 3} , y _{t, 4} } is input to the current map to derive the current brightness {x _{t, 1} , x _{t, 2} , x _{t, 3} , yt _{, 4} } for each current object.
Expression (1) shows the conversion relationship between the current luminance of the object and the image density.

For example, it is assumed that lane mark detection and headlight / taillight detection are correctly performed using an image acquired at the current time t. At this time, the road surface area and the lane mark area such as white line, yellow line, and botsdots are obtained from the result of the lane mark detection, and the area determined as the headlight or the taillight from the result of the headlight / taillight detection is determined. The image density {road surface: yt _{, 1} , lane mark: yt _{, 2} , tail light: yt _{, 3} , headlight: yt _{, 4} } is obtained. At this time, as an image density calculation method, an average value of pixel values in the target region may be obtained, or a target image density may be obtained from a histogram of pixel values in the target region. In addition, you may use the information calculated | required in the process of the object detection means.

After that, as shown in FIG. 3A, the brightness of each object {road surface: x _{t, 1} , lane mark: x _{t, 2} , tail light: x _{t, 3} , headlight from the exposure control map (formula (1)) : X _{t, 4} } is obtained.

  Also, the operation of FIG. 3 (b) luminance-image density conversion to optimum map selection is performed by setting the image density of a plurality of objects to the control target level at the next exposure from the current brightness of the plurality of objects obtained in FIG. 3 (a). The purpose is to derive an optimal exposure control map to be stored. There are various methods for selecting the optimum map, but the simplest method is to obtain the exposure control map number n that becomes the minimum in the following equation (2) as a round-robin method.

For example, in the above-described examples of lane mark detection and headlight / taillight detection, the estimated next-time image density is obtained by converting the luminance of each target using equation (1) in a certain exposure control map number n. {Road surface: ecm (n, xt _{, 1} ), Lane mark: ecm (n, xt _{, 2} ), Taillight: ecm (n, xt _{, 3} ), Headlight: ecm (n, xt _{, 4)} )}, The image density {road surface: y _{ref, 1} , lane mark: y _{ref, 2} , tail light: y _{ref, 3} , headlight: y _{ref, 4} } that is the respective control target n An operation to find out is required.

  When the next value of the optimum map number is obtained, the next value of each control factor previously associated with the exposure control map number is obtained. That is, from the table of FIG. 4, three types of exposure time / gain as exposure control signals, knee points as image correction signals, and next values of gamma coefficients are determined. One exposure control process is completed by directly transmitting the above two types of signals to the image sensor and DSP (Digital Signal Processor).

  Thus, by setting a non-linear exposure control map even for a single image, it is possible to obtain optimum exposure characteristics for each target.

  Next, the overall configuration of the vehicle to which the image processing apparatus of the present invention is applied will be described. FIG. 5 is a configuration diagram of an entire embodiment of an automobile which is an image processing system of the present invention.

  The image processing apparatus 101 includes a camera 102 that is an imaging element that captures the traveling direction of a vehicle, and an image analysis unit 103 that performs analysis processing on the captured image.

  A camera 102 that is an image sensor is installed in the vicinity of a rearview mirror so as to catch a road surface in the traveling direction of the vehicle or an object such as a vehicle. The image analysis unit 103 may be installed integrally with the camera, or may be installed alone in another place.

  The image captured by the camera 102 is input to the image analysis unit 103, which performs image analysis processing, and the analysis processing result is sent to the alarm device 104, the headlight control device 106, the brake via the vehicle LAN 105. It is output to the control device 108.

  The alarm device 104 has a built-in speaker, and outputs an alarm signal output by the image processing apparatus 101 as a lane mark detection as a sound. The alarm device 104 may be directly connected to the image processing device 101 without going through the vehicle LAN as long as it only generates sound (ON / OFF operation).

  The headlight control device 106 calculates the voltage to be applied to the high beam and the low beam of the headlight 107 from the position information of the oncoming vehicle headlight and the preceding vehicle taillight output from the image processing device 101, and calculates the calculated voltage to the headlight 107. Supply.

  Thus, the headlight control device 106 controls the irradiation distance of the headlight 107 from the distance to the vehicle ahead. Regarding the control of the headlight irradiation distance, the headlight 107 is configured to operate the filament or reflector portion of the headlight 107, and an optical axis angle control signal is transmitted from the headlight control device 106 to the headlight 107. It may be controlled by changing the optical axis.

  The brake control device 108 receives the presence / absence and position information of the lane mark output from the image processing device 101, the presence / absence and position information of the preceding vehicle and the oncoming vehicle, and controls the left and right and front and rear brakes 109.

  FIG. 6 is a diagram illustrating an internal configuration of the camera 102 and the image analysis unit 103 which are imaging elements.

  The image sensor 201 performs photoelectric conversion for converting an optical image into an electrical signal, sequentially reads out signal charges generated in each pixel of the image array, and transfers the signal charges to the DSP 202 as signal processing means.

  The DSP 202 performs various signal processing such as flaw defect correction, HDR image generation, gamma correction, and color matrix conversion. Details of processing of the image sensor 201 and the DSP 202 will be described later.

  The processing result of the DSP 202 is output as an RGB or YUV digital image signal and transferred to the image input I / F 203 of the image analysis unit 103. Although the image signal is transmitted continuously, the vertical / horizontal synchronization signal is also transmitted. Therefore, the image input I / F 203 can determine a frame or line break, and does not impair the positional relationship between pixels. You can capture images.

  The image captured by the image input I / F 203 is written in the memory 204 and processed or analyzed by the image processing unit 205. A series of processing is executed according to the program 206 written in the FROM.

  Here, a series of processes means image capturing control at the image input I / F 203, image processing control at the image processing unit 205, and capturing of vehicle information used for target detection at the external input / output I / F 208. , Control of the output of the result of object detection, and further control of transfer of an exposure control signal for appropriately setting the exposure condition of the image sensor performed by the CPU 207 and correction of the signal in the DSP 202. Control of the image correction signal for this purpose, and various calculation processes required for this. The calculation processing for performing exposure control in this embodiment is all executed by the CPU 207.

  Next, an exposure control map setting method will be described.

  FIG. 7 is a detailed view of the camera 102 as an image sensor, and shows the internal configuration of the image sensor 201 and the DSP 202.

The image sensor 201 incorporates a control register 309 for performing exposure control.
Exposure control is an exposure time that can be a control factor for controlling the exposure amount in order to improve and stabilize the detection performance of object detection in the image against changes in the brightness (brightness) of the subject that is the sensor input. And controlling the gain appropriately.

  However, in this embodiment, the target is to keep the image density of the target area in the image signal that is the final output signal constant, and the HDR is classified as image correction in the DSP 202 rather than exposure control. Variables used for knee correction and gamma correction at the time of image generation are treated as control factors as well as exposure time and gain.

  The exposure time set in the control register 309 is transferred to the drive control unit 301, and the pixel array 302 is set by the image array 302 performing signal charge reading and reset operations according to the drive timing generated by the drive control unit 301. Images can be taken with the exposure time set. Similarly, the set gain is transferred to the AMP 303 and reflected by amplifying the signal charge accumulated for the exposure time according to a set value. Since both the exposure time and the gain are related to the amplification factor of the output level of the image signal, only the slope of the output characteristic changes as shown in FIG.

  In this embodiment, the exposure time and gain are handled together in combination, and both can be changed in 16 steps. The signal charge amplified by the AMP 303 is converted into a digital image signal (digital RAW data) by the ADC 304 and transmitted to the DSP 202.

  At this time, three types of image signals having different exposure time and gain settings are transmitted to the DSP 202 in one frame period. In order to transmit a plurality of image signals in one frame period, a method in which one frame period is time-divided and signal charges are read and reset at a plurality of different exposure times, or one pixel is divided into a plurality of pixels. There is a method of outputting signal charges at different exposure times, a method of reading signal charges multiple times at different timings in one frame period without resetting in time division, and sequentially reading out the signal charges By applying different gains, a plurality of image signals having different exposure times and gain settings can be acquired. Any of the above methods may be used basically.

  The transmitted data is subjected to a correction process such as a scratch defect in the pre-processing unit 305 and then sent to the HDR image generation unit 306, where three types of digital image signals are combined into one based on the knee point. The knee point is a signal output level at which the generation source image is switched as shown in FIG. By adjusting the knee point as shown in FIG. 9B, it is possible to generate image signals having different output characteristics even when the image signals of the three types of exposure conditions are the same combination. A total of 20 level setting patterns for two knee points were prepared.

  Further, the knee slope can be changed as shown in FIG. 9C by changing the three types of exposure conditions. Adjustment in the process of generating the HDR image is called knee correction. In this embodiment, the signal output characteristic (photoelectric conversion characteristic) with respect to the optical input is linear, the gradation of the signal output is 12 bits, and the dynamic range of the optical input is from the three digital RAW data corresponding to 72 dB. Produces an image equivalent to 18 bits (108 dB).

  The generated image signal is sent to the gamma correction unit 307, and the signal level is corrected according to the gamma coefficient as shown in FIG. The gamma correction may be set using a lookup table. In this case, it is possible to set more finely, but in this embodiment, the change was made using the gamma coefficient, and 13 steps of change of γ: 0.25 to 4.0 were made possible.

  The corrected image signal is sent to the color conversion unit 308, converted from digital RAW data to digital YUV data or digital RGB data, and transmitted to the image analysis unit 103.

  The DSP 202 includes a control register 310 for performing various corrections on the image signal, and can set a knee point level in the HDR image generation unit and a gamma coefficient in the gamma correction unit.

  The exposure control signal and the image correction signal can be rewritten from the CPU 207, and the rewritten exposure time is reflected in the exposure of the next frame, and the next frame is applied to the transfer period. Therefore, the rewritten exposure time is reflected. The image can be used again by the CPU 207 after the memory transfer of the image signal is completed, and three frames after the previous exposure time is rewritten.

  Since the gain setting and various image correction settings are also updated in accordance with the exposure time setting, the exposure time, gain, knee point, and gamma coefficient to be set next time are calculated and rewritten every three frames.

  That is, the exposure control cycle is a three frame period. In the case of an image sensor of 30 fps, since an image is transferred at an interval of 33.3 ms per frame, the exposure control period is 100 ms.

  By changing these control factors to appropriate values, it is possible to non-linearly change the output characteristics of the image density with respect to the luminance of the object imaged by the camera.

  In this embodiment, a method of synthesizing an HDR image using a DSP from a plurality of image signals having different exposure conditions (exposure time and gain settings) is used, but the HDR image generation means is limited to this. Instead, there are various methods such as a method for improving the basic characteristics to increase the saturation level and the noise level, and a method for providing a logarithmic characteristic to the photodiode that performs photoelectric conversion.

  In this embodiment, HDR image generation is performed using an image sensor that transfers three image signals to the DSP in one frame (33 ms). For example, one image signal is transferred in one frame. Even if a general image sensor is used, one HDR image can be generated from three image signals in three frames (100 ms), and the exposure control timing is also a cycle of three frames (100 ms). Therefore, it can be realized without failure.

  The DSP 202 may be built in the image analysis unit 103 instead of the camera side, or the DSP 202 may not exist and the image processing unit 205 may supplement the processing of the DSP 202.

  As described above, the image processing apparatus of the present invention can be expected to have a great effect in a scene where there are a plurality of objects having greatly different luminances. For example, when detecting the lane mark and the headlights and taillights of the preceding vehicle at night by image processing as the detection target, when performing exposure control using background technology, the lane mark or road surface is controlled. In this case, since the brightness of the headlight and taillight varies depending on the vehicle type and distance, the image density changes variously on the captured image, and stable detection is difficult. For this reason, it is necessary to prepare a lane mark detection camera or image and a headlight / taillight detection camera or image separately. On the other hand, when exposure control is performed using the present invention, in one camera and image, the brightness change of the brightness with high brightness is maintained while keeping the image density of the low brightness lane mark and the road surface constant and maintaining a high contrast between the two. Since the image density of a large headlight and taillight can be controlled to be constant, it is possible to prevent an increase in cost due to an increase in cameras and an increase in processing time due to an increase in the number of images.

DESCRIPTION OF SYMBOLS 101 Image processing apparatus 102 Camera 103 Image analysis part 104 Alarm apparatus 105 LAN for vehicles
106 Headlight control device 107 Headlight 108 Brake control device 109 Brake 201 Image sensor 202 DSP
203 Image input I / F
204 Memory 205 Image processing unit 206 Program 207 CPU
208 External I / O I / F
301 Drive control unit 302 Image array 303 AMP
304 ADC
305 Pre-processing unit 306 HDR image generation unit 307 Gamma correction unit 308 Color conversion unit 309 Control register (image sensor side)
310 Control register (DSP side)

Claims (4)

The linear or non-linear exposure control output characteristics indicating the conversion relationship between the brightness of the object imaged by the vehicle-mounted image sensor and the image density detected from the imaged image are associated with a plurality of types of control factors, A plurality of stored storage means;
Current values of predetermined control factors, current values of image densities of a plurality of objects detected from images picked up by the in-vehicle image sensor, and a plurality of the exposure control output characteristics stored in the storage unit And image density luminance conversion means for outputting current values of the luminance of the plurality of objects based on
Current values of the brightness of the plurality of objects output from the image density / luminance conversion means, target values of image densities of the plurality of objects determined in advance, and the plurality of exposure control outputs stored in the storage means And outputting the plurality of types of control factors such that the image density of the plurality of objects detected from the image captured by the vehicle-mounted image sensor matches the target value of the image density based on the characteristics An image processing apparatus having factor output means. The image processing apparatus according to claim 1.
An image processing apparatus comprising control factor update means for updating the plurality of types of control factors output by the control factor output means from current control factors. The image processing apparatus according to claim 1 or 2,
The image processing apparatus, wherein the control factor is at least one of an exposure time, a gain, a knee point, and a gamma coefficient. The image processing apparatus according to any one of claims 1 to 3,
The exposure control output characteristic is an image processing apparatus having a linear output characteristic in which a conversion relationship between the luminance of an object imaged by an in-vehicle image sensor and an image density detected from the imaged image is shown.

Images