JP2019204988A - Image processing apparatus and image processing method - Google Patents

Abstract

To provide an image processing apparatus and an image processing method that can expand an object recognizable range.SOLUTION: An image processing apparatus 5 comprises: an infrared light radiation unit 1 that radiates infrared light toward the periphery of a vehicle V; an imaging unit 2 that picks up an infrared light image of the periphery of the vehicle and outputs an infrared light image signal; and a recognition unit 3 that recognizes a first area in the infrared light image where an image a first object is picked up on the basis of the infrared light image signal from the imaging unit 2, and outputs first position information in the infrared light image related to the first area. The imaging unit 2 has a brightness adjustment unit 234 that, on the basis of the first position information output from the recognition unit 3, adjusts a second brightness of a second area having a lower brightness than a first brightness of the first area in the infrared light image on the basis of the first brightness.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image processing apparatus and an image processing method including an infrared light irradiation unit that irradiates infrared light and an imaging unit that captures an infrared light image.

  An in-vehicle camera system having an in-vehicle camera that is mounted on a vehicle and mainly captures the front of the vehicle has been proposed and realized. In particular, in the latest EuroNCAP (European New Car Assessment Program), AEB (Autonomous Emergency Braking) for pedestrians at night is scheduled to be included in the evaluation items. The importance of an in-vehicle camera system having a camera that images the front is increasing.

  As a camera applied to such an in-vehicle camera system, a camera that images a return light from an object irradiated by a visible light headlight mounted on a vehicle has been used. Here, if the high beam headlight is always turned on for the purpose of improving the visibility and safety in the distance at night, the driver of the oncoming vehicle may be dazzled by irradiating the oncoming vehicle with this high beam headlight. It is difficult to always turn on the high beam headlight. For this reason, the visible light headlight has a certain limit in the light reachable range.

  Therefore, for the purpose of further improving nighttime visibility and safety, it has an in-vehicle camera that images the front of the vehicle capable of receiving near-infrared light, and a projector that projects near-infrared light in front of the vehicle. An in-vehicle camera system has been proposed (see, for example, Patent Document 1). Since humans cannot visually recognize near-infrared light, the driver of an oncoming vehicle is not dazzled by near-infrared light even when the projector is illuminated in front of the vehicle. As a result, it is possible to irradiate near-infrared light with a projector farther than the light from the visible headlight, and it is always possible to image subjects such as far away with the in-vehicle camera, and the visibility and safety of far away Improvement is expected.

Japanese Patent No. 5423363

  In a vehicle-mounted camera system having a projector that projects near infrared light, as typified by Patent Document 1, near infrared light projected from the projector is, for example, about 100 m away It was reaching. This also narrows down the near-infrared light projected from the projector, and as a result, the range in which the near-infrared light from the projector reaches, that is, the irradiation angle of the near-infrared light is also narrowed.

  Therefore, there is a possibility that the near-infrared light does not sufficiently reach an object located in a range where the near-infrared light from the projector does not reach, and this object may be difficult to recognize by the in-vehicle camera system.

  Therefore, an object of the present invention is to provide an image processing apparatus and an image processing method capable of expanding the recognizable range of an object.

  In order to achieve the above object, an image processing apparatus according to the present invention includes an infrared light irradiator that irradiates infrared light toward the periphery of a vehicle, and an infrared light image obtained by capturing an infrared light image around the vehicle. Based on the imaging unit that outputs the image signal and the infrared light image signal, the first region where the first object is imaged in the infrared light image is recognized, and the infrared light related to the first region is recognized. A recognition unit that outputs first position information in the image, and the imaging unit has a luminance lower than the first luminance of the first region in the infrared image based on the first position information The second area of the second area having a brightness adjustment section that adjusts the second brightness based on the first brightness.

  In the image processing apparatus of the present invention configured as described above, the brightness adjustment unit is configured to use the first brightness of the first region in the infrared light image based on the first position information output from the recognition unit. The second luminance of the second region having a lower luminance is adjusted based on the first luminance.

  By doing in this way, it becomes possible to expand the recognizable range of an object.

1 is a diagram illustrating a schematic configuration of an in-vehicle camera system to which an image processing apparatus according to an embodiment of the present invention is applied. It is a block diagram which shows the imaging part and recognition part of the image processing apparatus which is embodiment. It is a block diagram which shows schematic structure of the imaging part of the image processing apparatus which is embodiment. It is a block diagram which shows schematic structure of the signal processing part of the image processing apparatus which is embodiment. It is a block diagram which shows schematic structure of the recognition part of the image processing apparatus which is embodiment. 1 is a functional block diagram illustrating a schematic configuration of an image processing apparatus according to an embodiment. 3 is a flowchart for explaining the operation of the image processing apparatus according to the embodiment. 4 is a flowchart for explaining a brightness adjustment operation of the image processing apparatus according to the embodiment. It is a figure which shows an example of the image before the brightness adjustment by the image processing apparatus which is embodiment. It is a figure which shows an example of the image after the brightness adjustment by the image processing apparatus which is embodiment. It is a block diagram which shows the other example of the imaging part and recognition part of the image processing apparatus which is embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing a schematic configuration of an in-vehicle camera system to which an image processing apparatus according to an embodiment of the present invention is applied.

  In the description of this specification, the term “infrared” may be used in the same way as “near infrared”.

  The in-vehicle camera system S according to the present embodiment is a projector that projects light including near-infrared light toward the front of the vehicle V in which the in-vehicle camera system S is mounted (left side in the figure). Infrared light irradiation unit) 1, a camera (imaging unit) 2 that captures a near-infrared light image in front of the vehicle in the traveling direction and outputs a near-infrared light image signal, and a near-red light output from this camera Based on the external light image signal, a recognition unit 3 that recognizes an object captured in the near-infrared light image, and an ECU (Engine Control Unit) that receives a recognition result by the recognition unit 3 And a vehicle control unit 4.

  The image processing apparatus 5 according to the embodiment of the present invention includes at least a projector 1, a camera 2, and a recognition unit 3.

  The image processing apparatus 5 according to the present embodiment is an arithmetic element represented by an integrated circuit such as a programmable logic device such as a CPU or FPGA, an integrated circuit such as an ASIC, except for the projector 1 and an optical system 20 and an optical filter 21 of a camera 2 described later. And a large-capacity storage medium such as a hard disk drive, a storage medium such as a semiconductor storage medium such as ROM and RAM, and an interface circuit with an external device. Here, in the present embodiment, the camera 2 and the recognition unit 3 constituting the image processing apparatus 5 are described as having an integral arithmetic element, a storage medium, and an interface circuit. It may be configured.

  The camera 2 and the recognition unit 3 have a storage medium (not shown) and store a control program. The control program is executed by the arithmetic element when the in-vehicle camera system S is activated, and the image processing apparatus 5 has a functional configuration as shown in FIGS. In particular, since the image processing apparatus 5 according to the present embodiment performs high-speed image processing as will be described later, it is preferable that the image processing apparatus 5 includes an arithmetic element capable of high-speed calculation, such as an FPGA.

  The projector 1 projects light including near-infrared light forward in the traveling direction of the vehicle V. As an example, a single or a plurality of light-emitting diodes are generally mounted on a headlight mounted on the vehicle V. A near-infrared light source that emits near-infrared light such as (LED) is also provided. Of course, a configuration in which light having a wavelength band from visible light to near infrared light is emitted by a light source such as a single type of light emitting diode may be used.

  The projector 1 emits visible light and the like with a predetermined radiation angle for each of visible light and near-infrared light. As described above, the emission angle of near-infrared light is narrower than the emission angle of visible light, so that near-infrared light can reach farther than visible light. .

  In the image processing apparatus 5 according to the present embodiment, as shown in FIG. 2, the camera 2 captures a near-infrared light image in front of the vehicle V and outputs a near-infrared light image signal to the recognition unit 3. The recognizing unit 3 recognizes a region where an object such as a person is imaged in the near-infrared light image using a known image recognition method, and outputs the recognition result as position information of the region. The positional information output from the recognition unit 3 is transmitted to the vehicle control unit 4 by, for example, a CAN (Controller Area Network) and used for travel control of the vehicle V, while being fed back to the camera 2 to be described later, for example, brightness adjustment processing Used for.

  As shown in FIG. 3, the camera 2 includes an optical system 20, an optical filter 21, a sensor 22, and a signal processing unit 23.

  The optical system 20 includes an optical element such as a lens or a mirror, and forms an image of light emitted from a subject or reflected from the subject on an imaging surface (not shown) of a sensor 22 described later. In the case of the in-vehicle camera system S, the optical system 20 is generally a pan focus lens such as a narrow angle, a wide angle, and a fisheye. Further, a lens system including a zoom mechanism or an autofocus mechanism may be used, or a lens system including an aperture or a shutter may be used. In order to improve image quality and color reproducibility, a lens system including various filters such as an optical low-pass filter, a band separation filter, and a polarization filter may be used.

  The sensor 22 is composed of a plurality of pixels, and photoelectrically converts incident light based on the subject image formed on the imaging surface by the optical system 20 into an output voltage signal corresponding to the luminance. The photoelectrically converted output voltage signal is digitized through an amplifier (not shown) provided in the sensor 22 and further an AD converter (not shown) provided in the sensor 22 to generate an output signal RAW. The

  As the sensor 22, a photoelectric conversion element such as a CMOS image sensor or a CCD image sensor having sensitivity to at least near infrared light is used. In the present embodiment, the sensor 22 captures a near-infrared light image ahead of the traveling direction of the vehicle V at a predetermined frame rate (for example, 1/30 seconds, 1/60 seconds), and outputs an output signal to the signal processing unit 23. Send it out.

  For example, four types of filters (red filter R, green filter G, blue filter B, and transparent filter C) having different transmission wavelength bands are regularly arranged on each pixel constituting the sensor 22. These four types of filters constitute the optical filter 21. However, the configuration of the optical filter 21 may be a known one as long as the output signal RAW from the sensor 22 includes a near infrared light component.

  The signal processing unit 23 instructs the sensor 22 to perform imaging, receives the output signal RAW imaged by the sensor 22, performs various signal processing, and a near-infrared light image that is a result of this signal processing. Output a signal. As already described, the near-infrared light image signal output from the signal processing unit 23 is input to the recognition unit 3, and the position information of the region where the object is imaged is fed back from the recognition unit 3. It is used for luminance adjustment processing in the signal processing unit 23.

  Further, the signal processing unit 23 sends a so-called automatic exposure (AE) control signal to the sensor 22 based on the results of various signal processing. The sensor 22 performs an imaging operation with a shutter speed or the like according to the automatic exposure control signal based on the imaging timing from the signal processing unit 23.

  As illustrated in FIG. 3, the signal processing unit 23 includes an input control unit 230, a noise removal unit 231, a pixel interpolation unit 232, a white balance processing unit 233, a luminance adjustment unit 234, and an output control unit 235.

  The input control unit 230 receives the output signal RAW output from the sensor 22 and sends it to the noise removal unit 231 and the subsequent stage. Further, the input control unit 230 receives the luminance level of the near-infrared light image signal detected by the luminance adjustment unit 234 and the like, generates an automatic exposure control signal based on this, and sends it to the sensor 22.

  The noise removing unit 231 reduces and removes noise included in the output signal RAW from the sensor 22 by a known method.

  The pixel interpolation unit 232 separates the noise-removed output signal RAW that is the output of the noise removal unit 231 into four color signals (R, G, B, and C) corresponding to the respective colors that constitute the output signal RAW. At this time, the output of a pixel having no signal is 0 (blank). Next, the pixel interpolation unit 232 linearly interpolates the pixel values expected to be observed in the blank pixels using the neighboring pixel values. Then, four color signals (R, G, B, C) are generated in all the pixels. Further, the pixel interpolation unit 232 performs color signal correction processing by a known linear matrix calculation.

  The white balance processing unit 233 performs white balance processing on the color signal by multiplying each color signal (R, G, B, C) output from the pixel interpolation unit 232 by a correction coefficient corresponding to the color temperature. .

  The luminance adjustment unit 234 detects the luminance level of the output color signal from the pixel interpolation unit 232 and outputs the detected luminance level based on the detected luminance level and the like by using a known technique such as AGC (Automatic Gain Control). The gain (gain) of the luminance of the signal is adjusted, and further gamma processing is performed based on a specified gamma curve. In addition, the luminance adjustment unit 234 performs luminance adjustment processing unique to the present embodiment, which will be described in detail later.

  The output control unit 235 synthesizes the output color signal from the white balance processing unit 233 and the luminance signal adjusted by the luminance adjustment unit 234 and sends the synthesized signal to the recognition unit 3 as a near-infrared light image signal. Further, the output control unit 235 receives position information of the area where the object is imaged from the recognition unit 3 and sends the position information to the luminance adjustment unit 234.

  As shown in FIG. 5, the recognition unit 3 includes an input / output control unit 30 and a detection unit 31.

  The input / output control unit 30 receives a near-infrared light image signal that is an output from the camera 2 and sends it to the detection unit 31. Further, the input / output control unit 30 receives position information of an area where the object is imaged, which is an output from the detection unit 31, and sends it to the camera 2.

  When an object such as a person is imaged in the near-infrared light image captured by the camera 2, the detection unit 31 detects an area where the object is imaged, and in the near-infrared light image related to this area Output location information. An object detection (recognition) technique by the detection unit 31 may be a known technique. As an example, a technique of calculating a HoG (Histogram of Oriented Gradients) feature value and recognizing the object based on the feature value is given. Here, HoG is a histogram of the luminance gradient direction of the local region (cell), and can be used for extracting the contour of the object.

  As a result of recognizing the captured object, the detection unit 31 outputs position information (ROI information) in the near-infrared light image related to the region of interest (ROI) where the object is captured. This position information includes, as an example, position coordinate values of an object in the near-infrared light image, x_start (start value of horizontal coordinates), y_start (start value of vertical coordinates), x_end (end value of horizontal coordinates), and It is expressed by y_end (end value of vertical coordinate). If the area is rectangular, the coordinates (x_start, y_start) correspond to the lower left position of the rectangular area, and the coordinates (x_end, y_end) correspond to the upper right position of the rectangular area.

  In the case of object recognition using the HoG feature amount, the detection unit 31 can extract the outline of the object and detect the area surrounded by the outline, and thus the detection area (recognition area) is necessarily a rectangular area. However, in the image processing apparatus 5 according to the present embodiment, the detection unit 31 recognizes a rectangular area including the area where the object is imaged in the near-infrared light image, and outputs position information regarding the rectangular area. It shall be.

  In addition, the detection unit 31 outputs position information for a region (for example, several pixels) slightly expanded outside the region where the recognized object is imaged. The degree of expansion can be arbitrarily set. In addition, since it is considered that the change with the vehicle speed of the vehicle V is large between the central portion and the peripheral portion of the image, the number of pixels to be expanded in the peripheral portion may be larger than that in the central portion.

  Note that the detection unit 31 outputs x_start = y_start = x_end = y_end = 0 as position information when the object cannot be recognized in the near-infrared light image.

  Next, FIG. 6 is a functional block diagram showing a schematic configuration of the image processing apparatus 5 according to the present embodiment. Since each function configuration unit shown in FIG. 6 includes the already described contents, the already described contents are appropriately omitted.

  The image processing apparatus 5 according to the present embodiment includes at least an infrared light irradiation unit (projector) 1, an imaging unit (camera) 2, and a recognition unit 3.

  The infrared light irradiation unit 1 irradiates near infrared light toward the periphery of the vehicle V.

  The imaging unit 2 captures a near-infrared light image around the vehicle V and outputs a near-infrared light image signal. The imaging unit 2 includes a sensor 22 and a signal processing unit 23, and the signal processing unit 23 includes a luminance adjustment unit 234.

  The recognition unit 3 recognizes the first region where the first object is imaged in the near-infrared light image based on the near-infrared light image signal output from the imaging unit 2, and the first region First position information in the near-infrared light image relating to the region is output.

  Then, the luminance adjustment unit 234 is based on the first position information output by the recognition unit 3, and the second region having a luminance lower than the first luminance of the first region in the near-infrared light image. The second luminance is adjusted based on the first luminance.

  Here, the luminance adjustment unit 234 has substantially the same luminance as the first luminance in the near-infrared light image based on the first position information, and the first region and the second region. The brightness of the third region, which is different from each other, is made lower than the brightness of the third region related to the near-infrared light image signal.

  In addition, the recognition unit 3 recognizes the fourth region in which the second object having a luminance lower than the first luminance is imaged in the near-infrared light image, and the near-infrared regarding the fourth region. The second position information in the light image is output, and the brightness adjusting unit 234 increases the brightness of the fourth area based on the second position information from the brightness of the fourth area related to the near-infrared light image signal. .

  Further, the sensor 22 of the imaging unit 2 generates a near-infrared light image signal in units of frames, and the image processing device 5 includes a storage unit 6 in which the first position information and the first luminance are stored in units of frames. The luminance adjustment unit 234 reads the first position information and the first luminance related to the most recent frame stored in the storage unit 6 and relates the first luminance related to the frame for which the luminance adjustment is performed to the latest frame. Adjust to the first brightness.

  The details of the operations of the recognition unit 3 and the luminance adjustment unit 234 and the definitions of the first to fourth areas and the first and second luminances will be described in detail later.

  Next, an overview of operations of the recognition unit 3 and the luminance adjustment unit 234 of the image processing apparatus 5 according to the present embodiment will be described with reference to FIGS.

  As a result of the object recognition process by the recognition unit 3, when the position information (ROI information) is not output because the object is not captured in the near-infrared light image (the object cannot be recognized), the luminance adjustment unit 234 The gain of the output signal is adjusted by AGC or the like, and the result of further gamma processing is output. The near-infrared light image signal that has been subjected to general brightness adjustment or the like by the brightness adjustment unit 234 in this way is input to the recognition unit 3, and object recognition processing by the recognition unit 3 is performed again.

  In addition, since the object recognition processing by the recognition unit 3 is not performed even when the image processing device 5 is started up, the luminance adjustment unit 234 also performs gain adjustment of the output signal by AGC or the like, and further performs gamma processing. Output the result.

  In addition, the luminance adjustment unit 234 determines the region of the near-infrared light image based on the difference in luminance of the near-infrared light image signal in units of pixels. More specifically, the brightness adjusting unit 234 calculates the difference between the brightness of the near-infrared light image signal of the specific pixel and the brightness of the near-infrared light image signal of the pixel located next to the specific pixel in the horizontal direction. Based on this difference, a location where the luminance value exceeds the threshold is determined and a flag is set. Based on the determination result, the near-infrared light image is divided into a plurality of regions having different luminance values.

  From the viewpoint of making it less susceptible to noise, the brightness adjusting unit 234 can also set a flag when a plurality of pixels exceeding the above-mentioned threshold value continues, and also in the vertical direction (up and down direction) as well as the horizontal direction. The determination can also be made based on the difference in luminance value between adjacent pixels.

  Then, the luminance adjustment unit 234 performs luminance adjustment of this region based on the result of the region determination described above.

  First, the luminance adjustment unit 234 performs luminance adjustment for an area that is irradiated with near-infrared light from the projector 1 and is determined to be an area in which the object is imaged as a result of the object recognition processing by the recognition unit 3. Do not adjust the gain (that is, fix the gain). Whether or not the near-infrared light from the projector 1 is irradiated is determined depending on whether or not the region determined by the above-described region determination is a region having a luminance value equal to or higher than a certain value (that is, bright). Just do it. In such a region, the gain is fixed from the viewpoint of reliably performing the object recognition process. This area is the first area described above, the position information about the first area is the first position information, the object imaged in the first area is the first object, and the area Is the first luminance.

  Moreover, although the near-infrared light from the light projector 1 is irradiated, as a result of the object recognition process by the recognition part 3, the area | region where it is not determined that it is the area | region where the target object was imaged is enough. Since it is determined that there is, the luminance adjustment unit 234 decreases (decreases) the luminance gain. In other words, the brightness adjustment is performed in order to prevent the phenomenon called overexposure from occurring in the image due to excessively bright processing. This region is the third region described above. The amount to decrease the gain can be arbitrarily set.

  Furthermore, brightness adjustment is performed for a region that is not irradiated with near-infrared light from the projector 1 and is not determined to be a region in which the object is imaged as a result of the object recognition processing by the recognition unit 3. The unit 234 increases the luminance gain so that the luminance value is equivalent to the luminance value after the luminance adjustment is performed in the third region described above. Thus, the object recognition process by the recognition unit 3 can be performed more reliably. This region is the second region described above, and the luminance of the region before the luminance adjustment is the second luminance.

  And about the area | region which is not irradiated with the near-infrared light from the light projector 1, and was determined as an area | region where the target object was imaged as a result of the target object recognition process by the recognition part 3, about this area | region Since it is determined that the object is an imaged area even though the luminance value is low, there is a possibility that the recognition unit 3 determines that the noise is the object, and the recognition unit 3 correctly selects the object. The possibility of judging coexists. Therefore, the luminance adjustment unit 234 increases the luminance gain of this region by a gain value that is about half of the gain value increased for the second region described above. This area is the above-described fourth area, the position information about the fourth area is the second position information, and the object imaged in the fourth area is the second object.

  FIG. 9 and FIG. 10 schematically showing the near-infrared light images before and after the luminance (gain) adjustment by the luminance adjustment unit 234, with respect to the luminance adjustment by the luminance adjustment unit 234 described above. To explain.

  In FIG. 9, a region A1 irradiated with near-infrared light from the projector 1 is a third region, and a region A2 in which the object B1 is recognized by the recognition unit 3 in the region A1 is a first region. It is.

  The area A1 including the area A2 preferably secures a certain luminance from the viewpoint of more reliably recognizing the object B1. On the other hand, if the contrast with the brightness of areas not irradiated with near-infrared light (areas A3 and A4, which will be described later) is excessively increased, whiteout occurs, so that the brightness gain is reduced for area A2. With respect to A1, the luminance gain is not adjusted from the viewpoint of continuing reliable recognition in the image of the next frame.

  On the other hand, the region A3 where the near-infrared light from the projector 1 is not irradiated is the second region. As shown in FIG. 9, since the near-infrared light from the projector 1 reaches the far-infrared light by narrowing it to a narrow range, the proportion of the area A3 is relatively large. On the other hand, in order to recognize such a pedestrian as an object by narrowing near-infrared light to a narrow range like a pedestrian walking from the side of the road, the brightness of this area A3 Needs to be raised to some extent. Therefore, the brightness gain of the area A3 is increased so that the brightness of the area A3 is approximately the same as the brightness of the area A2 after the brightness adjustment.

  FIG. 10 shows a near-infrared light image after the luminance adjustment by the luminance adjustment unit 234. 9 and 10, it is shown that the luminance is darker as the density of hatched diagonal lines is higher. By increasing the luminance gain of the area A3, the object B2 hidden in the surrounding darkness in FIG. 9 is easily visible, and as a result, the object recognition process by the recognition unit 3 is facilitated. When the object B2 is recognized by the recognition unit 3, the area A4 in which the object B2 is recognized becomes the fourth area. Further, the contrast ratio between the region A2 and the region A3 can be suppressed, and occurrence of whiteout in the region A2 can be suppressed.

  Here, the luminance (gain) adjustment by the luminance adjustment unit 234 is performed by multiplying the luminance value of the pixel included in the target region by a certain value (this is a gain). That is, the brightness adjustment is performed by linear conversion. Alternatively, the luminance adjustment may be performed by adding a constant value (usually called an offset value) to the luminance value. Of course, brightness adjustment may be performed in which a constant value is multiplied and another constant value is added. Furthermore, luminance adjustment may be performed by creating a histogram of luminance values of pixels included in the target region and converting the histogram into a histogram having a desired shape (eg, histogram smoothing).

  Next, details of the operation of the image processing apparatus 5 according to the present embodiment will be described with reference to the flowcharts of FIGS. Of the operations of the respective parts constituting the image processing apparatus 5, operations already described in detail are appropriately omitted.

  The operation shown in the flowchart of FIG. 7 starts when the image processing apparatus 5 is started (basically at the same time as when the vehicle control unit 4 is started). First, in step S <b> 1, the sensor 22 of the camera 2 captures a near-infrared light image in front of the vehicle V and sends an output signal RAW to the signal processing unit 23. In step S <b> 2, the input control unit 230 receives the output signal RAW from the sensor 22.

  In step S <b> 3, the noise removal unit 231 performs noise removal processing on the output signal RAW from the sensor 22. In step S4, the pixel interpolation unit 232 performs linear interpolation and color correction processing.

  In step S5, the luminance adjustment unit 234 performs near-infrared light region determination on the signal that has undergone linear interpolation and color correction processing in step S4. Further, in step S6, the region determination result in step S5 is also determined. Refer to and adjust the brightness. A detailed flowchart of the brightness adjustment will be described later.

  In step S7, the white balance processing unit 233 performs white balance processing on the signal that has undergone linear interpolation and color correction processing in step S4.

  In step S8, the output control unit 235 combines the luminance signal adjusted in luminance in step S6 and the color signal subjected to white balance processing in step S7. In step S9, the output control unit 235 outputs the signal synthesized in step S8 to the recognition unit 3 as a near infrared light image signal.

  In step S <b> 10, the detection unit 31 of the recognition unit 3 performs a recognition process as to whether or not the object is imaged in the near-infrared light image signal by the object detection processing algorithm.

  In step S11, it is determined whether or not the detection unit 31 has recognized the object as a result of step S10. If it is determined that the object has been recognized, the program proceeds to step S12 and the object cannot be recognized. If it is determined that the program has occurred, the program proceeds to step S13.

  In step S12, the detection unit 31 outputs position information of the object, and in step S13, position information indicating that the detection unit 31 has not recognized the object (in the above example, x_start = y_start = x_end = y_end = 0). ) Is output. The position information output from the detection unit 31 in step S12 or step S13 is used for region determination in step S5.

  Next, FIG. 8 is a flowchart for explaining specific contents of the brightness adjustment operation by the brightness adjustment unit 234 in step S6 of the flowchart of FIG.

  First, in step S5 in FIG. 7, the luminance adjustment unit 234 performs region determination, and then in step S20, it is determined whether or not position information is output from the recognition unit 3. If it is determined that position information is output (YES in step S20), the program proceeds to step S21. If it is determined that position information is not output (NO in step S20), the program proceeds to step S32. To do.

  In step S21, a specific region for brightness adjustment is extracted using the result of region determination in step S5 of FIG. In step S22, the position information in the previous frame and the luminance related to the position information are read from the storage unit 6. And it is determined whether the area | region extracted in step S21 is an area | region where the recognition part 3 recognized the target object in the frame before one. If it is determined that the object is a recognized area (YES in step S22), the program proceeds to step S23. If it is determined that the object is not a recognized area (NO in step S22), the program The process proceeds to S24.

  In step S23, the brightness of the extracted area is adjusted to the brightness in the previous frame. Thereafter, the program proceeds to step S33.

  In step S24, a luminance value representative of the extracted area is calculated. The luminance value representing the extracted region can be set to an intermediate value or an average value of the histogram by creating a histogram of luminance values for each pixel in the region.

  In step S25, it is determined whether or not the luminance value calculated in step S24 exceeds the threshold value. If it is determined that the luminance value exceeds the threshold value (YES in step S25), the program proceeds to step S26, and is below the threshold value (step S25). If it is determined as NO), the program proceeds to step S29.

  In step S <b> 26, it is determined whether or not the region extracted in step S <b> 21 is a region where the object is recognized by the recognition unit 3. If it is determined that the object is a recognized area (YES in step S26), the program proceeds to step S27. If it is determined that the object is not a recognized area (NO in step S26), the program is executed. Goes to step S28.

  In step S27, the brightness value of the extracted area is not adjusted (the gain is fixed). Thereafter, the program proceeds to step S33. On the other hand, in step S28, the luminance value of the extracted area is lowered by a predetermined value α. This predetermined value α can be arbitrarily set. Thereafter, the program proceeds to step S33.

  On the other hand, also in step S29, it is determined whether or not the region extracted in step S21 is a region in which the object is recognized by the recognition unit 3. If it is determined that the object is a recognized area (YES in step S29), the program proceeds to step S30. If it is determined that the object is not a recognized area (NO in step S29), the program is executed. Moves to step S31.

  In step S30, the luminance value of the extracted area is increased by a predetermined value γ. This predetermined value γ is set to about half of the predetermined value β in step S31. Thereafter, the program proceeds to step S33. On the other hand, in step S31, the luminance value of the extracted area is increased by a predetermined value β. The predetermined value β is set so that the adjusted luminance value is equivalent to the luminance value lowered by the predetermined value α in step S28. Thereafter, the program proceeds to step S33.

  On the other hand, in step S32, general luminance adjustment is performed on the entire near-infrared light image. Thereafter, the program proceeds to step S32.

  In step S33, it is determined whether or not the luminance adjustment has been performed for all the regions determined, and if it is determined that the luminance adjustment has been performed for all the regions (YES in step S33), the program is the step of FIG. The process proceeds to S8, and if it is determined that there is an area where brightness adjustment has not been performed (NO in Step S33), the program proceeds to Step S34, and after the next area is extracted in Step S34, the process returns to Step S22.

  In the image processing apparatus 5 of the present embodiment configured as described above, the brightness adjustment unit 234 is configured to display the near-infrared light image based on the first position information output as a result of the object recognition processing by the recognition unit 3. The second brightness of the second area having a lower brightness than the first brightness of the first area is adjusted based on the first brightness.

  Thereby, the brightness | luminance of the area | region where the near-infrared light from the infrared light irradiation part 1 is not irradiated can be adjusted based on the brightness | luminance of the area | region where the near-infrared light is irradiated, and object recognition The image processing apparatus 5 that can expand the possible range can be realized.

  Here, the luminance adjustment unit 234 has substantially the same luminance as the first luminance in the near-infrared light image based on the first position information, and the first region and the second region. Since the brightness of the third region, which is different from each other, is lower than the brightness of the third region related to the near-infrared light image signal, the near-infrared light from the infrared light irradiation unit 1 is irradiated, but the object Therefore, the brightness of the area in which the image is not recognized can be lowered, thereby suppressing the occurrence of whiteout or the like.

  The recognizing unit 3 recognizes a fourth region in which the second object having a luminance lower than the first luminance is imaged in the near-infrared light image, and a near red color related to the fourth region. The second position information in the external light image is output, and the brightness adjusting unit 234 determines the brightness of the fourth area based on the second position information from the brightness of the fourth area related to the near-infrared light image signal. Therefore, the brightness adjustment can be performed in consideration of both the possibility that the object recognition processing result by the recognition unit 3 is based on noise and the possibility that the object is actually recognized.

  Further, the sensor 22 of the imaging unit 2 generates a near-infrared light image signal for each frame, the storage unit 6 stores the first position information and the first luminance for each frame, and the luminance adjustment unit 234 stores the memory. The first position information and the first luminance related to the latest frame stored in the unit 6 are read out, and the first luminance related to the frame on which the luminance adjustment is performed is adjusted to the first luminance related to the latest frame. Therefore, it is possible to maintain the luminance related to the object recognized by the recognition unit 3 in the latest frame, and thus it is possible to reliably perform the object recognition process by the recognition unit 3 continuously.

  The embodiment of the present invention has been described in detail above with reference to the drawings. However, the specific configuration is not limited to the embodiment and the example, and the design change is within a range not departing from the gist of the present invention. Are included in the present invention.

  As an example, in the above-described embodiment, the camera 2 and the recognition unit 3 are configured by an integral arithmetic element or the like, but the image processing ECU 7 is separated from the camera 2 as shown in FIG. It is also possible to provide the recognition unit 3 in the ECU 7. In FIG. 6, the recognition unit 3 may be disposed outside the image processing device 5.

S In-vehicle camera system V Vehicle 1 Floodlight (infrared light irradiation part)
2 Camera (imaging part)
3 Recognition Unit 5 Image Processing Device 6 Storage Unit 22 Sensor 23 Signal Processing Unit 234 Brightness Adjustment Unit

Claims (5)

An infrared light irradiation unit that irradiates infrared light toward the periphery of the vehicle;
An imaging unit that captures an infrared light image around the vehicle and outputs an infrared light image signal;
Based on the infrared light image signal, the first region in which the first object is imaged is recognized in the infrared light image, and the first region in the infrared light image relating to the first region is recognized. A recognition unit that outputs position information;
The imaging unit, based on the first position information, the second luminance of the second region having a luminance lower than the first luminance of the first region in the infrared light image, An image processing apparatus comprising: a luminance adjusting unit that adjusts based on the first luminance.   The brightness adjusting unit has a brightness substantially the same as the first brightness in the infrared light image based on the first position information, and the first area and the second area 2. The image processing apparatus according to claim 1, wherein the brightness of a third region different from any of the third regions is lower than the brightness of the third region related to the infrared light image signal. The recognizing unit recognizes a fourth region in which the second object having a luminance lower than the first luminance is imaged in the infrared light image, and the infrared light related to the fourth region. Output the second position information in the image,
The brightness adjustment unit increases the brightness of the fourth area based on the second position information, compared to the brightness of the fourth area related to the infrared light image signal. An image processing apparatus according to 1. The imaging unit generates the infrared image signal in frame units,
A storage unit that stores the first position information and the first luminance in units of frames;
The luminance adjustment unit reads the first position information and the first luminance related to the latest frame stored in the storage unit, and determines the first luminance related to the frame on which the luminance adjustment is performed. The image processing apparatus according to claim 1, wherein the first luminance relating to the frame is adjusted to the first luminance. An infrared light irradiation unit that irradiates infrared light toward the periphery of the vehicle;
An image processing method in an image processing apparatus having an imaging unit that captures an infrared light image around the vehicle and outputs an infrared light image signal,
Based on the infrared light image signal, the first region in which the first object is imaged is recognized in the infrared light image, and the first region in the infrared light image relating to the first region is recognized. Output location information,
Based on the first position information, the second luminance of the second region having a luminance lower than the first luminance of the first region in the infrared light image is set to the first luminance. An image processing method characterized by adjusting based on the above.