JP2019214348A - Object recognition device - Google Patents

Abstract

To provide an object recognition device which can reduce a processing load by using a camera having sensitivity on both a visible light region and a near-infrared region.SOLUTION: An object recognition device 100 includes: an imaging part 10 which has sensitivity on both a visible light region and a near-infrared region and images the periphery of a vehicle; an object shape determination part 21 which determines presence/absence of an object present in the periphery of the vehicle on the basis of an image obtained by imaging with the imaging part 10; a position calculation part 22 which calculates a position of the object on the basis of a determination result of the object shape determination part 21; a visual observation possibility determination part 23D which determines whether or not to recognize the object in the image by visual observation of a driver on the basis of a calculation result of the position calculation part 22; and an output determination part 24 which determines an output pattern on the basis of the determination result of the visual observation possibility determination part 23D.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an object recognition device.

  2. Description of the Related Art Conventionally, there is a pedestrian recognition device that recognizes a pedestrian in front of a vehicle using imaging results of a visible light camera and a far-infrared camera mounted on the vehicle (for example, see Patent Document 1). This pedestrian recognition device is used especially in a dark road environment such as at night or in a tunnel, and has a high pedestrian specific recognition rate based on an imaging result of a far-infrared camera and a low pedestrian specific recognition rate using a visible light camera. Is notified to the driver.

JP 2006-338594 A

  However, in the technique disclosed in Patent Document 1, it is necessary to use two types of cameras, a visible light camera and a far-infrared camera. That is, it was necessary to determine the recognition rate for each of the two types of camera images. Therefore, complicated processing such as simultaneous recognition of two types of camera images is required, and the processing load required for the system is increased.

  The present invention has been made in view of the above problems, and provides an object recognition device capable of reducing a processing load by using a camera having sensitivity in both a visible light region and a near infrared region. With the goal.

  The object recognition device according to the present invention has sensitivity in both the visible light region and the near-infrared region, and an imaging unit that captures an image around the vehicle, based on an image obtained by capturing the imaging unit. An object shape determination unit that determines the presence or absence of an object present around the vehicle, a position calculation unit that calculates the position of the object based on a determination result of the object shape determination unit, and a calculation result of the position calculation unit And a visual determination unit that determines whether or not the object in the image is visually recognizable by a driver, and an output determination unit that determines an output pattern based on a determination result of the visual determination unit. And characterized in that:

  According to the present invention, the processing load can be reduced by using a camera having sensitivity in both the visible light region and the near infrared region.

FIG. 2 is a hardware block diagram illustrating a state in which the object recognition device according to the first embodiment is mounted on a vehicle. FIG. 2 is a functional block diagram illustrating a functional configuration of the object recognition device according to the first embodiment. FIG. 3 is a diagram illustrating an example of an object in an image captured in a dark environment such as at night. It is a figure showing an example of pattern image information. FIG. 4 is a diagram illustrating an example of an icon displayed on an image. 5 is a flowchart illustrating an overall flow of a process performed in the first embodiment. 7 is a flowchart illustrating a detailed flow of an object shape determination process illustrated in FIG. 6. 7 is a flowchart showing a detailed flow of an object position determination process shown in FIG. FIG. 7 is a flowchart showing a detailed flow of a visual availability determination process shown in FIG. 6. 7 is a flowchart showing a detailed flow of a variable light distribution headlight irradiation process shown in FIG. 6. FIG. 2 is an explanatory diagram illustrating a scene in which the object recognition device according to the first embodiment is applied to a pedestrian (part 1). FIG. 3 is an explanatory diagram illustrating a scene in which the object recognition device according to the first embodiment is applied to a pedestrian (part 2). It is explanatory drawing which shows the irradiation image by the headlight provided with the function to switch high beam / low beam automatically (the 1). It is explanatory drawing which shows the irradiation image with the headlight provided with the function to switch high beam / low beam automatically (the 2). It is a figure showing an example of a bird's-eye view picture which makes a viewpoint above an own vehicle a viewpoint. FIG. 5 is an explanatory diagram illustrating a modification of the object recognition device according to the first embodiment.

  Hereinafter, specific embodiments of the object recognition device according to the present invention will be described with reference to the drawings.

(Description of hardware configuration of object recognition device)
FIG. 1 is a hardware block diagram illustrating a state in which the object recognition device 100 according to the first embodiment is mounted on a vehicle.

  The object recognition device 100 includes an imaging unit 10, an electronic control unit (ECU) 20, an augmented reality (AR) head-up display 31, a liquid crystal display 32, a near-infrared light 40, and a variable light distribution headlight 50. And

  The imaging unit 10 has sensitivity in a near-infrared region (for example, around 800 nm to 950 nm) and a visible light region (around 380 nm to 800 nm), and captures an image around the vehicle. The imaging unit 10 includes a lens 11, an imaging element 12, and a signal processing adjustment IC 13.

  The lens 11 refracts near-infrared light and visible light with a predetermined refractive power to form an image on the image sensor 12.

  The imaging element 12 is configured by a photoelectric conversion element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and converts the formed light into an image I. The image sensor 12 outputs an image signal converted into an electric signal.

  The signal processing adjustment IC 13 adjusts the gain of the image signal and converts it into a predetermined image signal. The converted image signal is output as an adjusted image signal.

  Note that the signal processing adjustment IC 13 outputs an imaging parameter adjustment signal for adjusting the aperture and the shutter speed to set the exposure amount to the lens 11 and the imaging element 12.

  Note that the imaging unit 10 is installed near the rear side of the windshield of the vehicle, for example, near the rear side of the room mirror, with the front of the vehicle facing forward.

  The ECU 20 includes, for example, a microcomputer, and outputs an imaging parameter adjustment instruction signal to the imaging unit 10 to set imaging parameters of the imaging unit 10. The ECU 20 processes the adjusted image signal captured using the imaging parameters to recognize an object existing around the vehicle. Here, the object is, for example, an animal (a deer, a raccoon, etc.), a motorcycle (a bicycle, a motorcycle, etc.), a pedestrian (an adult, a child, a wheelchair, a stroller, etc.), and other vehicles (small vehicles, ordinary vehicles, large vehicles). Etc.).

  When recognizing the object, the ECU 20 outputs an object recognition signal indicating the recognition to the AR head-up display 31 or the liquid crystal display 32.

  The ECU 20 outputs, to the near-infrared light 40, a near-infrared light amount setting signal for setting the irradiation light amount of the near-infrared light.

  The ECU 20 outputs an irradiation position setting signal for setting the irradiation position of visible light to the variable light distribution type headlight 50.

  The AR head-up display 31 is, for example, a combiner-type display using a transparent panel, and is arranged at a position where a driver can visually recognize the display.

  The liquid crystal display 32 is, for example, a rectangular flat panel display, and is arranged at a position where a driver can visually recognize the flat panel display.

  The near-infrared light 40 includes, for example, a near-infrared light device installed near the variable light distribution type headlight 50. The near-infrared light 40 may also be used as the variable light distribution headlight 50 of the vehicle.

  The near-infrared light 40 illuminates the traveling direction of the vehicle and irradiates the amount of near-infrared light indicated by the near-infrared light amount setting signal output from the ECU 20 toward the inside of the imaging range of the imaging unit 10. I do. The near-infrared light 40 outputs to the ECU 20 an illumination light extinguishing signal indicating whether or not the near-infrared light 40 is on. This illumination light extinguishing signal is used to confirm the lighting state of the near infrared light 40.

  The variable light distribution headlight 50 illuminates the traveling direction of the vehicle, has a plurality of LED light sources, and emits visible light toward a position specified by a visible light irradiation position setting signal output from the ECU 20. . The variable light distribution headlight 50 outputs to the ECU 20 an illumination light extinguishing signal indicating whether or not the variable light distribution headlight 50 is in a lighting state. This illumination light extinguishing signal is used to confirm the lighting state of the variable light distribution type headlight 50.

(Explanation of the functional configuration of the object recognition device)
FIG. 2 is a functional block diagram illustrating a functional configuration of the object recognition device 100 according to the first embodiment.

  The object recognition device 100 includes an imaging unit 10, an ECU 20, a display unit 30, and a variable light distribution headlight 50.

  The ECU 20 includes an object shape determination unit 21, an object position determination unit 22 (a position calculation unit), a visual availability determination unit 23, an output determination unit 24, a light distribution control unit 25, and a display control unit 26.

  The object shape determination unit 21 determines the presence or absence of an object existing around the vehicle based on an image I obtained by the imaging unit 10 capturing an image around the vehicle.

  The object shape determination unit 21 includes a matching determination unit 21A and a database 21B. The matching determination unit 21A performs a matching process between the image I obtained by imaging the periphery of the vehicle by the imaging unit 10 and the content stored in the database 21B. Specifically, the template stored in the database 21B is scanned while being superimposed on the image I. Then, the similarity is obtained at each position, an area similar to the template is searched for in the image I, and the type of the object (for example, raccoon or bicycle) is recognized.

  The object position determination unit 22 includes an object position measurement unit 22A, an object width / height measurement unit 22B, an object distance measurement unit 22C, and an object direction measurement unit 22D.

  The object position measurement unit 22A obtains position coordinates (X, Y) of the object extracted from the image I on the image I based on the recognition result of the object shape determination unit 21. The position coordinates (X, Y) are represented by two-dimensional coordinates with the position of the vehicle as the origin.

  The object width / height measurement unit 22B obtains a region (W × H) of the object on the image I based on the position coordinates (X, Y) measured by the object position measurement unit 22A using a known method. . Here, W indicates the width, and H indicates the height.

  The object distance measurement unit 22C converts, for example, the position coordinates (X, Y) into world coordinates, and obtains the distance from the origin of the world coordinates to the object.

  The object direction measuring unit 22D measures the direction of the object with respect to the traveling direction of the vehicle using a known method. Specifically, the object direction measurement unit 22D measures whether the object is facing the same direction as the traveling direction of the vehicle, rightward, leftward, or faces the traveling direction. For example, the orientation of a pedestrian's body is measured by applying a technology for detecting the orientation of a human face.

  The visual availability determination unit 23 includes a pixel number measurement unit 23A, a saturation detection unit 23B, an achromatic chromaticity calculation unit 23C, and a visual availability determination unit 23D.

  Based on the position coordinates of the object determined by the object position determination unit 22, the pixel number measurement unit 23A calculates the number of pixels in proportion to the area (W × H) of the region on the image I of the object using a known method. Ask. Here, W indicates the width, and H indicates the height.

  Next, with respect to the details of the saturation detecting unit 23B, a method of expressing a color by a numerical value will be described separately for an RGB format and a YUV format.

  In the case of the RGB format, the achromaticity described later increases as the amounts of the primary colors become equal, and R = G = B for a complete achromatic color, and R ≒ G ≒ B when the achromaticity is high. Using this characteristic, the saturation detection unit 23B calculates each value (0 to 255) of RGB (red, green, and blue) included in the region of the object.

  In the case of the YUV format, the achromaticity increases as the color information (U, V) other than the luminance V approaches 0, and in the case of a complete achromatic color, U = 0 and V = 0, and the achromaticity increases. In this case, U ≒ 0 and V ≒ 0. Using this characteristic, the saturation detecting unit 23B calculates Y (luminance), U (blue color difference component), and V (red color difference component) of the image data included in the object area.

  Note that the luminance V indicates the brightness of a color, and is represented by a value of 0 to 100 (%), and the brightness increases as the value increases. Note that 0% corresponds to black and 100% corresponds to white. The blue color difference component U and the red color difference component V are represented by 256 gradations of -128 to 128.

The achromaticity calculator 23C calculates the achromaticity (%) using the following equation (1).

  Here, the low chromaticity pixels include pixels whose achromaticity is higher than a predetermined value, in addition to completely achromatic pixels.

  FIG. 3 is a diagram illustrating an example of an object in an image captured in a dark environment such as at night. The visual availability determination unit 23D determines whether the achromaticity (%) exceeds a predetermined threshold (for example, 50%). When the achromaticity exceeds the threshold (achromaticity> 50%), it is determined that the object in the image I (the pedestrian shown on the right side in FIG. 3) is difficult to be visually recognized by the driver.

  On the other hand, when the achromaticity is equal to or less than the threshold (for example, achromaticity = 20%), it is determined that the object in the image I (a pedestrian shown on the left side in FIG. 3) can be visually recognized by the driver. You.

  The output determination unit 24 determines an output pattern based on the determination result of the visual availability determination unit 23D. An example of the output pattern is a light distribution pattern or a notification pattern.

  That is, when the achromaticity exceeds the threshold value in the visual availability determination section 23D, the output determination section 24 outputs the light distribution pattern to the light distribution control section 25 or outputs the notification pattern to the display control section 26. Or

  The light distribution pattern is determined by various conditions such as the type and specifications of the variable light distribution headlight 50, for example. Similarly, the notification pattern is determined based on various conditions such as, for example, the type and specification of the display unit 30.

  The light distribution control unit 25 controls the light distribution of the variable light distribution headlight 50 based on the determination result of the output determination unit 24. That is, the light distribution control unit 25 adjusts parameters such as the range, direction, and light amount to be irradiated by the headlight based on the light distribution pattern output by the output determination unit 24, and maintains the adjusted values. I do. The light distribution control unit 25 changes the irradiation direction of the variable light distribution type headlight 50 so as to include the direction in which the object exists so that the driver can recognize the object visually.

  The display control unit 26 displays the pattern image information indicating the presence of the object on the display unit 30 based on the determination result of the output determination unit 24. That is, the display control unit 26 displays the pattern image information as shown in FIG. 4 based on the notification pattern output by the output determination unit 24.

  The pattern image information includes an annular shape having a notch (hereinafter, simply referred to as a “ring”). The cutout is provided so as not to overlap with the pedestrian on the image I (FIG. 4). The ring RI shown in FIG. 4 indicates a ring as an object detection frame displayed near the foot of the pedestrian.

  The pattern image information includes an icon IC to be displayed on the image (FIG. 4). The icon IC shown in FIG. 4 is an icon as a highlighted display displayed above the pedestrian's head. Icon types representing the types of icon ICs include, for example, automobiles, pedestrians, motorcycles, and animals (see the lower left side of FIG. 5). Note that the right side of FIG. 5 shows a state in which the ring RI and the icon IC are superimposed on an object such as a pedestrian, a two-wheeled vehicle, and an automobile on an actual camera image.

  The variable light distribution type headlight 50 has an irradiation unit 51 (FIG. 2). The irradiation unit 51 controls the lighting state of the variable light distribution type headlight 50 based on the adjusted parameter values adjusted by the light distribution control unit 25.

  Next, the overall processing flow of the embodiment described above will be described with reference to the flowchart of FIG.

(Explanation of the overall processing flow of the embodiment)
(Step S1) It is determined whether or not the light switch is turned on and the variable light distribution type headlight 50 is turned on. When the light switch is on, the process proceeds to step S2, and when the light switch is off, step S1 is repeated.

  (Step S2) Near-infrared light is emitted from the near-infrared light 40.

  (Step S3) The imaging unit 10 simultaneously captures the near-infrared image and the color image in accordance with the irradiation of the near-infrared light 40.

  (Step S4) The object shape determination unit 21 determines the presence or absence of an object existing around the vehicle based on the image I obtained by the imaging unit 10 and performs an object shape determination process for recognizing the shape. The detailed flow of this processing will be described later.

  (Step S5) Based on the recognition result of the object shape determination processing, it is determined whether or not an object exists around the vehicle. When it is determined that the object exists, the process proceeds to step S6, and when it is determined that the object does not exist, the process returns to step S3.

  (Step S6) When it is determined that an object is present, the object position determination unit 22 performs an object position determination process. The detailed flow of this processing will be described later.

  (Step S7) The visual availability determination section 23 performs visual availability determination processing based on the position coordinates determined by the object position determination section 22. The detailed flow of this processing will be described later.

  (Step S8) Based on the determination result of the visual availability determination processing, the output determination unit 24 determines the output pattern according to whether or not there is an object that is difficult for the driver to view. When there is an object that is difficult to visually check, the process proceeds to step S9; otherwise, the process returns to step S3.

  (Step S9) When the output determination unit 24 determines that there is an object that is difficult to see visually, the variable light distribution headlight irradiation process is performed. The detailed flow of this processing will be described later.

  (Step S10) It is determined whether or not the ignition of the vehicle is off. When the ignition is off, the process of FIG. 6 is ended, and when the ignition is on, the process returns to step S1. Here, the vehicle speed of the vehicle may be detected, and the process of FIG. 6 may be terminated when the vehicle speed is 0, and the process may return to step S1 when the vehicle speed is not 0.

  Next, a detailed flow of the object shape determination processing shown in FIG. 6 will be described with reference to the flowchart in FIG.

(Explanation of the flow of object shape determination processing)
(Step S11) Template matching is performed between the image I captured in step S3 and the template stored in the database 21B to recognize the type of the object.

  (Step S12) It is determined whether or not all objects in the image I have been recognized. When all objects have been recognized, the process returns to the main routine (FIG. 6). Otherwise, the process returns to step S11 to recognize another object.

  Next, a detailed flow of the object position determination process shown in FIG. 6 will be described with reference to the flowchart in FIG.

(Explanation of the flow of the object position determination process)
(Step S21) The position of the object on the image I captured in step S3 is calculated.

  (Step S22) The area of the object on the image I is calculated.

  (Step S23) The distance from the object on the image I to the host vehicle is calculated.

  (Step S24) The direction of the object on the image I is calculated.

  (Step S25) Based on the results calculated in steps S21 to S24, an object recognition result is output. Thereafter, the process returns to the main routine (FIG. 6).

  Next, the detailed flow of the visual availability determination processing shown in FIG. 6 will be described with reference to the flowchart in FIG.

(Explanation of flow of visual availability determination processing)
(Step S31) The number of pixels proportional to the area of the object region is obtained.

  (Step S32) The color information (RGB value or YUV value) of each pixel is analyzed.

  (Step S33) The saturation of the color information is calculated.

  (Step S34) It is determined whether or not the saturation has been calculated for all the pixels in the object region. If the saturation has been calculated for all the pixels, the process proceeds to step S35. Otherwise, the process returns to step S32 to analyze the color information of another pixel.

  (Step S35) Achromaticity (%) is calculated. Thereafter, the process returns to the main routine (FIG. 6).

  Next, a detailed flow of the variable light distribution headlight irradiation processing shown in FIG. 6 will be described with reference to the flowchart in FIG. Here, the variable light distribution headlight irradiation processing will be described assuming the scenes of FIGS. 11 and 12. 11 and 12 show scenes in which the object recognition device 100 is applied to a pedestrian who is walking on a road at night.

(Explanation of the flow of the variable light distribution headlight irradiation process)
(Step S41) The position (x, y) of the pedestrian (FIG. 11) is calculated as a detected object that is difficult for the driver to see.

  (Step S42) The area of the rectangular area RA (FIG. 12) including the pedestrian is calculated.

  (Step S43) The range to be irradiated by the variable light distribution type headlight 50 is calculated.

  (Step S44) The direction to be irradiated by the variable light distribution type headlight 50 is calculated.

  (Step S45) The amount of light to be irradiated by the variable light distribution type headlight 50 is calculated.

  (Step S46) The lighting state of the variable light distribution type headlight 50 is controlled based on the results calculated in steps S43 to S45. Thereafter, the process returns to the main routine (FIG. 6).

  In the process shown in FIG. 6 described above, an example in which the variable light distribution headlight irradiation process (step S9) is performed when the output determination unit 24 determines that there is an object that is difficult to view (YES in step S8). Was. The present invention is not limited to this mode. As shown in FIG. 16, when the output determination unit 24 determines that there is an object that is difficult to visually recognize (YES in step S58), the display control unit 26 causes the display unit 30 to display the display unit 30. 5 and 15 are displayed (step S59).

  FIG. 13 and FIG. 14 are explanatory diagrams showing irradiation images of general headlights having a function of automatically switching between high beam and low beam.

  In the scene shown in FIG. 13, the left headlight without a pedestrian does not turn on the high beam.

  FIG. 14 shows a scene in which a headlight having a function capable of finely variable light distribution control is applied. In the scene shown in FIG. 14, the screen is divided in the vertical direction, and in step S8 shown in FIG. 6, it is determined where the pedestrian exists in the divided area. If it is determined that there is an object that is difficult for the driver to see, only the area where the pedestrian exists is illuminated by the headlight. In the scene shown in FIG. 14, care is taken not to illuminate the area other than the area where the pedestrian exists as much as possible so that the driver of the oncoming vehicle facing the own vehicle is not dazzled.

  In the above-described object recognition apparatus 100 of the first embodiment, the achromaticity is calculated using a camera having sensitivity in both the visible light region and the near infrared region.

  That is, there is no need to determine the achromaticity for each of the two types of camera images as in the related art. Therefore, there is no need for complicated processing such as recognizing two types of camera images at the same time. Therefore, the processing load required for the system can be reduced.

  As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, since the embodiments are merely examples of the present invention, the present invention is not limited only to the configurations of the embodiments, and the present invention is not limited thereto. Even if there is a change in the design or the like without departing from the gist, it is needless to say that the change is included in the present invention.

  In the first embodiment described above, an example in which the icon IC is displayed on the display unit 30 has been described (FIG. 5). The present invention is not limited to this mode, and an object that is visible to the naked eye of the driver and an object that is hardly visible to the naked eye may be displayed in different colors. The object visible to the naked eye may be displayed in, for example, a cool color such as blue-green, blue, or blue-violet, or a neutral color such as yellow-green, green, purple, or achromatic. Objects that are difficult to see with the naked eye may be displayed in conspicuous colors such as pink, red, yellow, and orange.

  In the first embodiment described above, an example in which the ring RI and the icon IC are superimposed on the camera image and displayed on an object such as a pedestrian (FIG. 5). The present invention is not limited to this mode, and the map may be displayed by superimposing the ring RI and the icon IC on the bird's-eye view image having the viewpoint above the own vehicle (FIG. 15). In the scene shown in FIG. 15, when the distance from the host vehicle to the object exceeds a certain distance (for example, about 50 m), a difference appears in how the driver views the object.

  In the first embodiment, an example in which a combiner-type display is used as the AR head-up display 31 has been described. The present invention is not limited to this mode, and the AR head-up display 31 may be, for example, a system that directly displays an image on a front window.

10 imaging unit 21 object shape determination unit 22 object position determination unit (position calculation unit)
23D: Visual availability determination unit 24: Output determination unit 25: Light distribution control unit 26: Display control unit 30: Display unit 31: AR head-up display 32: Liquid crystal display 50: Variable light distribution type headlight 100: Object recognition device IC: Icon (pattern image information)
RI ring (pattern image information)

Claims (4)

An imaging unit that has sensitivity in both the visible light region and the near-infrared region, and images the periphery of the vehicle,
An object shape determination unit that determines the presence or absence of an object present around the vehicle based on an image obtained by the imaging unit;
Based on the determination result of the object shape determination unit, a position calculation unit that calculates the position of the object,
A visual availability determination unit configured to determine whether the object in the image is visually recognizable by a driver based on a calculation result of the position calculation unit;
An object recognition device, comprising: an output determination unit configured to determine an output pattern based on a determination result of the visual availability determination unit. The object recognition device according to claim 1,
An object recognition device, comprising: a light distribution control unit that controls light distribution of a headlight based on a determination result of the output determination unit. The object recognition device according to claim 2,
The light distribution control unit,
An object recognizing device, wherein an irradiation direction of the headlight is changed so as to include a direction in which the object is present, based on a determination result of the output determination unit. The object recognition device according to claim 1,
An object recognition device, comprising: a display control unit that displays pattern image information indicating the presence of the object on a display unit based on a determination result of the output determination unit.