JP2023059552A - Driving support device and driving support method - Google Patents

Abstract

To provide a driving support device and a driving support method that facilitate detection of thermal-radiating objects through a curved mirror.SOLUTION: A driving support device 100 includes: an image capture control unit 141 that acquires a thermal image photographed by a far-infrared camera provided in a vehicle; a curved mirror detection unit 142 that detects a mirror surface range of the curved mirror in the thermal image; an image adjusting unit 143 that adjusts at least one of gain and contrast within the mirror surface range; an object detection unit 144 that detects thermal radiating objects within the mirror surface range in which at least one of gain and contrast is adjusted; and a notification control unit 145 that notifies a driver of the vehicle when the thermal radiating object is detected within the mirror surface range.SELECTED DRAWING: Figure 1

Description

TECHNICAL FIELD The present invention relates to a driving support device and a driving support method, and more particularly to technology for driving support using a far-infrared camera.

Driving support systems using visible light cameras and far-infrared cameras have become widespread, and techniques have been proposed for detecting objects reflected by curved mirrors using visible light cameras. Patent Literature 1 discloses a technique of detecting a curved mirror from a visible light image and detecting that an obstacle is reflected on the curved mirror. Further, Japanese Patent Application Laid-Open No. 2002-200000 describes that the mirror surface portion of a curved mirror is displayed in an enlarged manner to increase the resolution.

The driver of the vehicle can confirm the direction of the blind spot by the specular reflection in the curved mirror. However, when checking pedestrians who do not have a light source during a period of low light intensity (for example, evening to night), it is difficult to check the mirror surface range of the curved mirror visually or with a visible light camera. Also, when using a far-infrared camera that acquires a thermal image, the temperature difference of the object is acquired as a thermal image, so the area of the mirror surface range of the curved mirror becomes dark, and the thermal radiation object (e.g. pedestrian) is curved. It was difficult to detect through mirrors.

JP 2020-154698 A Japanese Patent Application Laid-Open No. 2018-007038

As described above, there is a problem that it is difficult to detect a heat emitting object through a curved mirror.

SUMMARY OF THE INVENTION The present invention has been made to solve such problems, and an object of the present invention is to provide a driving assistance device and a driving assistance method that facilitate detection of a heat emitting object through a curved mirror.

The present invention includes a photographing control unit that acquires a thermal image photographed by a far-infrared camera provided in a vehicle;
a curved mirror detection unit that detects a mirror surface range of the curved mirror in the thermal image;
an image adjustment unit that adjusts at least one of gain and contrast within the specular range;
an object detection unit that detects a thermally radiating object within the specular range for which at least one of gain and contrast is adjusted by the image adjustment unit;
and a notification control unit that notifies a driver of the vehicle when the heat emitting object is detected within the mirror surface range.

The present invention comprises a thermal image acquiring step of acquiring a thermal image captured by a far-infrared camera provided in a vehicle;
a curve mirror detection step of detecting a mirror surface range of the curve mirror in the thermal image;
an image adjustment step of adjusting at least one of gain and contrast within the detected specular range;
an object detection step of detecting a heat emitting object within the specular range in which at least one of gain and contrast is adjusted;
and a notification step of notifying the driver of the vehicle when the heat emitting object is detected within the mirror surface range.

ADVANTAGE OF THE INVENTION According to this invention, the driving assistance apparatus and driving assistance method which facilitate the detection of a thermal radiation object through a curved mirror can be provided.

1 is a block diagram showing the configuration of a driving support device according to Embodiment 1; FIG. 4 is a flow chart showing the flow of a driving assistance method according to the first embodiment; FIG. 2 is a block diagram showing the configuration of a driving support device according to a second embodiment; FIG.

(Embodiment 1)
FIG. 1 is a block diagram showing the configuration of a driving assistance device 100 according to the first embodiment. The driving support device 100 is mounted on a vehicle and is a device that supports driving of the vehicle driver. The driving support device 100 may be a night vision system. The driving assistance device 100 includes a far-infrared camera 110 , a display section 120 , an illuminance sensor 130 and a control section 140 . Moreover, the driving support device 100 may further include a speaker (not shown).

Far-infrared camera 110 captures a thermal image of the exterior of the vehicle. A far-infrared camera 110 is provided in a vehicle, and is installed, for example, in the front grill. The display unit 120 is a display device such as a display, and displays a thermal image obtained by the far-infrared camera 110 . The thermal image to be displayed has its gain and contrast adjusted by an image adjustment unit 143 of the control unit 140, which will be described later. Further, in the displayed thermal image, a heat emitting object, which will be described later, may be highlighted. The illuminance sensor 130 is a sensor that detects the illuminance outside the vehicle. Note that the driving support device 100 does not have to include the illuminance sensor 130 .

The control unit 140 includes a memory such as a CPU (Cnetral Processing Unit) that performs various processes in the driving support device 100 and a RAM (Random Access Memory), etc., and executes various processes according to programs. The control unit 140 includes a shooting control unit 141, a curved mirror detection unit 142, an image adjustment unit 143, an object detection unit 144, a notification control unit 145, and an illuminance detection unit 146 as functions. Each function of the control unit 140 may be realized by reading a program (not shown) into a memory such as a RAM (not shown) and executing it by a processor (not shown). Also, each function of the control unit 140 may be implemented as dedicated hardware.

Each function provided in the control unit 140 will be described below. The imaging control unit 141 controls imaging of a thermal image by the far-infrared camera 110 and acquires the thermal image captured by the far-infrared camera 110 .

The curved mirror detection unit 142 detects the mirror surface range of the curved mirror in the thermal image captured by the far-infrared camera 110 , that is, the thermal image acquired from the far-infrared camera 110 by the imaging control unit 141 . Specifically, the curved mirror detection unit 142 performs template matching processing, detection processing based on machine learning such as HOG (Histogram of Oriented Gradients) features, or ellipse detection processing using Hough transform to detect the curved mirror. The shape is detected, and the inside of the elliptical shape in the shape of the curved mirror is detected as the mirror surface range of the curved mirror.

The image adjustment unit 143 adjusts the thermal image acquired by the imaging control unit 141 from the far-infrared camera 110 . The image adjuster 143 adjusts at least one of gain and contrast within the specular range detected by the curved mirror detector 142 . The gain is, for example, a digital gain adjusted by software calculation of luminance value data. Note that the image adjustment unit 143 may perform processing for adjusting the gain on the far-infrared camera 110 side. Processing for adjusting contrast is also called processing for enhancing contrast. The process of adjusting at least one of gain and contrast includes a process of increasing at least one of gain and contrast and a process of decreasing at least one of gain and contrast. For example, there are cases where the gain is lowered to emphasize the contrast, and cases where the contrast is enhanced by partially adjusting the intensity of the contrast. Specifically, the image adjuster 143 adjusts the contrast and brightness within the specular range so that a heat emitting object, which will be described later, can be easily detected. After adjusting the contrast and brightness, the image adjustment unit 143 may further perform edge enhancement processing within the specular range.

The image adjustment performed by the image adjustment unit 143 allows the heat emitting object reflected on the mirror surface of the curved mirror to be visible. When the image-adjusted thermal image is displayed on the display unit 120, the driver can check the display unit 120 and quickly notice a pedestrian or the like reflected in the curved mirror.

The object detection unit 144 performs processing for detecting a thermal radiation object from the thermal image acquired by the imaging control unit 141 from the far-infrared camera 110 . The object detection unit 144 detects a thermal emission object within the specular range in which at least one of gain and contrast is adjusted by the image adjustment unit 143 . By heat emitting object is meant an object with heat radiation. Thermal emitting objects, also called hot objects, emit far infrared rays. A thermally radiating object may be an object that has a high temperature compared to surrounding environmental objects such as roads and buildings. Heat emitting objects are specifically pedestrians, cyclists (persons riding bicycles), animals and the like.

Detection of a heat emitting object is performed by machine learning, for example, HOG features. The object detection unit 144 may detect a person in the thermal image including the specular range using a dictionary that detects a person based on the thermal image. A person may be a pedestrian or a cyclist.

The notification control unit 145 notifies the driver of the vehicle when the imaging control unit 141 detects a thermal radiation object in the thermal image acquired from the far-infrared camera 110 . In addition, the notification control unit 145 notifies the driver of the vehicle when the image adjustment unit 143 detects a heat radiation object within the mirror surface range in which at least one of the gain and the contrast is adjusted. The notification control unit 145 may notify the driver of the vehicle particularly when a person is detected within the mirror surface range.

The notification control unit 145 notifies the driver of the vehicle of the presence of the heat emitting object through a display screen, sound, or the like. When the notification is made on the display screen, the notification control unit 145 displays a detection frame surrounding the detected thermal radiation object on the thermal image displayed on the display screen. In addition, the notification control unit 145 highlights the specular range and the heat emitting object within the specular range. For example, the notification control unit 145 may display an emphasis frame surrounding the specular range. When making a notification by sound, the driving support device 100 may output a warning sound from a speaker (not shown).

The illuminance detection unit 146 acquires information indicating the illuminance around the vehicle based on the detection result of the illuminance sensor. The illuminance detection unit 146 detects that the illuminance around the vehicle is less than a predetermined threshold based on the detection result of the illuminance sensor 130 . When the illuminance is less than the predetermined threshold, the driving support device 100 executes processing by the photographing control unit 141 , the curved mirror detection unit 142 , the image adjustment unit 143 , the object detection unit 144 and the notification control unit 145 . The illuminance detection unit 146 sets an illuminance threshold such that the illuminance around the vehicle is at night and the illumination of street lights and buildings is low, and detects that the illuminance around the vehicle is less than the predetermined threshold. The threshold is, for example, 10 lux.

Note that even when the control unit 140 does not include the illuminance detection unit 146, it can be included in the driving assistance device 100 according to the first embodiment. The driving support device 100 may start processing in response to an input from the driver, or may start processing in response to the state of the headlights of the vehicle as described later.

FIG. 2 is a flow chart showing the flow of the driving assistance method according to the embodiment. First, the imaging control unit 141 of the driving support device 100 starts processing for acquiring a thermal image from the far-infrared camera 110 (step S101). The process of step S101 is started when the engine and power of the vehicle are turned on. In addition, the process of step S101 acquires the on/off state of the headlights by the vehicle driver's operation or the vehicle's CAN (Controller Area Network) communication. Alternatively, it may be started based on the illuminance around the vehicle detected by the illuminance detection unit 146 . The thermal image captured by the far-infrared camera can be visually recognized by the driver of the vehicle at any time of the day or night. and so on. Further, in parallel with the processing from step S102 to step S106 in FIG. 2, notification of the object detection result in the entire thermal image is also executed.

Next, the curve mirror detection unit 142 of the driving support device 100 starts processing for detecting the mirror surface range of the curve mirror from the thermal image (step S102). When the specular range of the curved mirror is detected (Yes in step S103), the image adjusting unit 143 of the driving support device 100 performs image adjustment in the specular range within the thermal image (step S104). On the other hand, if the mirror surface range of the curved mirror is not detected (No in step S103), the process proceeds to step S107, which will be described later.

After performing the image adjustment in the mirror surface range of the curved mirror, the object detection unit 144 of the driving support device 100 performs processing to detect a thermal radiation object from the thermal image. If a heat emitting object is detected (Yes in step S105), the notification control unit 145 of the driving assistance device 100 notifies the driver of the vehicle that a heat emitting object exists (step S106). On the other hand, if no thermal radiation object is detected (No in step S105), it is determined whether or not to end acquisition of the thermal image (step S107). If the acquisition of the thermal image is not finished, the process returns to step S103, and if the acquisition of the thermal image is finished, the driving support method is finished.

When the illuminance around the vehicle is less than a predetermined threshold value, that is, at night and when the illuminance around the vehicle is particularly low, the driving support device 100 uses a curved mirror to detect a heat radiator having no light source such as a person. If the condition is such that the visual observation is difficult, the processing from step S102 to step S106 may be executed. As a result, the above-described driving support method can be executed in a time zone (for example, evening to nighttime) when it is difficult to detect an object from the curved mirror using visible light.

In this case, before step S102, there is a branch to determine whether the illuminance around the vehicle is less than the predetermined threshold. If so, the process proceeds to step S107.

Further, the driving support device 100 determines whether or not the vehicle is positioned around a curved mirror, particularly whether or not the curved mirror exists within a predetermined distance (for example, within 50 m) in the traveling direction of the vehicle. The processing from step S102 to step S106 may be executed according to the result. As a result, the mirror surface range of the curved mirror can be detected with high accuracy. The driving support device 100 determines whether or not a curved mirror exists around the vehicle based on GPS (Global Positioning System) information and map information of the vehicle, for example. The driving support device 100 may determine that a curved mirror exists around an intersection or a curve based on map information. Also, the map information may include information about the position of the curved mirror.

Further, when the vehicle is positioned around a curved mirror and a circular or elliptical object is detected at an assumed position, the curved mirror detection unit 142 described above detects the object in the thermal image. A region may be detected as the specular extent of a curved mirror. Thereby, the curve mirror detection unit 142 can accurately detect the mirror surface range of the curve mirror.

In this case, before step S102, a decision is made as to whether or not the vehicle is positioned near a curved mirror or whether or not a curved mirror exists within a predetermined distance in the direction in which the vehicle is traveling, and the result is Yes. If so, the process proceeds to step S102, and if determined to be No, the process proceeds to step S107.

The driving assistance device according to the first embodiment detects the mirror surface range of the curved mirror from the thermal image, and performs processing for adjusting at least one of gain and contrast. Therefore, the driving assistance device according to the first embodiment can facilitate detection of a heat emitting object through the curved mirror. As a result, it is possible to detect a thermally radiating object from the mirror surface range of the curved mirror and notify the driver.

(Embodiment 2)
FIG. 3 is a block diagram showing the configuration of the driving assistance device 100a according to the second embodiment. Comparing FIG. 1 and FIG. 3, a visible light camera 150 is added and the controller 140 is replaced with a controller 140a. Comparing the control unit 140 and the control unit 140a, the shooting control unit 141 is replaced with the shooting control unit 141a, and the curved mirror detection unit 142 is replaced with the curved mirror detection unit 142a.

The visible light camera 150 photographs in the same direction as the far infrared camera 110 . The imaging regions of the far-infrared camera 110 and the visible light camera 150 overlap. The imaging control unit 141 a obtains a thermal image from the far-infrared camera 110 and obtains a visible light image from the visible light camera 150 .

The curve mirror detection unit 142a first detects the mirror surface area of the curve mirror from the visible light image. Template matching processing, detection processing based on HOG features, or ellipse detection processing using Hough transform may be performed in the same manner as when the specular area of the curved mirror is detected from the thermal image in the first embodiment. When the imaging area of the visible light camera 150 is wider than the imaging area of the far-infrared camera 110, the mirror surface of the curved mirror is applied to the range of the imaging area of the visible light camera 150 that matches the imaging area of the far-infrared camera 110. Regions may be detected.

Then, the curved mirror detection unit 142a detects, from the thermal image, an area corresponding to the specular area in the visible light image as the specular area in the thermal image. The curved mirror detection unit 142 a outputs the specular area in the thermal image to the image adjustment unit 143 . The processes in the image adjustment unit 143, the object detection unit 144, the notification control unit 145, and the illuminance detection unit 146 are the same as those in the first embodiment, so descriptions thereof will be omitted.

The reason why the visible light image is used to detect the mirror surface range of the curved mirror is that the curved mirror itself may not have enough thermal contrast to detect the curved mirror with a far-infrared camera. In such a case, the mirror surface of the curved mirror can be appropriately detected based on the visible light image. Even when the specular range of a curved mirror is detected using a visible light image, a thermal object can be detected by adjusting the gain and contrast of the specular range in a thermal image, as in the first embodiment. Therefore, the driving assistance device according to the second embodiment can also facilitate detection of a thermally radiating object through the curved mirror, as in the first embodiment.

Finally, the effects of the driving assistance devices according to the first and second embodiments will be described in detail. When detecting a thermally radiating object through a curved mirror, there is a problem that far infrared rays from the thermally radiating object are attenuated. Therefore, the thermal radiation object beyond the curved mirror is dark in the thermal image, and it is difficult to detect the thermal radiation object from the area of the curved mirror.

The driving assistance apparatus according to the first and second embodiments detects the specular range of the curved mirror in the thermal image, and performs processing for adjusting the gain and contrast within the specular range. Therefore, the driving assistance device can easily detect a heat emitting object in the area of the curved mirror in the thermal image, and can detect a pedestrian or the like located in the driver's blind spot and notify the driver.

According to the driving assistance device according to the first and second embodiments, it is particularly difficult to check the curved mirror visually or with a visible light camera (for example, evening to night), so that pedestrians and the like can be seen from the curved mirror. It can be detected and notified to the driver.

The programs described above include instructions (or software code) that, when read into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or tangible storage medium. By way of example, and not limitation, computer readable media or tangible storage media may include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSD) or other memory technology, CDs -ROM, digital versatile disc (DVD), Blu-ray disc or other optical disc storage, etc. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example, and not limitation, transitory computer readable media or communication media include electrical, optical, or other forms of propagated signals.

100, 100a Driving support device 110 Far-infrared camera 120 Display unit 130 Illuminance sensor 140, 140a Control unit 141, 141a Shooting control unit 142, 142a Curved mirror detection unit 143 Image adjustment unit 144 Object detection unit 145 Notification control unit 146 Illuminance detection unit 150 visible light camera

Claims (5)

a photographing control unit for acquiring a thermal image photographed by a far-infrared camera provided in the vehicle;
a curved mirror detection unit that detects a mirror surface range of the curved mirror in the thermal image;
an image adjustment unit that adjusts at least one of gain and contrast within the specular range;
an object detection unit that detects a thermally radiating object within the specular range for which at least one of gain and contrast is adjusted by the image adjustment unit;
a notification control unit that notifies a driver of the vehicle when the heat emitting object is detected within the mirror surface range;
A driving support device. The object detection unit detects a person within the mirror surface range using a dictionary for detecting a person based on a thermal image,
The notification control unit notifies the driver of the vehicle when the person is detected within the mirror range.
The driving assistance device according to claim 1. The imaging control unit further acquires a visible light image captured by a visible light camera that captures images in the same direction as the far-infrared camera,
The curved mirror detection unit detects a mirror surface range of the curved mirror from the visible light image, and detects a range corresponding to the mirror surface range in the visible light image from the thermal image as the mirror surface range in the thermal image.
The driving assistance device according to claim 1 or 2. Further comprising an illuminance detection unit that detects that the illuminance around the vehicle is less than a predetermined threshold,
The driving support device is
When the illuminance detection unit detects that the illuminance around the vehicle is less than a predetermined threshold, the imaging control unit, the curved mirror detection unit, the image adjustment unit, the object detection unit, and the notification control perform processing by the department,
The driving assistance device according to any one of claims 1 to 3. a thermal image acquisition step of acquiring a thermal image captured by a far-infrared camera provided in the vehicle;
a curve mirror detection step of detecting a mirror surface range of the curve mirror in the thermal image;
an image adjustment step of adjusting at least one of gain and contrast within the detected specular range;
an object detection step of detecting a heat emitting object within the specular range in which at least one of gain and contrast is adjusted;
a notification step of notifying a driver of the vehicle when the heat emitting object is detected within the mirror surface range;
A driving support method executed by a driving support device.

Images