WO2015152304A1 - Driving assistance device and driving assistance system - Google Patents

Abstract

Provided is a driving assistance device equipped with: a detection means that detects conditions around a vehicle; a recognition means that recognizes objects around the vehicle on the basis of the detection result of the detection means; an analysis means that analyzes the objects recognized by the recognition means; a setting means that sets, on the basis of the analysis result of the analysis means, degrees of caution to be exercised with respect to the objects; a generation means that generates, on the basis of the degrees set by the setting means, an image for causing a driver to visually recognize the objects; and a display means that displays the image generated by the generation means.

Description

Driving support device and driving support system Cross-reference of related applications

This international application claims priority based on Japanese Patent Application No. 2014-72419 filed with the Japan Patent Office on March 31, 2014, and is based on Japanese Patent Application No. 2014-72419. The entire contents are incorporated herein by reference.

The present invention relates to a driving support device and a driving support system for notifying a driver of a vehicle of a danger corresponding to a situation around the vehicle.

2. Description of the Related Art Conventionally, a display device that can project an external situation or scenery on a vehicle window is known (see, for example, Patent Document 1).
The display device described in Patent Document 1 includes an observation device that observes the state (position, speed, etc.) of a vehicle and an accumulation device that accumulates image information of outside scenery in advance. Based on the information indicating the position of the observed vehicle, the image information of the scenery that will be visible outside the vehicle at the observed position is acquired from the storage device, and the image represented by the image information is displayed on the window of the vehicle. .

JP 2004-20223 A

However, according to the conventional image display device as described above, the landscape image stored in advance in the storage device is only displayed on the window of the vehicle. That is, it is impossible to detect the situation around the vehicle in real time and notify the driver of the situation about the situation. For this reason, the driver cannot recognize a risk (for example, a risk of a collision with an object around the vehicle) that occurs during driving.

It is desirable to be able to easily inform the driver of the danger according to the situation around the vehicle.

The driving support apparatus according to the first aspect of the present invention includes a detection unit that detects a situation around a vehicle, a recognition unit that recognizes an object around the vehicle based on a detection result of the detection unit, and the recognition unit. Analyzing means for analyzing the recognized object, setting means for setting a degree to be alerted based on the analysis result of the analyzing means, and causing the vehicle operator to visually recognize the object. And generating means for generating an image for use based on the degree set by the setting means, and display means for displaying the image generated by the generating means.

“Analysis” is intended to determine, determine, or estimate the type and state of an object by various analyses.
According to such a driving support device, the degree of alerting (hereinafter also referred to as the alert level) is set for each object according to the type and state of the object around the vehicle, and the alert level is set according to the alert level. The image will be displayed. For this reason, the driver | operator of a vehicle comes to be able to grasp | ascertain how much he should be alerted (it should be alerted) according to the kind and state, etc. of a target object.

This makes it possible to drive the vehicle properly. For example, when an image indicating that the alert level is high is displayed for an object, it may be possible to decelerate or perform a predetermined steering operation to avoid danger. “Danger” means the danger of a collision.

Further, the generation means may generate the following image as an image for allowing the vehicle operator to visually recognize the object.
-An image surrounding the object-An image pointing to the object (for example, an arrow image)
・ Images that schematically represent the object (images of object illustrations, etc.)
-Image of a message notifying the presence of an object The above-mentioned images may be displayed by the display means alone or in combination.

According to this, the driver can easily recognize the presence of the object.
Further, when combined and displayed, a plurality of images may be displayed at the same time or may be displayed with a time difference. When displayed with a time difference, specifically, the next image may be displayed a predetermined time after a certain image is displayed. For example, an image of an arrow indicating the target object may be displayed a predetermined time after the image surrounding the target object is displayed.

In one example, when an object is detected, an image surrounding the object is displayed first, and after that, when the distance from the host vehicle to the object becomes a predetermined distance or less, the object is displayed. An image of a pointing arrow may be displayed. According to such an aspect, the driver can be continuously supported so that the object can be easily recognized.

In addition, the driving support device includes a determination unit that determines whether or not the object recognized by the recognition unit is a person, and a storage unit that stores image data of symbolic symbols indicating the state of the person. The analyzing means analyzes the state of the object determined to be a human by the determining means among the objects, and the generating means indicates the analysis result based on the analysis result of the analyzing means. Read means for reading image data corresponding to a symbol that is a symbol and indicating the state of the person from the storage means, and the display means reads the image data when the image data of the symbol is read by the read means The symbol symbol represented by the data may be displayed.

According to this, an image can be displayed especially when the object is a person, and it is possible to further contribute to improving driving safety. Further, by displaying a symbol representing the state according to the state of the person, the driver can recognize the state of the person around the vehicle. For this reason, the driver can perform appropriate driving in consideration of the conditions of people around the vehicle.

In addition, the driving support device is a detection result of the line-of-sight detection unit that detects the line-of-sight detection unit that detects the line of sight of the driver of the vehicle and an image recognized by the driver among the images displayed by the display unit. And an identification unit that identifies the moving state of the driver's line of sight, and an erasing unit that erases an image identified by the identification unit as recognized by the driver.

According to this, when the driver recognizes the object, the image for visually recognizing the object can be erased. Therefore, the driver continues to display the image even if the driver recognizes the image (in other words, recognizes the existence and state of the object) (in other words, the driver is continuously warned). It can be avoided.

On the other hand, images that are not recognized by the driver are continuously displayed, and warnings are continued in that portion. For this reason, the effect of improving driving safety is not lost. According to this, usability and the effect of driving assistance can be made compatible at a high level.

In one example, the driving support device may include at least one imaging device as a detection unit. If an image around the vehicle is taken by the imaging device, the type and state of the object around the vehicle can be analyzed in more detail by image analysis. In addition, various image analysis techniques are known, and analysis can be performed relatively easily using conventional techniques.

When the detection unit includes a plurality of imaging devices, the driving support device (or the detection unit) may be configured to compare the captured images of the respective imaging devices and select a captured image to be adopted based on the comparison result. good. Further, it may be configured to extract a portion with high accuracy (a portion with little noise component or the like) from the data of each captured image and integrate such portions to generate one data. According to this, the accuracy of image analysis can be further increased. As a result, detection and grasping of the situation around the vehicle can be realized at a high level.

In addition, the detection unit reproduces the parallax (difference between the position of the image and the viewing direction) using a plurality of (specifically, two) imaging devices, and the three-dimensional information (specifically, depth) of the object based on the parallax. Information). According to this, the driving assistance apparatus can also set a warning level based on the three-dimensional information of the target object. In one example, the driving support device may display a stereoscopic image (3D image) of the object. In this case, the display unit may be configured to display a stereoscopic image (3D image) of the object based on the detection result (stereoscopic information) of the detection unit.

Further, the driving support device (or detection means) may be configured to identify a plurality of objects that overlap in the field of view of the imaging device from the three-dimensional information (depth information) of each object. For example, the recognition means may be configured to recognize a plurality of overlapping objects as separate objects from the three-dimensional information (depth information) of each object.

Here, a plurality of objects that are present in the same direction and partially overlap when viewed from the imaging device are recognized as one identical object only by image analysis based on two-dimensional information. On the other hand, based on three-dimensional information (depth information), such a plurality of objects can be identified as separate objects.

In this case, the generation means may be configured to generate an image for allowing the operator to visually recognize each of the plurality of objects that are partially overlapped. In this case, each of the plurality of objects may be easily recognized by changing the form of the image.

According to the above configuration, the driver can be notified more accurately of the situation around the vehicle. In addition, the driver can more easily grasp the situation around the vehicle. For example, it may be easier for the driver to recognize a separate object hidden behind the object.

Next, the analysis means may be configured to analyze whether or not the object exists on a travel route of the own vehicle (a vehicle on which the driving support device is mounted). Specifically, it is assumed that the road is recognized by the detection means and the recognition means, and the analysis means may analyze whether or not the object exists on the road. Further, when the course of the vehicle is estimated from the motion state of the vehicle or the like, it may be analyzed whether or not an object exists on the course.

In this case, the setting means may set the alert level relatively high for the object existing on the travel route. On the other hand, the warning level may be set relatively low for an object that does not exist on the travel route.

Further, the analysis means may be configured to analyze a distance from the own vehicle to the object.
For example, the analysis can be performed based on the detection result of the detection means. In one example, the distance can be calculated by image analysis of a captured image of an imaging device as a detection unit. Further, if the detection means includes a distance sensor, the distance to the object can be calculated based on the output result (output signal) of the distance sensor.

In this case, the setting means may set the alert level relatively high for an object having a relatively small distance from the host vehicle to the object. On the other hand, the warning level may be set relatively low for an object having a relatively large distance from the host vehicle to the object.

Further, the analysis means may be configured to analyze whether or not the person has a portable terminal such as a mobile phone, a smartphone, or a tablet when the object is a person. Specifically, an image analysis of a captured image of an imaging device as a detection unit may be performed. For example, the brightness of the display portion becomes bright during the operation of the mobile terminal, and the boundary of the display can be detected as an edge in the image analysis. Based on the fact that the display can be recognized by such edge detection, it may be configured to analyze whether or not the mobile terminal is operating (whether or not the mobile terminal exists).

Further, the analysis means may be configured to analyze whether or not a person is operating the mobile terminal. Analysis includes whether or not the mobile terminal is in operation, the relationship between the position of the human body part (particularly the position of the hand and face) and the position of the mobile terminal, the orientation of the face, and the positional relationship between both eyes relative to the mobile terminal. Etc. may be included. And based on those analysis, it may be determined whether the person is operating the portable terminal. The analyzing means may be configured to analyze whether a person is talking on the mobile terminal.

In this case, the setting means may set the alert level relatively high for the object that is operating the mobile terminal and the object that is talking. On the other hand, the warning level may be set relatively low for objects that are not operating the mobile terminal and objects that are not in a call.

Further, the analysis means may be configured to analyze (or estimate) whether or not the person recognizes the presence of the own vehicle when the person is operating the mobile terminal. For example, if a person's face part is analyzed and both eyes can be extracted, it is determined that the person is facing the direction of the own vehicle, and the person is determined to recognize the existence of the own vehicle. Also good. On the other hand, if both eyes cannot be extracted, it may be determined that the person is not facing the direction of the host vehicle and the person is not aware of the presence of the host vehicle.

In this case, the setting means may set the alert level relatively high for an object that does not recognize the presence of the host vehicle.
On the other hand, the setting means may set the alert level relatively low for an object that recognizes the presence of the host vehicle. Also, the alert level may be lowered.

Further, the analyzing means analyzes the reaction of the person after the driving support device issues a warning to the person, and whether or not the person recognizes the existence of the own vehicle (in other words, the It may be configured to analyze whether or not the existence has been noticed. For example, both eyes may be extracted as described above. Further, the movement of the face may be analyzed. For example, when it is detected that the person's face is facing the own vehicle, it may be determined that the person has noticed the existence of the own vehicle.

The analysis means may be configured to analyze the gender and age of the person using a face recognition technique.
Further, the analyzing means may be configured to analyze whether or not a person is using headphones. Further, the analyzing means may be configured to analyze (or estimate) whether or not the person recognizes the presence of the own vehicle when the person uses headphones.

Further, the analysis means may be configured to analyze whether or not a person is talking. The analysis means may be configured to analyze (or estimate) whether or not the person recognizes the presence of the own vehicle when the person is talking.

Further, the analysis means may be configured to analyze the movement state of the person. Specifically, the moving direction of the person may be determined. Further, it may be configured to analyze whether or not a person is approaching the own vehicle (in other words, whether or not the person is moving away).

In this case, the generation unit may be configured to generate an image representing the moving direction.
Further, the analyzing means may calculate a moving speed of the person.
In this case, the generation unit may be configured to generate an image representing the movement speed of the person.

Further, the analysis means may determine whether the person is a child or an adult based on the size (specifically, height) of the person. Specifically, it may be determined whether the student is junior high school student or younger or elementary school student or younger. For this determination, the average height of a predetermined age published as statistical data is used as a threshold value. Also good.

The setting means may set the alert level relatively high when the person is a child.
In another aspect, the present invention may be a system (driving support system) including the above-described driving support device.

In another aspect, the present invention provides a detection unit that detects a situation around the vehicle, a recognition unit that recognizes an object around the vehicle based on a detection result of the detection unit, and a recognition unit that recognizes the object. An analyzing means for analyzing the object, a setting means for setting a degree to be alerted based on an analysis result of the analyzing means, and a vehicle operator for visually recognizing the object. The driving support system may include a generating unit that generates the image based on the degree set by the setting unit, and a display unit that displays the image generated by the generating unit.

And this driving assistance system may be provided with the same composition as the composition with which the above-mentioned driving assistance device is provided.

It is a figure which shows the example of application to the vehicle of the driving assistance device of embodiment. It is a block diagram which shows the structure of the driving assistance device of 1st Embodiment. It is a flowchart showing the flow of the handling assistance process which control ECU performs. It is a flowchart showing the flow of the extraction process which control ECU performs. 3 is a flowchart showing the flow of analysis processing 1; 6 is a flowchart showing a flow of analysis processing 2; 10 is a flowchart showing the flow of analysis processing 3; 6 is a flowchart showing a flow of analysis processing 4; 10 is a flowchart showing the flow of analysis processing 5; 10 is a flowchart showing the flow of analysis processing 6; 10 is a flowchart showing a subroutine flow of analysis processing 6; 10 is a flowchart showing the flow of analysis processing 7; It is a flowchart showing the flow of a vehicle recognition determination process. It is a flowchart showing the flow of a display data generation process. It is a flowchart showing the flow of an emphasized image generation process. It is a flowchart showing the flow of a display process. It is a figure which shows the example of a display. It is a figure which shows the example of a display. It is a figure which shows the example of a display. It is a figure which shows the example of a display. It is a block diagram which shows the structure of the driving assistance device of 2nd Embodiment. It is a flowchart showing the flow of a recognition process (2). It is a flowchart showing the flow of a screen determination process. 10 is a flowchart showing the flow of analysis processing 8; It is a flowchart showing the flow of a display process (2). It is a figure explaining the effect | action of the driving assistance device of 2nd Embodiment. It is a block diagram which shows the structure of the driving assistance device of 3rd Embodiment. It is a figure explaining gaze detection. It is a flowchart showing the flow of a driving assistance process (2). It is a flowchart showing the flow of a correction determination process. It is a flowchart showing the flow of a display correction process. It is a flowchart showing the flow of a recognition determination process (2). 10 is a flowchart showing the flow of analysis processing 9; It is a figure which shows the modification 1. FIG. It is a figure which shows the modification 2. FIG. It is a figure which shows the modification 3. FIG. 4 is a flowchart showing the flow of analysis processing 10. It is drawing explaining the detection of raindrops. 10 is a flowchart showing the flow of analysis processing 11. 10 is a flowchart showing the flow of analysis processing 12; It is a flowchart showing the flow of a vehicle control process. It is drawing explaining the example of a display mode (1). It is drawing explaining the example of a display mode (2). It is drawing explaining the example of a display mode (3). It is drawing explaining the example of a display mode (4).

DESCRIPTION OF SYMBOLS 1,100,101 ... Driving assistance device, 2 ... Infrared radar, 3 ... Millimeter group radar, 4 ... Infrared camera, 5 ... Visible light camera, 6 ... Momentum amount detection unit, DESCRIPTION OF SYMBOLS 7 ... Head-up display (HUD), 8 ... Image projector, 9 ... Speaker unit, 10 ... Gaze detection unit, 11 ... Inter-vehicle communication unit, 12 ... Vehicle position sensor 20 ... Control ECU.

Embodiments to which the present invention is applied will be described below with reference to the drawings.
<First Embodiment>
1. Overall Configuration As shown in FIG. 1, the driving support apparatus 1 of the first embodiment includes an infrared radar 2, a millimeter wave radar 3, an infrared camera 4, a visible light camera 5, a momentum detection unit 6, and a head. An up display 7, a speaker unit 8, and a control ECU 20 are provided.

In addition, in FIG. 1, the image projector 9, the gaze detection unit 10, the inter-vehicle communication unit 11, and the vehicle position sensor 12 are shown.
Hereinafter, each structure with which the driving assistance apparatus 1 is provided is demonstrated based on FIG.1 and FIG.2.

[Infrared radar]
The infrared radar 2 is a radar that detects the surrounding situation using infrared rays (in other words, detects the presence / absence of an object (hereinafter referred to as an object) and the distance to the object).

As shown in FIG. 2, the infrared radar 2 includes an infrared transmission / reception unit 2a, a signal processing unit 2b, and an external interface 2c.
The infrared radar 2 irradiates infrared rays at the infrared transmission / reception unit 2a, and receives reflected light that is reflected by the object and returned. Then, the signal processing unit 2b calculates the distance to the object based on the time difference between the irradiation time of the infrared rays and the reception time of the reflected light. Data representing the calculated distance is transmitted to the control ECU 20 via the external interface 2c.

The distance that can be detected by the infrared radar 2 is about several tens of meters (for example, 20 to 30 m). In addition, the infrared radar 2 may be provided in a side part and a rear part in addition to the front part of the vehicle, as shown in FIG.

[Millimeter wave radar]
The millimeter wave radar 3 is a radar that detects surrounding conditions using millimeter wave radio waves.
As shown in FIG. 2, the millimeter wave radar 3 includes a millimeter wave transmission / reception unit 3a, a signal processing unit 3b, and an external interface 3c.

The millimeter radar 3 receives the reflected wave that is reflected by the object and irradiated with the millimeter wave by the millimeter wave transmitting / receiving unit 3a. Then, the signal processing unit 3b calculates the distance to the object based on the time difference between the irradiation time of the millimeter wave and the reception time of the reflected wave. Data representing the calculated distance is transmitted to the control ECU 20 via the external interface 3c.

The distance detectable by the millimeter wave radar 3 is up to about 150 m (or more). A resolution of about several tens of cm to 1 m is known.
In the first embodiment, an object at a short distance (up to several tens of meters) is detected by the infrared radar 2 described above, and a long distance (from several tens of meters to 150 m (or more)) is detected by the millimeter wave radar 3. Configured to detect the object.

[Infrared camera]
The infrared camera 4 is a camera that detects surrounding conditions by detecting infrared rays emitted from an object.

As shown in FIG. 2, the infrared camera 4 includes an infrared image sensor 4a, an image processing unit 4b, and an external interface 4c.
The infrared camera 4 detects light (infrared rays) in the infrared region with the infrared image sensor 4a. Then, the image processing unit 4b converts the infrared wavelength and intensity detected by the infrared image sensor 4a into an electrical signal, and generates an image based on the electrical signal. Data representing the generated image is transmitted to the control ECU 20 via the external interface 4c.

Since this infrared camera 4 forms an image by detecting infrared rays emitted from an object, the object can be detected even in the absence of ambient light (such as sunlight) or headlight light. Therefore, the object can be detected even at night.

In the first embodiment, as shown in FIGS. 1 and 2, as the infrared camera 4, two infrared cameras 4A and 4B arranged at different positions are provided. The parallax (difference in image position and viewing direction) is reproduced by the infrared camera 4A and the infrared camera 4B. Based on the fact that the parallax is correlated with the distance to the object, the distance to the object can be calculated according to the parallax.

Hereinafter, the infrared camera 4 refers to both the infrared cameras 4A and 4B unless otherwise specified.
[Visible light camera]
The visible light camera 5 is a camera that detects surrounding conditions by detecting reflected light of ambient light and headlight light.

As shown in FIG. 2, the visible light camera 5 includes a CCD image sensor 5a as an imaging device, an image processing unit 5b, and an external interface 5c.
The visible light camera 5 detects light by the CCD image sensor 5a, and photoelectrically converts light and darkness of the detected light into a charge amount. The charge amount data is transferred to the image processing unit 5b. The image processing unit 5b generates a color image by reproducing color and brightness based on the charge amount data for each pixel. The generated image information is transmitted to the control ECU 20 via the external interface 5c.

In the first embodiment, the visible light camera 5 is provided with two visible light cameras 5A and 5B arranged at different positions. The parallax is reproduced by the visible light camera 5A and the visible light camera 5B, and thereby a three-dimensional image can be generated. Further, as in the case of the infrared camera 4 described above, the distance to the object can be calculated.

Hereinafter, the visible light camera 5 refers to the visible light cameras 5A and 5B unless otherwise specified.
[Momentum detection unit]
The momentum detection unit 6 is a unit for detecting the momentum of the host vehicle, and includes a vehicle speed sensor 6a, a yaw rate sensor 6b, and a steering angle sensor 6c.

Specifically, the traveling speed of the host vehicle is detected by the vehicle speed sensor 6a, the yaw rate acting on the host vehicle is detected by the yaw rate sensor 6b, and the steering angle of the steering wheel is detected by the steering angle sensor 6c. The detection signal is transmitted to the control ECU 20.

Based on detection signals from the vehicle speed sensor 6a, the yaw rate sensor 6b, and the steering angle sensor 6c, the control ECU 20 calculates the amount of movement (traveling direction and displacement) of the host vehicle.
[Head-up display]
A head-up display (HUD: Head Up Display) 7 is a device that superimposes and displays an image on a vehicle window (in the example, a front window).

The HUD 7 has a laser projector 7a, performs signal processing by the laser projector 7a based on a signal from the control ECU 20, generates an image, and displays the image via an optical unit 7b including a mirror and a lens. .

The image is formed on the virtual image plane so as to be superimposed on the scenery outside the vehicle viewed through the front window. The virtual image plane is formed in front of the front window, so that the driver of the vehicle can recognize that the image is displayed in the viewable scenery.
[Speaker unit]
The speaker unit 8 is a device that emits sound (including sound) around the vehicle based on control by the control ECU 20.

[Control ECU]
The control ECU 20 is an electronic control device that includes a CPU 20a, a ROM 20b, a RAM 20c, a flash memory 20d, a communication interface 20e, and the like, and executes various processes.

2. Processing executed in driving support device 1 [Driving support processing]
Hereinafter, the outline of the processing executed by the control ECU 20 of the driving support device 1 will be described with reference to FIG.

The control ECU 20 repeatedly executes the driving support process of FIG. 3 at a predetermined cycle while the vehicle is traveling. Accordingly, the driving support device 1 detects the environment around the vehicle, recognizes an object (person, vehicle, etc.), and notifies the vehicle driver of the presence of the object (in other words, gives a warning).

In the driving support process, first, in S100, detection data is acquired from the momentum detection unit 6, and the momentum of the host vehicle is estimated based on the acquired vehicle speed, yaw rate, and steering angle.

Next, in S110, a signal from the infrared radar 2 is acquired.
Next, the process proceeds to S112, and a signal from the millimeter wave radar 3 is acquired.
In subsequent S114, based on the signal from the infrared radar 2 and the signal from the millimeter wave radar 3, it is determined whether or not an object exists in the detection range.

If it is determined in S114 that the object does not exist, the process proceeds to S116 to store a log indicating that the object does not exist, and then the process is terminated. This log may be stored in the flash memory 20d.

If it is determined in S114 that the object exists, the process proceeds to S118. In S118, based on the distance data to the object obtained from the infrared radar 2 signal and the millimeter wave radar 3 signal, it is determined whether or not the distance to the object is equal to or less than a preset threshold value α.

If it is determined in S118 that the distance to the object is equal to or less than the threshold value α, the process proceeds to S120. The threshold value α is appropriately set to a value that causes a risk of collision. A fixed value may be set as the threshold value α. Alternatively, based on the momentum of the host vehicle estimated in the above-described S100, a value that is determined to cause a collision risk in relation to the host vehicle traveling at the momentum is calculated by calculation. It may be set.

In S120, processing for warning the driver of the vehicle is executed. Specifically, the HUD 7 is controlled to display a warning superimposed on, for example, the front window of the vehicle. As the warning display, a message or a symbol indicating that there is a risk of collision may be displayed.

In subsequent S122, a collision avoidance command for avoiding a collision with the object is transmitted to an ECU (not shown) that controls the operation of the vehicle. Specifically, a collision avoidance command is transmitted to the brake control ECU, the steering control ECU, etc., and brake control and steering control for avoiding the collision are executed. Thereafter, the process ends.

If it is determined in S118 that the distance to the object is not less than the threshold value α, the process proceeds to S124.
In S124, image data is acquired from the infrared camera 4. In this case, either one of the image data of the infrared cameras 4A and 4B or both of them may be acquired. When both are acquired, an average value of the two image data may be calculated and used. Further, a highly accurate portion (a portion with less noise or the like) may be extracted from each of the two image data, and data combining them may be generated and used.

In subsequent S126, image data is acquired from the visible light camera 5. In this case, either image data of the visible light cameras 5A and 5B may be acquired, or both may be acquired. When both are acquired, an average value of the two image data may be calculated and used. Further, a highly accurate portion (a portion with less noise or the like) may be extracted from each of the two image data, and data combining them may be generated and used.

Subsequently, in S128, processing for recognizing and analyzing the object (hereinafter, recognition processing) is executed. Details of the recognition process will be described later.
Next, in S130, a display process for displaying information on objects around the vehicle based on the result of the recognition process in S128 is executed. In other words, this process is a process of notifying the driver of the presence of the object by displaying a predetermined image. Details of the display process will be described later.

[Recognition process]
Hereinafter, the recognition processing in S128 will be specifically described with reference to FIG.
When the control ECU 20 starts the recognition process of S128 (recognition process of FIG. 4), first, in S140, from the image data of the infrared camera 4 and the image data of the visible light camera 5, the analysis processing of the image represented by those data I do. Specifically, for each pixel constituting the image, processing for analyzing the luminance (brightness) of the pixel is executed. Note that either one or both of the image data of the infrared camera 4 and the image data of the visible light camera 5 may be used. In any case of image data, the following processing flow can be basically applied.

After S140, the process proceeds to S142, and an edge in the image (a portion where the amount of change in luminance (brightness / darkness) is larger than a predetermined threshold) is extracted. This process is based on the premise that the amount of change in luminance becomes large at the boundary between, for example, a person or a vehicle and the background.

In the subsequent S144, candidates for areas occupied by the same object are set based on the edge information extracted in S142. For example, as described above, it is assumed that the amount of change in luminance is large at the boundary between a person or vehicle and the background, but the amount of change in luminance is not always large at all boundaries, and the edges may be interrupted. obtain. In this processing, the range (area) of the same object determined by the edge is set (estimated) while recognizing the break of the edge from the data of the peripheral edge.

Next, the process proceeds to S146, and pattern matching is performed on the region set in S144 (more specifically, on the estimated object) with a pattern stored in advance and a past learning value (learned pattern), and the object Estimate what is In this pattern matching, a person (including a person riding a bicycle or the like), a vehicle, an animal (pet or the like), and an installation (a guardrail, a sign, a traffic light, a signboard or the like) can be recognized.

In subsequent S148, based on the result of pattern matching in S146, it is determined whether or not the extracted object is a person.
If it is determined in S148 that it is “others” (specifically, it is assumed that it is not a person or a vehicle), the process is terminated. This means that, among the extracted objects, objects classified as “others” are not further analyzed and no warning is displayed. The driver may also be notified (displayed) of objects classified as “others”.

If it determines with it being a vehicle in S148, it will transfer to S150. In S150, it is determined whether or not to notify the driver of the own vehicle of the extracted vehicle information (whether or not to warn).
For example, it may be configured to be able to set in advance whether or not to notify the driver of the vehicle of vehicle information. And in S150, you may determine based on the setting. Alternatively, the positional relationship between the vehicle and the host vehicle, the relative speed, and the like may be detected, and the risk may be determined based on the detected position.

If it is determined in S150 that no warning is given, the process is terminated.
On the other hand, if it determines with warning in S150, it will transfer to S156 and will set the flag (vehicle warning flag) showing that it will warn about a vehicle. Thereafter, the process proceeds to S154.

Next, when it is determined in S148 that the person is a person, the process proceeds to S152.
In S152, an analysis process for further analyzing the extracted situation, state, etc. of the person is executed. Details of the analysis processing will be described later.

After S152, the process proceeds to S154. In S154, warning display data is generated based on the analysis processing result in S152 and the determination result in S150. The data generated here is used in the display process in S130. Specifically, in S130, the data generated in S154 is transmitted to the HUD 7, and a warning image is displayed on the HUD 7.

[Analysis processing]
Hereinafter, the analysis processing in S152 will be specifically described with reference to FIGS. In the first embodiment, the analysis processes 1 to 7 in FIGS. 5 to 12 (and FIG. 13) are executed in parallel or sequentially in a predetermined order. The analysis processes 1 to 7 are executed for each object recognized as “person” in S148 described above.

In the analysis processes 1 to 7, a warning level for the object (person) is set according to the analysis result. The alert level is data used in the process of S154. Specifically, it is data for determining in what manner the warning is displayed to the driver of the vehicle. The alert level is represented by a numerical value. As the numerical value increases, display data is generated so that a warning is displayed in a manner that is more easily recognized by the driver of the vehicle.

It should be noted that at least one of the analysis processes 1 to 7 in FIGS. 5 to 12 (and FIG. 13) may be executed as the analysis process in S152.
[Analysis process 1]
The analysis process 1 in FIG. 5 is a process of analyzing a position (location) where an object (person) exists and setting a warning level according to the position.

In the analysis process 1, first, in S160, for the object recognized as a person in S148 described above, a process for analyzing the position where the object (person) exists is executed. Specifically, the distance from the host vehicle, the relative position with respect to other objects, and the like are analyzed by image analysis of a captured image by the infrared camera 4 or the visible light camera 5. The distance from the host vehicle can be calculated using the parallax of the infrared cameras 4A and 4B or the parallax of the visible light cameras 5A and 5B.

Here, as described above, the signal from the infrared radar 2 acquired in S110 and the signal from the millimeter radar 3 acquired in S112 include information on the distance to the object. The distance calculated in S160 may be calculated or corrected.

Alternatively, instead of using parallax, the distance may be calculated from the signals acquired in S110 and S112.
After S160, the process proceeds to S162, and it is determined whether an object (person) exists on the traveling route of the host vehicle.

It should be noted that the road on which the host vehicle is traveling is recognized by image analysis at the stage of the processing of S160 described above. Further, for the process of S170 described later, a sidewalk may be added and recognized. In S162, in addition, the movement direction (traveling direction) of the host vehicle is estimated based on the data on the amount of movement of the host vehicle acquired in S100. Based on these processes, it is determined whether or not the object (person) exists on the recognized road and on the estimated traveling direction of the own vehicle, so that the object (person) Determine if it exists above.

If it is determined in S162 that the object (person) exists on the travel route of the host vehicle, the process proceeds to S164.
In S164, the value of the alert level for the object (person) is incremented by 3 points. Thereafter, the process ends. In the first embodiment, the value of the alert level is incremented in the range of 1 to 3. When not incrementing, “+0” is described in the flowchart. This value is an example, and any value may be set as appropriate.

The numerical value of the alert level is stored in the flash memory 20d in association with the object (person).
If it is determined in S162 that the object (person) does not exist on the travel route of the host vehicle, the process proceeds to S166.

In S166, it is determined whether or not the object (person) exists on the road on which the host vehicle is traveling.
If it is determined in S166 that the object (person) exists on the road, the process proceeds to S168.

In S168, the alert level is incremented by 2 points. Thereafter, the process ends.
If it is determined in S166 that the object (person) does not exist on the road, the process proceeds to S170.

In S170, it is determined whether an object (person) exists on the sidewalk.
If it is determined in S170 that the object (person) exists on the sidewalk, the alert level is incremented by one point.

On the other hand, if it is determined in S170 that the object (person) does not exist on the sidewalk, it is determined that the object (person) exists in the vehicle or indoors (in other words, a relatively safe place), and the process proceeds to S174. Then, the process ends without incrementing the alert level.

[Analysis process 2]
The analysis process 2 will be described with reference to FIG.
The analysis process 2 in FIG. 6 is a process of calculating the distance from the host vehicle to the object (person) and setting the alert level according to the calculated distance.

In the analysis process 2, first, in S180, the distance from the own vehicle to the object (person) is calculated. The calculation method is as described above.
Next, the process proceeds to S182, and it is determined whether or not the distance calculated in S180 is equal to or less than a predetermined threshold value β. The value of β can be set as appropriate.

If it is determined in S182 that the distance is equal to or smaller than the predetermined threshold β, it is determined that the degree of risk is higher, and the process proceeds to S184.
In S184, the alert level is incremented by one point. Thereafter, the process ends.

On the other hand, if it is determined in S182 that the distance is not less than or equal to the predetermined threshold β, it is determined that the degree of risk is lower, and the process proceeds to S186, and the process is terminated without incrementing the alert level.

[Analysis process 3]
The analysis process 3 will be described with reference to FIG.
The analysis process 3 in FIG. 7 is a process for analyzing whether or not an object (person) is carrying and operating a portable terminal (and whether or not the vehicle is recognized) and setting a warning level based on the result. It is.

In the analysis process 3, first, in S190, it is determined whether or not the object (person) is carrying (gripping) the portable terminal. The presence / absence (presence) of the portable terminal is recognized by image analysis (pattern matching) in the processing of S140 to S146 described above. In particular, when the mobile terminal is in operation, the luminance (brightness) of the display screen portion is high, and edge extraction with relatively high accuracy is possible. In this case, recognition by pattern matching becomes easy. Further, even when the mobile terminal is not in operation, edge extraction can be performed based on a difference in luminance (brightness and darkness) from a human hand when the portable terminal is held by the human hand. Therefore, in any case, recognition by pattern matching is possible.

If it is determined in S190 that the object (person) is not carrying (gripping) the portable terminal, it is determined that the risk is not high, and the process proceeds to S192, and the process ends without incrementing the alert level. .

On the other hand, if it is determined in S190 that the object (person) is carrying (holding) the portable terminal, the process proceeds to S194.
In S194, it is determined whether or not the mobile terminal is operating. Here, based on the result of image analysis (the processes of S140 to S146), it is determined whether or not the mobile terminal is operating from the brightness (brightness and darkness) in the area recognized as the mobile terminal. This is intended to make a determination on the assumption that the luminance (brightness) of the portion of the display screen in the portable terminal is high when the portable terminal is in operation.

If it is determined in S194 that the mobile terminal is not operating, the process proceeds to S196.
In S196, the alert level is incremented by 1 point based on the determination that the object (person) is holding the mobile terminal although the mobile terminal is not operating. Thereafter, the process is terminated.

On the other hand, if it is determined in S194 that the mobile terminal is operating, the process proceeds to S198. Note that the process of S194 may be omitted. Specifically, if it is determined in S190 that the object (person) is carrying (gripping) the portable terminal, the process may proceed to S198 without executing the process of S194.

In S198, it is determined whether or not the object (person) is operating the mobile terminal. Here, based on the result of image analysis, the position of the portable terminal, the position of each part (hand, face) in the object (person), the orientation of the face, and the like are analyzed and comprehensively determined from the information.

If it is determined that the operation is not being performed in S198, the process proceeds to S196.
On the other hand, if it determines with operating in S198, it will transfer to S200.
In S200, a process of determining whether or not the object (person) recognizes the presence of the host vehicle (hereinafter, a recognition determination process) is executed.

FIG. 13 shows the recognition determination process.
In the recognition determination process in S200 (recognition determination process in FIG. 13), first, in S400, a face area in the object (person) is extracted.

Next, the process proceeds to S402, and the extracted face is analyzed. More specifically, “eyes” are extracted by edge detection and pattern matching.
In step S404, it is determined whether both eyes have been detected.

If it is determined in S404 that both eyes have been detected, it is determined that the host vehicle is likely to exist within the field of view of the object (person). Based on this determination, the object (person) A simple determination is made that the presence is recognized, and the flow proceeds to S406.

In S406, a recognition flag indicating that the object (person) recognizes the presence of the host vehicle is set. Thereafter, the process is terminated.
On the other hand, if it is determined in S404 that both eyes cannot be detected, it is determined that the host vehicle may not exist within the field of view of the object (person). Based on this determination, the object (person) A simple determination is made that the presence of the vehicle is not recognized, and the flow proceeds to S408.

In S408, an unrecognized flag indicating that the object (person) has not recognized the presence of the own vehicle is set. Thereafter, the process is terminated.
After the recognition determination process in S200 (recognition determination process in FIG. 13), the process proceeds to S202 in FIG.

In S202, a determination process based on the recognition flag set in S406 or the non-recognition flag set in S408 (specifically, a determination process as to whether or not the object (person) recognizes the presence of the host vehicle). )I do.

If it is determined in S202 that the object (person) recognizes the presence of the host vehicle (the recognition flag is set), the process proceeds to S204.
In S204, the alert level is incremented by 2 points based on the determination that the object (person) recognizes the existence of the own vehicle while the portable terminal is being operated. Thereafter, the process is terminated.

On the other hand, if it is determined in S202 that the object (person) has not recognized the presence of the host vehicle (the unrecognition flag is set), the process proceeds to S206.
In S206, the alert level is incremented by 3 points based on the determination that the object (person) is operating the mobile terminal and does not recognize the presence of the host vehicle.

Subsequently, the process proceeds to S208, and a warning setting process described below is performed. The warning setting process includes a flag for displaying an image for notifying (warning) the driver that the object (person) has not recognized the existence of the own vehicle, and an alarm process for the object (person). This is a process of setting a flag to the effect. This flag is stored in association with the target object (person).

When the warning setting process is executed, in S154 of FIG. 4, a warning image for notifying (warning) the driver that the object (person) has not recognized the presence of the own vehicle is generated. In the process of S130 in FIG. 3, the vehicle is superimposed and displayed on the front window. In addition, an alarm is issued to the object (person) through the speaker unit 8 (see FIGS. 1 and 2) by a separate process.

After S208, the process ends.
[Analysis process 4]
The analysis process 4 will be described with reference to FIG.

The analysis process 4 in FIG. 8 is a process of analyzing whether or not the object (person) is using the headphones (and whether or not the vehicle is recognized), and setting a warning level based on the result. .

In the analysis process 4, first, in S210, it is determined whether or not the object (person) is using headphones or earphones (hereinafter simply referred to as headphones). Here, the determination is made based on the result of image analysis (the processing of S140 to S146).

If it is determined in S210 that the object (person) does not use the headphones, it is determined that the risk is low, the process proceeds to S212, and the process is terminated without incrementing the alert level.

On the other hand, if it is determined in S210 that the object (person) is using the headphones, the process proceeds to S200, and then the process proceeds to S202.
The processing of S200 and S202 is the same as the processing of S200 and S202 described with reference to FIG. 7, and description thereof is omitted here.

If it is determined in S202 that the object (person) recognizes the presence of the own vehicle (the recognition flag is set), the process proceeds to S218.
In S218, the alert level is incremented by 1 point based on the determination that the object (person) recognizes the presence of the host vehicle and uses the headphones. Thereafter, the process is terminated.

On the other hand, if it is determined in S202 that the object (person) does not recognize the presence of the host vehicle (the unrecognition flag is set), the process proceeds to S220.
In S220, the warning level is incremented by 3 points based on the determination that the object (person) is using headphones and does not recognize the presence of the host vehicle.

Next, the process proceeds to S208. The process of S208 is the same as the process of S208 of FIG. 7, and a description thereof is omitted here. Thereafter, the process ends.
[Analysis process 5]
The analysis process 5 will be described with reference to FIG.

The analysis process 5 in FIG. 9 is a process for analyzing whether or not the object (person) is talking or talking (and whether or not the vehicle is recognized) and setting a warning level based on the result. is there.
In the analysis process 5, first, in S230, it is determined whether or not the object (person) is talking or talking. Here, the determination is made based on the result of image analysis (the processing of S140 to S146).

If it is determined in S230 that the object (person) is not talking or talking, it is determined that the risk is low, the process proceeds to S232, and the process is terminated without incrementing the alert level.

On the other hand, if it is determined in S230 that the object (person) is talking or talking, the process proceeds to S200, and then proceeds to S202.
The processing of S200 and S202 is the same as the processing of S200 and S202 described with reference to FIG. 7, and description thereof is omitted here.

If it is determined in S202 that the object (person) recognizes the presence of the own vehicle (the recognition flag is set), the process proceeds to S238.
In S238, the alert level is incremented by 1 point based on the determination that the object (person) recognizes the presence of the own vehicle while talking or talking. Thereafter, the process is terminated.

On the other hand, if it is determined in S202 that the object (person) has not recognized the presence of the host vehicle (the unrecognition flag is set), the process proceeds to S240.
In S240, the alert level is incremented by 3 points based on the determination that the object (person) is talking or talking and does not recognize the presence of the vehicle.

Next, the process proceeds to S208. The process of S208 is the same as the process of S208 of FIG. 7, and a description thereof is omitted here. Thereafter, the process ends.
[Analysis process 6]
The analysis process 6 will be described with reference to FIG.

The analysis process 6 in FIG. 10 is a process of analyzing the movement of an object (person) and setting a warning level based on the result.
In the analysis process 6, first, image data is re-acquired from the infrared camera 4 or the visible light camera 5 in S250.

Next, the process proceeds to S252, and data on the amount of movement of the host vehicle is reacquired from the momentum detection unit 6.
The processes of S250 and S252 are executed to track the movement of the object (person) in time series.

Next, in S254, tracking processing between a plurality of images (between frames) is executed for the object (person). Specifically, the similarity of objects (people) is calculated in the current image (current frame) and the image (frame) in the past, and objects (people) with high similarity are the same object (people). The same label is assigned because it is determined that there is a high possibility that As the similarity index, the size (area) of the region, luminance (brightness / darkness), movement amount, and the like are used. Here, the size (area) of the object (person), the amount of movement, and the like are corrected in consideration of the amount of movement of the host vehicle. The objects (people) with the same label are analyzed in time series, and the presence / absence of movement and the movement direction are calculated.

Next, the process proceeds to S256, and it is determined whether or not the movement of the object (person) can be analyzed. In other words, it is determined whether the re-acquired image data and exercise amount data are sufficient for recognizing or estimating the movement of the object (person).

If it is determined in S256 that analysis is impossible (in other words, the data is not sufficient), the process returns to S250 (and S252) again, and image data (and momentum data) is acquired again. Furthermore, a tracking process is performed in S254.

On the other hand, if it is determined in S256 that analysis is possible, the process proceeds to S258.
In S258, based on the result of the tracking process in S254, it is determined whether or not the object (person) is moving.

If it is determined in S258 that the object (person) has not moved, the process proceeds to S260, and the process ends without incrementing the alert level.
On the other hand, if it is determined in S258 that the object (person) is moving, the subroutine processing in S280 is entered.

FIG. 11 is a flowchart showing the flow of subroutine processing in S280.
When the subroutine processing of S280 (subroutine processing of FIG. 11) is started, first, in S282, it is determined whether or not the moving direction of the object (person) is the same as the moving direction of the host vehicle.

If it is determined in S282 that the moving directions are the same, the object (person) is moving with his back to the own vehicle (and the object (person) is likely not aware of the existence of the own vehicle). Based on the determination, the process proceeds to S284.

In S284, the alert level is incremented by 2 points. Thereafter, the process is terminated.
On the other hand, if it is determined in S282 that the moving directions are not the same, the own vehicle is likely to exist within the field of view of the object (person) (and the object (person) is aware of the existence of the own vehicle. Based on the determination that the possibility is high), the process proceeds to S286, and then the process is terminated without incrementing the alert level.

After the subroutine processing in S280 (subroutine processing in FIG. 11), the process proceeds to S262 in FIG.
In S262, it is determined whether or not the object (person) is meandering. As for the meanders, there are warnings about wobbling due to drinking, wobbling due to two-seater bicycles, and the like.

If it is determined in S262 that the object (person) is not meandering, the process proceeds to S264 based on the judgment that the object is not meandering but is moving, and the warning level is incremented by one point. Thereafter, the process is terminated.

On the other hand, if it is determined in S262 that the object (person) is meandering, the process proceeds to S266.
In S266, it is determined whether or not the object (person) is approaching the traveling route of the host vehicle.

If it is determined in S266 that the object (person) is not approaching the traveling route of the host vehicle, the process proceeds to S268 based on the determination that the object (person) is not approaching but is meandering, and the warning level is incremented by 2 points. Thereafter, the process is terminated.

On the other hand, if it is determined that the object (person) is approaching the travel route of the host vehicle, the process proceeds to S270, and the warning level is incremented by 3 points.
Next, the process proceeds to S272, and a flag for displaying an image indicating the moving direction of the object (person) is set. When this flag is set, an image indicating the moving direction of the object (person) is generated in the process of S154 in FIG. 4, and the image is displayed in the process of S130 in FIG. This series of processing is executed with the purpose of calling attention to the driver of the vehicle by displaying an image indicating the moving direction of the object (person).

[Analysis process 7]
The analysis process 7 will be described with reference to FIG.
The analysis process 7 in FIG. 12 is a process for simply determining whether or not the object (person) is a child and setting a warning level based on the result.

In the analysis process 7, first, in S290, it is determined whether or not the height (height) of the object (person) is equal to or less than a predetermined threshold value Ta. As the threshold Ta, an average height of a person (child) of an age to be discriminated may be assigned.

If it is determined in S290 that the value is equal to or less than the threshold value Ta, it is determined that the object (person) is a child, and the process proceeds to S292, where the alert level is incremented by 2 points. Thereafter, the process is terminated.

On the other hand, if it is determined in S290 that it is not less than or equal to the threshold value Ta, it is determined that the object (person) is not a child, the process proceeds to S294, and the process ends without incrementing the alert level.

[Display data generation processing]
Next, the display data generation process in S154 of FIG. 4 will be described with reference to FIG.
In the display data generation process, first, in S500, the object (person or vehicle) recognized in S140 to S148 in FIG. 3 is extracted.

Next, the process proceeds to S502, and for each object (specifically, each object recognized as “person”), the alert level set in the processing of FIGS. 5 to 13 is extracted.

In subsequent S504, processing for generating an image (emphasized image generation processing) for emphasizing the object (person) is executed for each object (person) according to the extracted alert level.

The enhanced image generation process will be specifically described with reference to FIG.
When the enhanced image generation process of S504 (enhanced image generation process of FIG. 15) is started, first, in S520, an image surrounding the object area is generated in accordance with the object area of each object (person). As an image surrounding the object region, a triangle, a quadrangle, a circle, an ellipse, or the like can be set as appropriate.

By generating an image that matches the object area, for example, a vertically long image can be generated for a standing object (person), and an image with a substantially equal aspect ratio can be generated for a sitting object (person), for example. In addition, a horizontally long image is generated for an object (person) that has fallen down. By displaying such an image, the driver can intuitively grasp the state of the object (person) (such as standing, sitting, or falling).

Next, the process proceeds to S522, and the display mode of the image generated in S520 is set according to the alert level extracted in S502 described above. Specifically, the thickness of the frame line, the color of the line, etc. are set. It is also set whether to blink display.

For example, the higher the alert level, the thicker the line. Further, when the alert level is high, the line color may be set to a color (red, yellow, other fluorescent color, etc.) that attracts the driver's attention. Further, when the alert level is high, the image may be set to blink.

Each object (person) may be classified as a group with a high point, an intermediate group, or a low group according to the set alert level point. And the display mode of an image may be set for every group.

Next, in S524, it is determined whether images have been set for all objects (people).
If it is determined in S524 that no image has been set for all objects (persons) (an unset object (person) exists), the process returns to S520 (and S522).

On the other hand, if it is determined in S524 that images have been set for all objects (persons), the processing ends.
When the enhanced image generation process of S504 (enhanced image generation process of FIG. 15) ends, the process proceeds to S506 of FIG.

In S506, it is determined whether or not a flag for warning the vehicle (hereinafter, vehicle warning flag) is set. This flag is set in the process of S156 in FIG.
If it is determined in S506 that the vehicle warning flag is set, the process proceeds to S508, and an image for emphasizing the object (vehicle) is generated in association with the target object (vehicle). Specifically, an image surrounding the object area of the object (vehicle) recognized in the processes of S140 to S148 in FIG. 4 is generated. As an image surrounding the object region, a triangle, a quadrangle, a circle, an ellipse, or the like can be set as appropriate. After the processing of S508, the process proceeds to S510.

If it is determined in S506 that the vehicle warning flag is not set, the process proceeds to S510.
In S510, whether or not a flag (hereinafter referred to as an unrecognized notification flag) for displaying an image for notifying (warning) the driver that the object (person) has not recognized the presence of the host vehicle is set. Determine whether. This flag is set in the processing of S208 in FIGS.

If it is determined in S510 that the unrecognized notification flag is set, the process proceeds to S512, and a warning image for the driver is generated in association with the target object (person). This warning image is an image for notifying (warning) the driver that the object (person) has not recognized the presence of the own vehicle. The image is not limited to a mark such as a symbol, but may be a message, for example. Alternatively, an image surrounding the face portion of the object (person) may be used. The color of the warning image may be a color (red, yellow, other fluorescent color, etc.) that further prompts the warning. After S512, the process proceeds to S514.

If it is determined in S510 that the unrecognition notification flag is not set, the process proceeds to S514.
In S514, it is determined whether or not a flag for displaying an image indicating the moving direction of the object (person) (hereinafter referred to as a moving display flag) is set. This flag is set in the process of S272 of FIG.

If it is determined in S514 that the movement display flag is set, the process proceeds to S516, and an image (for example, an arrow mark) indicating the movement direction of the object (person) is generated in association with the target object (person). To do. Next, the process proceeds to S518.

If it is determined in S514 that the movement display flag is not set, the process proceeds to S518.
In S518, it is determined whether there is another object (whether there is an object for which an image to be emphasized is to be generated).

If it is determined in S518 that there is an object, the process returns to S502.
On the other hand, if it is determined in S518 that there is no object, the process ends.
[Display processing]
Next, the display process of S130 in FIG. 3 will be described with reference to FIG.

When the display process of S130 (display process of FIG. 16) is started, first, in S540, an alignment adjustment signal for initializing and adjusting the display position and the imaging position by the HUD 7 is transmitted to the HUD 7. This causes the HUD 7 to execute adjustment (initialization) of the display position and the imaging position.

Next, the process proceeds to S542, and a signal representing an image generated in the display data generation process of S154 (display data generation process shown in FIGS. 14 and 15) is transmitted to the HUD 7. Thereby, the image is superimposed and displayed on the front window of the vehicle via the HUD 7. Note that the signal representing the image includes data of coordinate values (coordinate values based on the display area by the HUD 7) at which the image is to be displayed.

Specifically, the control ECU 20 includes information on coordinate axes (hereinafter referred to as camera coordinate axes) based on the imaging areas of the infrared camera 4 and the visible light camera 5 and coordinate axes (hereinafter referred to as HUD coordinate axes) based on the display area of the HUD 7. Have both information. Then, the coordinate value (the coordinate value indicating the position to be displayed by the HUD 7) on which the image generated by the image analysis of the infrared camera 4 or the visible light camera 5 is to be displayed is changed from the coordinate value on the camera coordinate axis on the HUD coordinate axis. Calculated by converting to coordinate values.

After S542, the process proceeds to S544, and it is determined whether there is an additional display image. Specifically, it is determined whether or not a new image is generated by the display data generation process of S154.
If it is determined in S544 that there is an additional image, the process of S542 is executed again.

On the other hand, if it is determined in S544 that there is no additional image, the process ends.
3. Next, the operation of the first embodiment (an example of a display mode) will be described with reference to FIGS.

First, the example of FIG. 17 will be described. In the example of FIG. 17, an object H that is a person and objects V0, V1, and V2 that are vehicles are extracted and recognized.
In this example, an image for emphasis is superimposed and displayed for both a person and a vehicle.

Specifically, for an object H, an oval frame image (hereinafter also simply referred to as a frame) W surrounding the object H is displayed.
For the objects V0, V1, and V2, elliptical frames X0, X1, and X2 that surround the objects V0, V1, and V2 are displayed.

The display mode may be different between the frame W and the frames X0, X1, and X2. For example, as shown in FIG. 17, the frame W may be displayed as a solid line, and the frames X0, X1, and X2 may be displayed as a broken line.

Here, in FIG. 17, the object V0 and the object V2 are seen partially overlapping (part of the object V2 is behind the object V0).
In such a situation, based on the parallax of the infrared cameras 4A and 4B or the parallax of the visible light cameras 5A and 5B, the positional relationship between the object V0 and the object V2 may be grasped from the stereoscopic information (depth information). Specifically, the control ECU 20 is configured not to recognize the object V0 and the object V2 as different objects based on the three-dimensional information (depth information) but as different objects.

In this case, as shown in FIG. 17, a frame X0 corresponding to the object V0 and a frame X2 corresponding to the object V2 are drawn. Then, the frame X0 and the frame X2 may also include stereoscopic information (depth information). Specifically, as shown in FIG. 17, the frame X2 may be displayed so that a part of the frame X2 is hidden behind the object V0.

In FIG. 17, a mode display area R is shown at the upper right of the drawing. This area is an area for displaying a target for displaying an image for emphasis (specifically, a target symbol mark). In the mode display area R, a vehicle symbol Mv and a human symbol Mp are shown. This indicates a mode in which an image for emphasis (frame W and frames X0, X1, and X2 in FIG. 17) is displayed for the vehicle and the person.

As described above, according to the driving support apparatus 1, an image (frame W) for detecting an object (person, other vehicle) or the like around the host vehicle and causing the driver to visually recognize the detected object is displayed on the object. It is superimposed and displayed in association. For this reason, it is easier for the driver to recognize the object.

Next, an example of FIGS. 18A and 18B will be described. In the example of FIGS. 18A and 18B, four people are visually recognized as shown in FIG. 18A.
The driving support device 1 analyzes the state of each person by image analysis for each of the four persons, and displays a warning image corresponding to the analyzed state.

As shown in FIG. 18B, the four persons will be described as objects H1, H2, H3, and H4, respectively.
First, a frame W (W1, W2, W3, W4) for emphasis is displayed for each of the objects H1 to H4. The frame W may be generated and displayed so as to surround the areas of the objects H1 to H4. For example, when the object H is a person standing, the frame W may be configured to be generated and displayed vertically. Further, when the object H is a person and is sleeping (falling down) for some reason, the frame W may be configured to be generated and displayed horizontally. When the object H is a sitting person, the frame W may be generated and displayed so that the aspect ratio of the frame W is approximately equal.

The display modes of the frames W1 to W4 differ depending on the alert level.
The objects H1 and H2 exist in the travel route of the host vehicle.
In particular, it is assumed that the distance from the host vehicle is equal to or less than a predetermined threshold value β for the object H1. As a result, when the alert level is set relatively high for the object H1, the frame W1 is displayed in a display mode that is more emphasized. For example, the frame W1 may be configured with a double frame. Further, it may be displayed in a more conspicuous color such as a fluorescent color.

Suppose that the distance from the host vehicle is greater than a predetermined threshold β for the object H2. Accordingly, when the alert level is set relatively low (compared to the object H1) for the object H2, the degree of emphasis of the frame W2 may be suppressed in comparison with the frame W1. The frame W2 may be displayed blinking, for example.

Further, when the objects H1 and H2 are in conversation, the state is detected by the driving support device 1.
When the driving support apparatus 1 determines that the objects H1 and H2 are in conversation, the driving support apparatus 1 displays the conversation symbols M1 and M2 indicating that the objects are in conversation in the vicinity of the frames W1 and W2 in association with the objects H1 and H2. You may do it. Data of conversation symbols M1 and M2 is stored in the flash memory 20d. Note that the data of the conversation symbols M1 and M2 may be stored in the ROM 20b. The same applies to a portable terminal symbol M3, a headphone symbol M4, an unrecognized symbol M1 ′, and a recognized symbol M2 ′ described later.

Also, the vicinity is a position adjacent to (or in contact with) the frame W regardless of whether it is up, down, left, or right with respect to the frame W. In addition, in the case of the vicinity, it may be within the area of the frame W. For example, the conversation symbols M1 and M2 may be displayed superimposed on the areas of the frames W1 and W2. The meaning of “near” is the same in the following.

If it is determined that the object H1 is not aware of the presence of the host vehicle, an unrecognized symbol M1 ′ representing that fact may be superimposed on the vicinity of the frame W1 in association with the object H1.

On the other hand, if it is determined that the object H2 recognizes the existence of the own vehicle (or notices the existence of the own vehicle), a recognition symbol M2 ′ representing that fact is superimposed on the vicinity of the frame W2 in association with the object H2. You may display.

Furthermore, if it is determined that the object H3 is operating the mobile terminal h3 (object h3), a mobile terminal symbol M3 indicating that may be displayed in a superimposed manner in the vicinity of the frame W3 in association with the object H3.

If the object H4 determines that the headphone h4 (object h4) is being used, the headphone symbol M4 indicating that may be displayed in the vicinity of the frame W4 in association with the object H4.

In addition, the driving support device 1 may set an auxiliary display area P1 for displaying auxiliary information. As shown in FIG. 18B, for example, the number of extracted objects (number of people, etc.) may be displayed in the auxiliary display area P1. Further, the number of persons displayed in the auxiliary display area P1 may be the number of persons displaying an image for emphasis.

In this case, when an image for emphasis is added and displayed, the numerical value of the number of people may be incremented according to the addition. On the other hand, when the image for emphasis is erased, the numerical value of the number of people may be decremented and displayed in accordance with the erase. Further, when the number of people is the same but the object is changed, the fact may be notified by flashing a numerical value or the like.

The symbols M1 to M4, M1 ′, and M2 ′ may be deleted or changed according to changes in the states of the objects H1 to H4.
As described above, according to the driving support device 1, when the object is a person, information indicating the state of the person is displayed, so the driver recognizes not only the presence of the person but also the state of the person. become able to. For this reason, it becomes possible for the driver to realize driving according to the conditions of people around the vehicle. That is, it can contribute to improving driving safety.

Next, the example of FIG. 19 will be described. In the example of FIG. 19, four objects H5, H6, H7, and H8 are extracted.
Objects H5 and H6 are pedestrians walking on a pedestrian crossing, and objects H7 and H8 are people riding bicycles.

Objects H5, H6, H7, and H8 are surrounded by frames W5, W6, W7, and frame 8 for emphasis, respectively.
The display mode of W5, W6, W7, and the frame 8 may be different depending on the distance from the host vehicle to the objects H5, H6, H7, and H8, for example. For example, the line thickness may be different.

In the example of FIG. 19, the object H5 is closest to the own vehicle, and the frame 5 corresponding to the object H5 is displayed with the thickest line. On the other hand, the object H8 is farthest from the host vehicle, and the frame 8 corresponding to the object H8 is displayed by the thinnest line.

In addition, in the example of FIG. 19, an arrow image (hereinafter, also simply referred to as an arrow) Y indicating the traveling direction of the object is displayed in association with each object H. Specifically, arrows Y5 and Y7 are directed to the right side in the drawing, indicating that the objects H5 and H7 are traveling toward the right side. An arrow Y6 is directed toward the left side in the drawing, and indicates that the object H6 is traveling toward the left side. Further, the arrow Y8 is directed toward the own vehicle, indicating that the object H8 is approaching the own vehicle.

The arrow Y may indicate the moving speed of each object H.
Specifically, the magnitude of the moving speed may be indicated by the length of the arrow Y. For example, in the example of FIG. 19, arrows Y5, Y6, and Y7 whose lengths are easily compared are targeted. In FIG. 19, it can be recognized that the length of the arrow Y7 is the longest and the moving speed of the object H7 is the highest. On the other hand, it can be recognized that the arrow Y6 is the shortest and the moving speed of the object H6 is the lowest. It can be recognized that the length of the arrow Y5 is intermediate between the arrows Y7 and Y6, and the moving speed of the object H5 is between the moving speed of the object H7 and the moving speed of the object H6.

Also, as indicated by arrows Y5, Y6, Y7, arrow gradation portions G5, G6, G7 may be drawn, and the magnitude of the moving speed may be indicated by the length, density, etc. of the gradation portions.

Also, the magnitude of the moving speed may be indicated by the position of the arrow Y with respect to the frame W (or the object H). In the example of FIG. 19, arrows Y5, Y6, and Y7 that can be compared in the height direction with respect to the frame W (or the object H) are targeted. In FIG. 19, the arrow Y7 is shown further upward in the range in the height direction of the frame W7 (and the object H7). The arrow Y5 is shown around the middle in the range in the height direction of the frame W5 (and the object H5). The arrow Y6 is shown below in the range in the height direction of the frame W6 (and the object H6).

In such a relationship, it may be indicated that the moving speed of the object H7 is the highest at the position of the arrow Y7 (the position in the vertical direction) shown above in relation to the object H. Further, it may be indicated that the moving speed of the object H6 is the lowest at the position of the arrow Y6 (vertical direction position) shown below in relation to the object H. In addition, even if the position of the arrow Y5 (vertical direction position) indicated at an intermediate position in relation to the object H indicates that the moving speed of the object H5 is intermediate between the object H7 and the object H6. good.

Thus, according to the driving support device 1, since the information indicating the direction in which the person around the vehicle is moving and the moving speed thereof is displayed, it is easy for the driver to predict the movement of the person. For this reason, it can contribute to improving the safety of driving.

Next, the example of FIG. 20 will be described. The example of FIG. 20 is a display example at night.
In the example of FIG. 20, the object H9 is extracted and recognized.
A frame W9 for emphasizing the object H9 is superimposed and displayed. The frame W9 may be drawn with a white color or a fluorescent color so that the frame W9 is easily visible at night.

In the upper part of the frame W9, an arrow symbol M9 is displayed to enhance the effect of attracting the driver's attention. Unlike the example of FIG. 19, the arrow symbol M <b> 9 faces the object H <b> 9 rather than the moving direction of the object H <b> 9 and strengthens the presence of the object H <b> 9.

In other words, the arrow symbol M9 is arranged so that when the line of sight is moved in the direction of the arrow, the object H9 is naturally recognized (so that it comes to the center of the field of view).
The driving support device 1 may be configured to display the frame W9 and the arrow symbol M9 at the same time when the object H9 is detected. For example, the frame W9 may be displayed and the arrow symbol M9 may be additionally displayed after a predetermined time has elapsed. According to the latter configuration, the enhancement effect can be further enhanced.

** Further, a caution symbol M9 'is displayed on the left side of the frame W9. The attention symbol M9 'can also be displayed to enhance the effect of attracting the driver's attention, like the arrow symbol M9.

Note that the display position of the attention symbol M9 '(and the arrow symbol M9) may be any position as long as it is easily recognized in relation to the background. For example, in the example of FIG. 20, the frame W9 (and the object H)) may be displayed in the lower left area Ra. Alternatively, it may be displayed in the region Rb directly below the frame W9 (and the object H)). Preferably, the background is displayed in a region where the luminance (brightness / darkness) does not vary (in other words, a region where the luminance (brightness / darkness) is constant).

In the example of FIG. 20, auxiliary display areas P2 to P4 may be set.
A symbol mark h9 representing the object H9 is displayed in the auxiliary display area P2. By additionally displaying such symbol marks, the driver's attention can be further increased.

Further, instead of displaying the frame W9, the arrow symbol M9, or the attention symbol M9 ', the symbol mark h9 may be displayed in the auxiliary display area P2. Such a mode (mode) may be set by the driver using an input device operated by the driver.

The distance is displayed in the auxiliary display area P3. This distance represents the distance from the host vehicle to the object H9.
In the auxiliary display area P4, a symbol m9 that is the same symbol as the attention symbol M9 ′ is displayed. The attention symbol M9 ′ and the symbol m9 may be displayed in conjunction with each other. For example, the symbol m9 may be automatically displayed when the attention symbol M9 ′ is displayed. Further, when the attention symbol M9 ′ is deleted, the symbol m9 may also be deleted.

Thus, according to the driving assistance device 1, it is possible to appropriately support the driver visually recognizing the surrounding situation at night or the like when the visibility is lowered for the driver. For this reason, it can contribute to improving driving safety even at night.

In the first embodiment, the infrared radar 2, the millimeter radar 3, the infrared camera 4, and the visible light camera 5 correspond to an example of a detection unit, and the processes of S114, S128, and S140 to S148 correspond to an example of a recognition unit. The processing of S152 corresponds to an example of analysis means, and S164, S168, S172, S174, S184, S186, S192, S196, S204, S206, S208, S212, S218, S220, S232, S238, S240, S260, The processing of S264, S268, S270, S284, S286, S292, S294, and S660 corresponds to an example of a setting unit, the processing of S154 corresponds to an example of a generation unit, and the processing of HUD7 and S130 corresponds to an example of a display unit. To do.

Further, the process of S148 corresponds to an example of a determination unit, and the ROM 20b or the flash memory 20d corresponds to an example of a storage unit.
The conversation symbols M1, M2, the mobile terminal symbol M3, and the headphone symbol M4 correspond to examples of symbolic symbols.

Second Embodiment
A second embodiment of the present invention will be described with reference to FIGS.
The driving support device 100 (see FIG. 21) of the second embodiment is different from the driving support device 1 (see FIG. 2) of the first embodiment in that an image projecting device 9 is provided.

The driving support device 100 is different from the driving support device 1 in the following points.
First, instead of the recognition process of FIG. 4, the recognition process (2) of FIG. 22 is executed.

Moreover, the analysis process 8 of FIG. 24 is performed.
Further, instead of the display process of S130 in FIG. 3 (display process of FIG. 16), the display process (2) of FIG. 25 is executed.

[Image projection device]
The image projecting device 9 is a device for projecting an image to an area in an environment outside the vehicle, which can project an image as a screen. A region where an image can be projected as a screen can be detected by analyzing an image captured by the infrared camera 4 or the visible light camera 5.

The image projector 9 has a laser projector 9a, performs signal processing by the laser projector 9a based on a signal from the control ECU 20 (in other words, generates a display image signal), and includes an optical system including a mirror, a lens, and the like. An image is projected through the unit 9b.

[Recognition process (2)]
Next, the recognition process (2) performed by the driving support device 100 will be described with reference to FIG.

22 is different from the recognition process in FIG. 4 in that the screen determination process in S550 is executed. Since the processing of S140 to S156 is the same as the recognition processing of FIG. 4, the description is omitted here.

In the recognition process (2) of FIG. 22, when the control ECU 20 determines that the object is “others” in S148, the control ECU 20 executes a screen determination process in S550.
[Screen judgment processing]
The flow of the screen determination process is shown in FIG.

The screen determination process is a process for determining whether or not an image can be projected onto the area of the object determined to be “other”.
When starting the screen determination process of S550 (screen determination process of FIG. 23), the control ECU 20 first determines whether or not the area of the object region is equal to or larger than the predetermined area S in S560. This process is a process for determining whether or not the image has a sufficient area (area) for projecting an image.

If it is determined in S560 that the area of the object region is not equal to or larger than the predetermined area S, it is determined that projection is impossible, and the process is terminated.
On the other hand, if it is determined in S560 that the area of the object region is equal to or larger than the predetermined area S, the process proceeds to S562.

In S562, it is determined whether the distance from the host vehicle to the object is a distance at which an image can be projected.
If it is determined in S562 that the distance is not projectable, the process ends.

On the other hand, if it is determined in S562 that the distance can be projected, the process proceeds to S564.
In S564, processing for estimating the flatness of the surface of the object region (flatness estimation processing) is executed. In the flatness estimation process, image analysis of the object area is performed, and the flatness of the surface of the object area is estimated.

Here, the difference and variation in the brightness (brightness and darkness) of the image increases as the degree of unevenness of the surface of the object region increases. On the other hand, the smaller the unevenness of the surface of the object area, the smaller the difference and variation in the luminance (brightness) of the image. On the premise of such a feature, the flatness is estimated by analyzing the difference (brightness and darkness) in the luminance (brightness and darkness) of the image of the object region.

After S564, the process proceeds to S566, and it is determined whether or not the estimated flatness is equal to or less than a predetermined threshold F (assuming that the smaller the flatness value is, the flatter it is).
If it is determined in S566 that the flatness is not less than or equal to the predetermined threshold value F, the process is terminated.

On the other hand, if it is determined in S566 that the flatness is equal to or less than the predetermined threshold value, the process proceeds to S568.
In S568, a process for estimating the color of the surface of the object area (color estimation process) is executed. The color estimation process is executed to determine whether an image can be projected onto the surface of the object area.

Here, the characteristic that the absorption rate of infrared rays varies depending on the color of the surface irradiated with infrared rays is used. White objects have a relatively low infrared absorption rate (in other words, infrared reflectance is relatively high. On the other hand, black objects have a relatively high infrared absorption rate (in other words, infrared reflectance). Is relatively low).

Based on such characteristics, the infrared absorption rate of the object can be calculated by analyzing the intensity of the reflected infrared light irradiated by the infrared sensor 2 in consideration of the distance to the object. Based on the calculation result, the color of the object can be estimated.

Therefore, in S568, the color of the surface of the object area is estimated using the infrared sensor 2 by the method described above.
After S568, the process proceeds to S570, and based on the estimation result in S568, it is determined whether or not the color of the surface of the object region is a color capable of projecting an image.

If it is determined in S570 that projection is not possible, the process ends.
On the other hand, if it determines with projection being possible in S570, it will transfer to S572.
In S572, a flag (projectable flag) indicating that an image can be projected as a screen is set for the target object region.

Next, the process proceeds to S574, and information on the target object region (information such as coordinates, range, area, etc.) is stored. Thereafter, the process ends.
[Analysis process 8]
The control ECU 20 further executes an analysis process 8 as one of the analysis processes of S152 in FIG.

FIG. 24 shows the flow of the analysis process 8.
When the analysis process 8 is started, the control ECU 20 first determines in S580 whether or not a person exists in the blind spot of the vehicle (other vehicle) from the extracted positional relationship between the person and the vehicle. In this determination, the traveling direction of the other vehicle, the person and the objects (obstacles) around the other vehicle are extracted, and it is comprehensively determined whether or not the person exists in the view area of the driver of the other vehicle.

If it is determined in S580 that there is no blind spot, the process is terminated.
On the other hand, if it is determined in S580 that the object is in the blind spot, the process proceeds to S582, and a flag (projection execution flag) for projecting an image to the external environment of the vehicle is set. Thereafter, the process ends.

[Display processing (2)]
Next, the display process (2) will be described with reference to FIG.
In the display process (2) of FIG. 25, the processes of S540 to S544 are the same as the processes of S540 to S544 in FIG.

In the display process (2), if it is determined in S544 that there is no additional display image, the process proceeds to S590.
In S590, it is determined whether the projection enable flag and the projection execution flag are set. The projectable flag is a flag set in the above-described processing of S572 (see FIG. 23). The projection execution flag is a flag set in the process of S582 described above (see FIG. 24).

If it is determined in S590 that the projection enable flag and the projection execution flag are set, the process proceeds to S592.
In S592, the information (specifically, information on the coordinate value, range, area, etc.) of the object area that can be projected as a screen and stored in S574 is transmitted to the image projection device 9.

Next, the process proceeds to S594, and image data to be projected on the image projection device 9 is transmitted to the image projection device 9. This image data may be a part or all of the data of the image captured by the infrared camera 4 or a part or all of the data of the image captured by the visible light camera 5. Moreover, the data produced | generated by the process (refer FIG. 22) of S154 may be included.

With the processing in FIG. 25 (more specifically, the processing in S590 to S594), the image projecting device 9 can project an image onto a predetermined region (region in which an image can be projected) in the environment around the vehicle. .

The operation of the second embodiment will be described with reference to FIG.
In FIG. 26, the driving support apparatus 100 is mounted on a vehicle (own vehicle) K1. There is another vehicle K2 around the host vehicle K1. There are also objects H10 and H11.

The object H10 is a bag and the object H11 is a person (here, a duo). In FIG. 26, only the top of one person's head is visible.
When viewed from the direction of the other vehicle K2, the object H11 is hidden behind the object H10. That is, a positional relationship is formed in which the object H10 cannot be visually recognized by the driver of the other vehicle K2.

On the other hand, it is assumed that the presence of the object H11 can be visually recognized and detected from the host vehicle K1.
The driving support device 100 of the host vehicle K1 detects the objects H10 and H11 by analyzing the image data of the infrared camera 4 or the image data of the visible light camera 5. Further, the other vehicle K2 is detected.

For the object H10, a screen determination process is executed to determine whether an image can be projected as a screen.
Further, the positional relationship between the objects H10 and H11 and the other vehicle K2 is analyzed, and it is determined whether or not the object H11 is within the field of view of the driver of the other vehicle K2.

The driving support apparatus 100 transmits the image data of the object H11 to the image projection apparatus 9, and projects the image of the object H11 on a predetermined area (screen area that can project an image as a screen) Sc1 in the object H10. .

Although the object H11 existing behind the object H10 cannot be visually recognized from the driver of the other vehicle K2, the driver displayed on the screen area Sc1 of the object H10 indicates that the driver of the other vehicle K2 has the presence of the object H11. Can be recognized.

As described above, according to the second embodiment, it is possible to notify surrounding people (for example, a driver or a pedestrian of another vehicle) of the surroundings of the host vehicle.
<Third Embodiment>
A third embodiment of the present invention will be described with reference to FIGS.

The driving support device 101 (see FIG. 27) of the third embodiment is different from the driving support device 1 (see FIG. 2) of the first embodiment in that it includes a line-of-sight detection unit 10.
The driving support device 101 is different from the driving support device 1 in that the driving support processing (2) in FIG. 29 is executed instead of the driving support processing in FIG.

[Gaze detection unit]
The line-of-sight detection unit 10 is a device that is mounted in a vehicle and detects the line of sight by tracking the movement of the eyeball (pupil) of the driver of the vehicle by image recognition.

The line-of-sight detection unit 10 includes a CCD image sensor 10a, an LED light source 10b, and an image processing unit 10c.
The LED light source 10b irradiates invisible near infrared rays. This near infrared ray is emitted toward the eyes of the driver. In this case, the near-infrared ray is reflected by the cornea of the eye, and the position of the reflection can be detected as a bright portion compared to the surroundings. Further, the reflection position has a feature that it maintains a constant position even if the line of sight changes (even if the position of the pupil changes).

The line-of-sight detection unit 10 detects an eye image by the CCD image sensor 10a, and analyzes the eye image by the image processing unit 10c.
In the image analysis, the above-described reflection position (near-infrared reflection position) in the cornea and the position of the pupil are detected.

28A and 28B show examples of image analysis. FIGS. 28A and 28B are schematic views showing examples of imaging of the driver's eyes. However, the line of sight (pupil position) is different.
The pupil is darker in the eye than other parts, and the corneal reflection is brighter in the eye than other parts. In this image analysis, the pupil and corneal reflection are detected and the positional relationship between them is analyzed using this feature.

The corneal reflection appears at the most prominent part of the entire cornea and the position thereof is almost constant, and the direction of the line of sight is detected (estimated) from the positional relationship of the pupil with respect to the position of the corneal reflection.

As shown in FIGS. 28A and 28B, the direction of the line of sight may be estimated based on the direction connecting the center position of corneal reflection and the center position of the pupil. Alternatively, the direction of the line of sight may be estimated based on research data on the relationship between the position of corneal reflection, the position of the pupil, and the direction of the line of sight, past learning values, and the like.

[Driving support processing (2)]
Next, driving support processing (2) executed by the driving support device 101 will be described with reference to FIG.

29 is different from the driving support process in FIG. 3 in that the correction determination process in S600 and the display correction process in S602 are executed after the process in S130. Since the processing of S100 to S130 is the same as the driving support processing of FIG. 3, the description thereof is omitted here.

In the correction determination process of S600, the control ECU 20 indirectly determines whether or not the driver has recognized the warning image based on the detection result of the line-of-sight detection unit 10, and reconstructs the warning image based on the determination result ( Correction).

The display correction process in S602 is a process in which the control ECU 20 corrects the display image based on the result of the correction determination process in S600.
The correction determination process will be specifically described with reference to FIG.

When the control ECU 20 starts the correction determination process of S600 (the correction determination process of FIG. 30), first, the control ECU 20 communicates with the line-of-sight detection unit 10 in S610.
Next, the process proceeds to S612, and analysis data by the line-of-sight detection unit 10 (in other words, data indicating movement of the line of sight) is acquired.

In subsequent S614, based on the analysis data acquired in S612, an object in a range in which the line of sight is assumed to be extracted is extracted.
Next, the process proceeds to S616, and the same process as the process of S612 and S614 is repeated a predetermined number of times. This process is executed for the purpose of monitoring (tracking) the movement of the driver's line of sight for a predetermined period.

In subsequent S618, for each of the objects extracted in S614 and S616, it is determined whether or not the driver's line of sight has moved more than a predetermined number of times to the area where the object exists. In this process, when the driver's line of sight is moved only once, it is not certain whether or not the object to be moved (in other words, the emphasized image) is surely recognized, and recognition is performed when the driver has moved a predetermined number of times. It is executed for the purpose of determining that

If it is determined in S618 that the user has not moved more than a predetermined number of times, the process is terminated.
On the other hand, if it is determined in S618 that the movement has been performed a predetermined number of times or more, a flag (image erasure flag) indicating that the image to be emphasized is erased in association with the object (person) in the area determined to have moved the predetermined number of times or more. ) Is set.

Next, the process proceeds to S504, and an emphasized image generation process is executed. The process of S504 is the same as the process of S504 in FIG.
In the emphasized image generation process of S504 in FIG. 30, except for the object (person) for which the flag for erasing the image to be emphasized is set, the other objects (persons) are newly processed in S520 to S524 (FIG. 15). ).

Thereby, in some cases, an object (person) belonging to a group with a lower alert level is moved up to a group with a higher alert level, and an image with a higher alert level can be set.

Returning to FIG. 30, after S504, the processing is terminated.
Next, the display correction process in S602 will be described with reference to FIG.
When the display correction process in S602 (display correction process in FIG. 31) is started, first, in S630, it is determined whether or not an image deletion flag is set. The image erasure flag is set in association with the object (person) in the processing of S620 described above.

If it is determined in S630 that the image deletion flag is not set, the process proceeds to S634.
On the other hand, if it is determined in S630 that the image deletion flag is set, the process proceeds to S632.

In S632, for the object (person) corresponding to the image erasure flag, a command for erasing the image for emphasizing the object (person) is generated and transmitted to the HUD 7. This causes the HUD 7 to stop displaying (erase) the display of the image to be erased. Next, the process proceeds to S634.

In S634, it is determined whether or not the image display mode has been reset. In other words, it is determined whether or not the process of S504 in FIG. 30 has been executed.
If it is determined in S634 that the image display mode has been reset (the process in S504 has been re-executed), the process proceeds to S636.

In S636, a signal representing an image to be displayed is transmitted to the HUD 7 based on the processing result of S504 in FIG. Thereby, the image is superimposed and displayed on the front window of the vehicle via the HUD 7. Thereafter, the process ends.

As described above, the driving support apparatus 101 according to the third embodiment displays an image that emphasizes objects around the vehicle in a superimposed manner on the front window, and determines whether the driver of the vehicle has recognized the image. Judgment is made by detecting the driver's line of sight. Then, the display of the image for which the determination result that the driver of the vehicle recognizes is obtained is stopped (in other words, deleted). Then, the display mode is reset. Thereby, another image (an image for emphasizing another object (person)) is further emphasized and displayed. In addition, an object (person) for which an image has not been set can be newly set and emphasized.

According to this, it becomes possible to make the driver recognize the existence of the object more effectively or efficiently.
For example, it is possible to avoid displaying an image for emphasizing the same object even though the driver recognizes it, and the utility is further improved.

In the third embodiment, the line-of-sight detection unit 10 corresponds to an example of a line-of-sight detection unit, the process of S618 corresponds to an example of an identification unit, and the processes of S620 and S602 correspond to an example of an erasure unit.

<Fourth embodiment>
A fourth embodiment of the present invention will be described.
In the fourth embodiment, the configuration of the driving support device is the same as the configuration of the driving support device 1 of the first embodiment (see FIG. 2).

On the other hand, the fourth embodiment is different in that the recognition determination process (2) in FIG. 32 is executed instead of the recognition determination process in FIG.
The recognition determination process (2) in FIG. 32 is different from the recognition determination process in FIG. 13 in that the processes of S650 to S658 are executed. Note that the processing of S400 to S408 is the same as the recognition determination processing of FIG.

In the recognition determination process (2) of FIG. 32, if the control ECU 20 determines that both eyes cannot be detected in the process of S404, the process proceeds to S650.
In S650, an alarm process for issuing an alarm to the object (person) is executed. Specifically, a process of emitting a predetermined sound (including sound) from the speaker unit 8 is executed. The alarm is issued to make the object (person) aware of the existence of the own vehicle. Also, it is issued to determine whether or not the object (person) has noticed the existence of the own vehicle. Note that an alarm may be given by turning on or blinking the headlamp of the vehicle.

Next, the process proceeds to S652, and the image data of the infrared camera 4 or the visible light camera 5 is acquired again.
In the subsequent S654, the face area of the same object (person) is re-extracted based on the image data reacquired in S652.

Next, the process proceeds to S656, and the extracted face is analyzed. More specifically, “eyes” are extracted by edge detection and pattern matching.
In step S658, it is determined whether both eyes have been detected.

If it is determined in S658 that both eyes have been detected, it is determined that the host vehicle is likely to exist within the field of view of the object (person), and the object (person) is determined based on this determination. A simple determination is made that the presence is recognized, and the flow proceeds to S406.

On the other hand, if it is determined in S658 that both eyes cannot be detected, it is determined that the host vehicle may not exist within the field of view of the object (person). Based on this determination, the object (person) A simple determination is made that the presence of the vehicle is not recognized, and the flow proceeds to S408.

As described above, according to the fourth embodiment, when it is determined that the object (person) does not recognize the presence of the own vehicle, the driving support device 1 issues an alarm to make the presence of the own vehicle noticeable. Thereby, an object (person) can be made to recognize presence of the own vehicle. Further, by reanalyzing the face of the object (person) after issuing the alarm, it is determined whether or not the object (person) notices the presence of the own vehicle. As a result, the danger (warning level) can be set appropriately in accordance with the state of the object (person). As a result, it is possible to display an image for warning the driver more appropriately.

<Fifth Embodiment>
A fifth embodiment of the present invention will be described.
In the fifth embodiment, the configuration of the driving support device is the same as the configuration of the driving support device 1 of the first embodiment (see FIG. 2).

On the other hand, the fifth embodiment is different from the driving support device 1 of the first embodiment in that the analysis process 9 of FIG. 33 is further executed.
Hereinafter, the analysis process 9 will be specifically described.

The processes of S400 to S404 in the analysis process 9 are the same as the processes of S400 to S404 in FIG. 13 (and the processes of S400 to S404 in FIG. 32).
Further, the processing of S650 to S658 in the analysis processing 9 is the same as the processing of S650 to S658 in FIG. Description of these processes is omitted.

The control ECU 8 executes the analysis process 9 in addition to the analysis processes 1 to 7 shown in FIGS. 5 to 12 (and FIG. 13) as the analysis process of S152 in FIG. 4, and in the analysis process 9, in S404 or S658 If it is determined that both eyes have been detected, the process proceeds to S660.

In S660, the alert level is decremented by 2 points for the target object (person) determined to have detected both eyes. The numerical value of 2 points is an example, and any value may be used.

The purpose of the process of S660 is that the object (person) can determine that the object (person) recognizes (or has noticed) the existence of the own vehicle by an affirmative determination in S404 or S658. The purpose is to lower.

As described above, according to the fifth embodiment, the warning level is lowered for the object (person) that recognizes the existence of the own vehicle, and thereby the other object (person) (for example, the existence of the own vehicle is recognized). The level of vigilance for objects (persons) that do not exist is relatively increased. As a result, a more emphasized image can be displayed for an object (person) to be more careful. Therefore, the presence of the object can be recognized more effectively or efficiently by the driver.

<Modification>
Hereinafter, another example of the display mode will be described.
[Modification 1]
Modification 1 will be described with reference to FIG. In the example of FIG. 34, the objects H7 and H8 are extracted and recognized.

In the example of FIG. 34, as in the example of FIG. 19, the moving direction of the object is detected and displayed by arrows (arrows Y7 and Y8).
In addition, in FIG. 34, the positions and paths of the objects H7 and H8 after a predetermined time are estimated and displayed.

For the object H7, the position surrounded by the frame W7 is the current position, and this is the initial position at time tA.
The driving assistance apparatus 1 repeats acquisition and analysis of the image data of the infrared camera 4 or the visible light camera 5, and executes a tracking process for tracking the movement of the object H7. Then, based on the tracking process, the moving speed and moving direction of the object H7 are estimated.

Then, the position and speed of the object H7 after a predetermined time tB are estimated, and an image of the object H7 is displayed so as to be superimposed on the estimated position. This image is displayed blinking.

Further, the position and speed of the object H7 after a predetermined time tC (tB <tC) are estimated using the time tA as a reference, and an image of the object H7 is displayed so as to be superimposed on the estimated position. This image is displayed blinking.

Further, the image after the predetermined time tB and the image after the predetermined time tC are continuously displayed as if the object H7 is moving.
In addition, a movement estimation arrow YF7 is displayed so as to follow a locus from the initial position at time tA to a position after a predetermined time tC. The movement estimation arrow YF7 is a movement path of the object H7 and indicates a movement path that is predicted (or is determined to have a high possibility of movement).

The same applies to the object H8. For the object H8, the position surrounded by the frame W8 is the current position, and this is the initial position at time ta.
By the tracking process, the position and speed of the object H8 after a predetermined time tb are estimated, and an image of the object H8 is displayed so as to be superimposed on the estimated position. This image is displayed blinking.

Further, the position and speed of the object H8 after a predetermined time tc (tb <tc) are estimated using the time ta as a reference, and an image of the object H8 is displayed so as to be superimposed on the estimated position. This image is displayed blinking.

Further, the image after the predetermined time tb and the image after the predetermined time tc are continuously displayed as if the object H8 is moving. The object H8 is approaching the host vehicle, and the image of the object H8 is displayed so as to gradually increase.

In addition, a movement estimation arrow YF8 is displayed so as to follow the locus from the initial position at time ta to the position after a predetermined time tc. The movement estimation arrow YF8 indicates a movement path of the object H8 and is predicted (or determined to have a high possibility of movement).

[Modification 2]
Modification 2 will be described with reference to FIG.
In the example of FIG. 35, the driving support apparatus 1 is mounted on a vehicle K3. The vehicle K4 is another vehicle. The vehicles K3 and K4 are traveling in the same traveling direction (from the bottom to the top (from the front to the back) in the drawing).

In front of the vehicle K3, there are pedestrians (objects) H12 and H13 that cross the pedestrian crossing.
For the driver of the vehicle K4, the objects H12 and H13 are behind the vehicle K3 and are difficult to see.

When the driving support device 1 of the vehicle K3 analyzes the positional relationship between the objects H12, H13 and the vehicle K4 and determines that the objects H12, H13 are difficult to visually recognize from the vehicle K3, the driving support device 1 in the vehicle Sc3, for example, in a region Sc2 in the rear window. Images of the objects H12 and H13 may be displayed. At this time, an image, a message, or the like that alerts the driver of the vehicle K4 may be displayed.

Accordingly, the driver of the vehicle K4 can easily recognize the existence of the objects H12 and H13, which is effective.
[Modification 3]
Modification 3 will be described with reference to FIG.

Here, in FIG. 17, an example in which an image for emphasis is displayed for a person and a vehicle has been described.
The example of FIG. 36 shows an example of displaying an image for emphasis on other objects around the vehicle in addition to the person and the vehicle. In addition, the name of the object is displayed.

Specifically, in the example of FIG. 36, in addition to pedestrians and vehicles, traffic lights, road signs, signboards, and road surface displays are extracted and recognized, and a frame W for emphasis is displayed in association with each other. I am doing so. In addition, a name display area N is set in association with each frame W, and the name of each object is displayed in the name display area N. Thus, it can be expected that the effect of attracting the driver's attention is also improved by displaying characters.

<Other embodiments>
In the above embodiment, an example in which the driving support device 1 includes the infrared radar 2, the millimeter wave radar 3, the infrared camera 4, and the visible light camera 5 has been described.

On the other hand, the infrared radar 2 may be omitted and the millimeter wave radar 3 may be configured to detect near and far objects.
In the above embodiment, an example in which the driving support device 1 includes the infrared radar 2, the millimeter wave radar 3, the infrared camera 4, and the visible light camera 5 has been described.

On the other hand, the mounting of the infrared radar 2 and the millimeter wave radar 3 may be omitted. In this case, for the distance to the object, information detected by each of the infrared camera and the visible light camera (information on the distance to the object) may be used.

In the above embodiment, an example in which the driving support device 1 includes the millimeter wave radar 3 has been described.
On the other hand, the driving support device 1 may include a laser radar instead of the millimeter wave radar 3. Alternatively, both a millimeter wave radar and a laser radar may be provided. The laser radar is a radar that detects a surrounding situation using laser light. Specifically, the laser radar scans the pulsed laser beam (two-dimensional scanning), and receives the laser beam that is reflected by the object and returned. Then, the laser radar measures the time difference between the emission time of the laser light and the reception time of the reflected light, and the intensity of the reflected light, and detects an object based on them. In the laser radar, in addition to a three-dimensional object, it is possible to detect a lane boundary line (a white line that forms a boundary such as a vehicle lane and a sidewalk).

In the above embodiment, the driving support devices 1, 100, 101 include the infrared radar 2, the millimeter wave radar 3, the infrared camera 4, the visible light camera 5, the momentum detection unit 6, the head-up display 7, the speaker. All of the unit 8, the image projection device 9, and the line-of-sight detection unit 10 may be mounted.

In the above embodiment, an example of displaying an image for emphasizing an object (an image surrounding the object) has been described, but an image of the object itself may be generated and displayed. In this case, the infrared cameras 4A and 4B or the visible light cameras 5A and 5B may be used to generate a stereoscopic image by capturing the object stereoscopically and display the stereoscopic image.

In the above embodiment, an illuminance sensor may be further provided, and the infrared camera 4 and the visible light camera 5 may be switched and used in accordance with detection data of the illuminance sensor. Specifically, when the illuminance is equal to or higher than a predetermined threshold (for example, during the day), the image data of the visible light camera 5 is used, and when the illuminance is lower than the predetermined threshold (for example, from evening to night, cloudy or rainy) In this case, the data of the infrared camera 4 may be used.

In the above embodiment, whether or not an object (person) is operating a mobile terminal may be determined by communicating with the mobile terminal.
For example, in the driving support devices 1, 100, and 101, Bluetooth (registered trademark) equipment or the like is mounted, and pairing with mobile terminals around the vehicle is attempted. When pairing is established, data communication may be performed with the mobile terminal, and data for determining whether or not the mobile terminal is operated may be acquired from the mobile terminal. In addition, when an application that detects that a user of the mobile terminal is moving and using the mobile terminal is installed on the mobile terminal, the mobile terminal is operated in conjunction with the application. May be detected. Further, a warning image may be transmitted from the driving support devices 1, 100, 101 to the mobile terminal, and the image may be displayed on the mobile terminal so that the user of the mobile terminal can recognize the presence of the vehicle.

In the above embodiment, the analysis processes 1 to 9 have been described. The example in which the analysis processes 1 to 9 are sequentially executed in parallel or in a predetermined order has been described.
On the other hand, after any one of the analysis processes 1 to 9 is executed, display data is generated based on the result of the analysis process (the process of FIG. 14 is executed), and then the next analysis process is executed. Display data may be newly generated based on the result of the analysis process (in other words, display data may be generated (corrected) every time the analysis process is executed).

In the above embodiment, the inter-vehicle communication unit may communicate with another vehicle to warn the driver of the other vehicle. Or you may make it receive warning information from another vehicle.

In the said embodiment, the position of the own vehicle may be acquired via the vehicle position sensor 12, and based on the acquired position, you may make it grasp | ascertain the positional relationship with an object.
In the above-described embodiment, the example in which the analysis processes 1 to 9 (FIGS. 5 to 12 (and FIG. 13), FIG. 24, and FIG. 33) are executed as the analysis process of S152 has been described. Here, another example of the analysis process will be described.
[Analysis process 10]
The analysis process 10 will be described with reference to FIG. The analysis process 10 is a process for analyzing the weather, and more specifically, a process for detecting rainy weather.

The analysis process 10 can be repeatedly executed by the control ECU 20 at a predetermined timing.
In the analysis process, first, in S670, image data of an image captured by the visible light camera 5 is acquired. The visible light camera 5 is provided in the vehicle so as to take an image of the surroundings of the vehicle from inside the vehicle through the window glass of the vehicle (see FIG. 1). Here, the image data including the image of the area of the window glass of the vehicle is acquired as the image data of the image captured by the visible light camera 5.

Next, the process proceeds to S672, and raindrop candidates (candidates for raindrops attached to the window glass) in the window glass region are detected from the image data acquired in S670. FIG. 38A shows an example of an image of raindrops. When raindrops adhere to the window glass of the vehicle, the raindrop portion becomes blurred and is detected with a transparency different from the transparency of the windowglass portion. At the boundary between the raindrop and the window glass, there appears a portion (edge) where the change in chromaticity (density) of adjacent pixels is steep. Based on such an edge, a raindrop region is detected.

Returning to FIG. 37, after S672, the process proceeds to S674, and it is determined whether or not the raindrop candidate detected in S672 is a raindrop by pattern matching.
Specifically, a raindrop image model is stored in advance in a storage device. As the storage device, a ROM 20b, a flash memory 20d, and the like can be considered, but other storage devices may be used.

FIG. 38B shows models 1, 2, 3,... N as examples of raindrop image models stored in advance in the storage device. As models 1, 2, 3,..., N, representative models representing raindrops can be arbitrarily selected and stored. The control ECU 20 may be provided with a function (learning function) for accumulating a raindrop image model.

Then, the raindrop candidate detected in S672 is compared with the models 1, 2, 3,... N stored in the storage device, and the area (number of pixels), color between the raindrop candidate and the model is compared. The degree of similarity is calculated for parameters such as degree (shading) and shape. Here, at least one of the parameters may be used. The raindrop candidate may be determined to be a raindrop if the at least one parameter matches a predetermined ratio.

Note that the processing in S672 and S674 can be executed for all raindrop candidates in the image data acquired in S670. Alternatively, a predetermined area may be extracted from the image data, and the processes of S672 and S674 may be executed only for the extracted area.

After S674, the process proceeds to S676, and the amount of raindrops (in other words, the amount of rainfall) is detected. For example, the amount of raindrops can be detected from the number of raindrops and / or the ratio of the area occupied by the raindrops in the image.

Next, the process proceeds to S678, and it is determined whether or not the amount of raindrops is a predetermined amount or more. If it is determined that the amount of raindrops is not equal to or greater than the predetermined amount (less than the predetermined amount), the process is terminated as it is. On the other hand, if it is determined that the raindrop amount is greater than or equal to the predetermined amount, the process proceeds to S680, and the warning level is incremented by one point.

According to such a configuration, raindrops can be detected with high accuracy, and as a result, rainy weather can be detected more reliably and accurately. And when it is detected that it is raining, the alert level is raised in the driving support devices 1, 100, 101, and objects around the vehicle are displayed by a more appropriate method (for example, a more conspicuous method). You can get it. The weather greatly affects the visibility of the driver, and it is usually difficult for the driver of the vehicle to recognize the surroundings of the vehicle particularly in case of rain. On the other hand, according to the present invention, it is possible to detect that it is raining and display it in an easier-to-see manner according to it, which can contribute to driving safety.
[Analysis process 11]
Incidentally, the weather may be recognized by a process different from the analysis process 10. Specifically, the weather may be recognized by the analysis process 11 shown in FIG.

In the analysis process 11 of FIG. 39, first, weather information is acquired by communication with the outside in S682. Specifically, the driving support devices 1, 100, 101 may be provided with a communication device for connecting to a communication line network (for example, the Internet network). Note that such a communication device is well known and will not be specifically described and illustrated. And the driving assistance apparatus 1,100,101 should just connect to an internet network through the communication apparatus, for example, and should acquire weather information.

Next, the process proceeds to S684, and illuminance data is acquired from an illuminance sensor (not shown) for detecting illuminance (outdoor illuminance) included in the vehicle.
Next, the process proceeds to S686, where temperature data is acquired from a temperature sensor (not shown) for detecting an external temperature provided in the vehicle.

Next, the process proceeds to S688, and it is comprehensively determined whether or not it is rainy based on the data acquired in S682 to S686.
For example, although it can be said that it is possible to grasp the weather only by the processing of S682 (for example, it is possible by acquiring weather forecast information), it is not guaranteed that the weather forecast is 100% accurate, Often, weather forecasts are not made for pinpoint areas.

Therefore, here, as an example, an illuminance sensor and a temperature sensor are used, and in addition to weather forecast data, outdoor illuminance and temperature are detected and used to determine whether it is rainy or not, thereby detecting the weather more accurately. be able to. In addition, the humidity may be detected and used.

After S688, the process proceeds to S690, and it is determined whether the weather is clear based on the processes of S682 to S688. If it is determined to be clear, the process proceeds to S694.
In S694, the alert level is maintained at the current alert level (in other words, the process for changing the alert level is not executed). Thereafter, the process is terminated.

If it is determined in S690 that the weather is not clear, the process proceeds to S692. In S692, it is determined whether or not the weather is cloudy. If it is determined that it is cloudy, then the flow shifts to S696. In S696, the alert level is incremented by one point. Thereafter, the process is terminated.

If it is determined in S692 that it is not cloudy, it is determined that it is raining, snowing, etc., the process proceeds to S698, and the warning level is incremented by 2 points.
According to the analysis process 11, the weather can be detected (or determined) with higher accuracy as described above, and thus more appropriate driving support according to the weather can be realized. Specifically, the warning level can be set appropriately according to the weather, the objects around the vehicle are highlighted according to the appropriately set warning level, and / or according to the danger It is possible to appropriately control the warning display.

[Analysis process 12]
Next, the analysis process 12 will be described with reference to FIG.
The driving support devices 1, 100, 101 of the present embodiment may execute the analysis process 12 in addition to the analysis processes 1 to 11 or instead of the analysis processes 1 to 11. The analysis process 12 detects that the user is operating the mobile communication terminal while moving (for example, walking) in conjunction with a function on the mobile communication terminal side such as a smartphone, a tablet, or a mobile phone (more specifically, Is a process of detecting such a portable communication terminal.

In cases where the user operates the mobile communication terminal while walking, it is pointed out that the user's line of sight, interest, attention, etc. are directed toward the mobile communication terminal, and attention is not directed to the surrounding situation, which is very dangerous. Has been.

In view of this point, a function and application for detecting that the user is operating the mobile communication terminal while moving may be provided on the mobile communication terminal side. Specifically, it is detected whether the position of the mobile communication terminal is fluctuated by the GPS function or the like, and thus whether the user is moving. And it is detected whether the portable communication terminal or the portable communication terminal is operated when the user is moving.

When it is detected that such movement and operation overlap, a warning to that effect is issued.
In an example, a warning is displayed on the display screen of the mobile communication terminal, a sound is generated, or an alarm (a sound alarm or a signal indicating a warning) is issued to surrounding terminals.

The analysis process 12 is based on the premise that the mobile communication terminal has the functions and applications as described above.
In the analysis process 12, first, in S700, a process for searching for portable communication terminals existing around is executed. In this search, it is possible to search by detecting a Bluetooth (registered trademark) signal or other wireless signal emitted from the mobile communication terminal. In one example, a pairing signal for pairing with Bluetooth (registered trademark) is transmitted from the driving support devices 1, 100, 101, and the presence or absence of a response signal to the pairing signal is detected. Alternatively, the presence / absence of a pairing signal transmitted from the mobile communication terminal is detected. Alternatively, in another example, the mobile communication terminal may be detected by image processing using image data from the visible light camera 5.

Next, the process proceeds to S702, and based on the process of S700, it is determined whether or not there is a mobile communication terminal around the driving support devices 1, 100, 101.
If it is determined that there is no portable communication terminal, the process is terminated as it is. On the other hand, if it is determined that a mobile communication terminal exists, the process proceeds to S704.

In S704, it is determined whether or not a warning signal warning that the user is operating the mobile communication terminal while moving is received from the mobile communication terminal detected in S700 and 702. In addition, it is preferable that the specification of this type of warning signal is determined so that it can be detected unconditionally in surrounding communication devices. For example, it is preferable that only the warning signal can be detected without having to establish pairing using a Bluetooth (registered trademark) function. However, since this is a specification and not the essence of the present invention, it will not be described in detail here.

If it is determined in S704 that a warning signal has not been received, the processing is terminated as it is.
On the other hand, if it is determined that a warning signal has been received, the process proceeds to S706, where a warning is displayed on the HUD 7. In step S708, the alert level is incremented by one point. Thereafter, the process is terminated.

According to such an analysis process 12, it is possible to detect a person who is operating the mobile communication terminal while moving, to raise the alert level, and to appropriately alert the driver.
[Vehicle control processing]
The driving support devices 1, 100, 101 of this example may control the operation of the vehicle according to the alert level. Such an example will be described with reference to FIGS. 41A and 41B.

The driving assistance devices 1, 100, and 101 repeatedly execute the vehicle control process of FIG. 41A at a predetermined timing.
In this vehicle control process, first, in S710, it is determined whether the alert level is equal to or higher than a predetermined level. If it is determined that the alert level is not equal to or higher than the predetermined level, the process is terminated as it is.

On the other hand, if it is determined that the alert level is equal to or higher than the predetermined level, the process proceeds to S712, and a control command for controlling the vehicle is output. Specifically, this control command can be output to an electronic control unit (ECU) that controls each part of the vehicle. And ECU which received the control command controls a controlled object. After the process of S712, the process is terminated.

Examples of the vehicle control by the vehicle control process include throttle control for controlling the opening of the throttle valve, braking control for controlling a braking device (brake), steering control for controlling the traveling route or traveling direction of the vehicle, and the like. be able to.

The throttle control may be control that suppresses the throttle opening (in other words, prohibits acceleration). The braking control may be control that causes the vehicle to decelerate by causing the brake to function. For example, the steering control may be a control for controlling the traveling path of the host vehicle so that the host vehicle is separated from the target object around the host vehicle and the possibility of a collision. In addition, an alarm may be given. For example, a vibration mechanism may be incorporated in the handle, and the vibration mechanism may be vibrated according to a warning level, and an alarm may be transmitted to the driver by vibration (hereinafter, this type of alarm is also referred to as alarm control).

And the driving assistance apparatus 1,100,101 has the table information which matched the warning level and the content of vehicle control as shown to FIG. 41B. This table information can be stored in advance in a storage device (ROM 20b or the like). According to the table information of FIG. 41B, vehicle control is realized as follows.

If the value of the alert level is from-(minus) to 0, vehicle control is not executed.
When the value of the warning level is 1 to 3, warning control and throttle control are performed.
When the value of the warning level is 4 to 6, alarm control, throttle control, and braking control are performed.

When the value of the alert level is 7 or more, alarm control, throttle control, braking control, and steering control are performed.
Here, it goes without saying that the classification of the alert level is an example. The warning level classification may be further subdivided in multiple stages or conversely. In addition, since the variation in the alert level may vary depending on the type of analysis processing to be executed (in this example, analysis processing 1 to 12 is exemplified), the table information is optimal depending on the type of analysis processing to be executed. It can be understood by those skilled in the art that it can be realized.

According to such a configuration, not only the display mode but also the operation of the vehicle is controlled according to the alert level, so that it can contribute to safe driving of the vehicle. Specifically, the display mode is controlled in accordance with the warning level, so that the driver of the vehicle is alerted and the driver can be encouraged to drive safely. Vehicle control is executed in order to realize safe driving without the driver's judgment entering the judgment process 101.

Thereby, in addition to the driving | operation by driver | operator himself, safe driving | running | working of a vehicle is implement | achieved by a higher level by assistance of the driving assistance apparatus 1,100,101. Even if the driving skill of the driver is immature and the expected value of safe driving by the driver himself / herself is low, safe driving of the vehicle can be realized by controlling the driving support devices 1, 100 and 101.
[Display control processing, display mode example]
The display control processing and display mode examples according to the present invention will be further described with reference to the drawings.

First, a description will be given based on FIGS. 42A and 42B.
42A and 42B show an example of controlling the display contrast in accordance with the alert level. Here, the contrast is intended to be a contrast between a target object (person, obstacle, etc.) to be highlighted and a display object other than the target object.

42. As shown in the table D4 in FIGS. 42A and 42B, the warning level may be associated with the contrast level at the time of display. This information can be stored in a storage device (ROM 20b or the like) as table information. When displaying an image with the HUD 7, the driving assistance devices 1, 100, 101 read the information of the table D4 from the storage device, and read the contrast information corresponding to the set alert level. Contrast (contrast ratio) is low when the warning level is-(minus) to 0, medium when the warning level is 1 to 3, high when the warning level is 4 to 6, and when the warning level is 7 or higher May be set to the highest.

FIG. 42A shows an example when the contrast is low. FIG. 42B shows the case where the contrast is highest.
In the example of FIG. 42A, the contrast between the object D0 as the highlight target and the surroundings (background, etc.) is low, and the difference in brightness between the object D0 and the surroundings is small, but the warning level is low. If the contrast is low, priority may be given to the merit of reducing the contrast. The advantages of reducing the contrast may be that eye fatigue may be suppressed in some cases, and that naturalness may be prioritized and may be close to the actual scenery.

42B, the contrast between the object D0 and its surroundings (background, etc.) is high, and the difference in brightness between the object D0 and its surroundings is large. For this reason, the object D0 is more emphasized and can be visually recognized more clearly and clearly. When the alert level is high, the object D0 may be more emphasized by setting the contrast high in this way.

Regarding contrast setting, shading may be set for each representative region in the image.
This point will be described using the tables D3 and D3 ′.
Tables D3 and D3 ′ contain information on the shades set for each of the representative regions in the image. In the tables D3 and D3 ′, the blocks Da and Da ′ indicate the density of the area of the object D0, the blocks Db and Db ′ indicate the intensity of the background planting area of the object D0, and the blocks Dc and Dc ′. Indicates the density of the ground surface of the background of the object D0, and the blocks Dd and Dd ′ indicate the density of the road.

In the tables D3 and D3 ′, the larger the positive value, the darker (closer to black), and the smaller the positive value, or the larger the negative value, the thinner (closer to white).
The relative relationship between the shades of the blocks may be automatically set according to the default value according to the contrast level (low, medium, high, maximum).

Or, the relative relationship between the shades of each block may be set manually by the user (driver). For example, a menu display D2 may be provided, and when the menu display D2 is selected, the screen shifts to various setting screens, and the contrast can be set on such a setting screen.

According to this, since the user (driver) himself can set the contrast that is most easily seen by the user (driver), convenience is improved. Further, the display mode can be optimized for each individual user (driver), and the effect of driving support can be maximized.

The display mode is adjusted according to the skill of the user (driver) (whether or not he is a good driver), age, gender, accident history, violation history, physical ability (mainly visual acuity), physical condition during driving, etc. May be.

For example, a mechanism for reading a driver's license may be provided, and some of the above information may be automatically acquired by reading the driver's license. Information that cannot be read from the driver's license, such as visual acuity and physical condition during driving, may be configured to be manually input.

42A and 42B, a display D1 is a display for indicating that the display control is functioning normally. If any abnormality is detected in the display control, the display content of the display D1 is changed to the content indicating that an abnormality has occurred.

In the embodiment of the present application, as described above, the target object is highlighted. As a highlighting mode, the frame is surrounded by a frame, highlighted, changed in display color, blinked, and adjacent symbols are displayed. Various modes are prepared, and such highlighting may be canceled. That is, the display mode can change in real time.

In this case, even if some abnormality occurs in the display control and an incorrect display is made, the user (driver) does not notice that the display is incorrect unless the user (driver) can grasp the abnormality. Will be recognized based on incorrect display. In other words, there is a possibility that misunderstandings may deviate from the originally intended results.

In view of this point, by providing a display for indicating that the display control is functioning normally like the display D1, the user (driver) can be confident that the displayed screen is a normal screen. When an abnormality occurs, the user (driver) can recognize that fact, and the occurrence of misidentification due to the apologized display can be suppressed.

Note that a generally known general method can be applied to the abnormality detection method (detection process), and detailed description thereof is omitted here.
However, even if an abnormality detection method (detection process) is common, various ideas can be considered for its execution timing.

In one example, it may be executed at the following timing.
(1) Timing at which the vehicle ignition switch is turned on (2) Timing at which the vehicle actually starts running after the vehicle ignition switch is turned on (for example, timing at which tire rotation is detected)
(3) Timing at which the vehicle stopped once after traveling (timing when the vehicle stopped at a signal, crossing store, etc.) (4) Arbitrary timing during traveling of the vehicle (including repeated execution)
The above timings (1) and (2) are timings when the driving is just started, and the abnormality detection processing is executed at such timing, so that the display D1 is displayed, so that the user (driver) ) Can give a sense of security to future driving.

In the timing (3), since the vehicle is stopped, the degree of danger is small and the processing load for display control is small, or the processing load for display control is omitted by deliberately omitting the display control. You can do that. If the abnormality detection process is executed at such timing, it is possible to suppress an excessive increase in processing load. In addition, it is possible to suppress the risk that display control, for example, causes any abnormality due to an increase in processing load (for example, risk that processing delay occurs). For this reason, it can contribute to driving support with high safety and reliability.

The timing of the above (4) is the timing during traveling, and unless the display control is hindered from the viewpoint of processing load or the like, the presence or absence of abnormality can be notified to the user (driver) in real time. It is safer for the driver.

Further, the display D1 may be always displayed or may be displayed at an arbitrary timing.
When displaying at an arbitrary timing, the display timing may be set in accordance with the execution timing of the abnormality detection process as exemplified in the above (1) to (4). Specifically, the display D1 may be displayed in synchronization with the timing at which the abnormality detection process is executed (the timing at which the process is completed).

Next, another display mode example will be described with reference to FIG.
FIG. 43 is an example in which a dangerous area in an area (a road or the like) where the vehicle travels is highlighted. This example is an example in the case where a landslide disaster occurs and a part of the road is cut off by landslides.

The driving support devices 1, 100, 101 include the visible light camera 5 as described above, and as described above, the detection target reflected in the image obtained by performing image processing on the image captured by the visible light camera 5, or the detection target An area occupied by the detection target can be recognized. Therefore, the driving assistance devices 1, 100, and 101 may be configured to detect a landslide area and display the emphasis display D6 in a superimposed manner on the landslide area.

In this case, in addition to the highlight display D6, a text display D5 may be displayed adjacent to the highlight display D6. These text display D5 and highlight display D6 may be displayed blinking. Further, the display color may be changed. For example, the display mode may be changed according to the scale of the disaster.

Furthermore, areas other than the areas where landslides may occur may be additionally highlighted. In this case, an area where no landslide countermeasure is taken (for example, an area not covered with concrete), an area where water and / or a slight amount of earth and sand, etc. are flowing on the surface is detected by image processing, and the area is detected. May be highlighted.

It may be probabilistic that you will encounter a sediment disaster. However, it is because of this that when the driver actually encounters the driver, the driver cannot understand. In other words, the driver may not be able to understand and may lead to a serious accident. Therefore, it is possible to further improve the reliability and safety of driving support by responding to such highlighting during a disaster.

Next, another display mode example will be described with reference to FIG.
FIG. 44 assumes a scene in which a river is running along a river. In such a scene, an example of displaying the safety level or risk level of a river level (in other words, the risk of flooding a river) is displayed. It is.

For example, the water level of a river may be detected and the degree of danger displayed as an indicator. Specifically, an indicator display D10 may be provided. A gradation display may be adopted as the indicator display D10.

A danger display D12 and a safety display D13 can be provided in the vicinity of the indicator display D10.
And according to the water level of a river, the present water level frame D14 is superimposed and displayed on the indicator display D10. The closer the display position of the current water level frame D14 is to the danger display D12, the higher the river water level is, and the more dangerous it is. The closer the display position of the current water level frame D14 is to the safety display D13, the safer the display is.

The text display D11 is provided adjacent to the current water level frame D14. As the text display D11, the degree of danger (or safety) may be displayed as text.
Emphasis display D15 can be superimposed and displayed on the area occupied by the river. The display mode such as the display color and pattern of the highlight display D15 is preferably matched with the display mode such as the display color and pattern of the region surrounded by the current water level frame D14.

In this case, the control ECU 20 of the driving assistance apparatus 1, 100, 101 extracts the data of the display mode in the region surrounded by the current water level frame D14, and then applies the display mode to the display mode of the highlight display D15. Processing will be executed.

According to such an example, the user (driver) can recognize the possibility of the occurrence of the disaster rather than recognizing the fact of the disaster as illustrated in FIG. In addition, the user (driver) can intuitively grasp the danger level through the indicator display D10 indicating the danger level. In addition, since the display mode of the highlight display D15 matches the display mode of the indicator D10 (the display mode of the region currently surrounded by the water level frame D14), the ease of recognition for the user (driver) is significantly improved. obtain.

Next, another display mode example will be described with reference to FIGS. 45A-45C.
First, about FIG. 45A, one of the meanings is to display a target object in three dimensions. In FIG. 45A, structures D20 and D22 are buildings that exist along the road on which the host vehicle travels. About this kind of building, you may display in three dimensions using a three-dimensional display technique. As a stereoscopic display technique, a plurality of projectors (generally a pair of left and right projectors) that project from different directions are prepared, and a stereoscopic vision is realized by displaying a left-eye image and a right-eye image, respectively. It has been known.

Alternatively, map data including stereoscopic image data may be used. That is, an image represented by the stereoscopic image data may be displayed.
According to the three-dimensional display, it is expected that the user (driver) can more easily see.

In FIG. 45A, symbol displays D21 and D23 are drawn in accordance with the attributes of the structures D20 and D22, respectively.
The attribute includes the type of structure. Types of structures include stores, government offices, and private houses.

The attribute includes attached information attached to the structure. For example, in the case of a store, the attached information includes information such as business hours, store size, average number of visitors, and location information.
Such attribute information is attached to the map data, for example, and the driving support device 1, 100, 101 side may be obtained from the map data.

In the example of FIG. 45A, if the structure D20 is a convenience store, for example, information such as being a store, being open 24 hours, and having many visitors in the morning and evening is included in the attributes of the structure D20. obtain.

Based on such attributes, the control ECU 20 displays a symbol display D21 that calls attention to the presence or absence of vehicles from the parking lot on the basis of information indicating that there are many visitors 24 hours a day, and the structure D20. Display in association with. Displaying in association with the structure D20 is understood to mean, for example, displaying near the structure D20, displaying adjacent to the structure D20, or displaying superimposed on the structure D20. Also good.

Further, it is assumed that the structure D22 is located at a point where the road is curved, and information (location information) that “is located at a curve point” is included in the attribute. For example, based on such attributes, the control ECU 20 displays a symbol display D23 that prompts traveling along a curve in association with the structure D22.

According to this, the user (driver) can also recognize a potential risk associated with the road that is running.
FIG. 45B shows an example in which the oncoming vehicle is running outside the central lane.

When such an oncoming vehicle is detected, the control ECU 20 estimates a route on which the oncoming vehicle may travel by calculation, and displays the emphasis display D25 superimposed on the estimated area. However, such display control may not be performed when the oncoming vehicle is traveling in the lane. For example, such display control may be executed when an oncoming vehicle is running out of the lane and there is a risk of collision.

FIG. 45C shows an example in which the host vehicle is running outside the lane.
In such a case, the control ECU 20 estimates a route on which the host vehicle may travel by calculation, displays a symbol of the host vehicle, and displays a highlight display D27 in a superimposed manner on the estimated region.

By displaying the symbol of the own vehicle, the user (driver) can objectively recognize the danger. In a traffic accident, there is a case where there is a problem in the belief that “no problem, no problem”, that is, subjective judgment. By making it possible to grasp the situation objectively, it is possible to eliminate subjective feelings and contribute to accident prevention.

However, such display control may not be performed when the host vehicle is traveling in a lane. For example, such display control may be executed when the host vehicle is running out of the lane and there is a risk of collision.

It is preferable that the display mode of the highlight display D25 and the display mode of the highlight display D27 are different in an easily distinguishable form. In this case, it is easy to recognize whether the oncoming vehicle protrudes from the lane and is dangerous, or whether the host vehicle protrudes from the lane and is dangerous.

Although one embodiment of the present invention has been described above, those skilled in the art can understand that various aspects can be within the scope of the present invention.
For example, the display screen may be configured with a touch panel. Then, by selecting an object on the touch panel, the object may be highlighted or canceled.

Further, when traveling in a place where traffic accidents frequently occur, the warning level may be additionally increased (a point of the warning level is additionally incremented).

Claims (3)

1. Detection means for detecting the situation around the vehicle;
   Recognition means for recognizing objects around the vehicle based on the detection result of the detection means;
   Analyzing means for analyzing the object recognized by the recognition means;
   For the object, based on the analysis result of the analysis means, setting means for setting the degree to be alerted;
   Generating means for generating an image for allowing a vehicle operator to visually recognize the object based on the degree set by the setting means;
   Display means for displaying the image generated by the generating means;
   A driving support apparatus comprising:
2. Determination means for determining whether or not the object recognized by the recognition means is a person;
   Storage means for storing image data of symbolic symbols indicating the state of the person,
   The analysis means analyzes the state of the object determined to be a human by the determination means among the objects,
   The generating unit includes a reading unit that reads out, from the storage unit, image data corresponding to the symbol representing the analysis result and the symbol representing the human state based on the analysis result of the analysis unit,
   When the image data of the symbol is read by the reading means, the display means displays the symbol represented by the image data.
   The driving support device according to claim 1, wherein
3. Line-of-sight detection means for detecting the line of sight of the driver of the vehicle;
   Identification means for identifying an image recognized by the driver among images displayed by the display means from a movement state of the driver's line of sight, which is a detection result of the line-of-sight detection means;
   An erasing unit for erasing an image identified as recognized by the driver by the identifying unit;
   The driving support device according to claim 1, further comprising:

Images