JP2020095688A - Driving assistance device - Google Patents

Abstract

To provide a driving assistance device that notifies a driver of a vehicle about danger in accordance with a circumstance around the vehicle.SOLUTION: A driving assistance device comprises: detection means for detecting a circumstance around a vehicle; identification means for identifying an object around the vehicle on the basis of a detection result of the detection means; analysis means for analyzing the object identified by the identification means; setting means for setting an alert level on the basis of an analysis result of the analysis means; generating means for generating an image for making a driver of the vehicle visually recognize the object on the basis of the level set by the setting means; and display means for displaying the image generated by the generating means.SELECTED DRAWING: Figure 2

Description

Cross-reference of related applications

This international application claims priority based on Japanese Patent Application No. 2014-72419 filed with the Japan Patent Office on March 31, 2014, and is based on Japanese Patent Application No. 2014-72419. The entire contents are incorporated into the present international application by reference.

The present invention relates to a driving support device and a driving support system that notify a driver of a vehicle of a danger according to a situation around the vehicle.

2. Description of the Related Art Conventionally, there is known a display device capable of projecting an external situation or a landscape on a window of a vehicle (for example, see Patent Document 1).
The display device described in Patent Document 1 includes an observation device for observing the state of the vehicle (position, speed, etc.) and a storage device for preliminarily accumulating image information of the outside scenery. Based on the information representing the observed position of the vehicle, the image information of the landscape that may be seen outside the vehicle at the observed position is acquired from the storage device, and the image represented by the image information is displayed on the window of the vehicle. ..

JP, 2004-20223, A

However, according to the conventional image display device as described above, the landscape image previously stored in the storage device is only displayed on the window of the vehicle. That is, it is impossible to detect the situation around the vehicle in real time and notify the driver of the vehicle of the situation. Therefore, the driver cannot recognize the danger that occurs during driving (for example, the risk of collision with an object around the vehicle).

It is desirable to be able to inform the driver of the vehicle of the danger according to the situation around the vehicle in an easily understandable manner.

The driving support device according to the first aspect of the present invention includes a detection unit that detects a situation around the vehicle, a recognition unit that recognizes an object around the vehicle based on a detection result of the detection unit, and the recognition unit. Analyzing means for analyzing the recognized object, setting means for the object based on an analysis result of the analyzing means, and setting means for setting a degree of caution, and allowing the driver of the vehicle to visually recognize the object. And a display unit for displaying the image generated by the generating unit, the generating unit generating the image for generating the image based on the degree set by the setting unit.

The term “analysis” means to judge, determine, or estimate the type and state of the target object by various analyses.
According to such a driving support device, a degree to be alerted (hereinafter also referred to as alert level) is set for each object according to the type and state of the object around the vehicle, and according to the alert level. Images will be displayed. For this reason, the driver of the vehicle can understand to what degree he should be warned (should be warned) according to the type and state of the object.

As a result, the vehicle can be driven appropriately. For example, when an image indicating that the alert level is high is displayed for the object, it may be possible to decelerate or perform a predetermined steering operation to avoid danger. The term "danger" refers to the risk of collision.

Further, the generation means may generate the following image as an image for allowing the driver of the vehicle to visually recognize the object.
-Image surrounding the object-Image pointing to the object (for example, arrow image)
-An image that schematically represents the target object (such as an image of an illustration of the target object)
Image of Message Informing Presence of Object The above-mentioned images may be displayed alone or in combination by the display means.

According to this, the driver can easily recognize the existence of the target object.
When the images are combined and displayed, a plurality of images may be displayed simultaneously or may be displayed with a time difference. When the images are displayed with a time difference, specifically, the next image may be displayed after a predetermined time from the display of one image. For example, an image of an arrow pointing to the target object may be displayed after a predetermined time has passed since the image surrounding the target object was displayed.

In one example, when an object is detected, first, an image surrounding the object is displayed, and then, when the distance from the vehicle to the object is equal to or less than a predetermined distance, the object is detected. An image of a pointing arrow may be displayed. According to such an aspect, the driver can be continuously supported so that the target object can be easily recognized.

The driving support device further includes a determination unit that determines whether or not the object recognized by the recognition unit is a person, and a storage unit that stores image data of a symbol representing the state of the person. The analyzing means analyzes the state of the object determined to be a person by the determining means among the objects, and the generating means indicates the analysis result based on the analysis result of the analyzing means. A reading means for reading image data corresponding to a symbol indicating the state of the person from the storage means is provided, and the display means displays an image of the image data of the symbol when the reading means reads the image data. It may be configured to display the symbolic symbol represented by the data.

According to this, an image can be displayed particularly when the object is a person, which can further contribute to improving driving safety. Further, by displaying a symbol indicating the state of the person according to the state of the person, the driver can recognize the state of the person around the vehicle. Therefore, the driver can perform an appropriate driving in consideration of the states of people around the vehicle.

In addition, the driving assistance device is a detection result of the line-of-sight detection unit that is an image recognized by the driver among the line-of-sight detection unit that detects the line-of-sight of the driver of the vehicle and the image displayed by the display unit. An identifying unit that identifies the driver's line-of-sight movement state, and an erasing unit that erases an image identified by the driver as recognized by the driver may be provided.

According to this, when the driver recognizes the target object, the image for visually recognizing the target object can be erased. Therefore, the image continues to be displayed even if the driver recognizes the image (in other words, recognizes the existence and state of the object) (in other words,
It is possible to prevent the driver from continuing to be warned).

On the other hand, the image not recognized by the driver is continuously displayed, and the warning is continued in that part. Therefore, the effect of improving driving safety is not lost. According to this, the usability and the effect of driving support can be compatible at a high level.

Further, in one example, the driving support device may include at least one imaging device as the detection unit. If the image of the surroundings of the vehicle is taken by the imaging device, the type and state of the object around the vehicle can be analyzed in more detail by image analysis. Moreover, various image analysis methods are known, and the analysis can be relatively easily performed using the conventional method.

When the detection unit includes a plurality of image pickup devices, the driving support device (or the detection unit) may be configured to compare the image pickup images of the image pickup devices and select the image pickup image to be adopted based on the comparison result. good. Further, it may be configured such that a highly accurate portion (a portion having a small noise component etc.) is extracted from the data of each captured image and such portions are integrated to generate one data. According to this, the accuracy of image analysis can be further improved. As a result, it is possible to detect and grasp the situation around the vehicle at a high level.

In addition, the detection unit reproduces parallax (difference in image position and viewing direction) using a plurality of (specifically, two) imaging devices, and based on the parallax, stereoscopic information (specifically, depth) of the object. Information) may be acquired. According to this, the driving assistance device can also set the alert level based on the three-dimensional information of the object. Further, in one example, the driving support device may display a stereoscopic image (3D image) of the target object. In this case, the display unit may be configured to display a stereoscopic image (3D image) of the object based on the detection result (stereoscopic information) of the detection unit.

Further, the driving support device (or the detection unit) may be configured to identify a plurality of objects overlapping in the field of view of the imaging device from the stereoscopic information (depth information) of each object. For example, the recognition unit may be configured to recognize a plurality of overlapping objects as separate objects from the stereoscopic information (depth information) of each object.

Here, a plurality of objects existing in the same direction and partially overlapping each other when viewed from the imaging device are recognized as one and the same object only by image analysis based on the two-dimensional information. On the other hand, based on the stereoscopic information (depth information), such a plurality of objects can be identified as separate objects.

In this case, the generation unit may be configured to generate an image for the driver to visually recognize each of the plurality of target objects that partially overlap each other. In this case, each of the plurality of objects may be easily recognized by changing the mode of the image.

According to the above configuration, the driver can be more accurately notified of the situation around the vehicle. In addition, the driver can more easily understand the situation around the vehicle. For example, it may be easier for the driver to recognize a separate object hidden behind the object.

Next, the analysis means may be configured to analyze whether or not the object is present on the traveling route of the host vehicle (vehicle on which the driving support device is mounted). Specifically, the road may be recognized by the detecting unit and the recognizing unit, and the analyzing unit may analyze whether or not the object is present on the road. Further, when the route of the vehicle is estimated from the motion state of the vehicle or the like, it may be analyzed whether or not an object exists on the route.

In this case, the setting means may set the warning level relatively high for the object existing on the traveling route. On the other hand, for objects that do not exist on the travel route, the alert level may be set relatively low.

Further, the analysis means may be configured to analyze the distance from the own vehicle to the object.
For example, the analysis can be performed based on the detection result of the detection means. In one example, the distance can be calculated by performing image analysis on the captured image of the image capturing device serving as the detection unit. Further, if the detection means includes a distance sensor, the distance to the object can be calculated based on the output result (output signal) of the distance sensor.

In this case, the setting means may set the warning level to be relatively high for an object whose distance from the host vehicle to the object is relatively small. On the other hand, the warning level may be set to be relatively low for an object whose distance from the own vehicle to the object is relatively large.

Further, when the object is a person, the analysis means may be configured to analyze whether or not the person has a mobile terminal such as a mobile phone, a smartphone, or a tablet. Specifically, the imaged image of the imaging device as the detection means may be subjected to image analysis. For example, the brightness of the display portion becomes bright during the operation of the mobile terminal, and the boundary of the display can be detected as an edge in the image analysis. It may be configured to analyze whether or not the mobile terminal is operating (whether or not the mobile terminal exists) based on the fact that the display can be recognized by such edge detection.

Further, the analysis means may be configured to analyze whether or not a person is operating the mobile terminal. The analysis includes whether or not the mobile terminal is in operation, the relationship between the positions of parts of the human body (particularly the positions of the hands and face) and the position of the mobile terminal, the face orientation, and the positional relationship between both eyes with respect to the mobile terminal. Etc. may be included in the analysis. Then, based on those analyzes, it may be determined whether or not a person is operating the mobile terminal. Further, the analysis means may be configured to analyze whether or not a person is talking on the mobile terminal.

In this case, the setting means may set the alert level to a relatively high level for an object that is operating the mobile terminal and an object that is in a call. On the other hand, the alert level may be set to be relatively low for an object not operating the mobile terminal and an object not in a call.

In addition, the analyzing unit may be configured to analyze (or estimate) whether or not the person recognizes the presence of the own vehicle when the person is operating the mobile terminal. For example, if a person's face is analyzed and both eyes can be extracted, it is determined that the person is facing the direction of the host vehicle, and it is determined that the person recognizes the presence of the host vehicle. Is also good. On the other hand, when both eyes cannot be extracted, it may be determined that the person is not facing the direction of the own vehicle, and it may be determined that the person does not recognize the existence of the own vehicle.

In this case, the setting means may set the warning level relatively high for the object for which the presence of the own vehicle is not recognized.
On the other hand, the setting means may set the warning level relatively low for the object whose presence of the own vehicle is recognized. Also, the alert level may be lowered.

The analyzing means analyzes the reaction of the person after the driving support device has issued an alarm to the person, and whether or not the person recognizes the existence of the own vehicle (in other words, It may be configured to analyze whether or not the existence is recognized). For example, both eyes may be extracted as described above. Also, the movement of the face may be analyzed. For example, when it can be detected that the face of the person is facing the host vehicle, it may be determined that the person is aware of the presence of the host vehicle.

Further, the analysis means may be configured to analyze a person's sex and age by using face recognition technology.
Further, the analysis means may be configured to analyze whether or not a person is using headphones. In addition, the analysis unit may be configured to analyze (or estimate) whether or not the person recognizes the presence of the own vehicle when the person uses the headphones.

Also, the analysis means may be configured to analyze whether or not a person is in conversation. Further, the analyzing means may be configured to analyze (or estimate) whether or not the person recognizes the presence of the own vehicle when the person is talking.

Further, the analysis means may be configured to analyze the moving state of the person. Specifically, the moving direction of the person may be determined. Further, it may be configured to analyze whether or not a person is approaching the own vehicle (in other words, whether or not the person is moving away).

In this case, the generating means may be configured to generate an image representing the moving direction.
Furthermore, the analysis means may calculate the moving speed of the person.
In this case, the generation means may be configured to generate an image representing the moving speed of the person.

Further, the analysis unit may determine whether the person is a child or an adult based on the size of the person (specifically, height). Specifically, it may be judged whether it is a junior high school student or less, or whether it is an elementary school student or less.For this judgment, the average height of a predetermined age published as statistical data is used as a threshold value. Is also good.

When the person is a child, the setting means may set the alert level to be relatively high.
Note that, in another aspect, the present invention may be a system (driving support system) including the driving support device described above.

Further, in another aspect, the present invention provides a detection unit that detects a situation around the vehicle, a recognition unit that recognizes an object around the vehicle based on a detection result of the detection unit, and a recognition unit that recognizes the target object. The analysis means for analyzing the object, the setting means for the object, based on the analysis result of the analysis means, to set the degree to be vigilant, in order to allow the driver of the vehicle to visually recognize the object The driving support system may include a generation unit configured to generate the image of No. 1 based on the degree set by the setting unit, and a display unit configured to display the image generated by the generation unit.

The driving support system may have the same configuration as that of the driving support device described above.

It is a figure which shows the example of application to the vehicle of the driving assistance device of embodiment. It is a block diagram which shows the structure of the driving assistance device of 1st Embodiment. It is a flow chart showing the flow of the luckman support processing which a control ECU performs. It is a flow chart showing a flow of extraction processing which a control ECU performs. 6 is a flowchart showing a flow of analysis processing 1. 6 is a flowchart showing a flow of analysis processing 2. 9 is a flowchart showing a flow of analysis processing 3. 9 is a flowchart showing a flow of analysis processing 4. 6 is a flowchart showing a flow of analysis processing 5. 9 is a flowchart showing a flow of analysis processing 6. 9 is a flowchart showing a flow of a subroutine of analysis processing 6. 6 is a flowchart showing a flow of analysis processing 7. It is a flowchart showing the flow of vehicle recognition determination processing. It is a flow chart showing a flow of display data generation processing. It is a flow chart showing a flow of emphasis image generation processing. It is a flowchart showing the flow of a display process. It is a figure which shows the example of a display. It is a figure which shows the example of a display. It is a figure which shows the example of a display. It is a figure which shows the example of a display. It is a block diagram which shows the structure of the driving assistance device of 2nd Embodiment. It is a flow chart showing a flow of recognition processing (2). It is a flow chart showing a flow of screen judgment processing. 9 is a flowchart showing a flow of analysis processing 8. It is a flow chart showing a flow of display processing (2). It is a figure explaining the effect|action of the driving assistance device of 2nd Embodiment. It is a block diagram which shows the structure of the driving assistance device of 3rd Embodiment. It is a figure explaining eye gaze detection. It is a flow chart showing a flow of driving support processing (2). It is a flow chart showing a flow of amendment judgment processing. It is a flow chart showing a flow of display amendment processing. It is a flow chart showing a flow of recognition judgment processing (2). 9 is a flowchart showing a flow of analysis processing 9. It is a figure which shows the modification 1. It is a figure which shows the modification 2. It is a figure which shows the modification 3. 6 is a flowchart showing a flow of analysis processing 10. It is a figure explaining detection of a raindrop. 6 is a flowchart showing a flow of analysis processing 11. 6 is a flowchart showing a flow of analysis processing 12. It is a flow chart showing a flow of vehicle control processing. It is drawing explaining an example of a display mode (1). It is drawing explaining an example of a display mode (2). It is drawing explaining an example of a display mode (3). It is drawing explaining an example of a display mode (4).

1, 100, 101... Driving support device, 2... Infrared radar, 3... Millimeter radar, 4... Infrared camera, 5... Visible light camera, 6... Momentum detection unit, 7... Head-up display (HUD), 8... Image projection device, 9... Speaker unit, 10... Line-of-sight detection unit, 11... Inter-vehicle communication unit, 12... Vehicle position sensor , 20... Control ECU.

Embodiments to which the present invention is applied will be described below with reference to the drawings.
<First Embodiment>
1. Overall Configuration As shown in FIG. 1, a driving assistance device 1 according to the first embodiment includes an infrared radar 2, a millimeter wave radar 3, an infrared camera 4, a visible light camera 5, a momentum detection unit 6, and a head. The up display 7, the speaker unit 8, and the control ECU 20 are provided.

In addition, in FIG. 1, an image projection device 9, a line-of-sight detection unit 10, an inter-vehicle communication unit 11, and a vehicle position sensor 12 are also shown.
Hereinafter, each configuration of the driving support device 1 will be described with reference to FIGS. 1 and 2.

[Infrared radar]
The infrared radar 2 is a radar that detects a surrounding situation using infrared rays (in other words, detects the presence or absence of an object (hereinafter, an object) and the distance to the object).

As shown in FIG. 2, the infrared radar 2 includes an infrared transmitter/receiver 2a, a signal processor 2b, and an external interface 2c.
The infrared radar 2 irradiates infrared rays at the infrared transmission/reception unit 2a, and receives the reflected light reflected by the object and returned. Then, the signal processing unit 2b calculates the distance to the object based on the time difference between the infrared irradiation time and the reflected light reception time. The data representing the calculated distance is transmitted to the control ECU 20 via the external interface 2c.

The distance that can be detected by the infrared radar 2 is up to several tens of meters (for example, 20 to 30 m). It should be noted that the infrared radar 2 may be provided on the side portion and the rear portion of the vehicle, as shown in FIG.

[Millimeter wave radar]
The millimeter wave radar 3 is a radar that detects a situation in the vicinity by using radio waves in the millimeter wave band.
As shown in FIG. 2, the millimeter wave radar 3 includes a millimeter wave transmitting/receiving unit 3a, a signal processing unit 3b, and an external interface 3c.

The millimeter wave radar 3 irradiates a millimeter wave at the millimeter wave transmission/reception unit 3a, and receives a reflected wave reflected by an object and returned. Then, the signal processing unit 3b calculates the distance to the object based on the time difference between the irradiation time of the millimeter wave and the reception time of the reflected wave. The data representing the calculated distance is transmitted to the control ECU 20 via the external interface 3c.

The distance that can be detected by the millimeter wave radar 3 is up to about 150 m (or more). A resolution of several tens of cm to 1 m is known.
In the first embodiment, the infrared radar 2 detects an object at a short distance (up to several tens of meters), and the millimeter wave radar 3 detects a long distance (up to several tens to 150 meters (or more)). Is configured to detect objects in.

[Infrared camera]
The infrared camera 4 is a camera that detects a surrounding situation by detecting infrared rays emitted from an object.

As shown in FIG. 2, the infrared camera 4 includes an infrared image sensor 4a, an image processing unit 4b, and an external interface 4c.
The infrared camera 4 detects light in the infrared region (infrared) with the infrared image sensor 4a. Then, the image processing unit 4b converts the wavelength and intensity of infrared rays detected by the infrared image sensor 4a into an electric signal, and generates an image based on the electric signal. The data representing the generated image is transmitted to the control ECU 20 via the external interface 4c.

Since this infrared camera 4 forms an image by detecting infrared rays emitted from an object, the object can be detected even in the absence of ambient light (sunlight or the like) or headlight light. Therefore, the object can be detected even at night.

In the first embodiment, as shown in FIGS. 1 and 2, two infrared cameras 4A and 4B arranged at different positions are provided as the infrared camera 4. Parallax (difference in image position and viewing direction) is reproduced by the infrared camera 4A and the infrared camera 4B. Then, based on the fact that the parallax has a correlation with the distance to the object, it is possible to calculate the distance to the object according to the parallax.

Note that, hereinafter, the infrared camera 4 refers to both the infrared cameras 4A and 4B unless otherwise specified.
[Visible light camera]
The visible light camera 5 is a camera that detects surrounding conditions by detecting reflected light of ambient light and headlight light.

As shown in FIG. 2, the visible light camera 5 includes a CCD image sensor 5a as an image sensor, an image processing unit 5b, and an external interface 5c.
The visible light camera 5 detects light with the CCD image sensor 5a and photoelectrically converts the detected light and darkness into the amount of electric charge. The charge amount data is transferred to the image processing unit 5b. The image processing unit 5b reproduces color and brightness based on the data of the charge amount of each pixel to generate a color image. The information of the generated image is transmitted to the control ECU 20 via the external interface 5c.

In the first embodiment, as the visible light camera 5, two visible light cameras 5A and 5B arranged at different positions are provided. The visible light camera 5A and the visible light camera 5B reproduce the parallax, which can generate a stereoscopic image. Further, similarly to the case of the infrared camera 4 described above, the distance to the object can be calculated.

In the following description, the visible light camera 5 refers to the visible light cameras 5A and 5B unless otherwise specified.
[Momentum detection unit]
The momentum detection unit 6 is a unit for detecting the momentum of the host vehicle, and includes a vehicle speed sensor 6a, a yaw rate sensor 6b, and a steering angle sensor 6c.

Specifically, the vehicle speed sensor 6a detects the traveling speed of the host vehicle, the yaw rate sensor 6b detects the yaw rate acting on the host vehicle, and the steering angle sensor 6c detects the steering angle of the steering wheel. The detection signal is transmitted to the control ECU 20.

The control ECU 20 calculates the momentum (the traveling direction and the displacement) of the host vehicle based on the detection signals from the vehicle speed sensor 6a, the yaw rate sensor 6b, and the steering angle sensor 6c.
[Head-up display]
A head-up display (HUD: Head Up Display) 7 is a device that displays an image by superimposing it on a window (a front window in an example) of a vehicle.

The HUD 7 has a laser projector 7a, performs signal processing on the laser projector 7a based on a signal from the control ECU 20, generates an image, and displays the image via an optical unit 7b including a mirror and a lens. ..

The image is formed on the virtual image plane by being superimposed on the scenery outside the vehicle visually recognized through the front window. The virtual image plane is formed in front of the front window, so that the driver of the vehicle can perceive the image as being displayed in the visually recognized scene.
[Speaker unit]
The speaker unit 8 is a device that emits sound (including sound) around the vehicle based on the control of the control ECU 20.

[Control ECU]
The control ECU 20 is an electronic control device including a CPU 20a, a ROM 20b, a RAM 20c, a flash memory 20d, a communication interface 20e, and the like, and executes various processes.

2. Processing Executed by Driving Assistance Device 1 [Driving Assistance Processing]
Hereinafter, the outline of the process executed by the control ECU 20 of the driving support device 1 will be described with reference to FIG.

The control ECU 20 repeatedly executes the driving support process of FIG. 3 in a predetermined cycle while the vehicle is traveling. Accordingly, the driving support device 1 detects the environment around the vehicle, recognizes the object (person, vehicle, etc.), and notifies the driver of the vehicle of the existence of the object (in other words, issues a warning).

In the driving support process, first, in S100, detection data is acquired from the momentum detection unit 6, and the momentum of the own vehicle is estimated based on the acquired vehicle speed, yaw rate, and steering angle of the own vehicle.

Next, in S110, the signal from the infrared radar 2 is acquired.
Next, the process proceeds to S112, and the signal from the millimeter wave radar 3 is acquired.
In subsequent S114, it is determined whether or not an object exists in the detection range based on the signal from the infrared radar 2 and the signal from the millimeter wave radar 3.

If it is determined that the object does not exist in S114, the process proceeds to S116, the log indicating that the object does not exist is stored, and then the process ends. This log may be stored in the flash memory 20d.

If it is determined that the object exists in S114, the process proceeds to S118. In S118, it is determined whether or not the distance to the object is equal to or less than a preset threshold value α based on the data of the distance to the object obtained from the signal of the infrared radar 2 and the signal of the millimeter wave radar 3.

If it is determined that the distance to the object is equal to or less than the threshold value α in S118, the process proceeds to S120. The threshold value α is appropriately set to a value that causes a risk of collision. A fixed value may be set as the threshold value α. Alternatively, based on the momentum of the own vehicle estimated in S100 described above, a value that is determined to cause a risk of collision in relation to the own vehicle traveling at the momentum is calculated and then calculated. It may be set.

In S120, a process of warning the driver of the vehicle is executed. Specifically, the HUD 7 is controlled to display a warning on the front window of the vehicle, for example. As the warning display, a message or symbol indicating that there is a risk of collision may be displayed.

In subsequent S122, a collision avoidance command for avoiding a collision with the object is transmitted to an ECU (not shown) that controls the operation of the vehicle. Specifically, a collision avoidance command is transmitted to the brake control ECU, the steering control ECU, etc., and brake control and steering control for avoiding a collision are executed. Then, the process is completed.

If it is determined that the distance to the object is not less than or equal to the threshold value α in S118, the process proceeds to S124.
In S124, image data is acquired from the infrared camera 4. In this case, the image data of either of the infrared cameras 4A and 4B may be acquired, or both of them may be acquired. When acquiring both, you may calculate and use the average value of two image data. Further, it is also possible to extract a highly accurate portion (a portion having a small amount of noise, etc.) from each of the two image data and generate and use data combining them.

Further, in subsequent S126, image data is acquired from the visible light camera 5. In this case, the image data of either of the visible light cameras 5A and 5B may be acquired, or both of them may be acquired. When acquiring both, you may calculate and use the average value of two image data. Further, it is also possible to extract a highly accurate portion (a portion having a small amount of noise, etc.) from each of the two image data and generate and use data combining them.

Subsequently, in S128, processing for recognizing and analyzing the object (hereinafter, recognition processing) is executed. Details of the recognition processing will be described later.
Next, in S130, a display process for displaying information of objects around the vehicle based on the result of the recognition process in S128 is executed. In other words, this process is a process of notifying the presence of the object to the driver by displaying a predetermined image. Details of the display process will be described later.

[Recognition processing]
The recognition process of S128 will be specifically described below with reference to FIG.
When the control ECU 20 starts the recognition process of S128 (the recognition process of FIG. 4), first, in S140, an analysis process of an image represented by the image data of the infrared camera 4 and the image data of the visible light camera 5 is performed. I do. Specifically, a process of analyzing the brightness (brightness) of the pixel is executed for each pixel forming the image. Either one of the image data of the infrared camera 4 and the image data of the visible light camera 5 may be used, or both of them may be used. Basically, the following processing flow can be applied to any image data.

After S140, the process shifts to S142 to extract an edge (where the amount of change in brightness (brightness) is larger than a predetermined threshold value) in the image. This is a process on the assumption that the amount of change in luminance is large at the boundary between a person or a vehicle and the background.

In subsequent S144, the candidates of the area occupied by the same object are set based on the edge information extracted in S142. For example, as described above, it is assumed that the amount of change in luminance is large at the boundary between a person or a vehicle and the background, but the amount of change in brightness is not necessarily large at all the boundaries, and edges may be interrupted. obtain. In this processing, while recognizing such edge discontinuity from the data of the peripheral edges, the range (area) of the same object defined by the edges is set (estimated).

Next, the process proceeds to S146, and for the area set in S144 (more specifically, for the estimated object), pattern matching is performed with a pattern to be stored in advance and a past learning value (learned pattern), Estimate what is. In this pattern matching, a person (including a person riding a bicycle or the like), a vehicle, an animal (a pet or the like), and an installation object (guardrail, sign, traffic light, signboard, etc.) can be recognized.

In subsequent S148, it is determined whether or not the extracted object is a person based on the result of the pattern matching in S146.
If it is determined to be “other” (specifically, a person other than a person and a vehicle is referred to here) in S148, the process ends. This means that, of the extracted objects, the objects classified as “other” are not further analyzed and no warning is displayed. It should be noted that the driver may be notified (displayed) of objects classified as “other”.

When it is determined in S148 that the vehicle is a vehicle, the process proceeds to S150. In S150, it is determined whether or not to notify the driver of the own vehicle of the extracted vehicle information (whether or not to warn).
For example, whether or not to notify the driver of the own vehicle of the vehicle information may be set in advance. Then, in S150, the determination may be made based on the setting. Alternatively, the positional relationship between the vehicle and the host vehicle, the relative speed, and the like may be detected, and the risk may be determined based on them, and if it is determined that the vehicle is dangerous, a warning may be determined.

If it is determined that the warning is not issued in S150, the process ends.
On the other hand, if it is determined that the warning is issued in S150, the process proceeds to S156, and a flag (vehicle warning flag) indicating that the vehicle is warned is set. Then, it transfers to S154.

Next, when it is determined in S148 that the person is a person, the process proceeds to S152.
In S152, an analysis process for further analyzing the situation, state, etc. of the extracted person is executed. Details of the analysis processing will be described later.

After S152, the process proceeds to S154. In S154, warning display data is generated based on the analysis processing result of S152 and the determination result of S150. The data generated here is used in the display process in S130. Specifically, in S130, the data generated in S154 is transmitted to the HUD 7, and the HUD 7 displays the warning image.

[Analysis processing]
Hereinafter, the analysis process of S152 will be specifically described with reference to FIGS. In the first embodiment, the analysis processes 1 to 7 of FIGS. 5 to 12 (and FIG. 13) are executed in parallel or sequentially in a predetermined order. Further, the analysis processes 1 to 7 are executed for each of the objects recognized as “people” in S148 described above.

Then, in the analysis processes 1 to 7, the alert level for the object (person) is set according to the analysis result. The alert level is data used in the process of S154. Specifically, it is data for determining in what manner a warning is displayed to the driver of the vehicle. The warning level is represented by a numerical value, and the higher the numerical value, the more the display data is generated so that the warning is displayed in a manner that the driver of the vehicle can easily recognize.

As the analysis process of S152, at least one of the analysis processes 1 to 7 of FIGS. 5 to 12 (and FIG. 13) may be executed.
[Analysis process 1]
The analysis process 1 of FIG. 5 is a process of analyzing a position (location) where an object (person) exists and setting a warning level according to the position.

In the analysis process 1, first, in S160, a process of analyzing the position where the object (person) exists with respect to the object recognized as a person in S148 described above is executed. Specifically, the distance from the own vehicle, the relative position to other objects, and the like are analyzed by image analysis of the captured image by the infrared camera 4 or the visible light camera 5. The distance from the own vehicle can be calculated using the parallax of the infrared cameras 4A and 4B or the parallax of the visible light cameras 5A and 5B.

Here, as described above, the signal from the infrared radar 2 acquired in S110 and the signal from the millimeter radar 3 acquired in S112 include information on the distance to the object. The distance calculated in S160 may be used for verification or correction.

Alternatively, instead of using the parallax, the distance may be calculated from the signals acquired in S110 and S112.
After S160, the process proceeds to S162, and it is determined whether or not the object (person) exists on the traveling route of the own vehicle.

It should be noted that at the stage of the processing of S160 described above, the road on which the vehicle is traveling is recognized by image analysis. In addition, because of the processing in S170 described below, a sidewalk may also be recognized. Then, in S162, in addition, the movement direction (traveling direction) of the own vehicle is estimated based on the data of the momentum of the own vehicle acquired in S100. Then, based on these processes, by determining whether or not the object (person) exists on the recognized road and in the estimated traveling direction of the own vehicle, the object (person) is traveling route. Determine if it exists above.

If it is determined in S162 that the object (person) is present on the traveling route of the own vehicle, the process proceeds to S164.
In S164, the value of the alert level for the object (person) is incremented by 3 points. Then, the process is completed. In addition, in the first embodiment, the value of the alert level is incremented in the range of 1 to 3. When not incrementing, it is described as "+0" in the flowchart. This value is an example, and any value may be appropriately set.

The value of the alert level is stored in the flash memory 20d in association with the object (person).
When it is determined in S162 that the object (person) does not exist on the traveling route of the own vehicle, the process proceeds to S166.

In S166, it is determined whether or not the object (person) is present on the road on which the vehicle is traveling.
When it is determined in S166 that the object (person) exists on the road, the process proceeds to S168.

In S168, the alert level is incremented by 2 points. Then, the process is completed.
If it is determined that the object (person) does not exist on the road in S166, the process proceeds to S170.

In S170, it is determined whether or not the object (person) exists on the sidewalk.
When it is determined in S170 that the object (person) exists on the sidewalk, the alert level is incremented by 1 point.

On the other hand, when it is determined in S170 that the object (person) does not exist on the sidewalk, it is determined that the object (person) exists in the vehicle, indoors, or the like (in other words, a relatively safe place), and the process proceeds to S174. Then, the processing is ended without incrementing the alert level.

[Analysis process 2]
The analysis process 2 will be described with reference to FIG.
The analysis process 2 of FIG. 6 is a process of calculating the distance from the own vehicle to the object (person) and setting the alert level according to the calculated distance.

In the analysis process 2, first, in S180, the distance from the own vehicle to the object (person) is calculated. The calculation method is as described above.
Next, the process proceeds to S182, and it is determined whether the distance calculated in S180 is less than or equal to a predetermined threshold β. The value of β can be set appropriately.

If it is determined in S182 that the distance is less than or equal to the predetermined threshold value β, it is determined that the risk is higher, and the process proceeds to S184.
In S184, the alert level is incremented by 1 point. Then, the process is completed.

On the other hand, if it is determined in S182 that the distance is not less than or equal to the predetermined threshold value β, it is determined that the risk is lower, the process proceeds to S186, and the process ends without incrementing the alert level.

[Analysis process 3]
The analysis process 3 will be described with reference to FIG.
The analysis process 3 of FIG. 7 is a process of analyzing whether or not an object (person) is carrying and operating a mobile terminal (and recognizing its own vehicle), and setting a warning level based on the result. Is.

In the analysis process 3, first, in S190, it is determined whether or not the object (person) is carrying (grasping) the mobile terminal. The presence/absence (presence) of the mobile terminal is recognized by the image analysis (pattern matching) in the processing of S140 to S146 described above. In particular, when the mobile terminal is in operation, the brightness (brightness) of the display screen portion becomes high, and the edge extraction can be performed with relatively high accuracy. In this case, recognition by pattern matching becomes easy. Further, even when the mobile terminal is not in operation, when the mobile terminal is held by the human hand, edge extraction can be performed based on the difference in brightness (brightness/darkness) with the human hand. Therefore, in any case, recognition by pattern matching is possible.

When it is determined in S190 that the object (person) does not carry (grip) the mobile terminal, it is determined that the risk is not high, the process proceeds to S192, and the process ends without incrementing the alert level. ..

On the other hand, if it is determined in S190 that the object (person) is carrying (grasping) the mobile terminal, the process proceeds to S194.
In S194, it is determined whether the mobile terminal is in operation. Here, based on the result of the image analysis (the processing of S140 to S146), it is determined whether or not the mobile terminal is operating based on the brightness (brightness) in the area recognized as the mobile terminal. This means that the determination is performed on the assumption that the brightness (brightness) of the display screen portion of the mobile terminal becomes high when the mobile terminal is in operation.

If it is determined in S194 that the mobile terminal is not operating, the process proceeds to S196.
In S196, the warning level is incremented by 1 point based on the determination that the object (person) is holding the mobile terminal although the mobile terminal is not operating. Then, after that, the process ends.

On the other hand, if it is determined in S194 that the mobile terminal is operating, the process proceeds to S198. The process of S194 may be omitted. Specifically, if it is determined in S190 that the object (person) is carrying (grasping) the mobile terminal, the process of S194 may be skipped and the process of S198 may be performed.

In S198, it is determined whether or not the object (person) is operating the mobile terminal. Here, based on the result of the image analysis, the position of the mobile terminal, the position of each part (hand, face) in the object (person), the orientation of the face, and the like are analyzed and comprehensively determined from the information.

If it is determined that the operation is not being performed in S198, the process proceeds to S196.
On the other hand, if it is determined in S198 that the operation is being performed, the process proceeds to S200.
In S200, a process of determining whether or not the object (person) recognizes the existence of the own vehicle (hereinafter, recognition determination process) is executed.

FIG. 13 shows the recognition determination process.
In the recognition determination process of S200 (recognition determination process of FIG. 13), first, in S400, the face region of the object (person) is extracted.

Next, in S402, the extracted face is analyzed. More specifically, “eyes” are extracted by edge detection, pattern matching, or the like.
Then, in S404, it is determined whether or not both eyes can be detected.

When it is determined in S404 that both eyes can be detected, it is determined that the own vehicle is likely to exist within the visual field of the object (person), and based on this determination, the object (person) is It is simply determined that the existence is recognized, and the process proceeds to S406.

In S406, a recognition flag indicating that the object (person) recognizes the existence of the own vehicle is set. Then, after that, the process ends.
On the other hand, if it is determined in S404 that both eyes cannot be detected, it is determined that the host vehicle may not exist within the range of the object (person)'s field of view, and based on this determination, the object (person) determines A simple determination is made that the presence of a vehicle is not recognized, and the flow shifts to S408.

In S408, an unrecognized flag indicating that the object (person) does not recognize the existence of the own vehicle is set. Then, after that, the process ends.
After the recognition determination process of S200 (recognition determination process of FIG. 13), the process proceeds to S202 of FIG.

In S202, a determination process based on the recognition flag set in S406 or the non-recognition flag set in S408 (specifically, a determination process of whether or not an object (person) recognizes the existence of the own vehicle) )I do.

When it is determined in S202 that the object (person) recognizes the presence of the own vehicle (the recognition flag is set), the process proceeds to S204.
In S204, the warning level is incremented by 2 points based on the determination that the object (person) recognizes the presence of the own vehicle while operating the mobile terminal. Then, after that, the process ends.

On the other hand, when it is determined in S202 that the object (person) does not recognize the existence of the own vehicle (the unrecognized flag is set), the process proceeds to S206.
In S206, the warning level is incremented by 3 points based on the determination that the object (person) is operating the mobile terminal and does not recognize the presence of the own vehicle.

Subsequently, the process shifts to S208, and a warning setting process described below is performed. The warning setting process includes a flag indicating that an image is displayed to notify (warn) the driver that the object (person) does not recognize the existence of the own vehicle, and an alarm process for the object (person). This is a process of setting a flag to the effect. This flag is stored in association with the target object (person).

When the warning setting process is executed, in S154 of FIG. 4, a warning image for notifying (warning) the driver that the object (person) does not recognize the existence of the own vehicle is generated, and the image is displayed. In the processing of S130 of FIG. 3, the image is displayed in a superimposed manner on the front window of the vehicle. In addition, an alarm is given to the object (person) through the speaker unit 8 (see FIGS. 1 and 2) by a separate process.

After S208, the process ends.
[Analysis process 4]
The analysis process 4 will be described with reference to FIG.

The analysis process 4 of FIG. 8 is a process of analyzing whether or not an object (person) is using headphones (and recognizing the own vehicle), and setting a warning level based on the result. ..

In the analysis process 4, first, in S210, it is determined whether or not the object (person) uses headphones or earphones (hereinafter, simply headphones). Here, the determination is made based on the result of the image analysis (the processing of S140 to S146).

When it is determined in S210 that the object (person) does not use the headphones, it is determined that the risk is low, the process proceeds to S212, and the process ends without incrementing the alert level.

On the other hand, if it is determined in S210 that the object (person) is using headphones, the process proceeds to S200, and then proceeds to S202.
The processing of S200 and S202 is the same as the processing of S200 and S202 described with reference to FIG. 7, and the description thereof is omitted here.

When it is determined in S202 that the object (person) recognizes the presence of the own vehicle (the recognition flag is set), the process proceeds to S218.
In S218, the warning level is incremented by 1 point based on the determination that the object (person) is using the headphones while recognizing the existence of the own vehicle. Then, after that, the process ends.

On the other hand, when it is determined in S202 that the object (person) does not recognize the existence of the own vehicle (the unrecognized flag is set), the process proceeds to S220.
In S220, the warning level is incremented by 3 points based on the determination that the object (person) uses the headphones and does not recognize the presence of the own vehicle.

Then, the process proceeds to S208. The process of S208 is the same as the process of S208 of FIG. 7, and the description thereof is omitted here. Then, the process is completed.
[Analysis process 5]
The analysis process 5 will be described with reference to FIG.

The analysis process 5 of FIG. 9 is a process of analyzing whether or not an object (person) is talking or talking (and recognizing the own vehicle), and setting a warning level based on the result. is there.
In the analysis process 5, first, in S230, it is determined whether or not the object (person) is talking or talking. Here, the determination is made based on the result of the image analysis (the processing of S140 to S146).

When it is determined in S230 that the object (person) is not talking or talking, the risk is determined to be low, the process proceeds to S232, and the process ends without incrementing the alert level.

On the other hand, if it is determined in S230 that the object (person) is in conversation or in a call, the process proceeds to S200, and then proceeds to S202.
The processing of S200 and S202 is the same as the processing of S200 and S202 described with reference to FIG. 7, and the description thereof is omitted here.

When it is determined in S202 that the object (person) recognizes the presence of the own vehicle (the recognition flag is set), the process proceeds to S238.
In S238, the warning level is incremented by 1 point based on the determination that the object (person) is in the conversation or talking while recognizing the existence of the own vehicle. Then, after that, the process ends.

On the other hand, when it is determined in S202 that the object (person) does not recognize the existence of the own vehicle (the unrecognized flag is set), the process proceeds to S240.
In S240, the warning level is incremented by 3 points based on the judgment that the object (person) is talking or talking and does not recognize the presence of the own vehicle.

Then, the process proceeds to S208. The process of S208 is the same as the process of S208 of FIG. 7, and the description thereof is omitted here. Then, the process is completed.
[Analysis process 6]
The analysis process 6 will be described with reference to FIG.

The analysis process 6 of FIG. 10 is a process of analyzing the movement of an object (person) and setting a warning level based on the result.
In the analysis process 6, first, in S250, image data is reacquired from the infrared camera 4 or the visible light camera 5.

Next, the process proceeds to S252, and the momentum data of the own vehicle is acquired again from the momentum detection unit 6.
The processes of S250 and S252 are executed to track the movement of the object (person) in time series.

Next, in S254, a tracking process between a plurality of images (between frames) is executed for the object (person). Specifically, the similarity between objects (people) in the current image (current frame) and the previous image (frame) is calculated, and objects (people) having a high similarity are the same object (people). And the same label is given. As the index of similarity, the size (area) of the region, the brightness (brightness), the movement amount, and the like are used. Further, here, the size (area) of the object (person), the movement amount, and the like are corrected in consideration of the momentum of the own vehicle. The objects (people) given the same label are analyzed in time series, and the presence/absence of movement and the movement direction are calculated.

Next, in S256, it is determined whether the movement of the object (person) can be analyzed. In other words, it is determined whether the re-acquired image data and the momentum data are sufficient to recognize or estimate the movement of the object (person).

When it is determined in S256 that the analysis is impossible (in other words, the data is not sufficient), the process returns to S250 (and S252) again, and the image data (and the exercise amount data) is acquired again. Furthermore, a tracking process is performed in S254.

On the other hand, if it is determined that the analysis is possible in S256, the process proceeds to S258.
In S258, it is determined whether the object (person) is moving based on the result of the tracking process of S254.

If it is determined in S258 that the object (person) has not moved, the process proceeds to S260, and the process ends without incrementing the alert level.
On the other hand, if it is determined in S258 that the object (person) is moving, the subroutine processing of S280 is started.

FIG. 11 is a flowchart showing the flow of the subroutine processing of S280.
When the subroutine process of S280 (subroutine process of FIG. 11) is started, first, in S282, it is determined whether or not the moving direction of the object (person) is the same as the moving direction of the own vehicle.

When it is determined in S282 that the moving directions are the same, the object (person) is moving with his/her back facing the own vehicle (and thus it is highly possible that the object (person) is not aware of the existence of the own vehicle). Based on the determination, the process proceeds to S284.

In S284, the caution level is incremented by 2 points. Then, after that, the process ends.
On the other hand, if it is determined in S282 that the moving directions are not the same, it is highly possible that the host vehicle is within the range of the field of view of the object (person) (and the object (person) is aware of the presence of the host vehicle). (The possibility is high), the process proceeds to S286, and the process is ended without incrementing the alert level.

After the subroutine process of S280 (subroutine process of FIG. 11), the process proceeds to S262 of FIG.
In S262, it is determined whether or not the object (person) meanders. As for meandering people, it is warranted that they may stagger due to drinking alcohol or stagger due to two-seater bicycles.

If it is determined in S262 that the object (person) is not meandering, it is determined that the object is not meandering but is moving, the process proceeds to S264, and the alert level is incremented by 1 point. Then, after that, the process ends.

On the other hand, if it is determined in S262 that the object (person) is meandering, the process proceeds to S266.
In S266, it is determined whether or not the object (person) is approaching the traveling route of the own vehicle.

When it is determined in S266 that the object (person) is not approaching the traveling route of the vehicle, it is determined that the object is meandering although it is not approaching, the process proceeds to S268, and the alert level is incremented by 2 points. Then, after that, the process ends.

On the other hand, if it is determined that the object (person) is approaching the traveling route of the own vehicle, the process proceeds to S270 and the alert level is incremented by 3 points.
Next, in S272, a flag indicating that an image indicating the moving direction of the object (person) is displayed is set. When this flag is set, an image showing the moving direction of the object (person) is generated by the process of S154 of FIG. 4, and the image is displayed by the process of S130 of FIG. This series of processes is executed for the purpose of alerting the driver of the vehicle by displaying an image showing the moving direction of the object (person).

[Analysis process 7]
The analysis process 7 will be described with reference to FIG.
The analysis process 7 of FIG. 12 is a process of simply determining whether or not the object (person) is a child, and setting the alert level based on the result.

In the analysis process 7, first, in S290, it is determined whether or not the height (height) of the object (person) is less than or equal to a predetermined threshold value Ta. As the threshold value Ta, the average height of a person (child) of the age to be discriminated may be assigned.

If it is determined in S290 that the value is equal to or less than the threshold value Ta, it is determined that the object (person) is a child, the process proceeds to S292, and the alert level is incremented by 2 points. Then, after that, the process ends.

On the other hand, if it is determined in S290 that it is not equal to or less than the threshold value Ta, it is determined that the object (person) is not a child, the process proceeds to S294, and the process ends without incrementing the alert level.

[Display data generation process]
Next, the display data generation process of S154 of FIG. 4 will be described with reference to FIG.
In the display data generation process, first, in S500, the object (person or vehicle) recognized in S140 to S148 of FIG. 3 is extracted.

Next, the process proceeds to S502, and the alert level set in the processing of FIGS. 5 to 13 is extracted for each of the objects (specifically, each of the objects recognized as “people”).

In subsequent S504, a process (emphasized image generation process) of generating an image for emphasizing the object (person) is executed for each object (person) according to the extracted warning level.

The emphasized image generation process will be specifically described with reference to FIG.
When the emphasized image generation process of S504 (the emphasized image generation process of FIG. 15) is started, first, in S520, an image surrounding the object area of each object (person) is generated in accordance with the object area. The image surrounding the object area may be appropriately set to a triangle, a quadrangle, a circle, an ellipse, or the like.

By generating an image that matches the object region, for example, a vertically long image can be generated for a standing object (person), and an image with substantially the same aspect ratio can be generated for a sitting object (person), for example. In addition, a horizontally long image is generated for an object (person) who is falling. By displaying such an image, the driver can intuitively understand the state of the object (person) (standing state, sitting state, falling state, or the like).

Next, the process proceeds to S522, and the display mode of the image generated in S520 is set according to the caution level extracted in S502. Specifically, the line thickness of the frame, the line color, and the like are set. Also, it is set whether or not to blink.

For example, the higher the alert level, the thicker the line may be. Further, when the alert level is high, the color of the line may be set to a color (red, yellow, other fluorescent color, etc.) that attracts the driver's attention. Further, when the alert level is high, the image may be set to blink.

Each object (person) may be classified into a high-point group, an intermediate group, and a low-point group according to the set alert level points. Then, the display mode of the image may be set for each group.

Next, in S524, it is determined whether images have been set for all objects (people).
If it is determined in S524 that images have not been set for all objects (people) (there are unset objects (people)), the process returns to S520 (and S522).

On the other hand, if it is determined in S524 that images have been set for all objects (people), the process ends.
When the emphasized image generation process of S504 (the emphasized image generation process of FIG. 15) is completed, the process proceeds to S506 of FIG.

In S506, it is determined whether or not a flag for warning the vehicle (hereinafter, vehicle warning flag) is set. This flag is set in the process of S156 in FIG.
When it is determined in S506 that the vehicle warning flag is set, the process proceeds to S508, and an image for emphasizing the object (vehicle) is generated in association with the target object (vehicle). Specifically, an image surrounding the object area of the object (vehicle) recognized in the processing of S140 to S148 of FIG. 4 is generated. The image surrounding the object area may be appropriately set to a triangle, a quadrangle, a circle, an ellipse, or the like. After the processing of S508, the process proceeds to S510.

If it is determined in S506 that the vehicle warning flag is not set, the process proceeds to S510.
In S510, whether or not a flag (hereinafter, unrecognized notification flag) for displaying an image for notifying (warning) the driver that the object (person) does not recognize the existence of the own vehicle is set. To determine. This flag is set in the process of S208 of FIGS.

If it is determined in S510 that the non-recognition notification flag is set, the process proceeds to S512, and a warning image for the driver is generated in association with the target object (person). This warning image is an image for notifying (warning) the driver that the object (person) does not recognize the existence of the own vehicle. The image is not limited to a mark such as a symbol, but may be a message, for example. Alternatively, it may be an image surrounding the face portion of the object (person). The color of the warning image may be a color (red, yellow, other fluorescent color, etc.) that prompts the warning. After the process of S512, the process proceeds to S514.

If it is determined that the non-recognition notification flag is not set in S510, the process proceeds to S514.
In S514, it is determined whether or not a flag indicating that an image indicating the moving direction of the object (person) is displayed (hereinafter, a moving display flag) is set. This flag is set in the process of S272 in FIG.

When it is determined that the movement display flag is set in S514, the process proceeds to S516, and an image (for example, an arrow mark) indicating the moving direction of the object (person) is generated in association with the target object (person). To do. Then, the process proceeds to S518.

If it is determined that the movement display flag is not set in S514, the process proceeds to S518.
In S518, it is determined whether or not there is another object (whether or not there is an object for which an image to be emphasized should be generated).

If it is determined in S518 that there is an object, the process returns to S502.
On the other hand, if it is determined in step S518 that there is no object, the process ends.
[Display processing]
Next, the display process of S130 in FIG. 3 will be described with reference to FIG.

When the display process of S130 (display process of FIG. 16) is started, first, in S540, an alignment adjustment signal for initializing and adjusting the display position and the imaging position by the HUD 7 is transmitted to the HUD 7. This causes the HUD 7 to adjust (initialize) the display position and the image formation position.

Next, the process proceeds to S542, and the signal representing the image generated in the display data generation process of S154 (display data generation process shown in FIGS. 14 and 15) is transmitted to the HUD 7. As a result, the image is displayed over the front window of the vehicle via the HUD 7. The signal representing the image includes data of the coordinate value at which the image should be displayed (the coordinate value based on the display area of the HUD 7).

Specifically, the control ECU 20 has information about coordinate axes (hereinafter, camera coordinate axes) based on the imaging areas of the infrared camera 4 and the visible light camera 5, and coordinate axes (hereinafter, HUD coordinate axes) based on the display area by the HUD 7. Have both information. Then, the coordinate value at which the image generated by the image analysis of the infrared camera 4 or the visible light camera 5 should be displayed (the coordinate value indicating the position to be displayed by the HUD 7) from the coordinate value on the camera coordinate axis to the HUD coordinate axis. It is calculated by converting to coordinate values.

After S542, the process proceeds to S544, and it is determined whether there is an additional display image. Specifically, it is determined whether or not a new image is generated by the display data generation process of S154.
If it is determined that there is an additional image in S544, the process of S542 is executed again.

On the other hand, if it is determined in S544 that there is no additional image, the process ends.
3. Operation of this Embodiment Next, the operation of the first embodiment (an example of a display mode) will be described with reference to FIGS. 17 to 20.

First, the example of FIG. 17 will be described. In the example of FIG. 17, the object H which is a person and the objects V0, V1 and V2 which are vehicles are extracted and recognized.
This example is an example of superimposing and displaying images for emphasis on both a person and a vehicle.

Specifically, for the object H, an image (hereinafter, also simply referred to as a frame) W of an elliptical frame surrounding the object H is displayed.
Further, for the objects V0, V1, V2, elliptical frames X0, X1, X2 surrounding the objects V0, V1, V2 are displayed.

The display modes of the frame W and the frames X0, X1, and X2 may be different. For example, as shown in FIG. 17, the frame W may be displayed with a solid line and the frames X0, X1, and X2 may be displayed with broken lines.

Here, in FIG. 17, the object V0 and the object V2 are partially overlapped with each other (a part of the object V2 is hidden behind the object V0).
In such a situation, the positional relationship between the object V0 and the object V2 may be grasped from the stereoscopic information (depth information) based on the parallax of the infrared cameras 4A and 4B or the parallax of the visible light cameras 5A and 5B. Specifically, the control ECU 20 is configured to recognize the objects V0 and V2 as different objects, not as the same object, based on the stereoscopic information (depth information).

In this case, as shown in FIG. 17, a frame X0 corresponding to the object V0 and a frame X2 corresponding to the object V2 are drawn. Then, the frame X0 and the frame X2 may also include stereoscopic information (depth information). Specifically, as shown in FIG. 17, the frame X2 may be displayed such that a part of the frame X2 is hidden behind the object V0.

In FIG. 17, a mode display area R is shown in the upper right of the drawing. This area is an area for displaying a target (specifically, a target symbol mark) for displaying an image for emphasis. In the mode display area R, a vehicle symbol Mv and a person symbol Mp are shown. This indicates that the vehicle and the person are in a mode for displaying images for enhancement (frame W and frames X0, X1, X2 in FIG. 17).

Thus, according to the driving support device 1, an image (frame W) for detecting an object (person, other vehicle) or the like around the own vehicle and allowing the driver to visually recognize the detected object is displayed as the object. It is displayed in association with each other. Therefore, the easiness of the driver to recognize the object is improved.

Next, an example of FIGS. 18A and 18B will be described. In the example of FIGS. 18A and 18B, four people are visually recognized as shown in FIG. 18A.
The driving assistance device 1 analyzes the state of each of the four persons by image analysis and displays a warning image according to the analyzed state.

As shown in FIG. 18B, four persons will be described as objects H1, H2, H3, and H4, respectively.
First, a frame W (W1, W2, W3, W4) for emphasis is displayed for each of the objects H1 to H4. The frame W may be generated and displayed so as to surround the areas of the objects H1 to H4. For example, when the object H is a standing person, the frame W may be vertically generated and displayed. Further, when the object H is a person and is sleeping (falling down) for some reason, the frame W may be configured to be horizontally generated and displayed. Further, when the object H is a sitting person, the frames W may be generated and displayed so that the aspect ratios of the frames W are approximately equal to each other.

The display modes of the frames W1 to W4 differ depending on the alert level.
The objects H1 and H2 are present in the travel route of the host vehicle.
Particularly, regarding the object H1, it is assumed that the distance from the own vehicle is equal to or less than a predetermined threshold β. As a result, when the alert level is set relatively high for the object H1, the frame W1 is displayed in a more emphasized display mode. For example, the frame W1 may be configured by a double frame. Further, it may be displayed in a more prominent color such as a fluorescent color.

For the object H2, it is assumed that the distance from the host vehicle is larger than the predetermined threshold β. Accordingly, when the alert level of the object H2 is set relatively low (compared to the object H1), the emphasis degree of the frame W2 may be suppressed in comparison with the frame W1. The frame W2 may be displayed in a blinking manner, for example.

Further, when the objects H1 and H2 are in conversation, the state is detected by the driving support device 1.
When the driving support device 1 determines that the objects H1 and H2 are in a conversation, the driving assistance device 1 superimposes and displays the conversation symbols M1 and M2 indicating the conversation in the vicinity of the frames W1 and W2 in association with the objects H1 and H2. You may. The data of the conversation symbols M1 and M2 are stored in the flash memory 20d. The data of the conversation symbols M1 and M2 may be stored in the ROM 20b. The same applies to a mobile terminal symbol M3, a headphone symbol M4, an unrecognized symbol M1', and a recognized symbol M2', which will be described later.

The vicinity is a position adjacent to (or in contact with) the frame W regardless of whether the frame W is above, below, left, or right. In addition, in the case of the vicinity, it may be within the area of the frame W. For example, the conversation symbols M1 and M2 may be displayed in an overlapping manner in the regions of the frames W1 and W2. The same applies to the meaning of “near”.

Further, when it is determined that the object H1 is not aware of the existence of the own vehicle, the unrecognized symbol M1′ representing that fact may be displayed in an overlapping manner near the frame W1 in association with the object H1.

On the other hand, when it is determined that the object H2 recognizes the existence of the own vehicle (or notices the existence of the own vehicle), the recognition symbol M2′ indicating that is associated with the object H2 and superimposed on the vicinity of the frame W2. You may display it.

Furthermore, when it is determined that the object H3 is operating the mobile terminal h3 (object h3), the mobile terminal symbol M3 indicating that is displayed may be displayed in an overlapping manner near the frame W3 in association with the object H3.

When it is determined that the object H4 is using the headphones h4 (object h4), a headphone symbol M4 indicating that effect may be displayed in the vicinity of the frame W4 in association with the object H4.

In addition, the driving assistance device 1 may set an auxiliary display area P1 for displaying auxiliary information. As shown in FIG. 18B, for example, the number of extracted objects (the number of persons, etc.) may be displayed in the auxiliary display area P1. Further, the number of people displayed in the auxiliary display area P1 may be the number of people displaying an image for emphasis.

In this case, when an image for emphasis is additionally displayed, the numerical value of the number of people may be incremented and displayed in accordance with the addition. On the other hand, when the image for emphasis is deleted, the numerical value of the number of people may be decremented and displayed according to the deletion. Further, when the number of people is the same but the objects are exchanged, the fact may be notified by blinking a numerical value or the like.

The symbols M1 to M4, M1′, M2′ may be erased or changed according to changes in the states of the objects H1 to H4.
As described above, according to the driving support device 1, when the object is a person, the information indicating the state of the person is displayed, so that the driver recognizes not only the presence of the person but also the state of the person. become able to. For this reason, it becomes possible for the driver to realize driving according to the state of people around the vehicle. That is, it can contribute to improving driving safety.

Next, the example of FIG. 19 will be described. In the example of FIG. 19, four objects H5, H6, H7, H8 are extracted.
Objects H5 and H6 are pedestrians walking on a pedestrian crossing, and objects H7 and H8 are people who ride a bicycle.

The objects H5, H6, H7, and H8 are surrounded by frames W5, W6, W7, and frame 8 for emphasis, respectively.
The display modes of W5, W6, W7 and the frame 8 may be different depending on the respective distances from the own vehicle to the objects H5, H6, H7, H8, for example. For example, the line thickness may be different.

In the example of FIG. 19, the object H5 is closest to the own vehicle, and the frame 5 corresponding to the object H5 is displayed with the thickest line. On the other hand, the object H8 is farthest from the own vehicle, and the frame 8 corresponding to the object H8 is displayed by the thinnest line.

In addition, in the example of FIG. 19, an image of an arrow (hereinafter, also simply referred to as an arrow) Y indicating the traveling direction of the object is displayed in association with each object H. Specifically, the arrows Y5 and Y7 point toward the right side in the drawing, indicating that the objects H5 and H7 are moving toward the right side. Further, an arrow Y6 is directed to the left side in the drawing, indicating that the object H6 is proceeding to the left side. The arrow Y8 is directed toward the host vehicle, indicating that the object H8 is approaching the host vehicle.

Further, the arrow Y may indicate the moving speed of each object H.
Specifically, the length of the arrow Y may indicate the magnitude of the moving speed. For example, in the example of FIG. 19, the arrows Y5, Y6, and Y7 whose lengths can be easily compared are targeted. In FIG. 19, it can be recognized that the arrow Y7 has the longest length and the moving speed of the object H7 is the highest. On the other hand, it can be recognized that the arrow Y6 is the shortest and the moving speed of the object H6 is the lowest. It can be recognized that the length of the arrow Y5 is intermediate between the arrows Y7 and Y6, and the moving speed of the object H5 is between the moving speed of the object H7 and the moving speed of the object H6.

Further, as indicated by arrows Y5, Y6, Y7, arrow gradation portions G5, G6, G7 may be drawn, and the magnitude of the moving speed may be indicated by the length and density of the gradation portions.

Also, the magnitude of the moving speed may be indicated by the position of the arrow Y with respect to the frame W (or the object H). In the example of FIG. 19, arrows Y5, Y6, and Y7 with which the positions in the height direction with respect to the frame W (or the object H) can be compared are targeted. In FIG. 19, the arrow Y7 is shown higher in the range of the frame W7 (and the object H7) in the height direction. The arrow Y5 is shown in the middle of the range of the frame W5 (and the object H5) in the height direction. The arrow Y6 is shown further downward in the range of the frame W6 (and the object H6) in the height direction.

In such a relationship, it may be indicated that the moving speed of the object H7 is the highest at the position of the arrow Y7 (position in the vertical direction) that is shown above in relation to the object H. Further, the position of the arrow Y6 (the position in the up-down direction) shown below in relation to the object H may indicate that the moving speed of the object H6 is the smallest. Further, even if it is shown that the moving speed of the object H5 is intermediate between the object H7 and the object H6 at the position of the arrow Y5 (position in the vertical direction) shown at the intermediate position in relation to the object H. good.

As described above, according to the driving support device 1, since information indicating the moving direction and the moving speed of the person around the vehicle is displayed, the driver can easily predict the movement of the person. Therefore, it is possible to contribute to improving the driving safety.

Next, the example of FIG. 20 will be described. The example of FIG. 20 is a display example at night.
In the example of FIG. 20, the object H9 is extracted and recognized.
Then, a frame W9 for highlighting the object H9 is displayed in a superimposed manner. The frame W9 may be drawn in a white-based color or a fluorescent color so that it can be easily visually recognized at night.

An arrow symbol M9 is displayed on the upper portion of the frame W9 in order to enhance the effect of attracting the driver's attention. Unlike the example of FIG. 19, the arrow symbol M9 faces the object H9 side instead of the moving direction of the object H9 and strengthens the existence of the object H9.

In other words, the arrow symbol M9 is arranged so that the object H9 is naturally recognized (moved to the center of the visual field) when the line of sight is moved in the direction of the arrow.
The driving support device 1 may be configured to display the frame W9 and the arrow symbol M9 at the same time when the object H9 is detected. Further, for example, the frame W9 may be displayed, and after a predetermined time has elapsed, the arrow symbol M9 may be additionally displayed. According to the latter configuration, the effect of emphasis can be further enhanced.

A caution symbol M9' is further displayed on the left side of the frame W9. The attention symbol M9' can also be displayed to enhance the effect of attracting the driver's attention, similarly to the arrow symbol M9.

The display position of the caution symbol M9' (and the arrow symbol M9) may be any position as long as it can be easily recognized in relation to the background. For example, in the example of FIG. 20, it may be displayed in the lower left area Ra with respect to the frame W9 (and the object H). Alternatively, it may be displayed in the region Rb directly below the frame W9 (and the object H). Preferably, in the background, it is preferable to display in a region where the brightness (brightness) does not vary (in other words, a region where the brightness (brightness) is constant).

Further, in the example of FIG. 20, the auxiliary display areas P2 to P4 may be set.
A symbol mark h9 representing the object H9 is displayed in the auxiliary display area P2. By additionally displaying such a symbol mark, it becomes possible to attract more attention of the driver.

Further, instead of displaying the frame W9, the arrow symbol M9, or the caution symbol M9′, the symbol mark h9 may be displayed in the auxiliary display area P2. Such a mode (mode) may be set by the driver using an input device or the like operated by the driver.

The distance is displayed in the auxiliary display area P3. This distance represents the distance from the host vehicle to the object H9.
In the auxiliary display area P4, a symbol m9, which is the same symbol as the caution symbol M9′, is displayed. The caution symbol M9′ and the symbol m9 may be displayed in conjunction with each other. For example, the symbol m9 may be automatically displayed when the caution symbol M9′ is displayed. Further, when the caution symbol M9' is erased, the symbol m9 may be erased.

As described above, according to the driving support device 1, it is possible to appropriately support the driver to visually recognize the surroundings at night when the visibility is reduced for the driver. For this reason, it is possible to contribute to improving driving safety even at night.

In the first embodiment, the infrared radar 2, the millimeter radar 3, the infrared camera 4, and the visible light camera 5 correspond to an example of a detection unit, and the processes of S114, S128, S140 to S148 correspond to an example of a recognition unit. However, the processing of S152 corresponds to an example of the analyzing means, and S164, S168, S172, S174, S184, S186, S192, S196, S204, S206, S208, S212, S218, S220, S232, S238, S240, S260, The process of S264, S268, S270, S284, S286, S292, S294, S660 corresponds to an example of a setting unit, the process of S154 corresponds to an example of a generating unit, and the process of HUD7 and S130 corresponds to an example of a display unit. To do.

Further, the process of S148 corresponds to an example of a determination unit, and the ROM 20b or the flash memory 20d corresponds to an example of a storage unit.
The conversation symbols M1 and M2, the mobile terminal symbol M3, and the headphone symbol M4 correspond to an example of symbol symbols.

<Second Embodiment>
The second embodiment of the present invention will be described with reference to FIGS.
The driving support device 100 (see FIG. 21) of the second embodiment is different from the driving support device 1 of the first embodiment (see FIG. 2) in that an image projection device 9 is provided.

The driving support device 100 is different from the driving support device 1 in the following points.
First, instead of the recognition processing of FIG. 4, the recognition processing (2) of FIG. 22 is executed.

Further, the analysis process 8 of FIG. 24 is executed.
Further, instead of the display processing of S130 (display processing of FIG. 16) in FIG. 3, display processing (2) of FIG. 25 is executed.

[Image projection device]
The image projection device 9 is a device for projecting an image in a region in the environment outside the vehicle, which is a screen and can project the image. The area where an image can be projected as a screen can be detected by analyzing the image captured by the infrared camera 4 or the visible light camera 5.

The image projection device 9 has a laser projector 9a, performs signal processing by the laser projector 9a based on a signal from the control ECU 20 (in other words, generates a signal of a display image), and an optical device including a mirror and a lens. The image is projected through the unit 9b.

[Recognition process (2)]
Next, the recognition process (2) executed by the driving support device 100 will be described with reference to FIG.

The recognition process (2) of FIG. 22 is different from the recognition process of FIG. 4 in that the screen determination process of S550 is executed. Since the processing of S140 to S156 is the same as the recognition processing of FIG. 4, description thereof will be omitted here.

In the recognition process (2) of FIG. 22, when the control ECU 20 determines in S148 that the object is “other”, the control ECU 20 executes the screen determination process of S550.
[Screen judgment processing]
The flow of the screen determination process is shown in FIG.

The screen determination process is a process of determining whether or not an image can be projected onto the area of the object determined to be “other”.
When the control ECU 20 starts the screen determination processing of S550 (screen determination processing of FIG. 23), first, in S560, it is determined whether or not the area of the object region is equal to or larger than a predetermined area S. This process is a process for determining whether or not the region (area) is sufficient for projecting an image.

If it is determined in S560 that the area of the object region is not equal to or larger than the predetermined area S, it is determined that projection is impossible, and the process ends.
On the other hand, if it is determined in S560 that the area of the object region is equal to or larger than the predetermined area S, the process proceeds to S562.

In S562, it is determined whether or not the distance from the own vehicle to the object is a distance at which an image can be projected.
If it is determined that the distance is not a projectable distance in S562, the process ends.

On the other hand, if it is determined in S562 that the distance is a projectable distance, the process proceeds to S564.
In S564, processing for estimating the flatness of the surface of the object area (flatness estimation processing) is executed. In the flatness estimation processing, image analysis of the object area is performed, and the flatness of the surface of the object area is estimated.

Here, the greater the degree of unevenness on the surface of the object area, the greater the difference and variation in the brightness (brightness) of the image. On the other hand, the smaller the degree of unevenness on the surface of the object area, the smaller the difference and variation in the brightness (brightness) of the image. Based on such characteristics, the flatness is estimated by analyzing the difference and variation in the brightness (brightness) of the image of the object area.

After S564, the process proceeds to S566, and it is determined whether the estimated flatness is less than or equal to a predetermined threshold F (note that the smaller the flatness value, the flatter).
If it is determined in S566 that the flatness is not less than or equal to the predetermined threshold F, the process ends.

On the other hand, if it is determined that the flatness is less than or equal to the predetermined threshold value in S566, the process proceeds to S568.
In S568, the process of estimating the color of the surface of the object area (color estimation process) is executed. The color estimation process is executed to determine whether an image can be projected on the surface of the object area.

Here, the feature that the absorptivity of the infrared rays differs depending on the color of the surface irradiated with the infrared rays is utilized. A white object has a relatively low infrared absorption rate (in other words, a relatively high infrared reflectance, while a black object has a relatively high infrared absorption rate (in other words, an infrared reflectance). Is relatively low).

Based on such characteristics, the infrared absorption rate of the object can be calculated by analyzing the intensity of the reflected infrared light emitted by the infrared sensor 2 in consideration of the distance to the object. Then, based on the calculation result, the color of the object can be estimated.

Therefore, in S568, the color of the surface of the object area is estimated by the above-mentioned method using the infrared sensor 2.
After S568, the process proceeds to S570, and based on the estimation result in S568, it is determined whether or not the color of the surface of the object area is a color capable of projecting an image.

If it is determined in S570 that projection is not possible, the process ends.
On the other hand, if it is determined in S570 that projection is possible, the process proceeds to S572.
In step S572, a flag (projectable flag) indicating that an image can be projected as a screen is set for the target object area.

Next, the process proceeds to S574, and the information of the target object area (information such as coordinates, range, area, etc.) is stored. Then, the process is completed.
[Analysis process 8]
The control ECU 20 further executes the analysis process 8 as one of the analysis processes of S152 in FIG.

FIG. 24 shows the flow of the analysis process 8.
When starting the analysis process 8, the control ECU 20 first determines in S580 whether or not a person is present in the blind spot of the vehicle (other vehicle) based on the positional relationship between the extracted person and vehicle. In this determination, the traveling direction of the other vehicle, a person and an object (obstacle) around the other vehicle, and the like are extracted, and it is comprehensively determined whether or not the person is present in the visual field of the driver of the other vehicle.

If it is determined in S580 that there is no blind spot, the process ends.
On the other hand, if it is determined in S580 that the vehicle is in the blind spot, the process proceeds to S582, and a flag (projection execution flag) for projecting an image on the external environment of the vehicle is set. Then, the process is completed.

[Display processing (2)]
Next, the display process (2) will be described with reference to FIG.
In the display process (2) of FIG. 25, the processes of S540 to S544 are the same as the processes of S540 to S544 in FIG. 16, and thus the description thereof is omitted here.

In the display process (2), if it is determined in S544 that there is no additional display image, the process proceeds to S590.
In S590, it is determined whether the projection possible flag and the projection execution flag are set. The projectable flag is a flag set in the above-described processing of S572 (see FIG. 23). The projection execution flag is a flag set in the above-described processing of S582 (see FIG. 24).

If it is determined in S590 that the projectable flag and the projection execution flag are set, the process proceeds to S592.
In step S<b>592, the information of the object area capable of projecting an image as a screen (specifically, information such as coordinate value, range, and area) stored in step S<b>574 described above is transmitted to the image projection device 9.

Next, in S594, the image data to be projected by the image projection device 9 is transmitted to the image projection device 9. This image data may be part or all of the data of the image captured by the infrared camera 4, or part or all of the data of the image captured by the visible light camera 5. Further, the data generated in the process of S154 (see FIG. 22) may be included.

By such processing of FIG. 25 (more specifically, processing of S590 to S594), the image can be projected by the image projection device 9 onto a predetermined area (area where an image can be projected) in the environment around the vehicle. ..

The operation of the second embodiment will be described with reference to FIG.
In FIG. 26, a driving assistance device 100 is mounted on a vehicle (own vehicle) K1. Another vehicle K2 exists around the host vehicle K1. Also, there are objects H10 and H11.

The object H10 is a fence, and the object H11 is a person (here, a duo). In FIG. 26, only the top of one person's head can be visually recognized.
The object H11 is hidden behind the object H10 when viewed from the direction of the other vehicle K2. That is, the positional relationship in which the driver of the other vehicle K2 cannot visually recognize the object H10 is formed.

On the other hand, the presence of the object H11 is visible and detectable from the host vehicle K1.
The driving support device 100 of the host vehicle K1 detects the objects H10 and H11 by analyzing the image data of the infrared camera 4 or the image data of the visible light camera 5. Further, the other vehicle K2 is detected.

For the object H10, screen determination processing is executed to determine whether or not an image can be projected as a screen.
Further, the positional relationship between the objects H10 and H11 and the other vehicle K2 is analyzed to determine whether or not the object H11 is within the visual field of the driver of the other vehicle K2.

The driving support device 100 transmits the image data of the object H11 to the image projection device 9, and causes the image of the object H11 to be projected onto a predetermined region (screen region that is a region where an image can be projected as a screen) Sc1 of the object H10. ..

Although the object H11 itself, which is present behind the object H10, cannot be visually recognized by the driver of the other vehicle K2, the driver of the other vehicle K2 indicates that the object H11 is present (human's presence) due to the image displayed in the screen area Sc1 of the object H10. Being able to recognize (presence).

As described above, according to the second embodiment, it is possible to notify a person around the vehicle (for example, a driver of another vehicle, a pedestrian, or the like) of the surrounding situation.
<Third Embodiment>
A third embodiment of the present invention will be described with reference to FIGS. 27 to 31.

The driving support device 101 (see FIG. 27) of the third embodiment is different from the driving support device 1 of the first embodiment (see FIG. 2) in that the eye gaze detection unit 10 is provided.
Further, the driving support device 101 differs from the driving support device 1 in that the driving support process (2) of FIG. 29 is executed instead of the driving support process of FIG. 3.

[Gaze detection unit]
The line-of-sight detection unit 10 is a device that is installed in the vehicle and detects the line of sight by tracking the movement of the eyeball (pupil) of the driver of the vehicle by image recognition.

The visual line detection unit 10 includes a CCD image sensor 10a, an LED light source 10b, and an image processing unit 10c.
The LED light source 10b emits invisible near infrared rays. The near infrared rays are emitted toward the eyes of the driver. In this case, the near infrared rays are reflected by the cornea of the eye, and the position of the reflection can be detected as a bright portion as compared with the surroundings. In addition, the position of reflection is characterized by maintaining a constant position even if the line of sight changes (even if the position of the pupil changes).

The line-of-sight detection unit 10 detects the image of the eye by the CCD image sensor 10a, and the image processing unit 10c analyzes the image of the eye.
In the image analysis, the above-mentioned reflection position (near infrared reflection position) on the cornea and the position of the pupil are detected.

28A and 28B show an example of image analysis. 28A and 28B are schematic diagrams each showing an example of imaging the eyes of the driver. However, the position of the line of sight (the position of the pupil) is different.
The pupil is darker in the eye than elsewhere, and the corneal reflex is brighter in the eye than elsewhere. Using this feature, in image analysis, the pupil and corneal reflex are detected and the positional relationship between them is analyzed.

The corneal reflex appears in the most raised part of the entire cornea and its position is almost constant, and the direction of the line of sight is detected (estimated) from the positional relationship of the pupil with respect to the position of the corneal reflex.

The direction of the line of sight may be estimated based on the direction connecting the central position of the corneal reflex and the central position of the pupil, as shown and suggested in FIGS. 28A and 28B. Alternatively, the direction of the line of sight may be estimated based on research data on the relationship between the position of the corneal reflex, the position of the pupil, and the direction of the line of sight, past learning values, and the like.

[Driving support processing (2)]
Next, the driving support process (2) executed by the driving support device 101 will be described with reference to FIG.

The driving support process (2) of FIG. 29 differs from the driving support process of FIG. 3 in that the correction determination process of S600 and the display correction process of S602 are executed after the process of S130. Since the processing of S100 to S130 is the same as the driving support processing of FIG. 3, the description thereof is omitted here.

In the correction determination process of S600, the control ECU 20 indirectly determines whether the driver recognizes the warning image based on the detection result of the line-of-sight detection unit 10, and reconstructs the warning image based on the determination result ( This is a process of correcting).

The display correction process of S602 is a process in which the control ECU 20 corrects the display image based on the result of the correction determination process of S600.
The correction determination process will be specifically described with reference to FIG.

When starting the correction determination process of S600 (correction determination process of FIG. 30), the control ECU 20 first communicates with the line-of-sight detection unit 10 in S610.
Next, the process proceeds to S612, and the analysis data by the line-of-sight detection unit 10 (in other words, the data indicating the movement of the line of sight) is acquired.

In subsequent S614, an object in the range where the line of sight is moved is extracted based on the analysis data acquired in S612.
Next, the process proceeds to S616, and the same process as S612 and S614 is repeated a predetermined number of times. This process is executed for the purpose of monitoring (tracking) the movement of the line of sight of the driver for a predetermined period.

In subsequent S618, for each of the objects extracted in S614 and S616, it is determined whether or not the driver's line of sight has moved a predetermined number of times or more to the area where the object exists. In this process, when the driver's line of sight moves only once, it is not certain whether or not the destination object (in other words, the emphasized image) is surely recognized, but it is recognized by having moved a predetermined number of times or more. It is executed for the purpose of determining that

When it is determined in S618 that the movement has not been performed the predetermined number of times or more, the processing is ended.
On the other hand, in S618, when it is determined that the image has been moved a predetermined number of times or more, the flag (image deletion flag) indicating that the image to be emphasized is deleted in association with the object (person) in the area determined to have moved the predetermined number of times or more. ) Is set.

Next, the process proceeds to S504, and the emphasized image generation process is executed. The process of S504 is the same as the process of S504 in FIG.
In the emphasized image generation processing of S504 in FIG. 30, except for the object (person) for which the flag for deleting the image to be emphasized is set, the other objects (person) are newly processed in S520 to S524 (FIG. 15). ).

As a result, in some cases, an object (person) that belonged to a group with a lower alert level is moved up to a group with a higher alert level, and thus an image with a higher alert level can be set.

Returning to FIG. 30, after S504, the process ends.
Next, the display correction process of S602 will be described with reference to FIG.
When the display correction process of S602 (display correction process of FIG. 31) is started, first, in S630, it is determined whether or not the image deletion flag is set. The image deletion flag is set in association with the object (person) in the process of S620 described above.

If it is determined that the image deletion flag is not set in S630, the process proceeds to S634.
On the other hand, when it is determined that the image deletion flag is set in S630, the process proceeds to S632.

In S632, for the object (person) corresponding to the image deletion flag, a command to erase the image for emphasizing the object (person) is generated and transmitted to the HUD 7. This causes the HUD 7 to stop (erase) the display of the image to be erased. Then, the process proceeds to S634.

In S634, it is determined whether the display mode of the image has been reset. In other words, it is determined whether or not the process of S504 in FIG. 30 has been executed.
If it is determined in S634 that the image display mode has been reset (the process of S504 has been re-executed), the process proceeds to S636.

In S636, a signal representing the image to be displayed is transmitted to the HUD 7 based on the processing result of S504 in FIG. As a result, the image is displayed over the front window of the vehicle via the HUD 7. Then, the process is completed.

As described above, the driving support apparatus 101 according to the third embodiment superimposes and displays an image emphasizing an object around the vehicle on the front window, and determines whether the driver of the vehicle recognizes the image. The determination is made by detecting the line of sight of the driver. Then, the display of the image for which the determination result that the driver of the vehicle has recognized is obtained is stopped (in other words, deleted). Then, the display mode is reset. Thereby, another image (an image for emphasizing another object (person)) is further emphasized and displayed. Further, it becomes possible to newly set and emphasize an object (person) for which an image has not been set.

According to this, it becomes possible to make the driver recognize the existence of the object more effectively or efficiently.
For example, with respect to the same object, it is possible to prevent the image for emphasizing the object from being continuously displayed even though the driver recognizes it, which further improves the practicality.

In the third embodiment, the line-of-sight detection unit 10 corresponds to an example of a line-of-sight detection unit, the process of S618 corresponds to an example of an identification unit, and the processes of S620 and S602 correspond to an example of an erasing unit.

<Fourth Embodiment>
A fourth embodiment of the present invention will be described.
In the fourth embodiment, the configuration of the driving support device is the same as the configuration of the driving support device 1 of the first embodiment (see FIG. 2).

On the other hand, the fourth embodiment is different in that the recognition determination process (2) of FIG. 32 is executed instead of the recognition determination process of FIG. 13.
The recognition determination process (2) of FIG. 32 is different from the recognition determination process of FIG. 13 in that the processes of S650 to S658 are executed. Note that the processing of S400 to S408 is the same as the recognition determination processing of FIG. 13, so description thereof will be omitted as appropriate.

In the recognition determination process (2) of FIG. 32, when the control ECU 20 determines that both eyes cannot be detected in the process of S404, the process proceeds to S650.
In S650, an alarm process for issuing an alarm to an object (person) is executed. Specifically, the process of emitting a predetermined sound (including voice) from the speaker unit 8 is executed. The alarm is issued to make the object (person) aware of the presence of the own vehicle. It is also issued to determine whether or not the object (person) notices the existence of the own vehicle. The warning may be given by turning on or blinking the headlamp of the vehicle.

Next, the process proceeds to S652, and the image data of the infrared camera 4 or the visible light camera 5 is acquired again.
In subsequent S654, the same object (based on the image data reacquired in S652)
Re-extract the face area in (person).

Next, in S656, the extracted face is analyzed. More specifically, “eyes” are extracted by edge detection, pattern matching, or the like.
Then, in S658, it is determined whether or not both eyes can be detected.

When it is determined in S658 that both eyes can be detected, it is determined that the own vehicle is likely to exist within the range of the field of view of the object (person), and based on this determination, the object (person) is It is simply determined that the existence is recognized, and the process proceeds to S406.

On the other hand, when it is determined in S658 that both eyes cannot be detected, it is determined that the own vehicle may not exist within the visual field of the object (person), and based on this determination, the object (person) is A simple determination is made that the presence of a vehicle is not recognized, and the flow shifts to S408.

As described above, according to the fourth embodiment, if the driving assistance apparatus 1 determines that the object (person) does not recognize the existence of the own vehicle, it issues an alarm to make the existence of the own vehicle notice. This allows the object (person) to recognize the presence of the own vehicle. Further, by re-analyzing the face of the object (person) after issuing the alarm, it is determined whether or not the object (person) notices the existence of the own vehicle. As a result, the risk (warning level) can be appropriately set according to the state of the object (person). As a result, it is possible to more appropriately display the image for warning the driver.

<Fifth Embodiment>
A fifth embodiment of the present invention will be described.
In the present fifth embodiment, the configuration of the driving support device is the same as the configuration of the driving support device 1 of the first embodiment (see FIG. 2).

On the other hand, the fifth embodiment differs from the driving support device 1 of the first embodiment in that the analysis process 9 of FIG. 33 is further executed.
The analysis process 9 will be specifically described below.

The processing of S400 to S404 in the analysis processing 9 is the same as the processing of S400 to S404 in FIG. 13 (and the processing of S400 to S404 in FIG. 32).
The processing of S650 to S658 in the analysis processing 9 is the same as the processing of S650 to S658 in FIG. A description of these processes will be omitted.

The control ECU 8 executes the analysis process 9 in addition to the analysis processes 1 to 7 shown in FIGS. 5 to 12 (and FIG. 13) as the analysis process of S152 of FIG. 4. In the analysis process 9, in S404 or S658. If it is determined that both eyes can be detected, the process proceeds to S660.

In S660, the caution level is decremented by 2 points for the target object (person) determined to be able to detect both eyes. The numerical value of 2 points is an example, and any value may be used.

The purpose of the processing of S660 is that it is possible to determine that the object (person) recognizes (or notices the existence of) the own vehicle by the affirmative determination in S404 or S658, and the alert level for the object (person) is high. The purpose is to lower.

As described above, according to the fifth embodiment, the warning level is lowered for the object (person) recognizing the existence of the own vehicle, and thus the other object (person) (for example, the existence of the own vehicle is recognized). The alert level for a non-existent object (person) will be relatively increased. As a result, a more emphasized image can be displayed for an object (person) that needs more attention. Therefore, the driver can be made to recognize the existence of the object more effectively or efficiently.

<Modification>
Hereinafter, another example of the display mode will be described.
[Modification 1]
Modification 1 will be described with reference to FIG. 34. In the example of FIG. 34, the objects H7 and H8 are extracted and recognized.

In the example of FIG. 34, the moving direction of the object is detected and displayed by arrows, as in the example of FIG. 19 (arrows Y7 and Y8).
In addition, in FIG. 34, the positions and paths of the objects H7 and H8 after a predetermined time have been estimated and displayed.

Regarding the object H7, the position surrounded by the frame W7 is the current position, and this is the initial position at time tA.
The driving support device 1 repeats the acquisition and analysis of the image data of the infrared camera 4 or the visible light camera 5, and executes the tracking process of tracking the movement of the object H7. Then, based on the tracking process, the moving speed and moving direction of the object H7 are estimated.

Then, the position and speed of the object H7 after a predetermined time tB is estimated, and the image of the object H7 is displayed so as to be superimposed on the estimated position. This image is displayed blinking.

Furthermore, the position and speed of the object H7 after a predetermined time tC (tB<tC) are estimated with reference to the time tA, and the image of the object H7 is displayed so as to be superimposed on the estimated position. This image is displayed blinking.

Further, the image after the predetermined time tB and the image after the predetermined time tC are continuously displayed as if the object H7 is moving.
In addition, movement estimation arrow YF7 is displayed so as to follow the locus from the initial position at time tA to the position after a predetermined time tC. The movement estimation arrow YF7 indicates the movement path of the object H7, which is predicted (or determined to be highly likely to move).

Next, the same applies to the object H8. Regarding the object H8, the position surrounded by the frame W8 is the current position, and this is the initial position at time ta.
By the tracking process, the position and speed of the object H8 after a predetermined time tb is estimated, and the image of the object H8 is displayed so as to be superimposed on the estimated position. This image is displayed blinking.

Furthermore, the position and speed of the object H8 after a predetermined time tc (tb<tc) are estimated with reference to the time ta, and the image of the object H8 is displayed so as to be superimposed on the estimated position. This image is displayed blinking.

The image after the predetermined time tb and the image after the predetermined time tc are continuously displayed as if the object H8 is moving. Note that the object H8 is approaching the host vehicle, and the image of the object H8 is displayed so as to gradually increase in size.

In addition, movement estimation arrow YF8 is displayed so as to follow the locus from the initial position at time ta to the position after a predetermined time tc. The movement estimation arrow YF8 indicates the movement path of the object H8, which is predicted (or determined to have a high possibility of movement).

[Modification 2]
Modification 2 will be described with reference to FIG.
In the example of FIG. 35, the driving assistance device 1 is mounted on the vehicle K3. The vehicle K4 is another vehicle. The vehicles K3 and K4 are traveling in the same traveling direction (from bottom to top (front to back) in the drawing).

Pedestrians (objects) H12 and H13 crossing a pedestrian crossing exist in front of the vehicle K3.
It is difficult for the driver of the vehicle K4 to see the objects H12 and H13 in the shadow of the vehicle K3.

When the driving support device 1 for the vehicle K3 analyzes the positional relationship between the objects H12, H13 and the vehicle K4 and determines that it is difficult to visually recognize the objects H12, H13 from the vehicle K3, for example, in a region Sc2 in the rear window of the own vehicle, Images of the objects H12 and H13 may be displayed. Further, at this time, an image, a message, or the like, which calls the attention of the driver of the vehicle K4, may be displayed.

This is effective because the driver of the vehicle K4 can easily recognize the existence of the objects H12 and H13.
[Modification 3]
Modification 3 will be described with reference to FIG.

Here, in FIG. 17, an example of displaying an image for emphasis on a person and a vehicle has been described.
The example of FIG. 36 shows an example in which images for enhancement are displayed not only on the person and the vehicle but also on other objects around the vehicle. In addition, the name of the object is displayed.

Specifically, in the example of FIG. 36, in addition to a pedestrian and a vehicle, a traffic signal, a road sign, a signboard, and a road surface display are extracted and recognized, and a frame W for emphasis is displayed in association with each other. I am trying. In addition, the name display area N is set in association with each frame W, and the name of each object is displayed in the name display area N. As described above, displaying the characters may be expected to improve the effect of attracting the driver's attention.

<Other Embodiments>
In the above embodiment, the example in which the driving support device 1 includes the infrared radar 2, the millimeter wave radar 3, the infrared camera 4, and the visible light camera 5 has been described.

On the other hand, the infrared radar 2 may be omitted, and the millimeter wave radar 3 may be configured to detect short-distance and long-distance objects.
In the above embodiment, the example in which the driving support device 1 includes the infrared radar 2, the millimeter wave radar 3, the infrared camera 4, and the visible light camera 5 has been described.

On the other hand, the mounting of the infrared radar 2 and the millimeter wave radar 3 may be omitted. In this case, for the distance to the object, information detected by the infrared camera and the visible light camera (information on the distance to the object) may be used.

In the above embodiment, the example in which the driving support device 1 includes the millimeter wave radar 3 has been described.
On the other hand, the driving support device 1 may include a laser radar instead of the millimeter wave radar 3. Alternatively, both the millimeter wave radar and the laser radar may be provided. The laser radar is a radar that detects a surrounding situation using laser light. Specifically, the laser radar scans (two-dimensionally scans) pulsed laser light, and receives the laser light reflected and returned by the object. Then, the laser radar measures the time difference between the emission time of the laser light and the reception time of the reflected light, and the intensity of the reflected light, and detects the object based on them. The laser radar can detect a lane boundary line (a white line that forms a boundary between a traffic zone and a sidewalk, etc.) in addition to a three-dimensional object.

In the above-described embodiment, the driving support devices 1, 100 and 101 include the infrared radar 2, the millimeter wave radar 3, the infrared camera 4, the visible light camera 5, the momentum detection unit 6, the head-up display 7, and the speaker. The unit 8, the image projection device 9, and the line-of-sight detection unit 10 may all be mounted.

In the above embodiment, an example of displaying an image for emphasizing an object (an image surrounding the object) has been described, but an image of the object itself may be generated and displayed. In this case, the infrared cameras 4A and 4B or the visible light cameras 5A and 5B may stereoscopically capture the object, generate a stereoscopic image, and display the stereoscopic image.

In the above embodiment, an illuminance sensor may be further provided, and the infrared camera 4 and the visible light camera 5 may be switched and used according to the detection data of the illuminance sensor. Specifically, when the illuminance is greater than or equal to a predetermined threshold (for example, during the day), image data of the visible light camera 5 is used, and when the illuminance is less than the predetermined threshold (for example, in the evening to midnight, cloudy, or rainy weather). In this case), the data of the infrared camera 4 may be used.

In the above embodiment, whether or not the object (person) is operating the mobile terminal may be determined by communicating with the mobile terminal.
For example, the driving support devices 1, 100, 101 are equipped with Bluetooth (registered trademark) devices or the like, and try pairing with a portable terminal around the vehicle. When pairing is established, data communication may be performed with the mobile terminal and data for determining whether or not the mobile terminal is being operated may be acquired from the mobile terminal. If an application that detects that the user of the mobile terminal is using the mobile terminal while moving is installed on the mobile terminal, the mobile terminal is operated in conjunction with the application. May be detected. Further, the driving support apparatus 1, 100, 101 may transmit a warning image to the mobile terminal, display the image on the mobile terminal, and let the user of the mobile terminal recognize the presence of the vehicle.

In the above embodiment, the analysis processes 1 to 9 have been described. Then, an example in which the analysis processes 1 to 9 are executed in parallel or sequentially in a predetermined order has been described.
On the other hand, after any one of the analysis processes 1 to 9 is executed, display data is generated based on the result of the analysis process (the process of FIG. 14 is executed), and then the next analysis process is executed. The display data may be generated again based on the result of the analysis processing (in other words, the display data may be generated (corrected) each time the analysis processing is executed).

In the above-described embodiment, the inter-vehicle communication unit may communicate with another vehicle to warn the driver of the other vehicle. Alternatively, the warning information may be received from another vehicle.

In the above embodiment, the position of the host vehicle may be acquired via the vehicle position sensor 12, and the positional relationship with the object may be grasped based on the acquired position.
Further, in the above embodiment, an example in which the analysis processes 1 to 9 (FIGS. 5 to 12 (and FIG. 13), FIG. 24, and FIG. 33) are executed as the analysis process of S152 has been described. Here, another example of the analysis process will be described.
[Analysis process 10]
The analysis process 10 will be described with reference to FIG. The analysis process 10 is a process for analyzing the weather, and more specifically, a process of detecting rainy weather.

The analysis process 10 can be repeatedly executed by the control ECU 20 at a predetermined timing.
In the analysis process, first, in S670, image data of an image captured by the visible light camera 5 is acquired. The visible light camera 5 is arranged in the vehicle so as to capture an image of the surroundings of the vehicle from the inside of the vehicle through the window glass of the vehicle (see FIG. 1 ). Here, as the image data of the image captured by the visible light camera 5, image data including the image of the region of the window glass of the vehicle is acquired.

Next, the process proceeds to S672, and raindrop candidates in the window glass area (raindrop adhered to the window glass) are detected from the image data acquired in S670. FIG. 38A shows an example of an image of raindrops. When the raindrops adhere to the window glass of the vehicle, the raindrop portion becomes bleeding and is detected with a transparency different from that of the window glass portion. At the boundary between the raindrop and the window glass, a portion (edge) where the chromaticity (density) of adjacent pixels changes sharply appears. The area of the raindrop is detected based on such an edge.

Returning to FIG. 37, after S672, the process proceeds to S674, and it is determined whether or not the raindrop candidate detected in S672 is a raindrop by pattern matching.
Specifically, the model of the image of the raindrop is stored in the storage device in advance. The storage device may be the ROM 20b, the flash memory 20d, or the like, but may be another storage device.

In FIG. 38B, models 1, 2, 3,... N are shown as an example of a model of a raindrop image stored in advance in the storage device. As the models 1, 2, 3,... N, representative models representing raindrops can be arbitrarily selected and stored. The control ECU 20 may be provided with a function (learning function) of accumulating raindrop image models.

Then, the raindrop candidate detected in S672 and the models 1, 2, 3,... N stored in the storage device are compared and collated, and the area (the number of pixels) and the color between the raindrop candidate and the model are compared. The degree of similarity is calculated for parameters such as degree (shading) and shape. Here, at least one of the parameters may be used. Then, if the at least one parameter is matched by a predetermined ratio, the raindrop candidate may be determined as the raindrop.

The processes of S672 and S674 can be executed for all of the raindrop candidates in the image data acquired in S670. Alternatively, a predetermined area may be extracted from the image data, and the processes of S672 and S674 may be executed only in the extracted area.

After S674, the process proceeds to S676, and the amount of raindrops (in other words, the amount of rainfall) is detected. For example, the amount of raindrops can be detected from the number of raindrops and/or the ratio of the area occupied by raindrops in the image.

Next, the process proceeds to S678, and it is determined whether or not the amount of raindrops is equal to or greater than a predetermined amount. When it is determined that the amount of raindrops is less than or equal to the predetermined amount (less than the predetermined amount), the process ends as it is. On the other hand, if it is determined that the amount of raindrops is greater than or equal to the predetermined amount, then the flow shifts to S680, where the alert level is incremented by 1 point.

With such a configuration, raindrops can be detected with high accuracy, and rainy weather can be detected more reliably and accurately. Then, when it is detected that it is raining, the alert level is raised in the driving support devices 1, 100 and 101, and the object around the vehicle is displayed by a more appropriate method (for example, a more conspicuous method). Will be able to do it. The weather greatly affects the visibility of the driver, and particularly in rainy weather, it is usually difficult for the driver of the vehicle to recognize the surroundings of the vehicle. On the other hand, according to the present invention, since it can be detected that it is raining and the display can be performed in a more easy-to-see mode accordingly, it is possible to contribute to driving safety.
[Analysis process 11]
By the way, the weather may be recognized by a process different from the analysis process 10. Specifically, the weather may be recognized by the analysis process 11 shown in FIG.

In the analysis process 11 of FIG. 39, first, in S682, weather information is acquired through communication with the outside. Specifically, the driving support devices 1, 100, 101 may be provided with a communication device for connecting to a communication line network (for example, the Internet network). Note that such a communication device is well known, and a detailed description and illustration thereof will be omitted. Then, the driving support devices 1, 100, 101 may connect to, for example, the Internet network via the communication device to acquire the weather information.

Next, the process proceeds to S684, and illuminance data is acquired from an illuminance sensor (not shown) for detecting illuminance (outdoor illuminance) provided in the vehicle.
Next, the process proceeds to S686, and temperature data is acquired from a temperature sensor (not shown) provided in the vehicle for detecting the external temperature.

Next, the process proceeds to S688, and based on the data obtained in S682 to S686, it is comprehensively determined whether or not it is raining.
For example, although it can be said that the weather can be grasped only by the process of S682 (for example, it is possible by acquiring the information of the weather forecast), there is no guarantee that the weather forecast is 100% accurate, and Weather forecasts for pinpoint areas are often not made.

Therefore, here, as an example, an illuminance sensor and a temperature sensor are used to detect the illuminance and temperature in the field in addition to the data of the weather forecast to determine whether or not it is raining, thereby more accurately detecting the weather. be able to. In addition, humidity may be detected and used.

After S688, the process proceeds to S690, and it is determined whether the weather is fine or not based on the processes of S682 to S688. If it is determined that the weather is fine, the process proceeds to S694.
In S694, the alert level is maintained at the current alert level (in other words, the process of changing the alert level is not executed). Then, after that, the process ends.

If it is determined that the weather is not fine in S690, the process proceeds to S692. In S692, it is determined whether the weather is cloudy. If it is determined that it is cloudy, the process proceeds to S696. In S696, the alert level is incremented by 1 point. Then, after that, the process ends.

If it is determined in S692 that it is not cloudy, it is determined that it is rain, snow, etc., and S698.
Move to and increase the alert level by 2 points.
According to the analysis processing 11, the weather can be detected (or determined) with higher accuracy as described above, and thus more appropriate driving assistance can be realized according to the weather. Specifically, the alert level can be appropriately set according to the weather, the object around the vehicle can be highlighted according to the appropriately set alert level, and/or depending on the danger. The warning display can be appropriately controlled.

[Analysis process 12]
Next, the analysis process 12 will be described with reference to FIG.
The driving support device 1, 100, 101 of the present embodiment may execute the analysis process 12 in addition to the analysis processes 1 to 11 or in place of the analysis processes 1 to 11. The analysis process 12 detects that the user is operating the mobile communication terminal while moving (for example, while walking) in cooperation with the function of the mobile communication terminal such as a smartphone, a tablet, or a mobile phone (more specifically, Is a process for detecting such a mobile communication terminal.

In the case where the user operates the mobile communication terminal while walking, the user's line of sight, interest, attention, etc. are directed toward the mobile communication terminal, and attention is not directed to the surroundings, which is extremely dangerous. Has been done.

In view of this point, there is a case where the mobile communication terminal side is provided with a function and an application for detecting that the user is operating the mobile communication terminal while moving. Specifically, it is detected by the GPS function or the like whether or not the position of the mobile communication terminal has changed, and thus whether or not the user is moving. Then, it is detected whether the mobile communication terminal or the mobile communication terminal is operated while the user is moving.

When it is detected that such movement and operation overlap, a warning to that effect is issued.
In one example, a warning is displayed on the display screen of the mobile communication terminal, a sound is emitted, or an alarm (a sound alarm, a signal indicating an alarm) is issued to surrounding terminals.

The analysis process 12 is premised on the mobile communication terminal having the functions and applications as described above.
In the analysis process 12, first, in S700, a process of searching for mobile communication terminals existing in the vicinity is executed. This search can be performed by detecting a Bluetooth (registered trademark) signal or another wireless signal emitted from the mobile communication terminal. In one example, a pairing signal for performing pairing with Bluetooth (registered trademark) is transmitted from the driving support apparatus 1, 100, 101, and the presence or absence of a response signal to the pairing signal is detected. Alternatively, the presence/absence of a pairing signal transmitted from the mobile communication terminal is detected. Alternatively, in another example, the mobile communication terminal may be detected by image processing using image data from the visible light camera 5.

Next, the process proceeds to S702, and based on the process of S700, it is determined whether or not a mobile communication terminal exists around the driving support devices 1, 100, 101.
When it is determined that the mobile communication terminal does not exist, the process ends as it is. On the other hand, if it is determined that there is a mobile communication terminal, the process proceeds to S704.

In S704, it is determined whether or not a warning signal that warns that the user is operating the mobile communication terminal while moving is received from the mobile communication terminals detected in S700 and 702. Note that it is preferable that the warning signal of this type has a specification so that it can be unconditionally detected in the peripheral communication device. For example, it is preferable that only the above-mentioned warning signals can be detected with each other without needing to establish pairing using the Bluetooth (registered trademark) function. However, since this is a specification and not the essence of the present invention, it will not be described in detail here.

If it is determined in step S704 that the warning signal has not been received, the processing ends.
On the other hand, when it is determined that the warning signal is received, the process proceeds to S706, and the HUD 7 displays the warning. Further, the processing shifts to S708, and the caution level is incremented by 1 point. Then, after that, the process ends.

According to the analysis process 12 as described above, it is possible to detect a person who is operating the mobile communication terminal while moving to raise the alert level, and appropriately alert the driver.
[Vehicle control processing]
The driving support device 1, 100, 101 of this example may control the operation of the vehicle according to the alert level. Such an example will be described with reference to FIGS. 41A and 41B.

The driving support devices 1, 100, 101 repeatedly execute the vehicle control process of FIG. 41A at a predetermined timing.
In this vehicle control process, first, in S710, it is determined whether or not the warning level is equal to or higher than a predetermined level. When it is determined that the warning level is not equal to or higher than the predetermined level, the processing is ended as it is.

On the other hand, when it is determined that the warning level is equal to or higher than the predetermined level, the process proceeds to S712, and a control command for controlling the vehicle is output. Specifically, this control command can be output to an electronic control unit (ECU) that controls each unit of the vehicle. Then, the ECU that receives the control command controls the control target. After the processing of S712, the processing ends.

Examples of vehicle control by this vehicle control processing include throttle control for controlling the opening of a throttle valve, braking control for controlling a braking device (brake), steering control for controlling a traveling route or traveling direction of a vehicle, and the like. be able to.

The throttle control may be a control that suppresses the throttle opening (in other words, prohibits acceleration). The braking control may be a control that causes a brake to function to decelerate the vehicle. The steering control may be, for example, control for controlling the traveling route of the own vehicle so that the own vehicle is separated from an object around the own vehicle, which is likely to collide. In addition, an alarm may be issued. For example, a vibration mechanism may be built in the steering wheel, the vibration mechanism may be vibrated according to the warning level, and an alarm may be transmitted to the driver with vibration (hereinafter, this type of alarm is also referred to as alarm control).

Then, the driving support devices 1, 100, 101 have table information as shown in FIG. 41B in which warning levels and vehicle control contents are associated with each other. This table information can be stored in advance in a storage device (ROM 20b or the like). According to the table information of FIG. 41B, vehicle control is realized as follows.

When the value of the warning level is from -(minus) to 0, vehicle control is not executed.
When the value of the warning level is 1 to 3, alarm control and throttle control are performed.
When the warning level value is 4 to 6, alarm control, throttle control, and braking control are performed.

When the value of the warning level is 7 or more, alarm control, throttle control, braking control, and steering control are performed.
Here, it goes without saying that the classification of the alert level is an example. The alert level may be further divided into multiple stages, or conversely may be coarsely divided. In addition, since the variation of the alert level may vary depending on the type of analysis processing (in this example, analysis processing 1 to 12 are illustrated), table information is optimal depending on the type of analysis processing to be executed. It can be understood by those skilled in the art that it can be realized.

According to such a configuration, not only the display mode but also the operation of the vehicle is controlled according to the alert level, so that it is possible to contribute to safe driving of the vehicle. Specifically, by controlling the display mode according to the alert level, the driver of the vehicle is alerted, and the driver himself/herself can be promoted to drive safely, while the driving support device 1, 100, The vehicle control is executed in order to realize safe driving without the driver's judgment being involved in the judgment processing of 101.

As a result, in addition to the driving by the driver himself, safe driving of the vehicle is realized at a higher level by the support of the driving support devices 1, 100, 101. Further, even if the driver's driving skill is immature and the expected value of the safe driving by the driver himself is low, the safe driving of the vehicle can be realized by the control of the driving support devices 1, 100, 101.
[Display control processing, display mode example]
The display control process and the display mode example according to the present invention will be further described with reference to the drawings.

First, a description will be given with reference to FIGS. 42A and 42B.
42A and 42B show an example of controlling the display contrast according to the alert level. Here, the contrast is intended to be a contrast between an object to be highlighted (a person, an obstacle, etc.) and a display object other than the object.

As shown in the table D4 in FIGS. 42A and 42B, the warning level may be associated with the contrast level at the time of display. This information can be stored in the storage device (ROM 20b or the like) as table information. When displaying an image by the HUD 7, the driving support devices 1, 100, 101 read the information in the table D4 from the storage device and read the contrast information corresponding to the set alert level. Contrast (contrast ratio) is low when the warning level is -(minus) to 0, medium when the warning level is 1 to 3, high when the warning level is 4 to 6, and high when the warning level is 7 or higher. May be set to be the highest.

FIG. 42A shows an example when the contrast is low. FIG. 42B shows the case where the contrast is the highest.
In the example of FIG. 42A, the contrast between the target object D0 as the highlighting target and its surroundings (background or the like) is low, and the difference in brightness between the target object D0 and its surroundings is small, but the alert level is If the contrast is low, the merit of reducing the contrast may be prioritized. The merit of reducing the contrast is that eye strain may be suppressed in some cases, and that nature may be prioritized and the scenery may become closer to the actual scenery.

In the example of FIG. 42B, the contrast between the target object D0 and its surroundings (background or the like) is high, and the difference in brightness between the target object D0 and its surroundings is large. Therefore, the object D0 is more emphasized and can be visually recognized more clearly. When the alert level is high, the object D0 may be more emphasized by setting the contrast high in this way.

Regarding the contrast setting, the light and shade may be set for each representative area in the image.
This point will be described using the tables D3 and D3′.
The tables D3 and D3' include information on the shade set for each of the representative areas in the image. In the tables D3 and D3′, blocks Da and Da′ indicate the shade of the area of the object D0, blocks Db and Db′ indicate the shade of the background planting area of the object D0, and the blocks Dc and Dc′. Indicates the shade of the ground surface of the background of the object D0, and the blocks Dd and Dd' indicate the shade of the road.

In the tables D3 and D3′, the larger the positive value is, the darker (closer to black) the smaller the positive value is, or the larger the negative value is, the lighter (closer to white).
The relative relationship between the light and shade of each block may be automatically set according to the default value according to the contrast level (low, medium, high, highest).

Alternatively, the relative relationship of shading of each block may be manually set by the user (driver). For example, the menu display D2 may be provided, and when the menu display D2 is selected, various setting screens may be displayed, and the contrast may be set on such a setting screen.

According to this, since the user (driver) himself/herself can set the contrast that is most visible to the user (driver), convenience is improved. Further, the display mode can be optimized for each individual user (driver), and the effect of driving support can be maximized.

The display mode is adjusted according to the skill of the user (driver) (whether or not he/she is a good driver), age, sex, accident history, violation history, physical ability (mainly visual acuity), or physical condition during driving. May be done.

For example, a mechanism for reading a driver's license may be provided, and some of the above information may be automatically acquired by reading the driver's license. Information that cannot be read from the driver's license, such as visual acuity and physical condition during driving, may be manually input.

42A and 42B, the display D1 is a display for indicating that the display control is functioning normally. When some abnormality is detected in the display control, the display content of the display D1 is changed to the content indicating that the abnormality has occurred.

In the embodiment of the present application, the object is highlighted as described above, but the modes of highlighting include enclosing with a frame, highlighting, changing the display color, blinking, and displaying adjacent symbols. Various modes such as “Yes” are prepared, and such highlighting may be canceled. That is, the display mode can change in real time.

In this case, even if an error occurs in the display control due to some abnormality, the user (driver) does not notice that the display is incorrect unless the user (driver) can understand the abnormality. You will end up making a recognition based on a false indication. In other words, there is a possibility that false positives that deviate from the originally intended result may occur.

In view of this point, by providing a display such as the display D1 for indicating that the display control is functioning normally, the user (driver) can be trusted that the screen being displayed is a normal screen. In addition, the user (driver) can recognize the fact that an abnormality has occurred, and it is possible to suppress the occurrence of misidentification due to the apologetic display.

Note that a generally known general method can be applied to the abnormality detection method (detection processing), and a detailed description thereof will be omitted here.
However, even if the abnormality detection method (detection processing) is general, various contrivances can be considered for the execution timing.

In one example, it may be possible to execute at the following timings.
(1) Timing when the vehicle ignition switch is turned on (2) Timing when the vehicle actually starts traveling after the vehicle ignition switch is turned on (for example, timing when tire rotation is detected)
(3) Timing at which the vehicle temporarily stops after traveling (timing at a traffic light, intersection, etc.) (4) Arbitrary timing during traveling of the vehicle (including repeated execution)
The timings of (1) and (2) above are timings at which the driving is just started, and the abnormality detection process is executed at such timing, and the display D1 is displayed, so that the user (driver) ) Can give a sense of security to future driving.

At the timing of the above (3), since the vehicle is stopped, the degree of risk is small and the processing load of the display control is small, or the processing load for the display control is set by intentionally omitting the display control. You may do that. If the abnormality detection process is executed at such a timing, it is possible to prevent the processing load from becoming excessively large. Further, it is possible to suppress the risk that some abnormality occurs in the display control due to the increase in the processing load (for example, the risk that processing delay occurs). Therefore, it is possible to contribute to safe and reliable driving support.

The timing of (4) above is a timing during traveling, and as long as the display control is not hindered from the viewpoint of processing load or the like, it is possible to notify the user (driver) of the presence or absence of an abnormality in real time. It is safer for the driver.

Further, the display D1 may be always displayed or may be displayed at an arbitrary timing.
When displaying at arbitrary timing, the display timing may be set in accordance with the execution timing of the abnormality detection processing as exemplified in (1) to (4) above. Specifically, the display D1 may be displayed in synchronization with the timing at which the abnormality detection processing is executed (the timing at which the processing ends).

Next, another display mode example will be described with reference to FIG.
FIG. 43 is an example in which a dangerous area in an area (a road or the like) in which a vehicle travels is highlighted. In this example, a sediment disaster occurs and a part of the road is fragmented by sediment.

As described above, each of the driving support devices 1, 100 and 101 includes the visible light camera 5, and as described above, the detection target obtained by performing image processing on the image captured by the visible light camera 5 or being reflected in the image, or the detection target. It is possible to recognize the area occupied by the detection target. Therefore, the driving support devices 1, 100, 101 may be configured to detect a landslide region and superimpose the highlighted display D6 on the landslide region.

In this case, in addition to the highlighted display D6, the text display D5 may be displayed adjacent to the highlighted display D6. The text display D5 and the highlight display D6 may be displayed in blinking. Further, the display color may be changed. For example, the display mode may be changed according to the scale of the disaster.

Further, in addition to the region where the landslide occurs, the region where the landslide may occur may be additionally highlighted. In this case, areas where landslide prevention measures have not been taken (for example,
It is also possible to detect by image processing a region in which water and/or a slight amount of sand or the like flows on the surface) by highlighting the region which is not covered with concrete).

Encountering sediment-related disasters may be low in probability. However, because of this, the driver may not be able to comprehend his/her understanding in the case of an actual encounter, in other words, the recognition may be delayed, and a serious accident may occur. Therefore, by supporting such highlighting at the time of a disaster, the reliability and safety of driving assistance can be further improved.

Next, another display mode example will be described with reference to FIG.
FIG. 44 assumes a situation of traveling on a bank along a river, and in such a case, an example of displaying the safety level or the risk degree of the river water level (in other words, the risk of flooding of the river) Is.

For example, the water level of the river may be detected and the degree of danger may be displayed as an indicator. Specifically, the indicator display D10 may be provided. Gradation display may be adopted as the indicator display D10.

A danger display D12 and a safety display D13 can be provided near the indicator display D10.
Then, according to the water level of the river, the current water level frame D14 is superimposed and displayed on the indicator display D10. The closer the display position of the current water level frame D14 is to the danger display D12, the higher the water level of the river is, and the more dangerous the current water level frame D14 is to the safety display D13.

The text display D11 is provided adjacent to the current water level frame D14. As the text display D11, the risk level (or safety level) may be displayed in text.
An emphasis display D15 can be displayed in an overlapping manner in the area occupied by the river. It is preferable that the display mode such as the display color and the pattern of the highlighted display D15 be the same as the display mode such as the display color and the pattern of the area currently surrounded by the water level frame D14.

In this case, the control ECU 20 of the driving assistance device 1, 100, 101 extracts the display mode data in the area currently surrounded by the water level frame D14, and then applies the display mode to the display mode of the highlighted display D15. Processing will be executed.

According to such an example, the user (driver) can recognize the potential possibility of a disaster rather than the fact of the disaster as in the case illustrated in FIG. 43. Moreover, the user (driver) can intuitively understand the risk level on the indicator display D10 indicating the risk level. Further, by making the display mode of the highlighted display D15 match the display mode of the indicator D10 (display mode of the area currently surrounded by the water level frame D14), the ease of recognition for the user (driver) is significantly improved. obtain.

Next, another display mode example will be described with reference to FIGS. 45A-45C.
First, regarding FIG. 45A, one of the purposes is to stereoscopically display an object. In FIG. 45A, structures D20 and D22 are buildings existing along the road on which the vehicle is traveling. This type of building may be displayed three-dimensionally by using a three-dimensional display technology. As a stereoscopic display technique, a method of realizing stereoscopic vision by preparing a plurality of projectors (generally a pair of left and right projectors) that project from different directions and displaying an image for the left eye and an image for the right eye respectively It has been known.

Alternatively, map data including stereoscopic image data may be used. That is, the image represented by the stereoscopic image data may be displayed.
The stereoscopic display is expected to be easier for the user (driver) to visually recognize.

Then, in FIG. 45A, for each of the structures D20 and D22, the symbol displays D21 and D23 are drawn according to their attributes.
The attribute includes the type of structure. The types of structures include stores, government offices, and private houses.

Further, the attribute includes attached information attached to the structure. For example, in the case of a store, the auxiliary information may include information such as business hours, store size, average number of visitors, and location information.
It is conceivable that such attribute information is, for example, attached to the map data, and is acquired from the map data on the driving support device 1, 100, 101 side.

In the example of FIG. 45A, if the structure D20 is a convenience store, in one example, information such as being a store, being open 24 hours, and having many guests in the morning and evening is included in the attributes of the structure D20. obtain.

Based on such an attribute, the control ECU 20 displays a symbol display D21 that calls attention to the entry and exit of vehicles from the parking lot, for example, based on the information that there are many customers who are open 24 hours a day, and the structure D20. Is displayed in association with. Displaying in association with the structure D20 is understood to mean, for example, displaying in the vicinity of the structure D20, displaying adjacent to the structure D20, and displaying in superposition on the structure D20. Is also good.

Further, it is assumed that the structure D22 is located at a point where the road is just curved, and the attribute "information (location information)" is included in the attribute. The control ECU 20 displays, for example, a symbol display D23 for promoting traveling along a curve in association with the structure D22 based on such an attribute.

According to this, the user (driver) can recognize the potential risk associated with the road on which the vehicle is traveling.
FIG. 45B shows an example in which an oncoming vehicle is running outside the central lane.

When such an oncoming vehicle is detected, the control ECU 20 estimates the route in which the oncoming vehicle may travel by calculation, and superimposes the highlighted display D25 on the estimated area. However, such display control may not be performed when the oncoming vehicle is traveling in the lane. For example, such display control may be executed when an oncoming vehicle is running out of the lane and there is a risk of collision.

FIG. 45C shows an example in which the host vehicle runs outside the lane.
In such a case, the control ECU 20 estimates a route in which the own vehicle may travel, displays the symbol of the own vehicle, and superimposes the highlighted display D27 on the estimated area.

By displaying the symbol of the own vehicle, the user (driver) can objectively recognize the danger. In a traffic accident, there is a case where there is a problem with the belief that "OK, no problem", that is, subjective judgment. By being able to grasp the situation objectively, it is possible to eliminate subjective feelings, which can contribute to accident prevention.

However, such display control may not be performed when the host vehicle is traveling in the lane. For example, such display control may be executed when the host vehicle is running out of the lane and there is a risk of collision.

It is preferable that the display mode of the highlighted display D25 and the display mode of the highlighted display D27 are different in a form that is easily distinguishable. In this case, it is easy to recognize whether the oncoming vehicle is over the lane and is dangerous, or whether the own vehicle is over the lane and is dangerous.

Although one embodiment of the present invention has been described above, those skilled in the art can understand that various aspects can belong to the scope of the present invention.
For example, the display screen may be configured with a touch panel. Then, by selecting an object on the touch panel, the object may be highlighted or may be canceled.

Further, when the vehicle travels in a place where traffic accidents frequently occur, the warning level may be additionally increased (the points of the warning level may be additionally incremented).

A driving support device according to a first aspect of the present invention is a detection unit that detects a situation around a vehicle, a recognition unit that recognizes an object around the vehicle by pattern matching based on a detection result of the detection unit, The analysis means for analyzing the object recognized by the recognition means, and the notification means for notifying the driver of the information indicating the analysis result of the analysis means, the recognition means, in addition to the pattern stored in advance The pattern learned and stored in the past is compared with the target object to recognize the target object. The analyzing unit may include a classifying unit that classifies an object to be analyzed and an object not to be analyzed among the objects recognized by the recognizing unit. The notification means may be configured not to notify the driver of information about the object classified as the object that is not analyzed. The analysis means analyzes the object recognized by the recognition means to determine whether or not the object is a person, and the first analysis processing analyzes the object to be a person. In this case, a second analysis process for further analyzing the state of the person is executed, and the driving support device sets a warning level for the person according to the result of the second analysis process. Further, the notifying means may be configured to notify the driver of information indicating the warning level set by the warning level setting means.
According to another aspect of the present invention, there is provided a driving support device including a detection unit that detects a situation around a vehicle, a recognition unit that recognizes an object around the vehicle based on a detection result of the detection unit, and the recognition unit. Analyzing means for analyzing the recognized object, setting means for the object based on an analysis result of the analyzing means, and setting means for setting a degree of caution, and allowing the driver of the vehicle to visually recognize the object. It is also possible to include a generation unit that generates an image for this purpose based on the degree set by the setting unit, and a display unit that displays the image generated by the generation unit .

Claims (3)

Detection means for detecting the situation around the vehicle,
Recognition means for recognizing an object around the vehicle based on the detection result of the detection means,
Analysis means for analyzing the object recognized by the recognition means,
For the object, based on the analysis result of the analysis means, setting means for setting the degree to be vigilant,
Generating means for generating an image for allowing the driver of the vehicle to visually recognize the object, based on the degree set by the setting means;
Display means for displaying the image generated by the generating means,
A driving support device comprising: Determination means for determining whether the object recognized by the recognition means is a person,
Storage means for storing image data of the symbol indicating the state of the person,
The analysis means, of the objects, analyzes the state of the object determined to be a person by the determination means,
The generating means includes a reading means for reading, from the storage means, image data corresponding to the symbol indicating the analysis result of the analyzing means, the symbol indicating the person's condition, based on the analysis result of the analyzing means.
When the image data of the symbol is read by the reading unit, the display unit displays the symbol represented by the image data.
The driving support device according to claim 1, wherein A line-of-sight detection means for detecting the line-of-sight of the driver of the vehicle,
Among the images displayed by the display unit, the image recognized by the driver, which is the detection result of the line-of-sight detection unit, an identification unit for identifying from the moving state of the line of sight of the driver,
Erasing means for erasing the image identified by the driver as recognized by the identifying means,
The driving support device according to claim 1, further comprising: