JP2022175359A - Vehicle information providing device, information providing method, and information providing program - Google Patents

Abstract

To provide a vehicle information providing device, a vehicle information providing method, and a vehicle information providing program capable of appropriately providing notification information output in response to VPA so that a VPA function can be used safely even while driving.SOLUTION: A VPA processing unit 11f makes a VPA operate to generate a response to a vehicle occupant and to output notification information based on the response. A situation determination unit 11b determines a surrounding situation of a vehicle or a situation of a driver. A display processing unit 13 or an audio processing unit 14 changes a form of the notification information according to the response of the VPA processing unit 11f based on a determination result of the situation determination unit 11b.SELECTED DRAWING: Figure 3

Description

The present invention relates to a vehicle information providing device, an information providing method, and an information providing program.

VPA is becoming popular as an HMI for implementing voice recognition in a vehicle. HMI is an abbreviation for Human Interface, and VPA is UI for artificial intelligence and an abbreviation for Virtual Personal Assistant. In recent years, VPA enables fluent voice communication with people using natural language.

In recent VPAs, user interfaces are increasing in which anthropomorphic characters make eye contact with the user and respond to the user's speech. The action of speaking to a character is a natural action for a person. Therefore, using the character as an interface makes it easier for the user to talk to the character.

Patent Literature 1 describes, "Using technology such as line-of-sight detection, it is possible to control whether or not to accept input from other media, recognition processing, or output presentation method, interruption, confirmation, etc. according to the gaze target of the user. especially in anthropomorphic interfaces, applied to simulate the use and role of non-verbal messages in human-to-human communication, e.g. allowing conversation to be initiated by looking at a face. It is stated that As described above, it is also recognized in Patent Document 1 that the action of speaking to a character is a natural action.

JP-A-10-301675

There are various problems when adapting the technology of the background art section to vehicle applications. When the VPA technology is applied to vehicle applications, for example, the driver's line of sight is forward during normal driving, and the visual information from the VPA is momentarily viewed only when necessary. Therefore, the driver cannot confirm the character for a long time.

Further, if the driver confirms the character in a situation where it should not be confirmed, the safety is impaired. In this way, the use of the VPA increases the danger for the driver because it may cause the driver to gaze at the line of sight while driving. Also, even when the VPA is used to notify the driver by voice, if the notification is not made at an appropriate timing, the danger to the driver may increase as described above. It is necessary to pay close attention to maintaining safe driving while establishing communication using VPA.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and its object is to provide a vehicle information system capable of appropriately notifying the VPA response so that the VPA function can be used safely even while driving. An object of the present invention is to provide a providing device, a vehicle information providing method, and a vehicle information providing program.

According to the first aspect of the present invention, the object is an information providing device that provides information on a notification mode by operating a VPA. The situation determination unit determines the surrounding situation of the vehicle or the situation of the driver. The information providing unit changes the notification mode by VPA based on the determination result of the situation determination unit. Therefore, it is possible to appropriately notify the voice information and visual information as a response by the VPA, and the VPA can be used safely even while driving.

External view of the cockpit system in the first embodiment FIG. 4 is a diagram schematically showing a control form of the vehicle information providing device; FIG. 1 is a block diagram schematically illustrating an electrical configuration of a vehicle information providing device; Flowchart outlining the process Explanatory diagram of the notification mode when the character is displayed at a position visible from the driver and at a position visible from the front passenger seat. Explanatory drawing of the notification mode when the character is displayed at a position invisible to the driver and visible from the front passenger seat. Flowchart for schematically explaining processing in the second embodiment Part 1 of Display Mode Example of Visual Information Including Characters Part 2 of Display Mode Example of Visual Information Including Characters Part 3 of display mode example of visual information including characters Part 4 of Display Mode Example of Visual Information Including Characters

Several embodiments of a vehicle information providing device, a vehicle information providing method, and a vehicle information providing program will be described below with reference to the drawings. In addition, the same code|symbol is attached|subjected and demonstrated to the site|part which is substantially common in each embodiment.

(First embodiment)
A first embodiment will be described with reference to FIGS. 1 to 6. FIG. As shown in FIG. 1, a vehicle information providing device 1 is configured by a cockpit system having a plurality of displays such as a pillar-to-pillar display 2, a center display 3, and an electronic mirror 4. FIG. However, the number of display units, the installation form, and the configuration are only examples, and the present invention is not limited to these. The pillar-to-pillar display 2 is hereinafter referred to as the PtoP display 2 .

As shown in FIGS. 1 and 2, the PtoP display 2 is configured such that a plurality of displays 2a are arranged side by side and become horizontally long. Each display 2a of the PtoP display 2 is composed of a liquid crystal display or an organic EL display. As shown in FIG. 1, the PtoP display 2 is a large display provided on the dashboard between the left and right pillars of the vehicle. The PtoP display 2 is configured so that a plurality of image contents can be continuously displayed side by side on a plurality of displays 2a, thereby improving the degree of freedom in arranging the contents. The PtoP display 2 is configured to be able to display various image contents such as meter images, images captured by the peripheral camera 23, entertainment images such as still images and moving images, and map images around the current position in full graphic display.

The meter image is displayed in a specific display 2a among the PtoP displays 2 within the driver's field of vision during normal driving. In the case of automatic driving, this is not the case. In addition, since the PtoP display 2 is configured to be long in the horizontal direction, the display contents can be confirmed not only by the driver and passengers in the front passenger seats but also by passengers in the rear seats.

On the other hand, the center display 3 is composed of, for example, a liquid crystal display or an organic EL display, and is installed between the driver's seat and the passenger's seat and below the PtoP display 2 . The center display 3 is provided in the vicinity of the center console so that both the driver and the passenger in the front passenger seat can easily see it, and can display various contents in full graphic display. An operation panel 21 is provided on the center display 3, and allows selection of contents to be displayed on the PtoP display 2, air conditioning operation, audio operation, and input operation of navigation functions.

The PtoP display 2 is installed side by side in the vertical direction with respect to the center display 3 while being spaced apart. If two screens are installed side by side in the vertical direction, it is possible to increase the display area that can be visually recognized by the occupant at one time. Further, in the cockpit system, the display screen of each display 2a of the PtoP display 2 is installed so as to be positioned further inward than the display screen of the center display 3. As shown in FIG. Here, the depth direction means that the display screen of the PtoP display 2 is located in the depth direction as viewed from the vehicle occupants such as the driver. is located in the traveling direction of the vehicle.

The electronic mirror 4 is configured using a display 4a and a door mirror camera 4b. See FIG. The display 4a of the electronic mirror 4 is composed of, for example, a liquid crystal display or an organic EL display. The displays 4a are installed one by one at the left and right positions of the driver, for example, under the A pillar in the vehicle interior. Each left and right display 4a of the electronic mirror 4 may be configured integrally with a specific display 2a in the PtoP display 2, for example, the displays 2a at both ends. The door mirror camera 4b is a peripheral camera 23 that captures an image of the rear side of the vehicle from the installation location of the door mirror of the vehicle. The display 4a of the electronic mirror 4 is provided at a position where the driver can easily check the rear side of the vehicle, and the display 4a is configured to display peripheral information of the vehicle captured by the door mirror camera 4b.

In addition, as shown in FIGS. 2 and 3 , a large number of ECUs 5 are configured in the vehicle and connected to an in-vehicle network 25 . The ECU 5 includes a display system ECU, a peripheral monitoring system ECU, a travel control system ECU, and a DCM that communicates with the outside of the vehicle. DCM is an abbreviation for Data Communication Module. The travel control system ECU includes a well-known vehicle control ECU, engine control ECU, motor control ECU, brake control ECU, steering control ECU, integrated control ECU, and the like. When the ECU of the travel control system receives an automatic control signal, it configures an automatic driving ECU that executes a predetermined level of corresponding driving support technology and automatic driving. The autonomous driving ECU is an autonomous driving electric control unit.
When an automatic control signal is input, the automatic driving ECU drives a driving actuator to execute a corresponding predetermined level of driving assistance and automatic driving. For example, Level I driving assistance includes automatic braking to avoid collisions with obstacles, follow-up driving that follows the preceding vehicle, or lane-departure prevention driving that controls the vehicle so that it does not stray from the lanes on both sides. can be executed. In level II automated driving, a combination of level I driving assistance, or automated driving under specific conditions, such as automatically overtaking a slow vehicle on an expressway or automatically merging on an expressway. You can run an automatic driving mode that In level II automated driving, the driver is obligated to monitor. In level III or higher automated driving, the system performs all driving tasks while being supervised by the system, but detailed explanations are omitted.

Each ECU 5 is mainly composed of a microcomputer having a processor, various storage units 6 such as a cache memory, a RAM and a ROM, I/O, and a bus connecting them. Each ECU 5 is communicably connected to other ECUs 5 provided in the vehicle through the communication control unit 7 and the in-vehicle network 25 .

In this embodiment, as shown in FIG. 2, a plurality of display system ECUs 5 constitute an HCU as a vehicle information providing apparatus 10, and the processing capacity of the physical resources inside the display system ECUs 5 is shared. Display processing is performed on the displays 2, 3, and 4a. The display system ECUs 5 may be connected to each other via a dedicated line, or may be configured by one ECU 5 . HCU is an abbreviation for Human Machine Interface Control Unit and indicates an information presentation control device. The storage unit 6 described above indicates a non-transitional material storage medium that non-temporarily stores computer-readable programs and data. A non-transitional material storage medium is implemented by a semiconductor memory or the like.

As shown in FIG. 3, the vehicle information providing device 10 includes a control device 11, an arithmetic device 12, a storage unit 6, a display processing unit 13, an audio processing unit 14, and an I/O unit for managing input or output from various devices. Control unit 15, communication control unit 7 for managing communication with other ECU 5, wireless control configured by connecting antenna 16a and enabling wireless connection with other portable terminal 27 by wireless LAN or Bluetooth (registered trademark) a portion 16; The display processing unit 13 and the audio processing unit 14 are configured as an information providing unit. Here, various sensor inputs or switch outputs will be explained through the I/O control unit 15, but inputs and outputs may also be performed from other ECUs 5 such as a perimeter monitoring ECU and a travel control ECU through the in-vehicle network 25. be.

The radio control unit 16 establishes a communication link with a portable terminal 27 carried by a vehicle occupant. The in-vehicle information providing apparatus 10 waits for an incoming call to the mobile terminal 27, and when the mobile terminal 27 receives an incoming call from the other party and the incoming call is answered, the speaker 18 and the microphone 17 are communicated with the other party through the mobile terminal 27. You can make hands-free calls through In addition, the microphone 17 has a directional sound input characteristic or is installed in a plurality near the driver's seat (hereinafter referred to as the D seat) and the front passenger seat (hereinafter referred to as the P seat). The unit 14 can determine from which seat, the D seat or the P seat, the voice is input. The voice processing unit 14 of the vehicle information providing device 10 can recognize voice input through the microphone 17 . Further, the vehicle information providing apparatus 10 can perform voice communication in natural language using artificial intelligence by the VPA processing section 11f.

Based on the control of the control device 11, the arithmetic device 12 displays the contents of images, sentences, characters, comments, or symbols (hereinafter referred to as images, etc.) stored in the storage unit 6 on the PtoP display 2 and the center display 3. A display area to be displayed on the display screen is calculated, and it is calculated in which area of the display screen of the PtoP display 2 or the center display 3 the contents of images and characters are to be displayed, and in which area they are to be superimposed and displayed. , the display area is output to the display processing unit 13 through the control device 11 together with the content such as the image.

The display processing unit 13 displays contents such as images in the aforementioned display areas in the display screens of the PtoP display 2 , the center display 3 and the electronic mirror 4 under the control of the control device 11 . Images and the like can be displayed on the display screens of the displays 2 and 3 for each display layer. Also, under the control of the control device 11 , the voice processing unit 14 receives received voice input from the microphone 17 and outputs transmitted voice from the speaker 18 .

When text or character content is input from the control device 11, the voice processing unit 14 converts it into voice, reads it out through the speaker 18, and outputs it. The voice processing unit 14 also detects whether the driver has spoken or the passenger in the front passenger seat has spoken based on the voice input from the microphone 17 , and outputs this detection signal to the control device 11 .

The position detector 19 detects a position with high accuracy using a well-known GNSS receiver such as GPS (not shown) and an inertial sensor such as an acceleration sensor or a gyro sensor. The position detector 19 outputs a position detection signal to the control device 11 through the I/O control section 15 . The position specifying unit 11a of the control device 11 realizes a function as an ADAS locator that sequentially measures the current position of the vehicle with high accuracy based on the map information input from the map data input device and the position detection signal of the position detector 19. do. ADAS stands for Advanced Driver Assistance Systems. In this case, the vehicle position is represented by a coordinate system consisting of latitude and longitude. In this coordinate system, for example, the X axis indicates longitude and the Y axis indicates latitude. It should be noted that positioning of the vehicle position can be performed, for example, on the basis of mileage information obtained based on the sensing results of a vehicle speed sensor mounted on the vehicle. configuration can be adopted. The control device 11 can perform so-called navigation processing based on the current position of the host vehicle.

The operation panel 21 is a touch panel configured on the center display 3 , and the I/O control unit 15 receives an operation input when there is an operation input by the passenger and outputs the operation input to the control device 11 . The control device 11 executes control based on operation signals from the operation panel 21 .

The occupant monitor 22 detects the state of the occupant in the vehicle or the operation state. The occupant monitor 22 is configured using, for example, a power switch, occupant state monitor, turn switch, automatic control switch, etc., and outputs various signals to the control device 11 . The occupant monitor 22 includes a steering sensor that detects whether the steering wheel is being gripped or steered by the driver, a seating sensor that detects whether the driver is seated, an accelerator pedal or brake pedal depression sensor, and the like. may contain

The power switch is turned on by a user in the vehicle interior to start the internal combustion engine or the electric motor, and outputs a signal corresponding to the operation. The occupant condition monitor includes a camera that detects the condition of the occupant in the D seat or the P seat by capturing the condition of the occupant with an image sensor and outputs an image signal. The driver's occupant condition monitor is called DSM. DSM is an abbreviation for Driver Status Monitor. The occupant condition monitor acquires an imaging signal obtained by irradiating the head of the driver with near-infrared light and capturing an image, analyzes the image as necessary, and outputs the signal to the control device 11 . An occupant condition monitor is used to detect the condition of an occupant such as a driver, especially during driving assistance or automatic driving. The turn switch is turned on by an occupant inside the vehicle when activating the direction indicator of the host vehicle, and outputs a turn signal for turning right or left according to the operation.

The automatic control switch outputs an automatic control signal according to the operation when it is turned on by an occupant inside the vehicle in order to command predetermined driving assistance or automatic control for the running state of the vehicle. By outputting this automatic control signal to the ECU of the travel control system, a corresponding predetermined level of driving assistance or automatic driving control can be executed.

The control device 11 can determine the behavior of the vehicle occupant, for example, in which direction the line of sight is directed, based on the sensor signal of the occupant monitor 22, and can also determine the operating state of the power switch, the operating state of the direction indicator, and the automatic control of the vehicle. command information, sensor information from various sensors, operation information, etc. can be input.

The peripheral camera 23 is a front view camera that images the front of the vehicle, a back camera that images the rear of the vehicle, a corner camera that images the front and rear sides of the vehicle, or a side camera that images the side of the vehicle. , or the door mirror camera 4a of the electronic mirror 4, which are controlled through the I/O control unit 15 as imaging signals of the front guide monitor, back guide monitor, corner view monitor, side guide monitor, and electronic mirror 4, respectively. Output to the device 11 .

A distance detection sensor 24 for detecting the distance to obstacles is installed in the vehicle as a peripheral monitoring sensor. The distance detection sensor 24 is composed of clearance sonar, LiDAR, radar using millimeter waves, or the like, and detects obstacles such as vehicles and people that are close to the front of the vehicle, the front side of the vehicle, the rear side of the vehicle, the rear of the vehicle, or the side of the vehicle. Detect the distance to

Each ECU 5 is equipped with SoC. SoC is an abbreviation for System On Chip. The mounted SoC incorporates the aforementioned microcomputer. The microcomputer incorporated in the SoC of the ECU 5 is configured to operate a variety of applications (hereinafter abbreviated as applications) on a pre-installed general-purpose OS and real-time OS. The providing device 10 can implement various functions by executing applications.

The aforementioned apps include image processing apps and other apps. A processor incorporated in the SoC performs drawing processing on the display screens of the displays 2a of the PtoP display 2, the center display 3, and the display 4a of the electronic mirror 4 in response to a drawing request from the image processing application. The control device 11 of the vehicular information providing device 10 functions as a position specifying portion 11a, a situation determination portion 11b, a line-of-sight detection portion 11c, a VPA processing portion 11f, and a division portion 11g by executing an application.

The position specifying unit 11 a specifies the current position of the vehicle based on the position detection signal from the position detector 19 . The aforementioned apps include navigation apps. The navigation application implements a navigation function and mainly draws a navigation screen including a map image displayed on the PtoP display 2 and the current position of the vehicle specified by the position specifying unit 11a.

The situation determination unit 11b determines the surrounding situation of the vehicle or the situation regarding driving. The situation determination unit 11b determines the situation around the vehicle, such as surrounding events that the driver should pay attention to, based on the image captured by the surrounding camera 23 and the signal of the surrounding monitoring sensor by the distance detection sensor 24 . Specifically, when judging the situation around the vehicle, the situation judgment section 11b preferably judges the position of surrounding vehicles, the degree of danger around the own vehicle, or the road situation.

In addition, when judging the driving situation, the situation judgment unit 11b judges whether the vehicle is being driven, whether the driving load is high, and the emotions of the vehicle occupants. When the situation determination unit 11b determines the emotion of the vehicle occupant, the occupant's biometrics are measured using a well-known biometric technique, and the facial expressions appearing on the surface of the occupant's body. It can determine psychological states such as sleepiness, positive emotions, negative emotions, fatigue, etc., and determine the situation of the occupant based on the acquired information. Also, the situation determination unit 11b may determine the emotion by determining the physical state such as the heart rate. Further, the situation determination unit 11b detects the blood flow in the head by irradiating the head of the occupant with light and measuring the reflectance of the light. The situation determination unit 11b can determine the situation of the occupant based on the detection result of the blood flow in the occupant's head.

Further, the control device 11 may recognize emotions including facial expressions such as laughter, anger, sadness, etc. by recognizing images captured by the situation determination section 11b using the passenger state monitor. When it is difficult to recognize facial expressions and the like on the occupant monitor 22 due to the vibration of the vehicle, emotions may be recognized based on the operating state of driving actuators and signals from various sensors.
For example, the control device 11 stores the operation history of the accelerator or the brake by the passenger in the storage unit 6, and based on the stored learning result, for example, when the operation timing of stepping on the brake is delayed by a predetermined time or more from the standard time, If it is determined, it is preferable to determine the feeling that the occupant is feeling tired.

Similarly, the control device 11 stores the turning operation history of the steering wheel by the occupant in the storage unit 6, and based on the stored learning result, determines that the turning operation speed is faster than the standard speed by a predetermined amount or more. In some cases, it may be determined that the occupant has performed a rough turning operation, and the occupant's feelings such as irritation may be recognized. In this way, the control device 11 learns the driving operation history and detects the tendency of changes in the operation of the driving actuators, so that it can be used as information for recognizing the emotions of the occupants. As a result, the control device 11 can determine the driving situation of the passenger based on the result of recognition of the passenger's emotion in the situation judgment section 11b. The line-of-sight detection unit 11c detects the line-of-sight of any vehicle occupant such as a driver or an occupant in a front passenger seat using the occupant state monitor of the occupant monitor 22 .

The VPA processing unit 11f operates by executing applications using AI technology such as machine learning and deep learning. When the VPA processing unit 11f operates, it recognizes the voice uttered by the vehicle occupant and generates a response based on the content of the voice recognition. The VPA processing unit 11f generates, as notification information, contents of sentences and characters of the utterance voice for outputting the voice as the above-mentioned correspondence, and passes it to the voice processing unit 14, and the voice processing unit 14 makes the speaker 18 notify it by voice. Furthermore, the VPA processing unit 11f generates visual information to be displayed on the displays 2, 3, etc. as the correspondence described above, passes it to the display processing unit 13, and the display processing unit 13 performs visual information based on the correspondence generated by the VPA processing unit 11f. Information is displayed on displays 2 and 3.

The VPA processing unit 11f can also cause the display 2 or 3 to display contents such as a pre-designed specific character A as visual information through the display processing unit 13. FIG. Further, when the visual information generated based on the operation of the VPA processing unit 11f is displayed on the PtoP display 2, the display 2a located in front of the D seat to the display 2a located in front of the P seat is displayed on the PtoP display 2. The arrangement of character A in can be freely changed according to the situation.

In this embodiment, the VPA processing unit 11f displays the character A on the display 2, but in addition to the character A, comments and non-characters displayed beside the character A may be displayed as visual information. . Further, the VPA processing section 11f may cause the displays 2 and 3 to display map information and character information around the current position as visual information through the display processing section 13. FIG. As a result, the VPA processing unit 11f can provide information necessary for the vehicle occupant and realize communication with the vehicle occupant.
Also, the control device 11 can chat with the VPA processor 11f. For example, the control device 11 can provide information such as today's weather and trivia about today by the function of the VPA processing section 11f. The VPA processing unit 11f may ask the occupant a question about his or her mood while chatting, obtain an answer to the question from the occupant, and determine the contents of the answer to recognize the occupant's emotion. In this case as well, the control device 11 can determine the driving situation of the passenger based on the result of recognition of the passenger's emotions by the situation judgment section 11b.

The app also includes a detection app. The detection application detects gestures of the occupants in the vehicle based on the imaging signal from the occupant monitor 22, and detects finger operations and line of sight. The control device 11 can detect the line of sight of the occupant in the vehicle with the line of sight detection unit 11c, and can detect facial expressions and gestures with other detection applications. The detection application implements a function as the line-of-sight detection unit 11c.

Further, when the line of sight of the vehicle occupant is detected by the line of sight detection unit 11c, the VPA processing unit 11f operates based on the line of sight detection result as if the character A were conversing with the vehicle occupant while meeting the line of sight. can be displayed on display 2 or 3 at the same time.

A display area for drawing the image content is allocated to the application that draws the image content among the respective applications. These display areas are secured on the storage unit 6 in such a size that necessary image contents can be drawn.
Also, various contents such as character A can be animated. Here, an animation action means a gradual change in the position or size of an image showing content, rotation of an image, overall movement of the user interface with a swipe operation, or gradual fading of an image. It is a display mode in which the image is displayed in or faded out, or the color of the image is changed.

The control device 11 also functions as a classification section 11g that classifies vehicle occupants to make visual information displayed by the display processing section 13 visible or invisible based on the determination result of the situation determination section 11b. At this time, the display processing unit 13 or the sound processing unit 14 changes the notification mode of the notification information according to the correspondence of the VPA processing unit 11c based on the classification result of the classification unit 11g. Further, the display processing unit 13 has a function as a position setting unit 13a that sets a position visible to a vehicle occupant classified as visible and invisible to a vehicle occupant classified as invisible. At this time, the display processing unit 13 displays the visual information at the position set by the position setting unit 13a among the displays 2, 3, and 4. FIG.

The action and operation of the above configuration will be described. When the power switch of the vehicle is turned on, the occupant monitor 22 accepts this operation, and the vehicle information providing device 10 executes the processing shown in FIG. The control device 11 determines whether or not the vehicle is being driven by the situation determining section 11b based on the information input in S1. Here, it is determined whether or not the vehicle is being driven by the driver.

For example, when it is determined that the vehicle is stopped while the driver is driving based on the various sensor information described above, or when the navigation application is being executed by the control device 11, setting of the navigation destination or current location is received from the operation panel 21. If so, it is determined that the driver is not driving. Further, it is determined that the vehicle is not being driven by the driver during automatic operation of level III or higher, and NO is made in S1.

When the vehicle occupant is not driving, the voice processing unit 14 receives speech from the vehicle occupant through the microphone 17, and the VPA processing unit 11f recognizes and analyzes the voice and generates a corresponding response. The VPA processing unit 11f causes the speaker 18 to output a voice response to the passenger when the analysis is finished and the response is to be performed. Further, the VPA processing unit 11f displays visual information including the character A on the display 2 or 3 while changing the display mode of the character A. FIG. At this time, the display position is set and displayed on a part of the display 2a that is easily visible to the driver sitting in the D seat.

That is, in a scene where safety is ensured, for example, when the vehicle is stopped, the display processing unit 13 displays the character A at a position where it is easy for the vehicle occupants including the driver sitting in the D seat and the passenger in the front passenger seat to see the character A, for example, the center of the PtoP display 2. is set and displayed on the display 2a. The easy-to-see position is preferably a position within the central visual field or the peripheral visual field when the driver looks at the central display 2 a of the display 2 . It may be displayed at a predetermined position that is easily visible to all vehicle occupants sitting in the rear seats, as well as the driver and the occupants in the front passenger seat.

Thus, when the response by the VPA processing unit 11f is uttered from the speaker 18 and the character A is displayed, the situation and reaction of the driver when the driver sees the character A can be determined by the situation determination unit 11b. A line of sight can be detected using the line of sight detection unit 11c. In particular, the reaction of the driver can be checked in connection with the changing mode of the character A. Furthermore, the occupant monitor 22 can be used to similarly check the situation and reactions of other vehicle occupants. Furthermore, it is desirable for the control device 11 to change the mode of the notification information after the situation determination section 11b confirms the driver's situation and reaction according to the response of the VPA processing section 11f. Moreover, it is preferable that the control device 11 changes the form of the notification information based on the result of detecting the driver's line of sight by the line-of-sight detection unit 11c in response to the VPA processing unit 11f. For example, it is desirable that the control device 11 changes the display mode of the audio information notified by the audio processing unit 14, the visual information notified by the display processing unit 13, and the display position of the character A.

Conversely, when the control device 11 determines that the vehicle is driving in S1, the display processing unit 13 preferably displays the character A at a position not visible from the D seat in S3. For example, as shown in FIG. 6, the display processing unit 13 may display the character A in a predetermined area 2b of the display 2 that is hidden from the driver's line of sight by the steering wheel. Further, the display processing unit 13 may hide the character A so that it cannot be seen by all passengers in the vehicle.

In other words, while the driver is driving, the display processing unit 13 should display the character A at a position that is difficult to see or invisible from the driver sitting in the D seat so as not to provide visual information to the driver. This can prevent inattentive driving.

Further, the display processing unit 13 may display the character A at a position visible to the passenger in the P seat, or may display the character A at a position invisible to the passenger. If the character A is displayed at a position that can be visually recognized by the occupant of the P seat, the occupant of the P seat can enjoy the response by the VPA processing unit 11f while confirming the character A. Also, when the passenger in the P seat sees the character A, the reaction and line of sight of the passenger can be detected.

Also, the display processing unit 13 may display the character A on none of the displays 2, 3, or the like. In particular, during driving, it is preferable to move or erase the character A to a position invisible to the driver.

Also, when confirming the utterance through the microphone 17 while driving, the control device 11 determines in S4 of FIG. 4 whether or not the utterer is the driver. At this time, the control device 11 may change the contents of the display processing by the display processing section 13 based on the detection signal of the audio processing section 14 that detects whether the speech is made by the driver or by the occupant in the front passenger seat. In the case of the driver's utterance, the response by the VPA processing unit 11f should be only voice, and visual information including the character A should not be displayed in S5.

Further, when the voice processing unit 14 detects that an occupant in the front passenger seat has spoken, the voice processing unit 14 determines NO in S4 and responds with visual information. At this time, the display processing unit 13 preferably displays visual information including the character A at a position that is invisible or difficult to see from the driver. The display processing unit 13 preferably displays the visual information including the character A on the display 2a at a position where the passenger in the front passenger seat can easily see, for example, the display 2a just before the P seat.

In such a case, it is considered that only the passenger in the front passenger seat is interested in the response of the VPA processing section 11f. Therefore, if the display processing unit 13 determines that the character A is not attractive to the driver, the display processing unit 13 may display the character A at a position visible to the driver. Further, after the display processing unit 13 displays the character A on the displays 2 and 3, the situation determination unit 11b determines the driver's situation, or the line-of-sight detection unit 11c detects the line of sight of the driver or the passenger in the P seat. By doing so, the response to the response of the VPA processing unit 11f is confirmed, and the control device 11 may change the mode of the notification information, for example, the display mode of the audio information, the visual information, and the display position of the character A. .

This embodiment is conceptually put together. According to the present embodiment, based on the determination result of the driver's situation determined by the situation determination unit 11b, the manner of notification information according to the response of the VPA processing unit 11f, for example, the display manner of voice information, visual information, character A I'm trying to change the display position. Also, based on the detection result of the line-of-sight detection unit 11c, the response to the response of the VPA processing unit 11f is confirmed, and the mode of notification information, such as the display mode of voice information, visual information, and the display position of character A, is changed. ing. As a result, it is possible to use notification information based on the correspondence of the VPA processing unit 11f, and to safely utilize the functions of the VPA processing unit 11f even during driving. In addition, natural conversation can be established.

(Second embodiment)
A second embodiment will be described with reference to FIGS. 7 to 9. FIG. A part different from the above-mentioned embodiment is explained. As shown in FIG. 7, when the power switch of the vehicle is turned on, the occupant monitor 22 accepts this operation, and the vehicle information providing device 10 executes the processing shown in FIG. The control device 11 determines whether or not the vehicle is in operation at S11.

Using the same conditions as in the first embodiment, if the situation determination unit 11b determines that the vehicle is not driving, it responds with visual information in S12. At this time, as in the first embodiment, the display processing unit 13 displays on a part of the display 2a that is easily visible to the driver sitting in the D seat. That is, in a scene where safety is ensured, for example, while the vehicle is stopped, the display processing unit 13 displays the character A at a position where the vehicle occupants including the driver sitting in the D seat can easily see the character A, for example, the central display 2a of the PtoP display 2. to display.

Conversely, when the control device 11 determines that the vehicle is being driven in S11, the situation determination unit 11b determines whether or not the degree of driving load is higher than a predetermined level during operation. The control device 11 receives the results detected by the distance detection sensor 24 and the peripheral monitoring sensor of the peripheral camera 23, and determines that the degree of driving load is high by determining that there are more obstacles in the surroundings than a predetermined number by the situation determination unit 11b. do. In addition, the control device 11 determines that the degree of driving load is low when it is determined by the situation determination unit 11b that there are few vehicles in the vicinity while traveling on the highway, and conversely determines that the degree of driving load is high when it is determined that there are many. do. Further, the control device 11 determines that the degree of driving load is high when the peripheral camera 23 or the distance detection sensor 24 detects that another person is walking next to the vehicle in a narrow alley or the like.

In addition, the degree of driving load changes according to the level of automatic driving. In general, the higher the level of automated driving, the lower the driving load. In addition, even during automatic driving, the positional relationship between the own vehicle and the other vehicle changes greatly when the own vehicle and the other vehicle are changing lanes. , and, if necessary, during a predetermined period before and after that, it is determined that the degree of driving load is high. On the contrary, using a certain level of driving support technology or automatic driving technology, for example, when following driving with ACC, or when driving steadily on an open highway with lane keeping assist (LAS), relatively stable and safe It is determined that the degree of operating load is low during steady operation.

When the display processing unit 13 determines in S13 that the degree of driving load is high, the display processing unit 13 displays visual information including the character A in a position that is not visible or difficult to see from the D seat in S14, and displays the character A according to the correspondence of the VPA processing unit 11f. It is good to change the aspect of Further, the display processing unit 13 may hide the visual information including the character A so that it cannot be seen by all passengers in the vehicle.

Further, when the display processing unit 13 determines that the degree of driving load is low in S13, the display processing unit 13 displays visual information including the character A at a position easily visible to the driver in S15.

As a result, during relatively stable and safe driving, such as during follow-up driving by ACC, or during steady driving on an open highway by lane keeping assist (LAS), the driver and character A can You can communicate while exchanging eye contact.

Further, when the line of sight of the driver confirming the character A is detected by the line of sight detection unit 11c, the display processing unit 13 detects the captured image of the forward vision camera and the image of the peripheral monitoring sensor in the central visual field or the peripheral visual field centered on this line of sight. Information useful for confirming the safety of the surrounding area may be displayed in parallel. This makes it possible to achieve both driving and communication. It is also possible to devise ways to prevent the driver from drawing too much attention while driving.

Further, the control device 11 determines whether or not the driver is gazing at the character A in S16. If the line-of-sight detection unit 11c determines that the driver is not gazing, the controller 11 responds with visual information in S17. As a result, the visual information including the character A is continuously displayed. However, it is desirable to use the character A to display so as to encourage safe behavior.

Conversely, the control device 11 uses the line-of-sight detection unit 11c to detect that the driver is staring, that is, the line-of-sight detection unit 11c detects that the driver has begun to look, and waits for a predetermined period of time. If it is detected that the character A is still watching, the visual information including the character A is nullified in S18.

Further, the control device 11 waits for a predetermined period of time after detecting that the line-of-sight detection unit 11c has started to look at the driver. If the visual information cannot be displayed as a result, it is preferable to display information regarding driving to the vehicle occupants, for example, information that encourages the driver to take safe driving behavior at the display position of the character A or its peripheral position. Then, the driver can be urged to drive safely so as not to deviate too much from the driving behavior.

For example, the driver may check the information of the next-generation in-vehicle information communication system. The next-generation in-vehicle information communication system is called an In-Vehicle Infotainment system or an IVI system. The upper diagram of FIG. 8 shows the image content of the front imaging camera of the surrounding camera 23 displayed on one display 2a of the PtoP display 2 as the surrounding information of the current position, and the map image including the destination mark B on the other display 2a. is displayed as an example.

When the line-of-sight detection unit 11c detects that the driver is looking at the destination mark B, the display processing unit 13 displays visual information including the character A in the vicinity of the destination mark B on the other display 2a in the driver's field of view. The character A is displayed to look at the destination mark B. The control device 11 may detect the driver's reaction to the display of the character A by means of the line-of-sight detection section 11c, and provide supplementary information, such as displaying a comment, based on the detection result.

Further, when the red light 100b is recognized by the forward monitoring camera when the vehicle stops temporarily due to a red light 100b or a traffic jam, the display processing unit 13 flickers the red light 100b included in the recognition image together with the recognition image. It is preferable to change the display mode of the character A so as to attract the driver's line of sight to the red light 100b.

The display processing unit 13 changes the character A to make a gesture of worrying about the front of the vehicle, and displays the character A near the area where the captured image of the front view camera is displayed, thereby directing the driver's line of sight to the front. can be induced to

When the line-of-sight detection unit 11c detects that the driver is temporarily confirming or glancing at the character A, the display processing unit 13 is arranged in parallel with the character A to encourage safe driving behavior. It is good to display the information of

For example, as shown in FIG. 9, when the line-of-sight detection unit 11c detects the line-of-sight and determines that the character A is being gazed at, the control device 11 determines YES in S16 of FIG. The character A is displayed ahead of the line of sight, and the number of seconds the line of sight is away from the front is counted and displayed. This count is preferably displayed in the visual field NV beyond the line of sight, such as the central visual field or the peripheral visual field. As a result, it is possible to prevent the driver from staring at the visual information, and to alert the driver to take safe actions.

Other cases that are likely to occur in vehicles will be described below.
<First example>
When the driver is driving alone, when the control device 11 determines from the occupant monitor 22 that the driver is driving alone in the vehicle interior, the VPA processing unit 11f responds only with sound, and the display processing unit 13 controls the display 2, Character A to 3 is hidden. After that, when the driver utters "○○, lower the temperature" while checking the road ahead, the content of this utterance is input through the microphone 17, and the controller 11 controls the air conditioner (not shown) in response to the VPA processing unit 11f. Activate the device and change the set temperature. After that, as a response of the VPA processing unit 11f, a reply of "Yes, the set temperature has been set to 22 degrees" is made only by voice. Since the character A is not displayed, the driver can drive safely without looking aside for conversation.

<Second case>
While the driver is driving with another person on the front passenger seat, the control device 11 determines from the passenger monitor 22 that at least two passengers, the driver and the front passenger seat, are in the vehicle. As shown in FIG. 6, the display processing unit 13 displays the character A in a position invisible to the driver by displaying the character A on a straight line from the line of sight of the driver through the steering wheel. At this time, the display processing unit 13 performs display processing so that the character A is installed at a position visible to the passenger on the front passenger seat.

When the driver utters "○○, lower the temperature" while checking the road ahead, the voice is recognized, and after operating the air conditioner device according to the response of the VPA processing unit 11f, "Yes, the set temperature is 22 degrees." I made it.” Since the character A cannot be confirmed from the driver, it can be executed safely without looking aside. Also, when the passenger in the P seat inputs "○○, lower the temperature" from the speech microphone 17, the control device 11 detects that the passenger in the P seat has spoken, and the display processing unit 13 performs the following operations as shown in FIG. , character A is displayed in front of the P-seat occupant, and the display mode is changed so that the direction of the P-seat occupant is seen. Then, the control device 11 responds, "Yes, the set temperature is set to 22 degrees." At the same time, the display processing unit 13 displays the set temperature "22 degrees" next to the character A. The passenger in the P seat can make eye contact with the character A and have a conversation including visual information, which facilitates communication.

<Third application example>
While the driver is driving with another passenger on the P seat, the control device 11 determines from the occupant monitor 22 that at least two people, the driver and the passenger on the P seat, are in the vehicle. At this time, the display processing unit 13 hides the character A on the displays 2 and 3 . When the driver utters "○○, lower the temperature", voice recognition is performed and the air conditioner adjusts the set temperature. After that, the voice processing unit 14 causes the speaker 18 to utter the processing result of the VPA processing unit 11f, "Yes, the set temperature has been set to 22 degrees."

Since the display processing unit 13 does not display the character A on the displays 2 and 3, the vehicle occupant does not look aside for conversation. When the passenger in the P seat utters "○○, lower the temperature", the voice processing unit 14 inputs voice. In response to the VPA processing unit 11f, the controller 11 responds from the speaker 18 with "Yes, the set temperature has been set to 22 degrees." At the same time, as shown in FIG. 11, the display processing unit 13 displays the set temperature "22 degrees" in the area in front of the passenger seated in the P seat on the display 2 . The P-seat occupant can easily understand the result of the response because it can be confirmed visually. Since this is a response to an occupant in the P seat, it is unlikely that the driver will look aside even if the position is visible to the driver.

<Fourth application example>
The display processor 13 displays the character A on the display 2 or 3 while the vehicle is stopped. When an arbitrary vehicle occupant utters "○○, lower the temperature", the VPA processing unit 11f responds to adjust the set temperature of the air conditioner. After that, the display processing unit 13 causes the character A to be displayed so as to look in the direction in which an arbitrary vehicle occupant is positioned.

In addition, the VPA processing unit 11f replies "Yes, the set temperature is set to 22 degrees", and the display processing unit 13 is positioned next to the character A display area to display the set temperature. Since the vehicle occupant who speaks can make eye contact with the character A, they can enjoy the conversation.

According to this embodiment, the same effects as those of the above-described embodiment can be obtained. Also, based on the judgment result of judging the surrounding situation of the vehicle by the situation judging section 11b, the form of the notification information, for example, the display form of the voice information, the visual information, and the display position of the character A are changed by the correspondence of the VPA processing section 11f. I am trying to As a result, it is possible to use notification information based on the correspondence of the VPA processing unit 11f, and to safely utilize the functions of the VPA processing unit 11f even during driving. In addition, natural conversation can be established.

(Other embodiments)
The present invention is not limited to the above-described embodiments, but can be implemented in various modifications, and can be applied to various embodiments without departing from the scope of the invention.

In the above-described embodiment, the vehicle occupant speaks to the VPA to cause some action on the side of the vehicle occupant, and after taking this action, the VPA processing section 11f generates a response and responds. is not limited to For example, the VPA processing unit 11f may spontaneously take action on the vehicle occupants.

In this case, the notification information based on the response of the VPA processing section 11f is, for example, the driving condition of the driver, or the entertainment information to the passenger in the P seat. At this time, it is desirable to send notification information based on the correspondence of the VPA processing section 11f to specific vehicle occupants.

In the first embodiment, if the speaker is a driver, YES is determined in S4 of FIG. 4, and the VPA processing unit 11f responds without visual information. However, it is not limited to this. do not have. Regardless of whether the speaker is the driver or not, the visual information such as the character A may be displayed as it is on the display 2a to respond. Also, the character A based on the action of the VPA processing section 11f may be always displayed on the display 2a or the like. In this case, as described in the first embodiment, it is desirable to display visual information such as the character A in the predetermined area 2b of the display 2 that is hidden from the driver's line of sight by the steering wheel. Then, even if the driver speaks while driving, the character A cannot be confirmed by the driver, so inattentive driving can be prevented.

The controller 11, display processor 13, and audio processor 14 approaches described in this disclosure constitute a processor and memory programmed to perform one or more functions embodied by a computer program. It may be realized by a dedicated computer provided by Alternatively, the control device 11, the display processing unit 13, the audio processing unit 14, and the techniques described in the present disclosure are implemented by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Also good. Alternatively, the control device 11, display processing unit 13, audio processing unit 14, and techniques described in the present disclosure may be implemented using a processor and memory programmed to perform one or more functions, and one or more pieces of hardware. It may also be implemented by one or more dedicated computers configured in combination with a processor configured by logic circuits. The computer program may also be stored as computer-executable instructions on a computer-readable non-transitional tangible storage medium.

Although the present invention has been described with reference to the embodiments described above, it is understood that the invention is not limited to such embodiments or constructions. The present invention includes various modifications and modifications within the equivalent range. In addition, various combinations and configurations, as well as other combinations and configurations including one, more, or less elements thereof, are within the scope and spirit of the invention.

In the drawing, 11b is a situation determination unit, 11c is a line-of-sight detection unit, 11f is a VPA processing unit, 11g is a classification unit, 13 is a display processing unit (information providing unit), 13a is a position setting unit, 14 is an audio processing unit (information provider).

Claims (16)

a VPA processing unit (11f) that operates a VPA (Virtual Personal Assistant) to generate responses to vehicle occupants and outputs notification information based on the responses;
A situation determination unit (11b) that determines the surrounding situation of the vehicle or the situation related to driving by the vehicle occupant;
information providing units (13, 14) for changing the form of the notification information according to the response of the VPA processing unit based on the determination result of the situation determination unit;
A vehicle information providing device. 2. The vehicular information providing apparatus according to claim 1, wherein when judging the surrounding situation of the vehicle, the situation judging section judges the position of surrounding vehicles, the degree of danger of the surroundings, or road conditions. 2. When judging the driving situation, the situation judgment unit judges whether the vehicle is being driven, judges whether the driving load is high, or judges the emotions of the vehicle occupants. The vehicle information providing device described. A line-of-sight detection unit (11c) for detecting the line-of-sight of the vehicle occupant,
4. The vehicle according to any one of claims 1 to 3, wherein the information providing unit confirms the response of the VPA processing unit to the response based on the detection result of the line-of-sight detection unit and changes the mode of the notification information. information provider. 5. The vehicle according to any one of claims 1 to 4, wherein the information providing unit includes a display processing unit (13) that performs display processing of visual information based on the correspondence of the VPA processing unit as a form of the notification information. Information provider. a classification unit (11g) for classifying the vehicle occupant to make the visual information displayed by the display processing unit visible or invisible based on the determination result of the situation determination unit;
6. The vehicular information providing device according to claim 5, wherein the information providing unit changes the reporting mode of the reporting information according to the correspondence of the VPA processing unit based on the classification result of the dividing unit. A position setting unit (13a) for setting the position visible to the vehicle occupant classified as visible and invisible to the vehicle occupant classified as invisible,
7. The vehicle information providing device according to claim 6, wherein the display processing section displays the visual information at the position set by the position setting section. 8. The vehicle information providing device according to any one of claims 1 to 7, wherein the situation determination unit determines a situation regarding driving by the vehicle occupant based on a recognition result of recognizing the emotion of the vehicle occupant. 6. The display processing unit according to claim 5, wherein the display processing unit changes and displays the visual information to a position where the visual information can be visually recognized by both the driver as the vehicle occupant and the occupant in the front passenger seat based on the determination result of the situation determination unit. Vehicle information providing device. The display processing unit changes the position to a position where the visual information is not visible to the driver as the vehicle occupant and to a position where the visual information is visible to the passenger in the front passenger seat, based on the determination result of the situation determination unit. 6. The information providing device for a vehicle according to claim 5, wherein the information is displayed as follows. 6. The vehicle information providing device according to claim 5, wherein the display processing section renders the visual information invisible to all vehicle occupants according to the determination result of the situation determination section. A line-of-sight detection unit (11c) for detecting the line-of-sight of the vehicle occupant,
When the display processing unit displays the visual information at a position visible to the vehicle occupant according to the determination result of the situation determination unit, the vehicle occupant starts to see the visual information by the line-of-sight detection unit. 6. If the visual information cannot be displayed due to a change in the situation regarding the driving before waiting for a predetermined period of time after detecting that, the information regarding the driving is provided. vehicle information providing device. 13. The vehicle according to any one of claims 5, 6, 10 to 12, wherein the display processing unit displays the visual information based on the correspondence of the VPA processing unit in the vicinity of the displayed peripheral information of the current position. Information provider. 14. The vehicle information providing device according to any one of claims 1 to 13, wherein the information providing unit provides the driver with information prompting safe driving behavior. a process of operating a VPA (Virtual Personal Assistant) by a VPA processing unit (11f) to generate a response to vehicle occupants and output notification information based on the response;
a process of judging a surrounding situation of the vehicle or a situation related to driving of the vehicle occupant by a situation judging section (11b);
a process of changing the form of the notification information according to the response of the VPA processing unit by the information providing unit (13, 14) based on the determination result of the situation determination unit;
A vehicle information providing method comprising: In the vehicle information provision device,
a procedure for operating a VPA (Virtual Personal Assistant) by a VPA processing unit (11f) to generate a response to vehicle occupants and output notification information based on the response;
a procedure for judging a surrounding situation of the vehicle or a situation related to driving of the vehicle occupant by a situation judging section (11b);
a procedure for changing the aspect of the notification information according to the response of the VPA processing unit by the information providing unit (13, 14) based on the determination result of the situation determination unit;
Information provision program for vehicles to run.

Images