JP7264804B2 - Recommendation system, recommendation method and program - Google Patents

Description

The present invention relates to a recommendation system, a recommendation method, and a program for an automatic driving vehicle.

Conventionally, a recommendation system has been devised that provides "recommended information" that may be of interest to the user.
For example, Patent Literature 1 describes recommendations based on chat mail history.

Specifically, Patent Document 1 discloses a labor management system that notifies a target person of proposal information that encourages him to take paid leave. is presumed as an employee who encourages the target to take paid leave, and by notifying this employee that the target has been notified of the proposal information, a labor management system that makes it easier for the target to take paid leave technology is described.

In recent years, driving support technology called automatic driving has been earnestly proposed in order to reduce the burden on drivers and realize comfortable vehicle operation.
In international standards for automated driving, multiple levels are hierarchically set according to the weight and presence of tasks required by the driver and the degree of automation. In addition, even within a certain level, divisions (categories) are set hierarchically over multiple levels from the viewpoint that a group of requirements for realizing automation can be realized side by side.
For example, in level 4 (highly automated driving) and level 5 (fully automated driving) automated driving, the driver does not need to override the driving operation.

JP 2019-86808 A

As described above, during automatic driving at level 4 or higher, the driver does not need to be involved in driving operations. This allows the driver to take advantage of the vehicle's infotainment system as well as other occupants.

An object of the present invention is to enable execution of infotainment such as an AR (Augmented Reality) tour and to provide a recommendation system for hot spot information and the like.

In order to solve the above-mentioned problems, the recommendation system of the present invention includes: a voice acquisition unit that acquires a conversational voice between a vehicle occupant and another person outside the vehicle; a predetermined word from the conversational voice; A language recognition unit that recognizes the meaning, an emotion estimation unit that estimates the emotions of the occupant from the intonation of the spoken conversation or the meaning of the extracted words, and the words recognized by the language recognition unit are estimated by the emotion estimation unit. a recommended information selection unit that evaluates according to the feelings of the occupant and obtains recommended information for the occupant based on the evaluation of the words; a recommended information display unit that displays the recommended information obtained by the recommended information selection unit to the occupant; and an evaluation table for storing a score of the emotions of the occupant estimated by the emotion estimation unit for each stream information and a sum of the scores for words recognized by the language recognition unit, and the recommended information selection unit only the words recognized from the conversation voice of the other person to the occupant by the language recognition unit, and only the emotion of the occupant estimated by the emotion estimation unit based on the conversation voice of the occupant to the other person. Based on the evaluation of the word, recommended information for the crew is obtained, and when the recommended information is evaluated based on a predetermined number of stream information, the emotion of the crew for each stream information in the evaluation table is calculated. I deleted the score .

According to the recommendation system of the present invention, the driver and passengers can obtain useful information and enjoy their travel time even in a vehicle that is automatically driven at level 4 or higher.

1 is a system configuration diagram showing an example of a recommendation system according to an embodiment; FIG. It is a figure showing an example of the compartment front part structure of an automatic driving vehicle. FIG. 4 is a sequence diagram illustrating an outline of operation of the recommendation system; FIG. 10 is a flow chart explaining selection operation of a chat member of an automatic driving vehicle; FIG. 4 is a flow diagram illustrating chat control operation of an automatic driving vehicle; FIG. 10 is a flow diagram for explaining in detail a recommended information generating process in the recommendation server 2; FIG. 10 is a diagram illustrating the configuration of a recommendation information evaluation table; FIG. 10 is a flow diagram illustrating display of progress information on a recommended information display unit; FIG. 10 is a flow diagram illustrating display of recommended information on a recommended information display unit; FIG. 10 is a diagram showing a configuration of display information corresponding to words of recommended information; FIG. 10 is a diagram showing an example of display of progress information in evaluation of recommended information; It is a figure which shows an example of a display of recommended information.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
The recommendation system of the embodiment includes an automatic driving vehicle 1, a member terminal 3 owned by a plurality of acquaintances/friends of the crew member on board the automatic driving vehicle 1 (other than the crew member), and an automatic driving A vehicle 1 and a member terminal 3 are composed of a recommendation server 2 that is connected by a communication line.

The recommendation server 2 relays voice chat or video chat between the occupant of the automatic driving vehicle 1 and acquaintances and friends of the occupant, and evaluates the preferences of the occupant of the automatic driving vehicle 1 based on the content of the chat, Recommended information based on the evaluation is notified to the automatic driving vehicle 1 .
Then, the automatically driven vehicle 1 displays an image corresponding to the notified recommended information on the display of the vehicle.

FIG. 1 is a system configuration diagram showing an example of a recommendation system according to an embodiment.
The autonomous driving vehicle 1 autonomously controls the steering/driving unit 172 of the vehicle based on the detection results of the sensor unit 171 such as a camera, radar, and lidar that detects the distribution of targets in the external world (surroundings) of the vehicle, thereby driving the vehicle. An operation control unit 17 for automatic operation is provided.

The display unit 10 corresponds to the navigation image generated by the navigation unit 13 that guides the route of the automatic driving vehicle 1 based on the map information 131 and the recommended information notified from the recommendation server 2 generated by the recommended information display unit 11. to display the image you want. A detailed screen configuration of the display unit 10 will be described later.

The chat information acquisition unit 15 acquires chat information ( connection information, etc.).
Then, the chat operation unit 18 connects to the member terminal 3 based on the acquired chat information, and instructs the chat control unit 26 of the recommendation server 2, which will be described later, to start chatting with the crew member and acquaintances/friends of the crew member.

The TCU 12 (Telematics Control Unit) is a two-way radio communication unit for mobile communication, road-to-vehicle communication, and the like.
A chat voice between a crew member and an acquaintance/friend of the crew member is input/output by the voice input/output unit 14 , and stream information of the voice is communicated with the member terminal 3 via the TCU 12 by the chat operation unit 18 . The image of the video chat is captured by the in-vehicle camera 1711 and communicated with the member terminal 3 via the TCU 12 by the chat operation unit 18 , and the video stream from the member terminal 3 is displayed on the display unit 10 .
Also, the recommended information from the recommendation server 2 is received by the chat operation unit 18 via the TCU 12 , and the corresponding image is generated by the recommended information display unit 11 and displayed on the display unit 10 .

Control units such as the operation control unit 17, the chat information acquisition unit 15, the chat operation unit 18, and the recommended information display unit 11 are realized by one or more processors or hardware having equivalent functions, and are implemented by a CPU (Central Processing Unit) or the like. It may be a configuration in which an ECU (Electronic Control Unit), an MPU (Micro-Processing Unit), or the like, in which a processor, a storage device, and a communication interface are connected via an internal bus, are combined.
Each function of the control unit functions by executing a program of the CPU.

Next, the configuration of the recommendation server 2 will be described.
The recommendation server 2 is an information processing device in which a CPU, a storage device, and a communication interface are connected by an internal bus, and each function of the control unit described below functions by executing a program of the CPU.

The recommendation server 2 relays chat communication between the automated driving vehicle 1 and at least one member terminal 3, evaluates the preferences of the occupants of the automated driving vehicle 1 based on the content of the chat, and makes recommendations based on the evaluation. Information is notified to the automatic driving vehicle 1 .

The communication line relay unit 21 performs two-way communication with the automatic driving vehicle 1 by two-way radio such as mobile communication or road-to-vehicle communication, and two-way communication with the member terminal 3 via mobile communication, road-to-vehicle communication, or the Internet. Communicate and relay chat streams.
The voice acquisition unit 22 is a control unit that acquires, from the communication line relay unit 21, stream information of conversation voices between a vehicle occupant and an acquaintance of the vehicle occupant outside the vehicle.
The language recognition unit 23 is a control unit that extracts predetermined words from the stream information acquired by the voice acquisition unit 22 and recognizes the meaning of the extracted words.

The emotion estimating unit 24 is a control unit that estimates the emotion of the occupant from the intonation of the occupant's conversation voice with an acquaintance or the meaning of the extracted words.
The recommended information selection unit 25 is a control unit that evaluates the words recognized by the language recognition unit 23 according to the emotion estimated by the emotion estimation unit 24 and obtains recommended information for the passenger based on the evaluation.
The candidate word storage unit 27 is a storage unit that stores the words extracted by the language recognition unit 23 .

The chat control unit 26 obtains connection information (chat information) related to chatting with acquaintances/friends of the crew member requested based on the identification information of the crew member from the chat information acquisition unit 15 instructed via the communication line relay unit 21. It is a control unit that acquires connection information of acquaintances/friends from an SNS (Social Networking Service) server 4 based on an acquisition request.
Further, the chat control unit 26 notifies the SNS server 4 of the start of chat communication with the designated member terminal 3 based on the instruction from the chat information acquisition unit 15 .
Details of the recommendation server 2 will be described later.

The SNS server 4 includes a chat function unit 41 that performs two-way message communication including voice between a plurality of terminals in real time, and includes connection information of members exchanging message communication as a member list 42 .
In this embodiment, the case where the recommendation server 2 is configured separately from the SNS server 4 will be described, but the recommendation server 2 may be one function of the SNS server 4 .

FIG. 2 is a diagram showing an example of the vehicle interior front structure of the automatic driving vehicle 1. As shown in FIG.
As shown in FIG. 2, the display unit 10 of the automatic driving vehicle 1 is provided in the vehicle width direction (Y direction ) includes a horizontally long information panel 51, a right panel 55R provided on the driver's seat side in the vehicle width direction, and a left panel 55L provided on the passenger seat side in the vehicle width direction.

The meter panel 54 displays, for example, a speedometer, a tachometer, an odometer, shift position information, lighting status information of lights, and the like.
The information panel 51 displays, for example, map information around the automatically driven vehicle 1, current position information of the automatically driven vehicle 1 on the map, and traffic information (including signal information) regarding the current driving route/scheduled route of the automatically driven vehicle 1. , traffic participant information about traffic participants (including pedestrians, bicycles, motorcycles, other vehicles, etc.) around the automated driving vehicle 1, and various information such as messages sent to traffic participants. Also, on the information panel 51, an image corresponding to the recommended information notified from the recommendation server 2 generated by the recommended information display unit 11 is displayed.

On the right panel 55R, image information of the rear and lower sides of the vehicle on the right side captured by a camera (not shown) provided on the right side of the automatic driving vehicle 1 is displayed.
On the left panel 55L, image information of the rear and lower sides of the left side of the vehicle captured by a camera (not shown) provided on the left side of the automatic driving vehicle 1 is displayed.

Next, the outline of the operation of the recommendation system of the embodiment will be described with reference to the sequence diagram of FIG.
FIG. 3 shows the progress of message communication among the automatic driving vehicle 1, the recommendation server 2, and the member terminal 3. As shown in FIG.

In step S31, the chat control unit 26 of the recommendation server 2 is notified of the identification information (identifier) of the passenger (driver) identified by the passenger identification unit 16 from the automatic driving vehicle 1 as a member list request.
The chat control unit 26 of the recommendation server 2 notifies the SNS server 4 of the notified identification information (identifier), and selects members of the chat channel stored in the member list 42 registered in correspondence with the notified identifier. Names and connection information of member terminals 3 are acquired as a member list.

In step S32, the chat control unit 26 notifies the automatic driving vehicle 1 of the member list acquired from the SNS server 4 as a response to step S31.

The chat information acquisition unit 15 of the automatic driving vehicle 1 displays the acquired member list on the information panel 51 (display unit 1) and prompts the passenger (driver) to select a member to chat with.
Then, in step S33, the chat operation unit 18 notifies the chat control unit 26 of an instruction to connect with the selected member.

In step S34, the chat control unit 26 instructs the SNS server 4 to establish a chat connection with the member terminal 3 of the selected member, and the SNS server 4 notifies the member terminal 3 of a chat request.

In step S<b>35 , the chat control unit 26 receives a chat connection acceptance response of the member terminal 3 from the SNS server 4 . The member terminal 3 that has made an acceptance response waits for a chat with the automatic driving vehicle 1 .
If the member terminal 3 that has received the chat request in step S34 cannot connect to the chat, it responds to the chat control unit 26 that it cannot accept the request in step S35.

In step S36, the chat control unit 26 notifies the automatic driving vehicle 1 of a connection response as a response corresponding to the connection instruction in step S33.
The automatic driving vehicle 1 starts chatting with the member terminal 3 upon notification of the connection response from the chat control unit 26 .

The communication line relay unit 21 of the recommendation server 2 receives the voice stream 1 of the in-vehicle voice acquired by the voice input/output unit 14 of the automatic driving vehicle 1 in step S37, and sends it to the member terminal 3 as voice stream 1 in step S38. Forward. At this time, the communication line relay unit 21 relays the audio stream 1 in steps S37 and S38 with low delay.
Also, the communication line relay unit 21 acquires stream information in order to generate recommended information from the stream information of the audio stream 1 .

In step S310, the communication line relay unit 21 receives the audio stream 2 from the member terminal 3, and transfers it as the audio stream 2 to the automatic driving vehicle 1 in step S39. The automatically driven vehicle 1 that has received the audio stream 2 outputs the audio stream 2 to the interior of the vehicle through the audio input/output unit 14 . At this time, the communication line relay unit 21 relays the audio stream 2 in steps S310 and S39 with low delay.
Also, the communication line relay unit 21 acquires stream information in order to generate recommended information from the stream information of the audio stream 2 .

As described above, the communication line relay unit 21 of the recommendation server 2 relays an audio stream from the automated driving vehicle 1 to the member terminal 3, and relays an audio stream from the member terminal 3 to the automated driving vehicle 1. Acquire audio streams to generate recommendations.

The communication line relay unit 21 of the recommendation server 2 relays the voice stream 3 from the automatically driven vehicle 1 to the member terminal 3 in steps S311 and S312, and relays the voice stream 3 from the member terminal 3 to the member terminal 3 in steps S314 and S313. Relay audio stream 4 to vehicle 1 .

In step S315, the recommendation server 2 notifies the recommendation information display unit 11 of the automatic driving vehicle 1 of the progress information as recommendation information. This progress information, details of which will be described later, indicates the progress of selecting recommended information based on a plurality of audio streams acquired by the communication line relay unit 21 .

Then, the communication line relay unit 21 of the recommendation server 2 relays the voice stream 5 from the automatic driving vehicle 1 to the member terminal 3 in steps S316 and S317, and from the member terminal 3 in steps S319 and S318. It relays the audio stream 6 to the autonomous vehicle 1 .

Furthermore, the communication line relay unit 21 of the recommendation server 2 relays the voice stream 7 from the automatic driving vehicle 1 to the member terminal 3 in steps S320 and S321, and from the member terminal 3 in steps S323 and S322. It relays the audio stream 8 to the autonomous vehicle 1 .

In step S324, similarly to step S315, the recommendation server 2 notifies the recommended information display unit 11 of the automatic driving vehicle 1 of progress information indicating the progress of selection of the recommended information as recommendation information.
In the description of FIG. 3, the recommendation server 2 notifies the automatic driving vehicle 1 of progress information in step S315 based on the audio streams 1 to 4, and recommends information (progress) in step S324 based on the audio streams 5 to 8. information) to the automated driving vehicle 1.

In other words, although the recommendation server 2 notifies the automatic driving vehicle 1 of the recommendation information (progress information) for each of the four audio streams, the recommendation information (progress information) is not limited to four audio streams, and is provided for each predetermined number of audio streams. progress information).
Alternatively, recommendation information (elapsed information) may be notified every predetermined elapsed time instead of the number of audio streams.

After step S324, the recommendation server 2 continues to relay the audio stream and notify recommendation information (progress information).
Then, in step S325, when recommended information is selected based on the acquired audio stream, the recommendation server 2 notifies the recommended information display unit 11 of the automatic driving vehicle 1 of this as recommended information.

After step S325, the recommendation server 2 continues to relay the audio stream and select recommended information.
Then, in steps S326 and S327, when the recommendation server 2 finishes relaying the final voice stream N in response to the instruction to end the chat, in step S328, the recommended information with the best evaluation in the selection of recommended information is automatically set as recommended information. The recommended information display unit 11 of the driving vehicle 1 is notified.
As described above, the process of notifying the recommended information based on the message communication between the automatic driving vehicle 1, the recommendation server 2, and the member terminal 3 is completed.

Although FIG. 3 shows the case of one-to-one automatic driving vehicle 1 and member terminal 3, the same applies to one-to-n, that is, automatic driving vehicle 1 and a plurality of member terminals 3. FIG.
The recommendation server 2 relays audio streams from the autonomous driving vehicle 1 and the plurality of member terminals 3, and analyzes the audio streams to generate recommendation information.

Next, detailed operations of the control units of the automatic driving vehicle 1 and the recommendation server 2 will be described.
4 and 5 are flow charts for explaining the chat member selection operation and the chat control operation of the automatic driving vehicle 1. FIG.

First, referring to FIG. 4, the operation of the chat information acquiring unit 15 until the passenger (driver) decides on the member to chat with will be described.
In step S<b>41 , the chat information acquisition unit 15 acquires the identification information (identifier) of the passenger (driver) using the passenger identification unit 16 .

Specifically, the occupant identification unit 16 acquires a captured image of the occupant (driver) using the vehicle interior camera 1711, performs face authentication based on authentication information registered in advance for each occupant identifier, and performs face authentication of the occupant (driver). Acquire the identification information (identifier) of
In addition, the passenger identification unit 16 identifies the passenger (driver) by voiceprint authentication based on the authentication information registered in advance for each passenger identifier based on the sound of the passenger (driver) acquired by the voice input/output unit 14. , the identification information (identifier) may be acquired.
The identification information (identifier) to be acquired may be identification information of a plurality of passengers.

In step S42, the chat information acquisition unit 15 notifies the chat control unit 26 of the recommendation server 2 via the TCU 12 of the passenger (driver) identification information (identifier) acquired in step S41 as a member list request.
Then, the chat information acquisition unit 15 acquires the member names and the connection information of the member terminals 3 registered in the member list 42 of the SNS server 4 from the chat control unit 26 as a chat member list.

In step S43, the chat information acquisition unit 15 displays the acquired member list on the information panel 51 (display unit 10) by the recommended information display unit 11, prompts the crew to select a member to chat with, and selects at least one member to chat with. to get
More specifically, the chat information acquisition unit 15 determines members to chat with by touching the screen of the information panel 51 of the passenger or by voice instruction.

When obtaining the chat member list in step S42, the member list of the member list 42 may be filtered according to predetermined conditions such as the position of the automatic driving vehicle 1 and the chat frequency, and the obtained member list may be obtained.

Next, the chat control operation of the automatic driving vehicle 1 by the chat operation unit 18 will be described with reference to FIG. 5 .
In step S<b>51 , the chat operation unit 18 acquires the chat member list (member name, connection information of the member terminal 3 ) selected by the chat information acquisition unit 15 .
Then, in step S52, the chat operation unit 18 repeats the processing from step S53 to step S56 for the acquired members.

In step S53, the chat operation unit 18 notifies the recommendation server 2 of a chat request with the member terminal 3 via the TCU 12 according to the connection information of the member terminal 3 in the member list.

In step S54, the chat operation unit 18 determines whether or not the chat is possible from the member terminal 3 via the recommendation server 2. If the response is "chat possible" ("possible" in S54), the process proceeds to step S55. . If the response is "chat not possible" ("not possible" in S54), the process proceeds to step S56.

In step S55, the chat operation unit 18 starts chatting with the member terminal 3 via the recommendation server 2, and displays the member name on the information panel 51 (display unit 10) via the recommended information display unit 11. , to notify the crew to start chatting. Then, the process proceeds to step S56.

In step S56, the chat operation unit 18 determines whether or not the processing of steps S53 to S55 has been performed for the acquired members, and if so, the process proceeds to step S57. If not, the iterative process continues.

In step S57, the chat operation unit 18 determines whether there is an instruction to end the chat from the crew member or an instruction to end the chat from the chat control unit 26. If there is an instruction to end the chat (Yes in S57), the process proceeds to step S512. If there is no end instruction (No in S57), it is assumed that the chat is continuing, and the process proceeds to step S58.

In step S58, if the voice input/output unit 14 is currently inputting voice, the chat operation unit 18 acquires the detected voice of the passenger as stream information. Then, the chat operation unit 18 outputs an audio stream to the member terminal 3 and to the recommendation server 2 (communication line relay unit 21) that relays the voice stream.

In step S59, if the TCU 12 is receiving an audio stream, the chat operation unit 18 causes the audio input/output unit 14 to output the received stream information as audio into the vehicle.

In step S510, when the chat operation unit 18 receives recommendation information as progress information from the recommendation server 2 via the TCU 12, the recommended information display unit 11 displays the progress information on the information panel 51 (display unit 10). indicate.
Display of progress information by the recommended information display unit 11 will be described later.

In step S511, when the chat operation unit 18 receives recommendation information as recommended information from the recommendation server 2 via the TCU 12, the recommended information display unit 11 displays the recommended information on the information panel 51 (display unit 10). indicate.
Display of recommended information (progress information or recommended information) by the recommended information display unit 11 will be described later.

In the flowchart of FIG. 5, steps S58, S59, S510, and S511 are indicated as sequential steps, but steps S58, S59, S510, and S511 are performed as needed until the chat ends in step S57.

In step S512, the chat operation unit 18 receives recommendation information as recommended information from the recommendation server 2 via the TCU 12, displays the recommended information on the information panel 51 (display unit 10) by the recommended information display unit 11, End the process.
Display of recommended information by the recommended information display unit 11 will be described later.

Next, referring to FIG. 6, a process of generating recommended information in the recommendation server 2 will be described in detail.
FIG. 6 shows the processing of the recommendation server 2 during the chat between the automatic driving vehicle 1 and the member terminal 3 after step S37 of the operation sequence of FIG.

In step S61, the chat control unit 26 of the recommendation server 2 determines whether or not the chat between the automatic driving vehicle 1 and the member terminal 3 is ongoing. Repeat process. If the chat is not ongoing, the process proceeds to step S613.

In step S62, the communication line relay unit 21 of the recommendation server 2 bi-directionally relays the audio stream of the automatic driving vehicle 1 and the audio stream of the member terminal 3.
In step S63, the voice acquisition unit 22 of the recommendation server 2 acquires the voice stream of the automatic driving vehicle 1 and the voice stream of the member terminal 3 as stream information. Then, the following processing is performed on the acquired stream information.

In step S<b>64 , the language recognition unit 23 of the recommendation server 2 recognizes and decodes the stream information into a text string, extracts predetermined words from the text string, and registers the extracted words in the candidate word storage unit 27 .
Although the details will be described later, the candidate word storage unit 27 stores a recommendation information evaluation table, and if the extracted word is a new word, an entry is created in the recommendation information evaluation table.

More specifically, the language recognition unit 23 includes an acoustic model that expresses the relationship between phonemes and acoustic features in a statistical model, and a language model that evaluates whether character strings and word strings are appropriate as Japanese. , and a pronunciation dictionary, which is a dictionary for associating the words of the language model with the acoustic model, the stream information consisting of speech waveforms is acoustically analyzed and recognized and decoded into a text string.
Then, the recognition-decoded text string is subjected to morphological analysis to extract words (morphemes) related to destination candidates for vehicle movement, such as place names, food names, types of entertainment, facility names (store names). Here, multiple words may be extracted.

The language recognition unit 23 may search the Internet for words highly correlated with a plurality of words extracted by morphological analysis in one piece of stream information, and use these words as words related to destination candidates.

In step S65, the emotion estimating unit 24 of the recommendation server 2 finds the intonation of the voice waveform of the stream information and performs emotion analysis. Alternatively, the emotion estimation unit 24 refers to the emotion dictionary for each morpheme of the text string of the stream information analyzed by the language recognition unit 23, and calculates the emotion value of the text string.

The emotion dictionary contains active words and expressions such as "delicious", "understanding", "serious", and "amazing", as well as negative words and expressions such as "bad" and "expensive". is registered, and neutral words and expressions that are neither active nor negative are registered.
For example, active words/expressions have an emotional value of +1, negative words/expressions have an emotional value of -1, and neutral words/expressions have an emotional value of 0, and are added for each morpheme in the text string. By doing so, the emotion value of the text string of the stream information is calculated.

Then, the emotion estimating unit 24 estimates the emotion of the crew member who uttered the stream information or the member of the chat partner from the emotion analysis of the stream information.
More specifically, the emotion estimator 24 estimates that the emotion of the crew member or the member of the chat partner who issued the stream information with a large voice inflection is active. Also, the emotion estimation unit 24 estimates that the emotion of the crew member or the member of the chat partner who sent the stream information with the emotion value of "+" is active.

In step S66, the recommended information selection unit 25 obtains a score according to the estimated emotions (active/negative/neutral) of the crew members or members for each word extracted from the stream information in step S64, and creates a recommended information evaluation table. Update.

In step S64, if a plurality of words are extracted from the stream information and registered in the evaluation table, in step S65, the emotion estimating unit 24 calculates the emotion value based on the stream information of a predetermined range including the extracted words. or the voice inflection of streamed information in a predetermined range to estimate the emotions of the crew or members.

Also, in step S65, the emotion estimation unit 24 performs emotion analysis on the text string of the stream information analyzed by the language recognition unit 23 for the audio stream from the autonomous vehicle 1 to the member terminal 3, or analyzes the voice stream information. Emotion analysis is performed by obtaining the intonation of the waveform, and the emotion analysis of the occupants is performed. Then, in step S66, the recommended information selection unit 25 obtains a score according to the emotion of the occupant of the automatically driven vehicle 1 estimated based on the voice stream from the automatically driven vehicle 1 to the member terminal 3. The evaluation table of the recommended information created based on the words extracted from the stream information of the audio stream to the driving vehicle 1 may be updated. At this time, the words in the evaluation table of the recommended information to be updated are the same words as the words extracted from the voice stream subjected to the emotion analysis of the occupant in step S64.
As a result, the accuracy of recommended information can be improved.

Here, the configuration of the recommendation information evaluation table in the candidate word storage unit 27 will be described with reference to FIG.
The recommended information evaluation table includes a word string 71 that stores words extracted from stream information, a rating string 72 that stores ratings according to estimated crew or member emotions for each word of stream information, and a score string for each word. and an evaluation point sequence 73 obtained by summing up the evaluation points.

If the word extracted in step S64 is not registered in the word string 71, the word string 71 is newly registered.
A new score string is added to the score string 72 each time stream information is acquired in step S63, and the score is recorded in the column corresponding to the word and stream information in step S66.
The score string 73 is obtained by adding the total score of the score string 72 to the existing score for each word.

In addition to obtaining the score for each word according to the member's emotion (active/negative/neutral), the length of the stream information (conversation length) extracted from this word is evaluated, and the longer the score, the higher the score. can be increased.

Returning to FIG. 6, in step S67, the recommended information selection unit 25 of the recommendation server 2 determines whether or not the recommended information has been evaluated based on a predetermined number of pieces of stream information. That is, it is determined whether or not it is time to notify the progress information indicating the intermediate progress of the evaluation of the recommended information. If it is time to notify the progress information (Yes in S67), the process proceeds to step S68, and if it is not the time to notify the progress information (No in S67), the process proceeds to step S610.

As for the notification timing of the progress information in step S67, the progress information may be notified every predetermined elapsed time other than every predetermined number of stream information (audio streams).

In step S68, the recommended information selection unit 25 notifies the recommended information display unit 11 of the automatic driving vehicle 1 of the evaluation progress information of the recommended information as recommendation information, corresponding to step S315 in FIG.
More specifically, the recommended information selection unit 25 selects an active word whose evaluation score in the evaluation score column 73 is greater than “0” in the recommended information evaluation table of FIG. and Specifically, the evaluation points of "car cafe", "cat cafe", and "yoga" in FIG. 7 are used as progress information.

In step S<b>69 , the recommended information selection unit 25 deletes the score column 72 in the recommended information evaluation table of FIG. 7 . This is because the score for each piece of stream information is added to the score column 73, so the score column 72 does not need to be stored. In other words, the score column 72 is temporarily stored until the score is calculated.
By clearing the temporary storage in this way, the data capacity of the candidate word storage unit 27 can be compressed.

In step S610, the recommended information selection unit 25 determines whether or not there is a word in the evaluation table of recommended information shown in FIG. "Yes"), the process proceeds to step S611, and if not ("No" in S610), the process proceeds to step S612.

In step S611, corresponding to step S325 in FIG. 3, the recommended information selection unit 25 selects words whose evaluation point values in the evaluation point sequence 73 are equal to or greater than a predetermined threshold in the recommended information evaluation table in FIG. 7 as recommended information. , the recommended information display unit 11 of the automatic driving vehicle 1 is notified of the recommended information (recommended information). When the evaluation points of multiple words are equal to or higher than the threshold, each word is treated as recommended information.

In step S612, the chat control unit 26 of the recommendation server 2 determines whether or not the chat between the automatic driving vehicle 1 and the member terminal 3 is ongoing. Repeat process. If the chat ends, the process proceeds to step S613.

In step S613, the recommended information selection unit 25 displays the recommendation information ( recommended information).
Specifically, the recommended information selection unit 25 selects “car cafe” having the highest evaluation value in the evaluation point sequence 73 in the recommended information evaluation table of FIG. 7 as the recommended information.

In the above description, the emotion estimating unit 24 refers to the emotion dictionary and analyzes the emotion of the stream information. However, the text string obtained by decoding the stream information may be subjected to emotion estimation using a supervised machine learning model. .
Furthermore, stream information may be subjected to emotion analysis directly by AI (Artificial Intelligence) such as deep learning.

Next, the display of recommended information as progress information in the recommended information display unit 11 of the automatically driven vehicle 1 will be described in detail with reference to FIG. 8 .
In step S<b>81 , the recommended information display unit 11 acquires evaluation progress information of recommended information from the recommended information selection unit 25 of the recommendation server 2 as recommendation information.
This progress information consists of words and their scores associated with at least one set of recommended information.

In step S<b>82 , the recommended information display unit 11 acquires an image in front of the vehicle using an exterior camera (not shown) of the automatically driven vehicle 1 .
Then, in step S83, the recommended information display unit 11 generates a 3D city map image 111 (see FIG. 11A) based on the front image of the vehicle, and displays it on the information panel 51 (display unit 10). This 3D city map image 111 is generated by, for example, extracting building shapes from the front image of the vehicle and rendering the wall surfaces.

Further, instead of steps S82 and S83, the recommended information display unit 11 acquires the city map information of the map information 131 by the navigation unit 13 based on the current position information of the autonomous vehicle 1, and displays the 3D city map image. 111 may be used.

In step S84, the recommended information display unit 11 repeats steps S85 and S86 by the number of words in the progress information for each word in the progress information acquired as recommendation information.

In step S85, the recommended information display unit 11 displays words related to the recommended information, and generates a balloon image (circular image) having a display area proportional to the evaluation value of the word on the display unit 10. FIG.
Then, in step S86, the recommended information display unit 11 overlay-displays the balloon image on the 3D city map image displayed on the information panel 51. FIG.
Note that the balloon image may be an image corresponding to a word related to the recommended information instead of the word related to the recommended information.

In step S87, after repeatedly displaying the balloon images for the number of words in the progress information, the recommended information display unit 11 ends the display of the recommendation information as the progress information of the evaluation of the recommendation information.

FIG. 11A is a diagram showing an example of display of progress information in evaluation of recommended information, which is one type of recommendation information displayed on the information panel 51. FIG.
Balloon images 113 a , 113 b , 113 c , 113 d and 113 e for each word of progress information are overlaid on the 3D city map image 111 .
Since the balloon image 113a has a larger display area than the other balloon images (113b to 113e), it can be seen that the evaluation of the recommended information is high.

Next, display of recommended information as recommended information in the recommended information display unit 11 of the automatic driving vehicle 1 will be described in detail with reference to FIG. 9 .
In step S<b>91 , the recommended information display unit 11 acquires recommended information indicating destination candidates as recommended information from the recommended information selection unit 25 of the recommendation server 2 .
This recommended information consists of words corresponding to at least one recommended information.

In step S92, the recommended information display unit 11 repeats step S93 by the number of words for each word of the recommended information acquired as recommendation information.

In step S93, the recommended information display unit 11 acquires the name/address information/display image corresponding to the words of the recommended information, and uses them as recommended data.
To acquire the name/address information/display image corresponding to the word of the recommended information, the navigation unit 13 searches the map information 131 using the word of the recommended information as a key to acquire the name/address information/display image. Alternatively, it may be obtained by searching the Internet through the recommendation server 2 using the word of the recommended information as a key.

FIG. 10 is a diagram showing the configuration of recommendation data corresponding to words of recommendation information.
The recommendation data is composed of a plurality of sets of name/address/image information corresponding to each word of the recommendation information.
The name is character string information indicating a store name, facility name, etc., the address is character string information indicating its location, and the image is image data information indicating an image of the store or facility.

Returning to FIG. 9, in step S94, the recommended information display unit 11 repeats the number of words in the recommended information acquired as the recommended information, and then proceeds to step S95.

In step S<b>95 , the recommended information display unit 11 acquires from the map information 131 the map information including the points indicated by the address information of all the recommended data that are destination candidates.

In step S96, the recommended information display unit 11 acquires the front image 112 (see FIG. 11B) of the vehicle by the outside camera (not shown) of the automatic driving vehicle 1, and displays it on the information panel 51 (display unit 10).

In step S97, the recommended information display unit 11 repeats steps S98 and S99 for each word of the recommended information acquired as recommendation information.

In step S98, the recommended information display unit 11 pins (marks) the position corresponding to the address of the recommended data in the front image obtained in step S96 based on the map information obtained in step S95.

In step S99, the recommended information display unit 11 draws a lead line from the pin stand (mark) and draws and displays image data showing an image related to the recommended information on the front image.

In step S910, after repeating the display of the image data of the recommended information for the number of words of the recommended information, the recommended information display unit 11 terminates the display of the recommended information.

FIG. 11B is a diagram showing an example of display of recommended information, which is one of the recommendations displayed on the information panel 51. As shown in FIG.
Display images 114a and 114b corresponding to the words of the recommended information are overlaid on the front image 112 of the vehicle.
Since the location of the recommended information is pinned (marked), it can be easily grasped as a destination candidate.

1 Automated Driving Vehicle 10 Display Unit 13 Navigation Unit 131 Map Information 11 Recommended Information Display Unit 15 Chat Information Acquisition Unit 16 Passenger Identification Unit 12 TCU
14 voice input/output unit 17 operation control unit 171 sensor unit 1711 vehicle interior camera 172 vehicle steering/driving unit 18 chat operation unit 3 member terminal 4 SNS server 2 recommendation server 21 communication line relay unit 22 voice acquisition unit 23 language recognition unit 24 emotion Estimation unit 25 Recommended information selection unit 27 Candidate word storage unit 26 Chat control unit

Claims (6)

a voice acquisition unit that acquires a conversation voice between a vehicle occupant and another person outside the vehicle;
a language recognition unit that extracts predetermined words from the conversational voice and recognizes the meaning of the extracted words;
an emotion estimating unit for estimating the emotion of the occupant from the intonation of the conversation voice or the meaning of the extracted words;
a recommended information selection unit that evaluates the words recognized by the language recognition unit according to the emotion of the occupant estimated by the emotion estimation unit, and obtains recommended information for the occupant based on the evaluation of the word;
a recommended information display unit that displays the recommended information obtained by the recommended information selection unit to the occupant,
an evaluation table for storing a score of the emotion of the occupant estimated by the emotion estimation unit for each stream information and a sum of the scores for words recognized by the language recognition unit;
The recommended information selection unit
Only the words recognized from the conversational voice of the other person to the occupant by the language recognition unit are evaluated according to only the emotion of the occupant estimated by the emotion estimation unit based on the conversational voice of the occupant to the other person. and obtaining recommended information for the occupant based on the evaluation of the word ,
When the recommended information is evaluated based on a predetermined number of stream information, the score of the occupant's emotion for each stream information in the evaluation table is deleted.
A recommendation system characterized by: 2. The recommendation system according to claim 1, wherein the recommended information display unit displays the image corresponding to the recommended information in proportion to the display area on the display unit according to the evaluation value. comprising a navigation unit of the vehicle;
3. The recommendation according to claim 1, wherein the recommended information selection unit recommends the destination based on the evaluation of the word, using the word as a related keyword of the destination candidate in the navigation unit. system. The navigation unit has map information,
4. The recommendation system according to claim 3, wherein the recommended information display unit displays the map information and displays facilities related to the word on the displayed map information. The recommender system processor
a voice acquisition unit that acquires a conversation voice between a vehicle occupant and another person outside the vehicle;
a language recognition unit that extracts predetermined words from the conversational voice and recognizes the meaning of the extracted words;
an emotion estimating unit for estimating the emotion of the occupant from the intonation of the conversation voice or the meaning of the extracted words;
Only the words recognized from the conversational voice of the other person to the occupant by the language recognition unit are evaluated according to only the emotion of the occupant estimated by the emotion estimation unit based on the conversational voice of the occupant to the other person. Then, recommended information for passengers is obtained based on the evaluation of the words, and when the recommended information is evaluated based on a predetermined number of stream information, the words recognized by the language recognition unit are evaluated for each stream information. a recommended information selection unit that deletes the occupant's emotion score for each stream information from an evaluation table that stores the occupant's emotion score estimated by the emotion estimating unit and the sum of the scores;
and a recommended information display unit for displaying the recommended information obtained by the recommended information selection unit to a passenger. a voice acquisition step of acquiring conversational voices between the occupants of the vehicle and others outside the vehicle;
a language recognition step of extracting predetermined words from the conversational voice and recognizing the meaning of the extracted words;
an emotion estimation step of estimating the emotions of the occupant from the intonation of the conversation voice or the meaning of the extracted words;
In the language recognition step, only the words recognized from the conversational voice of the other person to the passenger are evaluated based on the conversational voice of the passenger to the other person only according to the emotion of the passenger estimated in the emotion estimation step. and a recommended information selection step of obtaining recommended information for the occupant based on the evaluation of the word;
When the recommended information is evaluated based on a predetermined number of pieces of stream information, the score of the occupant's emotion estimated in the emotion estimation step for each stream information and the sum of the scores for the word recognized in the language recognition step deleting the rating of the occupant's emotion for each stream information in a rating table that stores
and a recommended information display step of displaying the recommended information obtained in the recommended information selection step to the passenger.

Images