JP2022137792A - Communication support device and communication support method - Google Patents

Abstract

To provide a communication support device and a communication support method for allowing a listener to easily understand utterance contents of an utterer by improving accuracy for estimating the degree of understanding of the listener.SOLUTION: A communication support device acquires utterance contents of a first user from voice of the first user collected by a sound collection device, acquires state data of a second place, showing a state of the second place at which a second user is present and which is different from a first place at which the first user is present, estimates the degree of understanding of the second user of utterance contents of the first user on the basis of the state data of the second place, and controls an output device provided at the second place so as to present utterance contents of the first user to the second user in a presentation mode based on the degree of understanding of the second user.SELECTED DRAWING: Figure 1

Description

The present invention relates to a communication support device and a communication support method.

2. Description of the Related Art A speech communication system that improves the intelligibility of speech output to a listener is known (for example, Patent Document 1). In the speech communication system described in Patent Document 1, the background noise of the listener is analyzed, and using the analysis result of the background noise, it is determined whether or not the listener can easily understand the output speech under the current background noise. judge.

Japanese Patent Publication No. 2002-507291

The listener's comprehension is affected not only by the listener's background noise, but also by the speaker's utterance content. In the speech communication system described in Patent Document 1, even if it is determined that the listener can easily understand the speech under the current background noise, the listener may not be able to easily understand the speech depending on the content of the utterance. In other words, in the voice communication system described in Patent Document 1, since the listener's degree of understanding is not related to the utterance content, the accuracy of estimating the listener's degree of understanding is low, and it is difficult for the listener to understand the utterance content of the speaker. There's a problem.

The problem to be solved by the present invention is to provide a communication support device and a communication support method that make it easier for the listener to understand the utterance content of the speaker by improving the accuracy of estimating the listener's degree of understanding.

The present invention acquires the utterance content of the first user from the first user's voice collected by a sound collector, and obtains the content of the first user's utterance at a second place where the second user is different from the first place where the first user is. obtaining situation data at a second location indicating the situation of the second user, estimating the degree of understanding of the second user with respect to the utterance content of the first user based on the situation data of the second place, and obtaining the degree of understanding of the second user The above problem is solved by controlling the output device provided at the second location so as to present the speech content of the first user to the second user in a presentation mode based on the above.

According to the present invention, the degree of understanding of the second user is estimated with respect to the utterance content of the first user based on the situation of the place where the second user is. The accuracy of estimation is improved, and the second user can easily understand the content of the first user's speech.

1 is a block diagram of a communication system according to this embodiment; FIG. FIG. 4 is an explanatory diagram showing an example of vehicle situation data acquired by a situation acquisition unit; 4 is a flowchart showing voice output processing by the communication system according to the embodiment; 4 is a flowchart showing voice output processing by the communication system according to the embodiment; FIG. 4 is an explanatory diagram showing an example of a presentation mode for emphasizing the contents of a speech of a remote user;

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below with reference to the drawings.
In this embodiment, a communication support device and a communication support method according to the present invention will be described using an example in which they are applied to a communication system.

FIG. 1 is a block diagram of a communication system 100 according to this embodiment. The communication system 100 according to this embodiment is used in a scene where users A and B who are not in the same space have a conversation. In this embodiment, the user A is an example of a user who is in the vehicle 1 and is in the vehicle interior space 20, and the user B is an example of a user who is in the remote space 30, which is a place away from the vehicle 1. I will list and explain. The location of user A and the location of user B are different.

Communication system 100 may be used in situations where user A and user B are located at different locations, and the locations where user A and user B are located are not particularly limited. For example, user A does not have to be in the vehicle 1 . In the following description, for the sake of convenience, user A in the vehicle interior space 20 will be referred to as in-vehicle user A, and user B in the remote space 30 will be referred to as remote user B.

In addition, in the present embodiment, the state of each user when in-vehicle user A and remote user B converse is not particularly limited. For example, FIG. 1 shows a state in which the in-vehicle user A is operating the steering wheel of the vehicle 1 as the state of the in-vehicle user A. may be Also, for example, in FIG. 1, as the state of remote user B, the state in which remote user B is seated is shown, but remote user B may be standing. Further, in this embodiment, a scene in which in-vehicle user A and remote user B converse will be described as an example, but communication system 100 can be used in situations other than conversation. For example, the communication system 100 can be used in a situation where the remote user B speaks unilaterally to the in-vehicle user A, that is, a situation in which the voice flow is unidirectional.

Also, in this embodiment, a case where the communication system 100 is used in a system using virtual reality and augmented reality technology will be described as an example. As shown in FIG. 1, since the in-vehicle user A and the remote user B are in different spaces, the in-vehicle user A and the remote user B cannot converse in the same space. However, in a virtual reality technology that allows a user to perceive a virtual space, which is a world created by a computer, as a reality, the computer can create a virtual space that makes the remote user B feel as if they are in the vehicle interior space 20. can. In the virtual space, the remote user B perceives as if he were in the vehicle interior space 20 at least visually and aurally. For example, the remote user B can visually perceive the virtual space as the vehicle interior space 20 by wearing a device using virtual reality technology known at the time of filing the application, such as VR goggles. Further, the remote user B can perceive the virtual space as the in-vehicle space 20 through hearing through the communication system 100 according to the present embodiment.

On the other hand, in the augmented reality technology that allows the user to perceive a virtual augmented reality by superimposing visual information created by a computer, the computer provides the in-vehicle user A with the remote location user B It is possible to create a virtual space as if the user is in the vehicle interior space 20.例文帳に追加In the virtual space, the in-vehicle user A perceives, at least visually and aurally, as if the remote user B is in the in-vehicle space 20 . For example, in-vehicle user A wears a device using augmented reality technology known at the time of the filing of the present application, such as AR glasses, so that through vision, a computer-generated character ( avatar) can be recognized as if it were in the vehicle interior space 20. Further, the in-vehicle user A can hear the avatar of the remote user B as if it were in the in-vehicle space 20 with the communication system 100 according to the present embodiment. Each configuration of the communication system 100 will be described below with reference to FIG.

As shown in FIG. 1, the communication system 100 includes a peripheral information server 10, an in-vehicle space 20, a remote space 30, and a server 40. FIG. The remote space 30 is a space separated from the vehicle interior space 20 . The remote space 30 may be, for example, a room in a vehicle other than the vehicle 1 or a room in a building.

The peripheral information server 10 provides information indicating the situation of the place where the user A is in the vehicle. The information indicating the situation of the place where the in-vehicle user A is is the surrounding information of the vehicle 1 . As shown in FIG. 1, the surrounding information of the vehicle 1 provided by the surrounding information server 10 includes traffic information 11 including road information, accident information, and traffic information around the vehicle 1, weather and temperature around the vehicle 1, and traffic information 11 including traffic information. and communication information 13 including the types of wireless LAN standards that can be used around the vehicle 1 and the communication speed of the wireless LAN. The peripheral information server 10 transmits peripheral information of the vehicle 1 to the in-vehicle communication device 22 in response to a request from the in-vehicle controller 24 to be described later. Although the traffic information 11, the weather information 12, and the communication information 13 are exemplified as the information provided by the surrounding information server 10, the surrounding information server 10 does not need to provide all of these information, and the traffic information 11, weather information 12, and communication information 13.

Next, the vehicle interior space 20 will be described. The vehicle interior space 20 is a space indicating the interior of the vehicle 1 . A sensor 21 , an in-vehicle communication device 22 , an in-vehicle output device 23 , and an in-vehicle controller 24 are provided in the vehicle interior space 20 . Each device provided in the vehicle interior space 20 is connected by, for example, a CAN (Controller Area Network) or other vehicle-mounted network (Ethernet (registered trademark), etc.) in order to mutually transmit and receive information.

The sensor 21 is a sensor that detects the situation where the user A is in the vehicle. The situation of the place where the user A is in the vehicle includes at least one of the situation inside the vehicle 1, the situation outside the vehicle 1, the driving situation of the vehicle 1, and the situation of the user A inside the vehicle. As shown in FIG. 1, the sensor 21 is composed of a vehicle interior sensor 211 and a vehicle exterior sensor 221 . The in-vehicle sensor 211 is a sensor for detecting the condition of the interior of the vehicle 1, the traveling condition of the vehicle 1, and the condition of the user A inside the vehicle. The vehicle exterior sensor 221 is a sensor for detecting the conditions outside the vehicle 1 and the traveling conditions of the vehicle 1 .

As shown in FIG. 1 , the in-vehicle sensor 111 includes an in-vehicle camera 212 , an in-vehicle microphone 213 , a steering angle sensor 214 , and a pedal sensor 215 .

The in-vehicle camera 212 is an imaging device that images the interior of the vehicle 1 . The in-vehicle camera 212 detects the state of the interior of the vehicle 1 by capturing an image of the interior of the vehicle 1 . In addition, the in-vehicle camera 212 detects the situation of the user A in the vehicle by capturing an image of the user A in the vehicle. For example, the in-vehicle camera 212 captures images of the user A inside the vehicle and other occupants of the vehicle 1 . A captured image captured by the vehicle interior camera 212 is output to the vehicle controller 24 as captured image data of the vehicle interior space 20 . Note that the in-vehicle camera 212 may be any device or device that is mounted in the vehicle 1 and captures an image of the interior of the vehicle 1, and its form, number, characteristics, installation location, and the like are not particularly limited. For example, the in-vehicle camera 212 may be composed of a plurality of cameras or a plurality of types of cameras.

The in-vehicle microphone 213 is a sound collecting device that collects sounds inside the vehicle 1 . The in-vehicle microphone 213 detects the situation inside the vehicle 1 by collecting the sound inside the vehicle 1 . Also, the in-vehicle microphone 113 detects the driving situation of the vehicle 1 by collecting sound output from a device or system mounted on the vehicle 1 . Further, the in-vehicle microphone 213 detects the situation of the in-vehicle user A by collecting the voice of the in-vehicle user A. For example, the in-vehicle microphone 213 is used for the voice of the in-vehicle user A, the voice of other passengers in the vehicle 1, the engine sound and motor sound of the vehicle 1, the turn signal sound when the direction indicator is activated, and the wiper sound when the wiper is activated. It collects sounds such as wiper sounds, hazard lamp sounds when the hazard lamps are activated, sounds output from the car navigation system, and music played from the audio system. The sound collected by the in-vehicle microphone 213 is output to the in-vehicle controller 24 as collected sound data of the in-vehicle space 20 . The in-vehicle microphone 213 may be any device or device that is mounted in the vehicle 1 and collects the sound in the interior of the vehicle 1, and its form, number, characteristics, installation position, etc. are not particularly limited. For example, the in-vehicle microphone 213 may be composed of a plurality of microphones or a plurality of types of microphones.

The steering angle sensor 214 is a sensor that is provided on the steering shaft of the vehicle 1 and detects the steering angle of the steering wheel of the vehicle 1 . The steering angle sensor 214 detects the running condition of the vehicle 1 by detecting the steering angle of the steering wheel. The steering angle of the steering wheel detected by the steering angle sensor 214 is output to the in-vehicle controller 24 as steering angle data of the vehicle 1 .

The pedal sensor 215 is a sensor that is provided for each of the accelerator pedal and the brake pedal and detects the amount of depression of the accelerator pedal and the amount of depression of the brake pedal. The pedal sensor 215 detects the running condition of the vehicle 1 by detecting the amount of depression of the accelerator pedal and the amount of depression of the brake pedal. The amount of depression of the accelerator pedal and the amount of depression of the brake pedal detected by the pedal sensor 15 are output to the in-vehicle controller 24 as pedal depression data of the vehicle 1 .

The sensor described above is an example of the in-vehicle sensor 211, and the in-vehicle sensor 211 may include at least one of the sensors described above. The in-vehicle sensor 211 is not limited to the sensors described above, and may include a sensor that detects at least one of the indoor conditions of the vehicle 1, the traveling conditions of the vehicle 1, and the conditions of the user A inside the vehicle. . For example, the in-vehicle sensors 211 include a shift position sensor that detects the position of the shift lever of the vehicle 1, a temperature sensor that measures the room temperature of the vehicle 1, a vehicle speed sensor that detects the vehicle speed of the vehicle 1, and an acceleration sensor that detects the acceleration of the vehicle 1. A sensor, a gyro sensor for detecting the angular velocity of the vehicle 1, or the like may be included.

Further, as shown in FIG. 1 , the vehicle exterior sensor 221 includes an exterior camera 222 , an exterior microphone 223 , a millimeter wave radar 224 , an ultrasonic sonar 225 and a GPS 226 .

The vehicle exterior camera 222 is an imaging device that captures an image of the surroundings of the vehicle 1 . The exterior camera 222 detects the situation outside the vehicle 1 and the traveling situation of the vehicle 1 by imaging the surroundings of the vehicle 1 . For example, the vehicle exterior camera 222 images an object existing around the vehicle 1 . Objects captured by the vehicle exterior camera 222 include, for example, bicycles, motorcycles, road obstacles, traffic lights, and road markings. The captured image captured by the vehicle exterior camera 222 is output to the in-vehicle controller 24 as captured image data of the surroundings of the vehicle 1 . Note that the exterior camera 222 is mounted on the vehicle 1 and may be any device or device that captures an image of the surroundings of the vehicle 1, and its form, number, characteristics, installation location, and the like are not particularly limited. For example, the vehicle exterior camera 222 may be composed of a plurality of cameras or a plurality of types of cameras.

The vehicle external microphone 223 is a sound collecting device that collects sounds around the vehicle 1 . The external microphone 223 detects the situation outside the vehicle 1 and the traveling situation of the vehicle 1 by collecting sounds around the vehicle 1 . For example, the vehicle external microphone 223 collects running sounds and horn sounds of vehicles other than the vehicle 1, siren sounds of emergency vehicles, and advertising sounds output from advertising vehicles. The sound collected by the external microphone 223 is output to the in-vehicle controller 24 as collected sound data around the vehicle 1 . Note that the external microphone 223 may be any device or device that is mounted on the vehicle 1 and collects sounds around the vehicle 1, and its form, number, characteristics, installation position, etc. are not particularly limited. For example, the vehicle exterior microphone 223 may be composed of a plurality of microphones or a plurality of types of microphones.

The millimeter wave radar 224 is a sensor that detects objects existing around the vehicle 1 by using the millimeter wave radar. Specifically, the millimeter wave radar 224 detects the direction in which the object is positioned with respect to the traveling direction of the vehicle 1 and the distance from the vehicle 1 to the object. Moreover, when the object is moving, the millimeter wave radar 224 detects the relative speed of the object with respect to the speed of the vehicle 1 . The millimeter wave radar 224 detects objects existing around the vehicle 1 to detect the conditions outside the vehicle 1 and the traveling conditions of the vehicle 1 . An example of an object detected by the millimeter wave radar 224 is an example of an object imaged by the vehicle exterior camera 222, so the above description is used for an example of the object. Information on the object detected by the millimeter wave radar 224 is output to the onboard controller 24 as object data around the vehicle 1 .

The ultrasonic sonar 225 is a sensor that uses ultrasonic waves to detect an object existing in a relatively close range to the vehicle 1 . Specifically, the ultrasonic sonar 225 detects the direction in which the object is positioned with respect to the traveling direction of the vehicle 1 and the distance from the vehicle 1 to the object. The ultrasonic sonar 225 detects objects existing around the vehicle 1 to detect the conditions outside the vehicle 1 and the traveling conditions of the vehicle 1 . For example, ultrasonic sonar 225 detects obstacles in parking lots, parked vehicles, and the like. Information on the object detected by the ultrasonic sonar 225 is output to the vehicle-mounted controller 24 as object data around the vehicle 1 .

The GPS 226 acquires position information indicating the current position of the vehicle 1 (Global Positioning System, GPS). The GPS 226 acquires position information of the vehicle 1 by receiving information transmitted from a plurality of satellite communications with a receiver. The GPS 226 detects the situation outside the vehicle 1 and the traveling situation of the vehicle 1 by acquiring the position information of the vehicle 1 . The positional information of the vehicle 1 acquired by the GPS 226 is output to the in-vehicle controller 24 as the positional data of the vehicle 1 .

The sensor described above is an example of the vehicle exterior sensor 221, and the vehicle exterior sensor 221 may include at least one of the sensors described above. Further, the vehicle exterior sensor 221 is not limited to the sensors described above, and may include a sensor that detects at least one of the conditions outside the vehicle 1 and the running condition of the vehicle 1 . For example, the vehicle exterior sensor 221 may include a raindrop sensor that detects the amount of raindrops, a lidar that detects objects around the vehicle 1 with pulsed infrared rays, and the like.

The in-vehicle communication device 22 is a device having a wireless communication function for wirelessly transmitting and receiving data to and from the outside of the vehicle 1 . The in-vehicle communication device 22 is, for example, a telematics control unit (TCU). The in-vehicle communication device 22 connects to the Internet using a communication standard such as 4G/LTE and Wifi (registered trademark) by a wireless communication function, and communicates with a server or system provided outside the vehicle 1. to send and receive various data. In this embodiment, the in-vehicle communication device 22 transmits and receives data to and from the peripheral information server 10 , the remote communication device 31 and the server communication device 41 . The in-vehicle communication device 22 outputs data received from these communication devices or servers to the in-vehicle controller 24 . The in-vehicle communication device 22 also transmits data input from the in-vehicle controller 24 to these communication devices or servers.

The in-vehicle output device 23 is provided in the vehicle 1 and is a device that outputs the utterance content of the remote user B to the user A in the vehicle. As shown in FIG. 1, the in-vehicle output device 23 includes an in-vehicle display 23a for presenting the utterance content of the remote user B to the in-vehicle user A in a display format, In-vehicle speaker 23b to present is mentioned. Presentation mode data is input to the in-vehicle output device 23 from the in-vehicle controller 24 . The presentation mode data includes information on a device to be controlled controlled by the in-vehicle controller 24 to present the contents of the speech of the remote user B, and setting information on the device to be controlled. The in-vehicle output device 23 presents the utterance content of the remote user B to the in-vehicle user A in the presentation mode indicated by the presentation mode data.

The in-vehicle controller 24 has a ROM (Read Only Memory) that stores a program for executing a situation acquisition process for acquiring situation data of the vehicle 1 and an utterance content presentation process for presenting the utterance content of the remote user B to the in-vehicle user A. ), a CPU (Central Processing Unit) that executes programs stored in the ROM, and a RAM (Random Access Memory) that functions as an accessible storage device. The in-vehicle controller 24 has a situation acquisition unit 25 and a speech content presentation unit 26 . The in-vehicle controller 24 can perform the functions of the situation acquisition unit 25 and the utterance content presentation unit 26 by executing programs stored in the ROM.

The situation acquisition unit 25 acquires situation data of the vehicle 1 indicating the situation of the place where the user A is in the vehicle. In this embodiment, the situation acquisition unit 25 acquires situation data of the vehicle 1 from the surrounding information server 10 and the sensor 21 .

For example, the situation acquisition unit 25 obtains traffic information 11 around the vehicle 1, weather information 12 around the vehicle 1, and communication information 13 around the vehicle 1 from the surrounding information server 10 via the in-vehicle communication device 22. get. The status acquisition unit 25 can acquire various types of information from the surrounding information server 10 by transmitting an information request signal to the surrounding information server 10 . Further, for example, the situation acquisition unit 25 acquires data detected by each sensor constituting the sensor 21 from the sensor 21 mounted on the vehicle 1 . The data detected by the sensor 21 includes imaged image data of the vehicle interior space 20, collected sound data of the vehicle interior space 20, steering angle data of the vehicle 1, pedal depression data of the vehicle 1, imaged image data of the surroundings of the vehicle 1, vehicle 1 surrounding sound collection audio data, vehicle 1 surrounding object data, and vehicle 1 location data. The situation acquisition unit 25 is an example of the "situation acquisition unit" of the present invention.

FIG. 2 is an explanatory diagram showing an example of situation data of the vehicle 1 acquired by the situation acquisition unit 25. As shown in FIG. As shown in FIG. 2, the situation data of the vehicle 1 is classified into the situation data inside the vehicle 1, the situation data outside the vehicle 1, the travel situation data of the vehicle 1, and the situation data of the user inside the vehicle.

As shown in FIG. 2, data indicating the noise condition of the vehicle 1 and data indicating the communication condition of the vehicle 1 can be used as the condition data of the interior of the vehicle 1 . The data indicating the noise situation of the vehicle 1 includes road noise, volume of sound (radio, television) flowing in the vehicle interior space 20, and the like. The situation acquisition unit 25 acquires data indicating the noise situation of the vehicle 1 from the surrounding information server 10 and the in-vehicle microphone 213 . Data indicating the communication status of the vehicle 1 includes a bit rate indicating the line speed. The status acquisition unit 25 acquires data indicating the communication status of the vehicle 1 from the peripheral information server 10 .

As shown in FIG. 2, data indicating the road conditions around the vehicle 1, data indicating the surrounding conditions of the vehicle 1, and data indicating weather conditions around the vehicle 1 can be used as the situation data of the vehicle 1 outside the vehicle 1, as shown in FIG. The data indicating the road conditions around the vehicle 1 and the surrounding conditions of the vehicle 1 include traffic information on the road on which the vehicle 1 travels or its surroundings, the location where the vehicle 1 travels, information on objects existing around the vehicle 1, and the like. is mentioned. The situation acquisition unit 25 indicates the road situation around the vehicle 1 and the situation around the vehicle 1 from the surrounding information server 10, the vehicle exterior camera 222, the vehicle exterior microphone 223, the millimeter wave radar 224, the ultrasonic sonar 225, the GPS 226, and the like. Get data. Data indicating weather conditions around the vehicle 1 include data indicating the weather (“sunny”, “cloudy”, “rainy”, “foggy”, etc.). The condition acquisition unit 25 acquires data indicating weather conditions of the vehicle 1 from the surrounding information server 10 .

As the driving condition data of the vehicle 1, there is data indicating the driving condition of the vehicle 1, as shown in FIG. The data indicating the driving status of the vehicle 1 includes information indicating the state of the vehicle 1 (“parking backwards”, “parallel parking”, “running on highway”, “turning left or right”, “pausing”, etc.). ). The situation acquisition unit 25 acquires the driving situation of the vehicle 1 from the vehicle interior microphone 213, the steering angle sensor 214, the pedal sensor 215, the vehicle exterior camera 222, the vehicle exterior microphone 223, the millimeter wave radar 224, the ultrasonic sonar 225, the GPS 226, and the like. Get the data shown.

In addition, as the status data of the user A in the vehicle, data indicating the status of dialogue with the remote user B, and data indicating the reaction status of the user A in the vehicle, as shown in FIG. The data indicating the state of dialogue with the remote user B includes information indicating the level of understanding of the in-vehicle user A in the dialogue with the remote user B ("understood", "not understood", "say again", "hmm"). , “Uh,” etc.). The status acquisition unit 25 acquires data indicating the status of the conversation with the remote user B from the in-vehicle camera 212, the in-vehicle microphone 213, and the like. In addition, as the data indicating the reaction state of the in-vehicle user A, information indicating the mode of expression of the in-vehicle user A after the content of the speech of the remote user B is presented to the in-vehicle user A (change in appearance of the in-vehicle user A, line of sight, etc.). The status acquisition unit 25 acquires data indicating the reaction status of the in-vehicle user A from the in-vehicle camera 212, the in-vehicle microphone 213, and the like. The situation data of the vehicle 1 acquired by the situation acquisition unit 25 shown in FIG. 2 is an example, and the situation data of the vehicle 1 is not limited.

Returning to FIG. 1, the functions realized by the in-vehicle controller 24 will be described. The utterance content presentation unit 26 receives the utterance content data and the presentation mode data of the remote user B transmitted from the remote communication device 31 via the in-vehicle communication device 22 . The utterance content presentation unit 26 controls the in-vehicle output device 23 so as to present the utterance content of the remote user B to the in-vehicle user A in the presentation mode indicated by the presentation mode data. For example, the utterance content presenting unit 26 refers to the presentation mode data to specify the setting information of the in-vehicle display 23a and the setting information of the in-vehicle speaker 23b for presenting the utterance content of the remote user B to the in-vehicle user A. . The timing at which the utterance content data of the remote user B is input to the utterance content presentation unit 26 and the timing at which the presentation mode data is input to the utterance content presentation unit 26 may be the same or different. good. The utterance content presenting unit 26 is an example of the "utterance content presenting unit" of the present invention.

The remote space 30 will be described. As shown in FIG. 1, the remote space 30 is provided with a remote communication device 31 , a remote microphone 32 , a remote input device 33 , a remote output device 34 and a remote controller 35 . Each device provided in the remote space 30 is connected by, for example, Ethernet (registered trademark) or the like in order to mutually transmit and receive information.

The remote communication device 31 is a device having a wireless communication function for wirelessly transmitting and receiving data to and from the outside of the remote space 30 . Remote communication device 31 may be, for example, a router. The remote communication device 31 connects to the Internet using a communication standard such as 4G/LTE, Wifi (registered trademark), etc., using a wireless communication function, and connects to a server or system provided outside the remote space 30. Send and receive various data between In this embodiment, the remote communication device 31 transmits and receives data to and from the in-vehicle communication device 22 and the server communication device 41 . Data transmitted and received between each communication device will be described later.

The remote microphone 32 is a sound collecting device that collects sound at the location where the remote user B is located (the remote space 30). The remote microphone 32 is preferably installed at a position where the voice of the remote user B can be collected. The sound collected by the remote microphone 32 is output to the remote controller 35 . Note that the remote microphone 32 is provided in the remote space 30 and may be any device or device capable of collecting the voice of the remote user B, and its form, number, characteristics, installation location, etc. are not particularly limited. For example, the remote microphone 32 may consist of a plurality of microphones or a plurality of types of microphones.

The remote input device 33 is a device that functions as a human interface that the remote user B can operate. The remote user B can input settings related to the communication system 100 by operating the remote input device 33 . The remote input device 33 may be composed of separate devices such as a keyboard and a mouse, or may be composed of a display and an input unit integrally like a touch panel. The form, characteristics, installation location, etc. of the remote input device 33 are not particularly limited.

In this embodiment, the remote user B sets, as a setting related to the communication system 100, a mode of presentation to the in-vehicle user A, which indicates in what manner the content of the utterance of the remote user B is to be presented to the in-vehicle user A. set. For example, the remote user B inputs, through the remote input device 33, a device to be controlled to present the content of the speech of the remote user B, and detailed settings of the device to be controlled.

The remote output device 34 is provided in the remote space 30 and outputs to the remote user B information on the in-vehicle user A's degree of understanding of the remote user B's utterance content. As shown in FIG. 1, the remote output device 34 includes a remote display 34a for presenting the degree of understanding of the in-vehicle user A to the remote user B in a display format, and a remote display 34a for presenting the degree of understanding of the in-vehicle user A to the remote user B in a voice format. A remote speaker 34b presented to B. The comprehension level data of the in-vehicle user A is input from the remote controller 35 to the remote output device 34 . The remote output device 34 presents the comprehension level of the in-vehicle user A to the remote user B based on the comprehension level data of the in-vehicle user A.

The remote controller 35 performs an utterance content acquisition process for acquiring the utterance content of the remote user B, an understanding level presentation process for presenting the in-vehicle user A's level of understanding of the utterance content of the remote user B, and an input to the remote input device 33. A ROM (Read Only Memory) storing a program for executing a reception process for receiving the received information and a presentation mode determination process for determining the presentation mode to the in-vehicle user A, and executing the program stored in this ROM It is a computer composed of a CPU (Central Processing Unit) that functions as an accessible storage device and a RAM (Random Access Memory) that functions as an accessible storage device. The remote controller 35 has an utterance content acquisition unit 36 , an understanding level presentation unit 37 , a reception unit 38 , and a presentation mode determination unit 39 . The remote controller 35 can execute the functions of the utterance content acquisition unit 36, the comprehension level presentation unit 37, the reception unit 38, and the presentation mode determination unit 39 by executing programs stored in the ROM. Each function of the remote controller 35 will be described.

Voice data of the remote user B is input to the speech content acquisition unit 36 from the remote microphone 32 . The utterance content acquisition unit 36 acquires the utterance content of the remote user B from the voice of the remote user B. The utterance content acquisition unit 36 can specify the utterance content of the remote user B by performing voice recognition processing on the voice data of the remote user B. FIG. The utterance content of the remote user B is the content or theme that the remote user B is speaking. For example, the utterance content acquisition unit 36 can identify the content uttered by the remote user B based on specific words included in the voice of the remote user B or words frequently used by the remote user B. can. The utterance content acquisition unit 36 obtains data including the voice of the remote user B collected by the remote microphone 32 and the utterance content of the remote user B via the remote communication device 31. It is transmitted to the in-vehicle communication device 22 and the server communication device 41 as utterance content data. The speech content acquisition unit 36 is an example of the "speech content acquisition unit" of the present invention.

The comprehension level presentation unit 37 receives in-vehicle user A's comprehension level data for remote user B's utterance content transmitted from the server communication device 41 via the remote communication device 31 . The comprehension level presenting unit 37 controls the remote output device 34 so as to present the in-vehicle user A's comprehension level to the remote user B. FIG. For example, the comprehension level presentation unit 37 causes the in-vehicle user A's comprehension level data to be displayed on the remote display 34a. Further, for example, the comprehension degree presenting unit 37 causes the in-vehicle user A's comprehension degree data to be output by voice from the remote speaker 34b. As a result, the in-vehicle user A's level of understanding is presented to the remote user B, and the remote user B can understand the in-vehicle user A's level of understanding of the content of his/her own utterance. It should be noted that there is no particular limitation on how the comprehension level presenting unit 37 presents the comprehension level of the in-vehicle user A to the remote user B. FIG. For example, the in-vehicle user A's degree of understanding may be displayed as a numerical value or graph on the remote display 34a, or may be read out by machine voice via the remote speaker 34b. The comprehension degree presenting section 37 is an example of the "understanding degree presenting section" of the present invention.

The receiving unit 38 receives information input to the remote input device 33 . For example, when the remote user B inputs settings regarding the communication system 100 to the remote input device 33 , the reception unit 38 receives the setting information of the communication system 100 input by the remote user B. FIG. In this embodiment, the remote user B inputs an instruction to the remote input device 33 to set the mode of presentation to the in-vehicle user A as the information on the mode of presenting the contents of his or her speech to the in-vehicle user A. The receiving section 38 is an example of the "receiving section" of the present invention.

The presentation mode determination unit 39 determines the presentation mode for the in-vehicle user A based on the remote user B's instruction to set the presentation mode. For example, when remote user B issues a setting instruction to emphasize and present the content of his/her own utterance, the presentation mode determination unit 39 controls the device to be controlled and Determine device settings. For example, the presentation content determination unit 29 determines the in-vehicle display 23a as a device to be controlled, and determines specific content to be displayed on the in-vehicle display 23a. The presentation mode determining unit 39 transmits the determined presentation mode data to the vehicle interior space 20 via the remote communication device 31 .

Next, the server 40 will be explained. As shown in FIG. 1, the server 40 includes a server communication device 41, a database 42, and a server controller 43. Each device included in the server 40 is connected by, for example, Ethernet (registered trademark) or the like in order to mutually transmit and receive information.

The server communication device 41 is a device having a wireless communication function for wirelessly transmitting and receiving data between the in-vehicle communication device 22 and the remote communication device 31 . An example of the server communication device 41 is a router. The server communication device 41 connects to the Internet using a communication standard such as 4G/LTE, Wifi (registered trademark), etc., using a wireless communication function, and communicates with other servers and systems provided outside the server 40. Send and receive various data between

Data transmitted and received by the server communication device 41 will be described. The server communication device 41 receives the situation data of the vehicle 1 acquired by the situation acquisition unit 25 from the in-vehicle communication device 22 . The server communication device 41 outputs the received status data of the vehicle 1 to the database 42 and the server controller 43 .

The server communication device 41 also receives the data of the utterance content of the remote user B acquired by the utterance content acquisition unit 36 from the remote communication device 31 . The server communication device 41 outputs the received utterance content data of the remote user B to the database 42 and the server controller 43 .

Further, in-vehicle user A's comprehension degree data for remote user B's utterance content is input from server controller 43 to server communication device 41 . The server communication device 41 transmits the comprehension level data of the in-vehicle user A to the remote communication device 31 .

The database 42 stores various data of users who use the communication system 100 . The database 42 associates and stores the utterance content of the remote user B and the situation data of the vehicle 1 when the utterance content was presented to the in-vehicle user A. The situation data of the vehicle 1 when the contents of the speech of the remote user B are presented include the situation of the in-vehicle user A when the contents of the speech of the remote user B are presented to the user A in the vehicle.

The situation of in-vehicle user A when remote user B's utterance content is presented is the facial expression of in-vehicle user A when remote location user B's utterance content is presented, and the remote location user B's utterance content is presented. It includes at least one of the sound quality of the voice uttered by the in-vehicle user A when the remote location user A uttered the content and the content uttered by the in-vehicle user A when the utterance content of the remote user B was presented. The facial expression of in-vehicle user A when the content of speech of remote user B is presented is information obtained from captured image data including the facial expression of in-vehicle user A captured by in-vehicle camera 212 . In addition, the sound quality of the voice uttered by the in-vehicle user A when the content of the utterance of the remote user B is presented and the content uttered by the in-vehicle user A when the content of the utterance of the remote user B is presented are recorded by the in-vehicle microphone 213 . This is information obtained from voice data of user A inside the vehicle collected by the . Each data is input to the database 42 from the server communication device 41, and the database 42 updates the stored data each time the data is input.

For example, after in-vehicle user A and remote user B start using the communication system 100, it is assumed that an opportunity has been provided in advance for presenting the utterance content of remote-site user B to in-vehicle user A. On this occasion, the reaction (so-called reaction) of in-vehicle user A when remote user B's utterance content is presented is acquired as data by in-vehicle camera 212 and in-vehicle microphone 213 . For example, it is assumed that a remote user B utters a content related to the route of the vehicle 1 in order to guide the route of the vehicle 1 to the user A in the vehicle. When the in-vehicle user A hears the voice of the remote user B through the in-vehicle speaker 23b, the in-vehicle camera 212 captures the expression of the in-vehicle user A at that time, and the in-vehicle microphone 213 captures the voice of the in-vehicle user A at that time. Collect the voice of A. Moreover, the sensor 21 detects the situation of the vehicle 1 at that time. Each time the utterance content of the remote user B is presented to the in-vehicle user A, the in-vehicle camera 212 and the in-vehicle microphone 213 capture the reaction of the in-vehicle user A when the utterance content of the remote user B is presented as data. get. For example, when in-vehicle user A and remote user B finish using the communication system 100, image data captured by the in-vehicle camera 212, sound data collected by the in-vehicle microphone 213, and sound data detected by the sensor 21 The situation data of the vehicle 1 is transmitted to the server communication device 41 via the on-board communication device 22 . As a result, the database 42 can associate and store the utterance content of the remote user B and the situation data of the vehicle 1 when the utterance content is presented to the in-vehicle user A, thereby realizing a database. can be done. In this embodiment, the type and number of utterances of the remote user B are not particularly limited.

The server controller 43 includes a ROM (Read Only Memory) storing a program for executing a comprehension level estimation process for estimating the comprehension level of the in-vehicle user A with respect to the utterance content of the remote user B, and the program stored in the ROM. and a RAM (Random Access Memory) functioning as an accessible storage device. The server controller 43 has an understanding degree estimation unit 44 . The server controller 43 can perform the functions of the comprehension level estimation unit 44 by executing a program stored in the ROM. The function of the comprehension level estimation unit 44 will be described.

The comprehension level estimation unit 44 estimates the in-vehicle user A's level of comprehension of the utterance content of the remote user B based on the situation data of the vehicle 1 . An example of situation data of the vehicle 1 shown in FIG. 2 will be used for explanation.

For example, the comprehension level estimation unit 44 estimates the comprehension level of the in-vehicle user A based on the data indicating the noise situation of the vehicle 1 shown in FIG. For example, when the SN ratio between the voice of the remote user B and the background noise is equal to or less than a predetermined ratio, the intelligibility estimation unit 44 determines that the in-vehicle user A understands the utterance content of the remote user B in a noisy environment. , the in-vehicle user A's degree of understanding is estimated to be lower than when the SN ratio is higher than a predetermined ratio. Further, for example, the comprehension level estimation unit 44 estimates the comprehension level of the in-vehicle user A based on the data indicating the communication status of the vehicle 1 shown in FIG. For example, if the communication speed between the in-vehicle communication device 22 and the remote communication device 31 is less than or equal to a predetermined communication speed, the comprehension level estimation unit 44 determines that in an unstable communication environment where the voice is interrupted, the in-vehicle user A Assuming that remote user B understands the utterance content, the comprehension level of in-vehicle user A is estimated to be lower than when the communication speed is faster than a predetermined communication speed. Note that the predetermined ratio and the predetermined communication speed described above are an example of the ratio and an example of the communication speed set for estimating the degree of understanding of the in-vehicle user A, and even if they are experimentally determined values Alternatively, it may be a preset value.

Further, for example, the comprehension level estimation unit 44 estimates the comprehension level of the in-vehicle user A based on the data indicating the road conditions and surrounding conditions of the vehicle 1 shown in FIG. For example, when the road on which the vehicle 1 travels is congested, the comprehension level estimation unit 44 allows the in-vehicle user A, who is the driver, to understand the utterance content of the remote user B in an environment where the driver must concentrate on driving. Assuming that the road on which the vehicle 1 travels is not congested, the degree of understanding of the in-vehicle user A is estimated to be lower than when the road is not congested. Further, for example, the comprehension level estimation unit 44 estimates the comprehension level of the in-vehicle user A based on the data indicating the weather conditions around the vehicle 1 shown in FIG. For example, when the weather around the vehicle 1 is "rainy", the comprehension level estimation unit 44 determines that the in-vehicle user A who is the driver understands the utterance content of the remote user B in an environment where the driver needs to concentrate on driving. Assuming that the weather around the vehicle 1 is "sunny", the in-vehicle user A's degree of understanding is estimated to be lower than when the weather is "sunny".

Further, for example, the comprehension level estimation unit 44 estimates the comprehension level of the in-vehicle user A based on the data indicating the driving situation of the vehicle 1 shown in FIG. For example, when the vehicle 1 is parked facing backwards, the understanding level estimation unit 44 assumes that the in-vehicle user A who is the driver does not understand the utterance content of the remote user B in an environment where the driver must concentrate on driving. , the level of understanding of the in-vehicle user A is estimated to be lower than when the vehicle 1 is at a temporary stop.

Further, for example, the comprehension level estimation unit 44 estimates the comprehension level of the in-vehicle user A based on the data indicating the state of dialogue with the remote user B shown in FIG. For example, when the in-vehicle user A utters "Tell me again", the intelligibility estimation unit 44 assumes that the in-vehicle user A does not understand the content of the remote user B's utterance, and assumes that the in-vehicle user A does not understand the utterance content of the remote user B. The degree of understanding of the in-vehicle user A is estimated to be lower than in the case of uttering . Further, for example, the comprehension level estimation unit 44 estimates the comprehension level of the in-vehicle user A based on the data indicating the reaction state of the in-vehicle user A shown in FIG. For example, after the utterance content of remote user B is presented to in-vehicle user A, the intelligibility estimation unit 44 determines that, when in-vehicle user A makes a gesture of tilting his or her head, in-vehicle user A recognizes the utterance content of remote user B as It is assumed that the user A does not understand, and the degree of understanding of the user A in the car is estimated to be lower than in the case where the user A in the car makes a gesture of backhanding. Each of the methods for estimating the in-vehicle user A's degree of understanding described using the example of FIG. 2 is an example, and does not limit the method for estimating the in-vehicle user's A degree of understanding.

As an example of realizing the estimation processing by the comprehension level estimation unit 44 as described above, in order to estimate the comprehension level of the in-vehicle user A from the utterance content data of the remote user B and the situation data of the vehicle 1, A trained model that has been trained by performing machine learning using stored data is exemplified. A trained model has a trained neural network. The trained neural network was subjected to machine learning processing using teacher data so as to estimate the level of understanding of in-vehicle user A when utterance content data of remote user B and situation data of vehicle 1 were input. It is a neural network. When the utterance content data of the remote user B and the situation data of the vehicle 1 are input via the server communication device 41, the comprehension level estimation unit 44 uses a trained neural network to estimate the comprehension level of the in-vehicle user A. presume. The comprehension level estimation unit 44 transmits the in-vehicle user A's comprehension level data to the remote communication device 31 via the server communication device 41 .

Next, with reference to FIGS. 3 and 4, the flow from the communication system 100 presenting the utterance content of the remote user B to the in-vehicle user A to changing the presentation mode to the in-vehicle user A will be described. do. 3 and 4 are flowcharts showing voice output processing by the communication system 100. FIG. As shown in FIGS. 3 and 4, each step of the flowchart is executed by either the on-board controller 24, the remote controller 35, or the server controller 43. FIG.

In step S101, the remote controller 35 determines whether use of the communication system 100 has started. For example, use of the communication system 100 is started when remote user B wearing VR goggles performs an operation to activate the communication system 100 . When the remote controller 35 detects the above operation information by the remote user B, the remote controller 35 determines that the use of the communication system 100 has started. If the remote controller 35 makes an affirmative determination, the process proceeds to step S102. On the other hand, if the remote controller 35 does not receive the operation information of the remote user B and the remote controller 35 makes a negative determination, the voice output processing by the communication system 100 ends. Also, when remote user B performs an operation to end the communication system 100, the remote controller 35 makes a negative determination, and the voice output processing by the communication system 100 ends.

In step S102, the remote controller 35 determines whether or not the voice of the remote user B collected by the remote microphone 32 has been acquired. When voice data of remote user B is input from remote microphone 32, remote controller 35 determines that remote user B's voice has been acquired. On the other hand, if the voice data of the remote user B is not input from the remote microphone 32, the remote controller 35 determines that the voice of the remote user B has not been acquired. If the remote controller 35 makes an affirmative determination, the process proceeds to step S103, and if the remote controller 35 makes a negative determination, the process returns to step S101.

If the remote controller 35 makes an affirmative determination in step S102, the process proceeds to step S103. In step S103, the remote controller 35 acquires the content of the speech of the remote user B from the voice of the remote user B collected in step S102, and identifies the content and theme of the speech of the remote user B. . For example, the remote controller 35 acquires the utterance content of the remote user B from the voice of the remote user B by voice recognition processing.

In step S<b>104 , the remote controller 35 transmits the utterance content data of the remote user B acquired in step S<b>103 to the vehicle interior 20 via the remote communication device 31 . In step S<b>105 , the in-vehicle controller 24 receives the utterance content data of the remote user B transmitted from the remote communication device 31 in step S<b>104 via the in-vehicle communication device 22 .

In step S106, the in-vehicle controller 24 controls the in-vehicle output device 23 to present the utterance content of the remote user B to the in-vehicle user A in a predetermined presentation mode. By outputting the content of the speech of the remote user B from the in-vehicle output device 23, the content of the speech of the remote user B is presented to the user A in the vehicle. For example, the predetermined presentation mode includes a preset presentation mode that is initially set. The initial setting presentation mode includes a control target device to be controlled to present the content of the speech of the remote user B, and setting information of the device. Note that the predetermined presentation mode in this step is not limited to the default presentation mode. For example, when proceeding to step S106 again via step S118, which will be described later, the in-vehicle controller 24 can use the presentation mode in step S118 as the predetermined presentation mode.

In step S107, the in-vehicle controller 24 acquires the situation data of the vehicle 1 indicating the situation of the place where the user A is in the vehicle. For example, the in-vehicle controller 24 acquires traffic information 11 around the vehicle 1 , weather information 12 around the vehicle 1 , and communication information 13 around the vehicle 1 from the surrounding information server 10 . Further, for example, the in-vehicle controller 24 acquires data detected by each sensor constituting the sensor 21 from the sensor 21 mounted on the vehicle 1 .

In step S108, the in-vehicle controller 24 transmits the utterance content data of the remote user B received in step S105 and the status data of the vehicle 1 acquired in step S107 to the server 40 via the in-vehicle communication device 22.

In step S109, the server controller 43 receives, via the server communication device 41, the utterance content data of the remote user B and the situation data of the vehicle 1 transmitted from the in-vehicle communication device 22 in step S108.

In step S110, the server controller 43 estimates the understanding level of the in-vehicle user A using the utterance content data of the remote user B and the situation data of the vehicle 1 received in step S109. For example, the server controller 43 estimates the level of understanding of the in-vehicle user A using a trained neural network.

In step S111, the server controller 43 transmits the understanding level data of the in-vehicle user A estimated in step S110 to the remote space 30 via the server communication device 41. FIG.

In step S112, the remote controller 35 receives, via the remote communication device 31, the understanding level data of the in-vehicle user A transmitted from the server communication device 41 in step S111.

In step S<b>113 , the remote controller 35 determines whether it is necessary to change the manner in which the contents of the speech of the remote user B are presented to the in-vehicle user A. For example, the remote controller 35 causes the in-vehicle user A's comprehension level data received in step S112 to be displayed on the remote display 34a to present the in-vehicle user A's comprehension level to the remote user B. FIG. Remote user B, to whom in-vehicle user A's degree of understanding is presented, determines the necessity of changing the presentation mode according to in-vehicle user A's degree of understanding. When the remote user B determines that the presentation mode needs to be changed, the remote user B inputs an instruction to change the presentation mode and an instruction to set the presentation mode including information on the changed presentation mode via the remote input device 33. . When the remote user B inputs an instruction to set the presentation mode via the remote input device 33, the remote controller 35 determines that the presentation mode for the in-vehicle user A needs to be changed. On the other hand, if remote user B does not input a setting instruction even after a predetermined period of time has passed since in-vehicle user A's level of understanding was presented to in-vehicle user B, remote controller 35 presents the level of understanding to in-vehicle user A. It is determined that there is no need to change the mode. If the remote controller 35 makes an affirmative determination, the process proceeds to step S114, and if the remote controller 35 makes a negative determination, the process returns to step S101. The predetermined time is a predetermined time for determining whether or not it is necessary to change the mode of presentation to user A in the vehicle.

If the remote controller 35 makes an affirmative determination in step S113, the process proceeds to step S114. In step S114, the remote controller 35 determines the mode of presentation to the in-vehicle user A based on the processing result in step S113. For example, when the remote controller 35 receives a setting instruction from the remote user B in step S113, the remote controller 35 sends a message to the in-vehicle user A based on the changed presentation mode information input by the remote user B. Determines the presentation mode of The remote controller 35 determines a device to be controlled to present the utterance content of the remote user B to the in-vehicle user A, and the settings related to the device.

In step S<b>115 , the remote controller 35 transmits the presentation mode data determined in step S<b>114 to the vehicle interior 20 via the remote communication device 31 . In step S<b>116 , the in-vehicle controller 24 receives the presentation mode data transmitted from the remote communication device 31 in step S<b>115 via the in-vehicle communication device 22 .

In step S117, the in-vehicle controller 24 changes the mode of presentation to the in-vehicle user A based on the presentation mode data received in step S116. For example, if the presentation mode data received in step S116 includes a setting to emphasize the sound, the in-vehicle controller 24 sets the volume of the in-vehicle speaker 23b so that the volume of the in-vehicle speaker 23b increases.

In step S118, the in-vehicle controller 24 presents the utterance content of remote user B to in-vehicle user A in the presentation mode changed in step S117. For example, when emphasizing the utterance content of remote user B and presenting it to in-vehicle user A, the in-vehicle controller 24 causes the in-vehicle speaker 23b to output the voice of remote user B at the volume set in step S117. After the process of step S118 is completed, the process returns to step S107, and the voice output process by the communication system 100 shown in FIGS. 3 and 4 is repeatedly executed until a negative determination is made in step S101.

FIG. 5 is an explanatory diagram showing an example of a presentation mode for emphasizing the utterance content of the remote user B. As shown in FIG. As shown in FIG. 5, when the content of the utterance of the remote user B is accident information, the in-vehicle controller 24 causes the in-vehicle speaker 23b to output the voice of the remote user B notifying the accident information, and the utterance of the remote user B. In order to emphasize the content, a map indicating the location of the accident is displayed on the in-vehicle display 23a. When the utterance content of the remote user B is route guidance, the in-vehicle controller 24 causes the in-vehicle speaker 23b to output the voice of the remote user B guiding the route of the vehicle 1, and emphasizes the utterance content of the remote user B. In order to do so, a map showing the route of the vehicle 1 is displayed on the in-vehicle display 23a. Furthermore, when the utterance content of the remote user B is tourist information, the in-vehicle controller 24 increases the volume of the in-vehicle speaker 23b that outputs the tourist information by the voice of the remote user B, and displays the icons and characters indicating the tourist information on the vehicle. It is additionally displayed on the display 23a. Note that the presentation mode for emphasizing the utterance content of the remote user B shown in FIG. 5 is an example, and does not limit the control by the in-vehicle controller 24 .

As described above, in the present embodiment, the remote controller 35 includes the utterance content acquisition unit 36 that acquires the utterance content of the remote user B from the voice of the remote user B collected by the remote microphone 32, The in-vehicle controller 24 includes a situation acquisition unit 25 for acquiring situation data of the vehicle 1 indicating the situation of the vehicle 1 in which the in-vehicle user A is present, which is different from the remote space 30 in which the remote user B is present. , an understanding degree estimation unit 44 for estimating the understanding degree of the in-vehicle user A with respect to the utterance content of the remote user B based on the situation data of the vehicle 1 . The in-vehicle controller 24 includes an utterance content presenting unit 26 that controls the in-vehicle output device 23 so as to present the utterance content of the remote user B to the in-vehicle user A in a presentation mode based on the understanding level of the in-vehicle user A. . Since the in-vehicle user A's degree of understanding estimated based on the situation data of the vehicle 1 is for the utterance content of the remote user B, it is possible to improve the accuracy of estimating the in-vehicle user A's degree of understanding as a listener. can. As a result, it becomes easier for the in-vehicle user A, who is the listener, to understand the content of the speech of the remote user B, who is the speaker.

In this embodiment, the situation of the vehicle 1 includes at least one of the situation inside the vehicle 1, the situation outside the vehicle 1, the traveling situation of the vehicle 1, and the situation of the user A inside the vehicle. As a result, since the situation of the vehicle 1 can be grasped from a different viewpoint regarding the vehicle 1, even if the situation of the vehicle 1 changes, the degree of understanding of the in-vehicle user A can be estimated before and after the change of the situation of the vehicle 1. High accuracy can be maintained.

Furthermore, in this embodiment, the remote controller 35 controls the remote output device 34 provided in the remote space 30 so as to present the understanding level of the in-vehicle user A to the remote user B. A portion 37 is provided. As a result, the remote user B can grasp the level of understanding of the in-vehicle user A with respect to the contents of his or her utterance.

In addition, in the present embodiment, the remote controller 35 instructs the remote user B, who is presented with the level of understanding of the in-vehicle user A, to set the presentation mode from the remote input device 33 provided in the remote space 30. A reception unit 38 for receiving data, and a presentation mode determination unit 39 for determining a presentation mode for in-vehicle user A based on a presentation mode setting instruction from remote user B are provided. As a result, the remote user B can determine the mode of presentation to the in-vehicle user A after ascertaining the in-vehicle user A's level of understanding. It is possible to appropriately change the mode of presentation to in-vehicle user A, such as by emphasizing the content of user B's speech.

In addition, in the present embodiment, when the utterance content of the remote user B and the situation data of the vehicle 1 are input, the comprehension level estimation unit 44 estimates the comprehension level of the in-vehicle user A using a trained neural network. . The learned neural network is a neural network that has been subjected to machine learning processing using teacher data so as to estimate the degree of understanding of the in-vehicle user A when the utterance content of the remote user B and the situation data of the vehicle 1 are input. network. This makes it possible to estimate the in-vehicle user A's level of understanding with higher accuracy than estimating the in-vehicle user's A level of understanding based on rules.

Furthermore, in the present embodiment, the comprehension level estimation unit 44 acquires the utterance content of the remote user B by the utterance content acquisition unit 36, and based on the situation data of the vehicle 1 acquired by the situation acquisition unit 25, The degree of understanding of user A inside the vehicle is estimated. As a result, it is possible to change the mode of presentation to the in-vehicle user A so as to match the content uttered by the remote user B, so that the in-vehicle user A finds it difficult or impossible to understand the content of the utterance of the remote user B. can be suppressed.

It should be noted that the embodiments described above are described to facilitate understanding of the present invention, and are not described to limit the present invention. Therefore, each element disclosed in the above embodiments is meant to include all design changes and equivalents that fall within the technical scope of the present invention.

For example, in the above-described embodiment, the estimation method using the situation data of the vehicle 1 was described as an example of the method of estimating the in-vehicle user A's degree of understanding of the utterance content of the remote user B, but other methods have been described. Alternatively, in-vehicle user A's degree of understanding of remote user B's utterance content may be estimated by combining with an estimation method using situation data of vehicle 1 . For example, the understanding level estimation unit 44 calculates the complexity of the utterance content of the remote user B with respect to the utterance content of the remote user B, and based on the calculated complexity of the utterance content of the remote user B, In-vehicle user A's understanding of remote user B's utterance content may be estimated. The complexity of the utterance content of the remote user B is also referred to as the difficulty of the content uttered by the remote user B.

As a method for calculating the complexity of the utterance content of remote user B, for example, a technique for determining the complexity of text sentences known at the time of filing of the present application can be used. For example, the comprehension degree estimation unit 44 performs speech recognition processing on the voice of the remote user B, transcribes the voice of the remote user B, and converts the utterance content of the remote user B into text. The comprehension level estimation unit 44 calculates the complexity of the content of the utterance of the remote user B by calculating the complexity of the sentence in which the content of the utterance of the remote user B is expressed.

The comprehension degree estimation unit 44 calculates the complexity of the utterance content of the remote user B based on the words that the remote user B uses. The comprehension degree estimating unit 44 may calculate the complexity of the content of the utterance of the remote user B based on the length of each word included in the sentence in which the content of the utterance of the remote user B is expressed. For example, the comprehension degree estimating unit 44 calculates the degree of complexity of the utterance content of the remote user B higher as the length of each word included in the sentence expressing the utterance content of the remote user B is longer.

The comprehension level estimation unit 44 may also calculate the complexity of the utterance content of the remote user B based on the richness of the vocabulary used by the remote user B. FIG. For example, the comprehension degree estimating unit 44, when the text expressing the content of the utterance of the remote user B includes a technical term used in a specific field, compares with the case where the technical term is not included, The complexity of the utterance content of the remote user B is calculated to be high. Further, for example, the comprehension level estimation unit 44 divides a word that appears once in a sentence that expresses the utterance content of the remote user B by the total number of words contained in the entire sentence. The complexity of the utterance content of the local user B is calculated.

Further, the comprehension level estimation unit 44 may calculate the complexity of the utterance content of the remote user B based on the grammar that the remote user B uses. For example, the comprehension level estimation unit 44 calculates the complexity of the utterance content of the remote user B based on the number of parts of speech of words and the number of sentence elements such as subjects and predicates included in the sentence expressing the utterance content of the remote user B. Calculate degrees.

When the complexity of the utterance content of the remote user B is calculated by the above-described method, the comprehension level estimation unit 44 calculates the utterance of the remote user B based on the calculated complexity of the utterance content of the remote user B. The level of in-vehicle user A's understanding of the content is estimated. For example, the comprehension level estimation unit 44 estimates the comprehension level of the in-vehicle user A to be low because the higher the complexity of the utterance content of the remote user B, the more difficult it is for the in-vehicle user A to understand the utterance content of the remote user B. . On the other hand, the comprehension level estimation unit 44 estimates that the comprehension level of the in-vehicle user A is high, assuming that the lower the complexity of the utterance content of the remote user B, the easier it is for the in-vehicle user A to understand the utterance content of the remote user B. .

In this way, by estimating the degree of understanding of the in-vehicle user A with respect to the utterance content of the remote-site user B based on the complexity of the utterance content of the remote-site user B, the degree of understanding of the in-vehicle user A can be Since it is estimated based on the complexity of the object to be understood, the accuracy of estimating the in-vehicle user A's degree of understanding can be improved.

Further, for example, in the above-described embodiment, as a method of determining the mode of presentation to in-vehicle user A, the in-vehicle user A's level of understanding is presented to remote user B, and then, based on a setting instruction from remote user B, in-vehicle Although the method of determining the mode of presentation to user A has been described as an example, the mode of presentation to in-vehicle user A may be determined by other methods. For example, the remote controller 35 may include a presentation mode determination unit that determines the mode of presentation to the in-vehicle user A based on the level of understanding of the user A in the vehicle. As a result, the manner of presentation to the user A in the vehicle is automatically determined based on the degree of understanding of the user A in the vehicle, so that the manner of presentation to the user A in the vehicle can be changed smoothly. In addition, since the operation by the remote user B becomes unnecessary, the user's convenience can be improved.

Further, in the above-described modified example, for example, if the in-vehicle user A's degree of understanding of the utterance content of the remote user B is lower than a predetermined reference level of understanding, the presentation mode determination unit selects the remote A presentation mode that emphasizes the utterance content of the local user B may be determined. As a result, when the in-vehicle user A's degree of understanding is relatively low, the utterance content of the remote user B can be emphasized and presented to the in-vehicle user A, so that the in-vehicle user A can It becomes easier to understand the content of the utterance of the remote user B.

Further, for example, in the above-described embodiment, an estimation method using a trained neural network was described as an example of a method for estimating the degree of comprehension with respect to the utterance content by the intelligibility estimating unit 44. Comprehension may be estimated. For example, the comprehension estimation unit 44 includes an input layer and an output layer. A neural network that uses the degree of understanding of user A in the car for the content of the utterance, a machine learning unit that learns the neural network using actual values of input data and output data as teacher data, and a new neural network learned by the machine learning unit. It may have an estimating unit that receives input data and estimates the in-vehicle user A's degree of understanding as new output data for the input data. As a result, the comprehension degree estimating unit 44 learns the correlation between the utterance content of the remote user B and the situation of the vehicle 1 each time the utterance content of the remote user B and the situation data of the vehicle 1 are input. , the accuracy of estimating the in-vehicle user A's degree of understanding can be further improved.

Further, for example, in the above-described embodiment, as the timing for estimating the degree of understanding of the in-vehicle user A with respect to the utterance content of the remote user B, the timing after acquiring the utterance content of the remote user B has been described as an example. Before acquiring the utterance content of local user B, the in-vehicle user A's degree of understanding may be estimated. The comprehension degree estimation unit 44 determines whether the remote user B has spoken based on the situation data of the vehicle 1 acquired by the situation acquisition unit 25 before the speech content acquisition unit 36 acquires the content of the speech of the remote user B. It is also possible to estimate in advance the level of understanding of the in-vehicle user A with respect to the contents of the utterance of the remote user B in the case where the remote location user B speaks. For example, it is assumed that the comprehension degree estimating unit 44 identifies from the situation data of the vehicle 1 that the communication speed between the on-vehicle communication device 22 and the remote communication device 31 is equal to or lower than a predetermined communication speed. In this case, the comprehension level estimation unit 44 predicts that it is difficult for in-vehicle user A to understand the content of remote user B's utterance when remote user B speaks because of an unstable communication environment where voice is interrupted. As a result, the comprehension level of the in-vehicle user A is presumed to be lower than when the communication speed is higher than the predetermined communication speed. By estimating the degree of understanding of the in-vehicle user A before the remote user B speaks, it is possible to prevent the in-vehicle user A from having difficulty understanding or being unable to understand the speech content of the remote user B from the beginning. .

Further, for example, in the above-described embodiment, a case where machine learning is performed using situation data of the vehicle 1 as input data as a trained neural network possessed by the comprehension level estimation unit 44 has been described as an example. The neural network of the unit 44 may be made to learn by inputting situation data of each vehicle from not only the vehicle 1 but also a plurality of vehicles. Further, in each of the plurality of vehicles, the in-vehicle controller may be provided with an understanding level estimation unit. In this case, each vehicle is made to learn by inputting vehicle situation data, and each learning result is transmitted to the server. It may be configured to

Further, for example, in the above-described embodiment, the in-vehicle controller 24 includes the situation acquisition unit 25 and the utterance content presentation unit 26, and the remote controller 35 includes the utterance content acquisition unit 36, the comprehension level presentation unit 37, the reception unit 38, and the presentation mode. Although the case where the determining unit 39 is provided and the server controller 43 is provided with the comprehension degree estimating unit 44 has been described as an example, the configuration of the communication support device of the present invention is not limited to this. For example, the utterance content acquisition unit, the situation acquisition unit, the comprehension level estimation unit, the utterance content presentation unit, the understanding level presentation unit, the reception unit, and the presentation mode determination unit may be provided in a specific controller or device. For example, the in-vehicle controller 24 may contain these blocks, the remote controller 35 may contain these blocks, or the server 40 may contain these blocks. In the case of the configuration in which each block of the communication support device is aggregated in this way, the device having each block transmits and receives data to and from a device having none of the blocks via the communication device. Effects similar to those of the embodiment can be obtained.

DESCRIPTION OF SYMBOLS 100... Communication system 10... Peripheral information server 11... Traffic information 12... Weather information 13... Communication information 20... Vehicle space 21... Sensor 211... Vehicle interior sensor 212... Vehicle interior camera 213... Vehicle interior microphone 214... Steering angle sensor 215... Pedal sensor 221 External sensor 222 External camera 223 External microphone 224 Milliwave radar 225 Ultrasonic sonar 226 GPS
22 In-vehicle communication device 23 In-vehicle output device 23a In-vehicle display 23b In-vehicle speaker 24 In-vehicle controller 25 Situation acquisition unit 26 Speech content presentation unit 30 Remote space 31 Remote communication device 32 Remote microphone 33 Remote input device 34 Remote output device 34a Remote display 34b Remote speaker 35 Remote controller 36 Speech content acquisition unit 37 Comprehension level presentation unit 38 Reception unit 39 Presentation mode determination unit 40 Server 41 Server communication device 42 Database 43 Server controller 44 Understanding level estimation unit

Claims (13)

an utterance content acquisition unit that acquires the utterance content of the first user from the first user's voice collected by a sound collecting device;
a situation acquisition unit that acquires situation data of a second place that indicates the situation of a second place where a second user is present, which is different from the first place where the first user is present;
an understanding degree estimation unit that estimates the second user's degree of understanding of the utterance content of the first user based on the situation data of the second location;
Speech content presentation for controlling an output device provided at the second location so as to present the speech content of the first user to the second user in a presentation mode based on the degree of understanding of the second user. A communication support device comprising: the first location is a remote location away from the vehicle;
the first user is a remote user at the remote location;
the second location is the vehicle;
the second user is an in-vehicle user on board the vehicle;
2. The communication support device according to claim 1, wherein the situation data of the second location is vehicle situation data indicating the situation of the vehicle. The comprehension level estimating unit calculates the complexity of the utterance content for the utterance content of the remote user, and estimates the in-vehicle user's comprehension level based on the calculated complexity of the utterance content. Item 3. The communication support device according to item 2. 4. The communication according to claim 2, wherein the situation of the vehicle includes at least one of an indoor situation of the vehicle, an outdoor situation of the vehicle, a running situation of the vehicle, and a situation of the user inside the vehicle. support equipment. Claims 2 to 7, further comprising an understanding level presenting unit for controlling an output device provided at the remote location so as to present the understanding level of the in-vehicle user with respect to the utterance content of the remote location user to the remote location user. 5. The communication support device according to any one of 4. a receiving unit that receives, from the input device provided at the remote location, an instruction to set the presentation mode by the remote user presented with the level of understanding of the in-vehicle user;
6. The communication support device according to claim 5, further comprising a presentation mode determination unit that determines the presentation mode based on the setting instruction. 5. The communication support device according to any one of claims 2 to 4, further comprising a presentation mode determination unit that determines the presentation mode based on the in-vehicle user's degree of understanding of the utterance content of the remote user. The presentation mode determination unit emphasizes the utterance content of the remote user as the presentation mode when the understanding level of the in-vehicle user with respect to the utterance content of the remote user is lower than a predetermined standard comprehension level. 8. The communication support device according to claim 7, wherein the presentation mode is determined. The comprehension level estimation unit
An input layer and an output layer are included, and the input data to the input layer are the utterance content of the remote user and the situation data of the vehicle, and the output data from the output layer is the interior of the vehicle for the utterance content of the remote user. a neural network as the level of understanding of the user;
a machine learning unit for learning the neural network using actual values of the input data and the output data as teacher data;
The new input data is input to the neural network learned by the machine learning unit, and the in-vehicle user's degree of understanding of the utterance content of the remote user is estimated as the new output data for the input data. 9. The communication support device according to any one of claims 2 to 8, comprising an estimating unit that When the utterance content of the remote user and the situation data of the vehicle are input, the comprehension level estimating unit estimates the in-vehicle user's comprehension level of the utterance content of the remote user using a trained neural network. presume,
The trained neural network, when input with the utterance content of the remote user and the situation data of the vehicle, stores training data so as to estimate the in-vehicle user's degree of understanding of the utterance content of the remote user. 9. The communication support device according to any one of claims 2 to 8, which is a neural network subjected to machine learning processing. The comprehension degree estimating unit, after the utterance content acquisition unit acquires the utterance content of the remote user, based on the situation data of the vehicle acquired by the situation acquisition unit, 11. The communication support device according to any one of claims 2 to 10, which estimates the level of understanding of the in-vehicle user with respect to the utterance content. The comprehension degree estimating unit, before the utterance content is acquired by the utterance content acquisition unit, is based on the vehicle situation data acquired by the situation acquisition unit, and calculates the above-mentioned 11. The communication support device according to any one of claims 2 to 10, wherein said in-vehicle user's degree of understanding of the utterance content of the remote user is presumed. A communication support method executed by a controller, comprising:
Acquiring the utterance content of the first user from the first user's voice collected by a sound collecting device;
obtaining a context of the second location that indicates a context of a second location of the second user that is different from the first location of the first user;
estimating the degree of understanding of the second user with respect to the utterance content of the first user based on the situation data of the second location;
A communication support method for controlling an output device provided at the second location so as to present the utterance content of the first user to the second user in a presentation mode based on the degree of understanding of the second user. .

Images