WO2024079942A1

WO2024079942A1 - Information processing apparatus, control method, program, and storage medium

Info

Publication number: WO2024079942A1
Application number: PCT/JP2023/022911
Authority: WO
Inventors: 高志飯澤; 廣人根岸; 昂生加峯
Original assignee: パイオニア株式会社
Priority date: 2022-10-12
Filing date: 2023-06-21
Publication date: 2024-04-18

Abstract

An on-vehicle machine 1 or a server device 5 is an information processing apparatus used in a system that intermediates communication between a driver of a target vehicle which is a movable body and a user of an out-of-vehicle terminal 2 which is present outside the target vehicle, and comprises a calculation means, a detection means, and a display control means. The calculation means calculates a workload value which is an index value relating to the degree of driving load (workload) of the driver of the target vehicle. The detection means detects a cause of increasing the driving load when the workload value indicates that the driving load is higher than a prescribed reference. The display control means displays, on the out-of-vehicle terminal 2, information relating to the cause of increasing the driving load.

Description

Information processing device, control method, program, and storage medium

The present invention relates to a system that mediates communication between a driver of a mobile object and a user outside the mobile object.

Technologies that enable conversations between a vehicle driver and a person outside the vehicle have been known for some time. For example, Patent Document 1 discloses a conversation providing system that enables a call between the vehicle driver and a conversation partner by communicating with an in-vehicle system via a communication network.

JP 2017-138277 A

When the driver is talking to someone outside the vehicle, it is difficult for the person outside the vehicle to understand the driver's psychological state, so even when the driver is under high driving stress, the conversation cannot be interrupted, which can increase the driver's driving stress.

In consideration of the above-mentioned problems, one of the objectives of the present invention is to provide an information processing device that can effectively mediate communication between a driver of a moving body and a user outside the moving body.

The invention described in claim 1 is
An information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, comprising:
A calculation means for calculating an index value relating to a degree of a driving load of a driver of the moving body;
a detection means for detecting a cause of an increase in the driving load when the index value indicates that the driving load is higher than a predetermined standard;
a display control means for displaying information relating to the cause on a terminal device of the user;
The present invention is characterized in that the information processing device has the following features.

The invention described in claim 10 is as follows:
A control method executed by an information processing device used in a system that mediates communication between a driver of a moving body and a user present outside the moving body, comprising:
A calculation step of calculating an index value related to a degree of a driving load of a driver of the moving body;
a detection step of detecting a cause of an increase in the driving load when the index value indicates that the driving load is higher than a predetermined standard;
a display control step of displaying information about the cause on a terminal device of the user;
The control method is characterized by having the following features.

The invention described in claim 11 further comprises:
A program executed by a computer of an information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, comprising:
A calculation means for calculating an index value relating to a degree of a driving load of a driver of the moving body;
a detection means for detecting a cause of an increase in the driving load when the index value indicates that the driving load is higher than a predetermined standard;
The program causes a computer to function as a display control means for displaying information about the cause on a terminal device of the user.

1 illustrates a configuration example of a driving call system according to an embodiment. 2 shows an example of a schematic configuration of an in-vehicle device. 2 shows an example of a schematic configuration of an external terminal. 13 shows a first example of a display on an external terminal when a target vehicle is traveling. 13 shows a second example of a display on the external terminal when the target vehicle is traveling. 13 shows a third example of a display on the external terminal when the target vehicle is traveling. 4 is an example of a flowchart showing a procedure of a process executed by an in-vehicle device. 13 shows a fourth display example of the external terminal according to the modified example. 13 shows a configuration example of a driving call system according to a modified example. 2 illustrates an example of a schematic configuration of a server device.

In one preferred embodiment of the present invention, an information processing device used in a system that mediates communication between a driver of a mobile body and a user outside the mobile body includes a calculation means for calculating an index value relating to the degree of driving load on the driver of the mobile body, a detection means for detecting a cause of an increase in the driving load when the index value indicates that the driving load is higher than a predetermined standard, and a display control means for displaying information relating to the cause on the user's terminal device.

The information processing device described above can conveniently allow a user outside the vehicle to recognize the causes of the driver's increased driving load. This allows the user to communicate with the driver according to the causes of the increased driving load.

In one aspect of the information processing device, the calculation means calculates the index value based on the current state of at least one of the driving behavior of the moving body, the driver, or the driving environment. In a preferred example, the detection means calculates a score that evaluates the current state of at least one of the driving behavior of the moving body, the driver, or the driving environment for each element related to the driving load, and calculates the index value based on the score. This allows the information processing device to accurately calculate the index value of the driving load.

In another aspect of the information processing device, the detection means determines the factor that is the cause based on the score. This aspect enables the information processing device to accurately detect the cause of the increased driving load.

In another aspect of the information processing device, the display control means causes the terminal device to display information related to the cause and information indicating the magnitude of the index value. With this aspect, the information processing device can conveniently present the degree of driving load, together with the cause of the increased driving load, to a user outside the mobile body.

In another aspect of the information processing device, the display control means causes the terminal device to display a line representing the trajectory of the driver's line of sight on an image captured from the moving body of the outside of the moving body. This aspect allows the user of the terminal device to conveniently grasp the direction in which the driver is not paying much attention, and the user of the terminal device can assist the driver in checking for safety in the direction in which the driver is not paying much attention by communicating with the driver. In a preferred example, the display control means may change the display mode of the line depending on the index value. In another preferred example, the display control means may display the line when the index value indicates that the driving load is higher than a predetermined standard, and may hide the line when the index value indicates that the driving load is equal to or lower than the predetermined standard.

In another aspect of the information processing device, the display control means highlights in the captured image an object to which the driver should direct his or her gaze. This aspect enables the user of the terminal device to identify an object to which the driver is not directing his or her gaze and to communicate with the driver to encourage the driver to direct his or her gaze to the object.

In another preferred embodiment of the present invention, a control method is executed by an information processing device used in a system that mediates communication between a driver of a mobile body and a user outside the mobile body, the control method comprising a calculation step of calculating an index value relating to the degree of driving load on the driver of the mobile body, a detection step of detecting a cause of increased driving load when the index value indicates that the driving load is higher than a predetermined standard, and a display control step of displaying information relating to the cause on the user's terminal device. By executing this control method, the information processing device can allow a user outside the mobile body to appropriately recognize the cause of increased driving load on the driver.

In yet another embodiment of the present invention, a program executed by a computer of an information processing device used in a system that mediates communication between a driver of a mobile body and a user outside the mobile body causes the computer to function as a calculation means for calculating an index value relating to the degree of driving load on the driver of the mobile body, a detection means for detecting a cause of increased driving load when the index value indicates that the driving load is higher than a predetermined standard, and a display control means for displaying information relating to the cause on the user's terminal device. By executing this program, the computer of the information processing device can conveniently allow a user outside the mobile body to recognize the cause of increased driving load on the driver. Preferably, the program is stored in a storage medium.

Below, a preferred embodiment of the present invention will be described with reference to the drawings.

(1) System Configuration Fig. 1 shows a configuration example of a driving call system according to a first embodiment. The driving call system has an in-vehicle device 1 installed in a vehicle and an external terminal 2 used by a person outside the vehicle. In the driving call system, the in-vehicle device 1 and the external terminal 2 communicate with each other to mediate a voice call between the driver of the vehicle and the user of the external terminal 2, and the user of the external terminal 2 talks to the driver of the vehicle while checking various information obtained from the vehicle on the external terminal 2.

The on-board device 1 moves with the vehicle and performs processes to realize a call between the driver of the vehicle and the user of the external terminal 2. Hereinafter, the vehicle equipped with the on-board device 1 is also referred to as the "target vehicle". The on-board device 1 performs data communication with the external terminal 2 via a communication network 3 such as the Internet or a dedicated communication network. Data exchanged between the on-board device 1 and the external terminal 2 includes voice data generated during a call between the driver of the target vehicle and the user of the external terminal 2, and display instruction data for displaying information about the target vehicle or the driver. In this embodiment, the on-board device 1 transmits display instruction data including information about the driver's driving load (workload) (also referred to as "workload-related information") to the external terminal 2.

The vehicle-mounted device 1 may be a navigation device that is installed in the target vehicle and provides route guidance to a set destination, or may be a mobile terminal such as a smartphone. The vehicle-mounted device 1 may also be incorporated in the target vehicle. The vehicle-mounted device 1 is an example of an "information processing device." The target vehicle is an example of a "mobile body."

The external terminal 2 is a terminal operated by a person outside the target vehicle, and performs data communication with the in-vehicle device 1 via the communication network 3. The external terminal 2 is, for example, a mobile terminal such as a smartphone. For example, the external terminal 2 establishes communication with the in-vehicle device 1 while the target vehicle is being driven, and exchanges voice data with the in-vehicle device 1 for a call between the driver and the user of the external terminal 2. The external terminal 2 also receives display instruction data including workload-related information from the in-vehicle device 1 during the call, and displays the driver's workload state, etc. based on the display instruction data. In this way, the external terminal 2 allows the user of the external terminal 2 to recognize the driver's workload state, and preferably provides information that can be used to determine a convenient timing for talking to the driver. The external terminal 2 is an example of a "terminal device".

The communication between the in-vehicle unit 1 and the external terminal 2 may be realized by being relayed by a server device (not shown). Even if the in-vehicle unit 1 and the external terminal 2 establish direct communication, the in-vehicle unit 1 and/or the external terminal 2 may exchange information (e.g., communication address information) necessary for the in-vehicle unit 1 and the external terminal 2 to establish communication with the server device. In these cases, the server device performs the processes necessary for the in-vehicle unit 1 and the external terminal 2 to perform data communication (including authentication processes for the in-vehicle unit 1 and the external terminal 2).

(2) Device Configuration Fig. 2 shows an example of a schematic configuration of the vehicle-mounted device 1. The vehicle-mounted device 1 mainly has a communication unit 11, a storage unit 12, an input unit 13, a control unit 14, a sensor group 15, a display unit 16, and a sound output unit 17. The elements in the vehicle-mounted device 1 are connected to each other via a bus line 10.

The communication unit 11 performs data communication with other terminals based on the control of the control unit 14. For example, the communication unit 11 may receive map data for updating the map DB (Database) 4 from a map management server (not shown).

The storage unit 12 is composed of various types of memory such as RAM (Random Access Memory), ROM (Read Only Memory), and non-volatile memory (including hard disk drives, flash memories, etc.). The storage unit 12 stores programs for the in-vehicle device 1 to execute predetermined processes. The above-mentioned programs may include application programs for making calls, and application programs for causing the external terminal 2 to display maps and images captured in the target vehicle. The storage unit 12 is also used as a working memory for the control unit 14. The programs executed by the in-vehicle device 1 may be stored in a storage medium other than the storage unit 12.

The memory unit 12 also stores a map DB (Data Base) 4. Various data necessary for route guidance is recorded in the map DB 4. The map DB 4 is data necessary for map display based on a specified position such as the current position of the target vehicle. The map DB 4 is a database that includes, for example, road data that represents a road network by a combination of nodes and links, and facility data that indicates facilities that are candidates for a destination, a stop-off point, or a landmark. The map DB 4 may be updated based on information received by the communication unit 11 from a map management server under the control of the control unit 14.

The input unit 13 is a button, a touch panel, a remote controller, a voice input device, etc. that the user can operate. The display unit 16 is a display or the like that displays information based on the control of the control unit 14. The sound output unit 17 is a speaker or the like that outputs sound based on the control of the control unit 14.

The sensor group 15 includes various sensors that sense the state of the target vehicle or the environment outside the vehicle. The sensor group 15 includes an exterior camera 51, a driver camera 52, a vehicle behavior detector 53, and a biosensor 54.

The exterior camera 51 is one or more cameras that capture images outside the target vehicle, such as the area in front of the target vehicle, and generates images captured at predetermined time intervals (also called "exterior image"). The driver camera 52 is a camera that is installed so that the driver's face is included in the capture range, and generates images captured at predetermined time intervals (also called "driver image").

The vehicle behavior detector 53 generates detection signals indicating the behavior of the target vehicle, such as the current position, vehicle speed, acceleration, steering angle, etc. The vehicle behavior detector 53 includes, for example, a GNSS (Global Navigation Satellite System) receiver, a gyro sensor, an IMU (Inertial Measurement Unit), a vehicle speed sensor, an acceleration sensor, a steering angle sensor, etc.

The biosensor 54 is one or more sensors that generate biosignals that indicate the driver's biological phenomena, such as heart rate and amount of sweat.

The sensor group 15 may include various external sensors (including cameras, lidars, radars, ultrasonic sensors, infrared sensors, sonar, etc.) and internal sensors in addition to the exterior camera 51, driver camera 52, vehicle behavior detector 53, and biosensor 54.

The control unit 14 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the entire vehicle-mounted device 1. The control unit 14 functions as a "calculation means," a "detection means," a "display control means," and a computer that executes programs, etc.

The processing executed by the control unit 14 is not limited to being realized by software programs, but may be realized by any combination of hardware, firmware, and software. The processing executed by the control unit 14 may also be realized by using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcomputer. In this case, the program executed by the control unit 14 in this embodiment may be realized by using this integrated circuit.

2 is an example, and various modifications may be made to the configuration shown in FIG. 2. For example, instead of the map DB 4 being stored in the memory unit 12, the control unit 14 may receive map information from a map management server (not shown) via the communication unit 11. In another example, at least one of the input unit 13, the display unit 16, and the sound output unit 17 may be provided in the target vehicle as an external device of the vehicle-mounted unit 1, and may supply the generated signal to the vehicle-mounted unit 1. Also, at least some of the sensors in the sensor group 15 may be sensors mounted on the target vehicle. In this case, the vehicle-mounted unit 1 may acquire information output by a sensor mounted on the target vehicle from the target vehicle based on a communication protocol such as CAN (Controller Area Network).

FIG. 3 shows an example of the schematic configuration of the external terminal 2. The external terminal 2 mainly has a communication unit 21, a memory unit 22, an input unit 23, a control unit 24, a sensor group 25, a display unit 26, and a sound output unit 27. The elements in the external terminal 2 are connected to each other via a bus line 20.

The communication unit 21 performs data communication with other terminals under the control of the control unit 24. The storage unit 22 is composed of various types of memory such as RAM, ROM, and non-volatile memory. The storage unit 22 stores programs for the external terminal 2 to execute predetermined processes. The above-mentioned programs may include application programs for calling the driver, displaying information related to the driving of the target vehicle of the in-vehicle unit 1 (including displaying maps and various captured images), and displaying information related to the driver's workload when communication with the in-vehicle unit 1 is established. The storage unit 22 is also used as a working memory for the control unit 24. The programs executed by the external terminal 2 may be stored in a storage medium other than the storage unit 22.

The input unit 23 is a button, a touch panel, a remote controller, a voice input device, etc. that the user can operate. The display unit 26 is a display or the like that displays information based on the control of the control unit 24. The sound output unit 27 is a speaker or the like that outputs sound based on the control of the control unit 24. The sensor group 25 includes an internal sensor that senses the state of the external terminal 2, and an external sensor that senses the state of the external world of the external terminal 2.

The control unit 24 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the entire external terminal 2. Note that the configuration of the external terminal 2 shown in FIG. 3 is an example, and various modifications may be made to the configuration shown in FIG. 3.

(3) Display of Workload-Related Information Next, a display of workload-related information regarding the driving load (workload) of the driver of the target vehicle will be described. In summary, when the driver's workload is equal to or greater than a predetermined level, the in-vehicle device 1 identifies the cause of the increased workload (also called a "workload increase cause") and displays the identified workload increase cause on the external terminal 2. This allows the in-vehicle device 1 to make the user of the external terminal 2 aware of the workload increase cause, and the user of the external terminal 2 to take appropriate action according to the workload increase cause.

(3-1) Calculation of Workload Value First, a method for calculating the workload value, which is a value (score) indicating the degree of the workload, will be described. Hereinafter, for the sake of convenience, the workload value indicates that the higher the workload value, the higher the degree of the driver's workload.

The vehicle-mounted device 1 calculates a workload value based on a score (also called an "element-specific score") that evaluates the current state for each element related to the workload (also called "workload-related element"). The workload-related element is an element that represents the state related to the target vehicle, the driver, or the driving environment. Examples of workload-related elements include elements related to the environment of the road (driving road) on which the target vehicle is traveling (also called "road environment elements"), elements related to driving behavior (driving operations) that affect the workload (also called "driving behavior elements"), and elements related to the driver's physical condition (also called "driver physical condition elements"). In other words, the workload-related element is an element that is a candidate for the cause of an increase in workload. Note that the workload-related element may be an element obtained by further subdividing at least one of the road environment elements, driving behavior elements, and driver physical condition elements.

Therefore, for example, while communication for a call is established with the external terminal 2, the vehicle-mounted device 1 calculates a score (also called an "element-specific score") that evaluates the current state of the target vehicle, driver, or driving environment for each workload-related element. The vehicle-mounted device 1 then determines the total value, average value, or other representative value of the calculated element-specific scores as the workload value. Note that the workload-related elements for which the element-specific score is calculated may be all of the road environment elements, driving behavior elements, and driver physical condition elements, or any one of these, or any two of them. In addition, the element-specific score may preferably be weighted according to the degree of influence of each workload-related element on the workload. In this case, for example, the weighting coefficient multiplied by the element-specific score of an element with a high degree of influence on the workload is a larger value, and information indicating these weighting coefficients is stored in advance in the storage unit 12, etc.

Here, when the vehicle-mounted device 1 calculates the element-specific score of the road environment element, for example, the vehicle-mounted device 1 calculates the element-specific score by taking into account at least one of the congestion degree, width, speed limit, and visibility of the road to be traveled. In this case, for example, the vehicle-mounted device 1 refers to the road information in the map DB 4 or the road traffic information acquired by the communication unit 11, and identifies each degree of the congestion degree, width, speed limit, visibility, etc. of the road to be traveled, and converts each identified degree into an element-specific score by referring to a predetermined formula or table. The predetermined formula or table is stored in the storage unit 12 or the like in advance. Note that the road environment element may be subdivided into multiple workload-related elements. In this case, for example, the vehicle-mounted device 1 may regard each of the congestion degree, width, speed limit, and visibility of the road to be traveled as a workload-related element, and calculate the element-specific score for each of these.

When calculating the element-specific scores of the driving behavior elements, the vehicle-mounted device 1 determines whether the current driving behavior corresponds to a driving behavior such as a right turn, a left turn, merging, or a temporary stop, and calculates the element-specific scores according to the determined driving behavior. In this case, for example, the vehicle-mounted device 1 determines the element-specific scores of the driving behavior elements by referring to a table or the like that associates the corresponding driving behavior with the element-specific scores of the driving behavior elements. Here, the vehicle-mounted device 1 may determine the current driving behavior based on information obtained by the route guidance process (for example, the current position of the target vehicle and the route to the set destination), or may determine the current driving behavior based on a signal indicating the state of the turn signal or the steering state obtained from the target vehicle. The vehicle-mounted device 1 may also determine the element-specific scores of the driving behavior elements based on the presence or absence of auto-cruise driving (or the presence or absence of other automatic driving functions). The driving behavior elements may be subdivided into multiple workload-related elements. For example, whether or not a right turn was made, whether or not a left turn was made, whether or not a merge was made, whether or not a temporary stop was made, and whether or not auto-cruise was made are each set as workload-related elements, and the vehicle-mounted device 1 determines the element-specific score for each of these workload-related elements by referring to a predetermined table or the like, depending on the presence or absence of the corresponding driving action.

When calculating the element-specific score of the driver's physical condition element, the vehicle-mounted device 1 calculates the element-specific score by taking into account at least one of the driver's continuous driving time, drowsiness level (alertness level), and other bioindicators that affect driving. In this case, the vehicle-mounted device 1, for example, identifies the drowsiness level or other bioindicators that affect driving based on the signal output by the biosensor 54. The vehicle-mounted device 1 then refers to a predetermined formula or table and converts the identified continuous driving time and one or more bioindicators into an element-specific score of the driver's physical condition element. Note that the driver's physical condition element may be subdivided into multiple workload-related elements. In this case, for example, the vehicle-mounted device 1 may regard each of the drowsiness level and the continuous driving time as workload-related elements and calculate the element-specific scores for these, respectively.

By doing the above, the vehicle-mounted device 1 can determine the workload value by taking into account various factors that affect the workload.

(3-2) Detection of Cause of Increase in Workload Next, a method for detecting the cause of an increase in workload will be described.

The vehicle-mounted device 1 executes a process for detecting the cause of an increase in workload when the calculated workload value becomes greater than a predetermined threshold value (also called the "workload threshold value"). The workload threshold value is, for example, determined in advance and stored in the storage unit 12. The workload threshold value is an example of a "predetermined standard."

Then, when the workload value becomes greater than the threshold value, the vehicle-mounted device 1 determines the proportion of the workload value that the element-specific score of each workload-related element accounts for (also called the "workload value occupancy proportion"). Next, the vehicle-mounted device 1 adds up the workload value occupancy proportions in order from the workload-related element with the highest workload value occupancy proportion until the total workload value occupancy proportion reaches a predetermined proportion. Then, when the total workload value occupancy proportion reaches a predetermined proportion, the vehicle-mounted device 1 detects the workload-related element with the added workload value occupancy proportion as the cause of the workload increase. The above-mentioned predetermined proportion is, for example, determined in advance and stored in the memory unit 12.

For example, the above-mentioned specified ratio is 30%. In this case, the vehicle-mounted device 1 determines whether the workload value occupancy ratio of the workload-related element with the first workload value occupancy ratio is 30% or more. If the workload value occupancy ratio of the workload-related element with the first workload value occupancy ratio is 30% or more, the vehicle-mounted device 1 detects the workload-related element as a cause of the workload increase. On the other hand, if the workload value occupancy ratio of the workload-related element with the first workload value occupancy ratio is not 30% or more, the vehicle-mounted device 1 determines whether the total value of the workload value occupancy ratios of the workload-related elements with the first and second workload value occupancy ratios is 30% or more. If the total value of the workload value occupancy ratios of the workload-related elements with the first and second workload value occupancy ratios is 30% or more, the vehicle-mounted device 1 detects these workload-related elements as a cause of the workload increase. On the other hand, if the workload value occupancy ratios of the workload-related elements with the first and second highest workload value occupancy ratios are not 30% or more, the vehicle-mounted device 1 determines whether the total value of the workload value occupancy ratios of the workload-related elements with the first to third highest workload value occupancy ratios is 30% or more. The vehicle-mounted device 1 repeatedly executes this process until the total value of the workload value occupancy ratios of the workload-related elements is 30% or more, and at the point when the total value is 30% or more, detects the workload-related element with the added workload value occupancy ratio as the cause of the workload increase.

As a result, the vehicle-mounted device 1 can appropriately detect one or more workload-related factors that cause an increase in the workload. In other words, one or more of the following factors are identified as causes of increased workload: the environment of the road during driving (degree of congestion, width, speed limit, visibility), driving behavior, and the driver's physical condition (continuous driving time, degree of drowsiness based on biometric information, etc.).

(3-3) Display Example Fig. 4 shows a first display example of the external terminal 2 when the target vehicle of the in-vehicle device 1 is traveling. The external terminal 2 has established communication with the in-vehicle device 1, and displays the display screen shown in Fig. 4 on the display unit 26 based on the display instruction data received from the in-vehicle device 1.

Here, the display screen shown in FIG. 4 has an outside-vehicle image display area 60, a workload-related information display area 61, and a map display area 62.

The external terminal 2 displays the latest external image (video) generated by the external camera 51 on the external image display area 60. In this case, the vehicle-mounted device 1 transmits display instruction data including the latest external image generated by the external camera 51 to the external terminal 2.

In addition, the external terminal 2 displays a display window 71 and an indicator 72 in the workload-related information display area 61 based on the workload-related information included in the display instruction data sent by the vehicle-mounted device 1.

The display window 71 displays the level of the workload value calculated by the in-vehicle unit 1 and the cause of the workload increase detected by the in-vehicle unit 1. Here, since the workload value calculated by the in-vehicle unit 1 is at the highest level (for example, the highest level when divided into five levels), the display window 71 displays "High workload", which indicates the highest level. In addition, since the workload value exceeds the workload threshold, the in-vehicle unit 1 detects the cause of the workload increase (here, turning right at an intersection), and the external terminal 2 displays "Turn right at intersection" in the display window 71.

Indicator 72 also indicates the degree of workload, and the greater the workload value calculated by in-vehicle device 1, the further to the right the gauge extends. Here, since the workload value is close to the maximum value, the gauge of indicator 72 extends to the near right end. The color of the gauge may change depending on the workload value. For example, the higher the workload value, the more noticeable the color of the gauge may be. The "High Workload" display in display window 71 and indicator 72 are examples of "information indicating the magnitude of the index value."

The external terminal 2 also displays a map of the vicinity of the current position of the target vehicle on the map display area 62. The in-vehicle device 1 generates display instruction data for the external terminal 2 to display the above-mentioned map based on the state of the target vehicle estimated based on the vehicle behavior detector 53, the map DB 4, and information related to route guidance to the destination, and transmits the display instruction data to the external terminal 2. Note that here, a current position mark 73 indicating the current position of the target vehicle and a route line 74 indicating the guided route along which the in-vehicle device 1 will guide the driver of the target vehicle are superimposed on the map on the map display area 62.

In this way, in the first display example, when the workload is high, the in-vehicle device 1 can conveniently make the user of the external terminal 2 aware of the fact that the workload is high and the cause of this (here, turning right at an intersection). This allows the user of the external terminal 2 to, for example, determine that the driver of the in-vehicle device 1 should concentrate on driving, and take measures such as refraining from talking to the driver about matters other than driving (for example, requests to go shopping, etc.) until the driver has completed the right turn at the intersection.

FIG. 5 shows a second display example of the external terminal 2 when the target vehicle of the in-vehicle device 1 is traveling. The external terminal 2 has established communication with the in-vehicle device 1, and displays the display screen shown in FIG. 5 on the display unit 26 based on the display instruction data received from the in-vehicle device 1. The external terminal 2 has an external-vehicle captured image display area 60, a workload-related information display area 61, and a map display area 62 on the display screen.

The external terminal 2 displays the latest external image (video) generated by the external camera 51 in the external image display area 60 based on the display command data received from the vehicle-mounted device 1. The external terminal 2 also displays a map of the area around the current position of the target vehicle in the map display area 62 based on the display command data received from the vehicle-mounted device 1.

The external terminal 2 also displays a display window 71A and an indicator 72A indicating the degree of workload on the workload-related information display area 61 based on the workload-related information included in the display instruction data received from the vehicle-mounted device 1. Here, since the workload value calculated by the vehicle-mounted device 1 is equal to or lower than the workload threshold, the vehicle-mounted device 1 does not detect the cause of the workload increase. Therefore, the external terminal 2 displays "low workload" in the display window 71, indicating that the workload value level is the lowest level, and does not display the cause of the workload increase. Note that, here, since the vehicle-mounted device 1 detects from the vehicle that the current driving mode of the target vehicle is auto-cruise driving, which puts the driver in a low workload state, the workload-related information includes the fact that the vehicle is in auto-cruise driving. Based on the workload-related information received from the vehicle-mounted device 1, the external terminal 2 displays the fact that the vehicle is in auto-cruise driving in the workload-related information display area 61. The external terminal 2 also displays an indicator 72A including a gauge whose length corresponds to the level of the workload value in the workload-related information display area 61.

In this way, in the second display example, when the workload is low, the in-vehicle device 1 conveniently lets the user of the external terminal 2 know that the workload is low. This allows, for example, the user of the external terminal 2 to determine that the driver of the in-vehicle device 1 is relatively available, and to choose the right timing to talk to the driver about necessary matters.

FIG. 6 shows a third display example of the external terminal 2 when the target vehicle of the in-vehicle device 1 is traveling. The external terminal 2 has established communication with the in-vehicle device 1, and displays the display screen shown in FIG. 6 on the display unit 26 based on the display instruction data received from the in-vehicle device 1. The external terminal 2 has an external-vehicle captured image display area 60, a workload-related information display area 61, and a map display area 62 on the display screen.

The external terminal 2 displays the latest external image (video) generated by the external camera 51 on the external image display area 60 based on the display instruction data received from the vehicle-mounted device 1, in the same manner as in the first and second display examples. The external terminal 2 also displays a map of the area around the current position of the target vehicle on the map display area 62 based on the display instruction data received from the vehicle-mounted device 1, in the same manner as in the first and second display examples.

Based on the workload-related information received from the vehicle-mounted device 1, the external terminal 2 displays a display window 71B showing the workload level and the cause of workload increase, and an indicator 72B visually showing the workload level, in the workload-related information display area 61. Here, since the workload value calculated by the vehicle-mounted device 1 is greater than the workload threshold, the cause of the workload increase is displayed in the display window 71B. Specifically, the vehicle-mounted device 1 detects the drowsiness level of the driver's physical condition element and the visibility of the road environment element as causes of the workload increase, and as a result, "drowsiness + low visibility" is displayed in the display window 71B. Also, "workload somewhat high", which indicates the level of the workload according to the workload value calculated by the vehicle-mounted device 1, is displayed in the display window 71B. The external terminal 2 also displays an indicator 72B including a gauge whose length corresponds to the level of the determined workload value, in the workload-related information display area 61.

In this way, in the third display example, when the workload is high due to the driver feeling drowsy, the in-vehicle device 1 can conveniently make the user of the external terminal 2 aware of the high workload and its cause. This allows the user of the external terminal 2 to take action, such as proactively speaking to the driver of the in-vehicle device 1 to wake him up.

Note that the display layouts in the first to third display examples are merely examples, and various modifications may be applied. For example, the external terminal 2 may display an image captured by the driver instead of an image captured outside the vehicle, based on an operation of the user of the external terminal 2 on the external terminal 2. In another example, the external terminal 2 may provide only one of the external image display area 60 and the map display area 62 on the display screen. In yet another example, the external terminal 2 may provide only the workload-related information display area 61 on the display screen.

(4) Processing Flow Fig. 7 is an example of a flowchart showing the procedure of processing executed by the vehicle-mounted device 1. The vehicle-mounted device 1 executes the processing of the flowchart shown in Fig. 7 when communication with the external terminal 2 is established and a call is started.

First, the vehicle-mounted device 1 calculates the score of each workload-related element related to the workload of the driver of the target vehicle (i.e., element-specific score) (step S101). In this case, the vehicle-mounted device 1 calculates element-specific scores for road environment elements, driving behavior elements, driver physical condition elements, or elements that are further subdivided from these, based on the map DB 4 and data output by the sensor group 15.

Next, the vehicle-mounted device 1 calculates a workload value based on the scores of each workload-related element (element-specific scores) (step S102). In this case, the vehicle-mounted device 1 may calculate a workload value by adding up or averaging (including using a weighting factor) the element-specific scores, or may calculate the workload value by substituting each element-specific score into a formula for calculating the workload value.

Next, the vehicle-mounted device 1 determines whether the workload value is greater than the workload threshold (step S103). If the workload value is greater than the workload threshold (step S103; Yes), the vehicle-mounted device 1 detects the cause of the workload increase (step S104). The vehicle-mounted device 1 then supplies display instruction data including the workload value and workload-related information related to the detected cause of the workload increase to the external terminal 2 (step S105). As a result, the external terminal 2 performs display related to the workload value and the cause of the workload increase based on the display instruction data. Note that if the display instruction data includes at least one of an image taken outside the vehicle, an image of the driver, and map display information for displaying a map around the current position, the external terminal 2 may perform display based on at least one of the image taken outside the vehicle, the image of the driver, and the map display information.

On the other hand, if the workload value is equal to or less than the workload threshold (step S103; No), the in-vehicle device 1 supplies display instruction data including workload-related information on the workload value to the external terminal 2 (step S106). As a result, the external terminal 2 executes display related to the workload value based on the display instruction data. Note that if the display instruction data includes at least one of an image taken outside the vehicle, an image of the driver, and map display information, the external terminal 2 may perform display based on at least one of an image taken outside the vehicle, an image of the driver, and map display information.

Then, the vehicle-mounted unit 1 determines whether the call has ended (step S107). If the vehicle-mounted unit 1 determines that the call has ended (step S107; Yes), it ends the processing of the flowchart. On the other hand, if the vehicle-mounted unit 1 determines that the call has not ended (step S107; No), it returns the processing to step S101.

(5) Modifications Next, preferred modifications of the above-described embodiment will be described. The following modifications may be combined and applied to the above-described embodiment.

(Variation 1)
When the vehicle-mounted device 1 displays the outside-of-vehicle captured image on the external terminal 2, the vehicle-mounted device 1 may superimpose a line (gaze tracking line) representing the recognized trajectory of the driver's gaze on the outside-of-vehicle captured image.

FIG. 8 shows a fourth display example of the external terminal 2 relating to the first modification. The external terminal 2 has established communication with the in-vehicle device 1, and displays the display screen shown in FIG. 8 on the display unit 26 based on the display instruction data received from the in-vehicle device 1. Here, as an example, the external terminal 2 provides an external-vehicle captured image display area 60, a workload-related information display area 61, and a map display area 62 on the display screen, similar to the first to third display examples, based on the display instruction data received from the in-vehicle device 1.

Here, on the outside-vehicle image display area 60, gaze tracking lines 66 indicating the trajectory of the gaze direction of the driver of the target vehicle within a most recent specified time are superimposed on the latest outside-vehicle image in correspondence with the gaze direction in the outside-vehicle image. In this case, the vehicle-mounted device 1 detects the gaze direction on the outside-vehicle image taken at the same time as the driver's image based on any gaze detection technology, and generates gaze direction data indicating the detected gaze direction. Then, the vehicle-mounted device 1 superimposes the gaze tracking lines 66 indicating the gaze direction of the driver within the most recent specified time indicated by the gaze direction data on the outside-vehicle image included in the display instruction data to be transmitted to the outside-vehicle terminal 2. The above-mentioned specified time is stored in advance in the storage unit 12, for example. Then, upon receiving the display instruction data, the outside-vehicle terminal 2 displays the outside-vehicle image with the gaze tracking lines 66 superimposed on it in the outside-vehicle image display area 60.

The vehicle-mounted device 1 may, for example, prestore face-gaze correspondence data that associates each position on the image captured outside the vehicle with a face image model when the driver is looking at that position, and generate gaze direction data based on the face-gaze correspondence data. In this case, the vehicle-mounted device 1 identifies the face image model that most closely matches the driver's face image extracted from the driver's captured image, and generates gaze direction data that indicates the position on the image captured outside the vehicle that corresponds to the identified face image model.

In this way, according to this modified example, the user of the external terminal 2 can easily grasp the direction in which the driver is not paying much attention (to the right in the example of FIG. 8), and the user of the external terminal 2 can assist the driver in checking for safety in the direction in which the driver is not paying much attention by talking to the driver. In addition, the driver can feel reassured as if a passenger is watching over him or her, and can feel a sense of unity as if the user of the external terminal 2 is actually riding with the driver.

In a preferred example, the vehicle-mounted device 1 may change the display mode (display color, display on/off, etc.) of the gaze tracking line according to the workload value. For example, the vehicle-mounted device 1 superimposes the gaze tracking line on the outside-of-vehicle captured image when the workload value is equal to or greater than a predetermined threshold, and hides the gaze tracking line when the workload value is less than the threshold. The above-mentioned predetermined threshold may be the same as the workload threshold, or may be different. In this way, the vehicle-mounted device 1 can display the gaze tracking line only when there is a high probability that the user of the outside-vehicle terminal 2 needs assistance in checking safety. On the other hand, during normal times when the user of the outside-vehicle terminal 2 does not need assistance in checking safety, the user of the outside-vehicle terminal 2 can enjoy the scenery from the outside-vehicle captured image. In another example, the vehicle-mounted device 1 may display the gaze tracking line in a color according to the workload value. In this case, the vehicle-mounted device 1 displays the gaze tracking line in a color that is more noticeable the higher the workload value, for example.

When displaying the gaze tracking line, the vehicle-mounted device 1 may highlight the object (also called the "gaze object") to which the driver should direct his/her gaze in the outside-vehicle captured image and display it on the outside-vehicle terminal 2. The gaze object includes, for example, a traffic light, a road sign, an obstacle, a pedestrian, etc. In this case, the vehicle-mounted device 1 uses any object recognition technology (including those using deep learning models such as instance segmentation) to extract objects corresponding to a predetermined type as the gaze object from the outside-vehicle captured image. The vehicle-mounted device 1 then processes the outside-vehicle captured image so as to highlight the area of the object on the extracted outside-vehicle captured image by edging or the like and superimpose the gaze tracking line, and transmits the processed outside-vehicle captured image to the outside-vehicle terminal 2 together with the display instruction data. The outside-vehicle terminal 2 that has received the display instruction data displays the outside-vehicle captured image with the gaze tracking line superimposed and the gaze object highlighted. This allows the user of the external terminal 2 to recognize the presence of an object that the driver is not looking at from the image captured outside the vehicle, and to advise the driver to pay attention to the object.

(Variation 2)
At least a part of the processing executed by the vehicle-mounted device 1 may be executed by a server device that performs data communication with the vehicle-mounted device 1 and the external terminal 2 .

FIG. 9 shows an example of the configuration of a driving call system according to the second modification. The driving call system has an in-vehicle device 1A, an external terminal 2, and a server device 5. The in-vehicle device 1A and the server device 5, and the external terminal 2 and the server device 5 each perform data communication via the communication network 3.

The vehicle-mounted device 1A has the same configuration as the vehicle-mounted device 1 described in the first embodiment above (see FIG. 2). Note that if the server device 5 performs processing based on the map DB 4, the vehicle-mounted device 1A does not need to have the map DB 4. The vehicle-mounted device 1A then transmits to the server device 5 an upload signal that includes information output by the sensor group 15 and input information input by the input unit 13.

The server device 5 relays data necessary for voice calls between the in-vehicle device 1A and the external terminal 2. During a voice call, the server device 5 generates display instruction data based on an upload signal or the like received from the in-vehicle device 1A, and transmits the generated display instruction data to the external terminal 2. Specifically, during a voice call, the server device 5 executes the process of the flowchart shown in FIG. 7 based on an upload signal or the like received from the in-vehicle device 1A.

FIG. 10 shows an example of the schematic configuration of the server device 5. The server device 5 mainly has a communication unit 41, a storage unit 42, and a control unit 44. The elements in the server device 5 are connected to each other via a bus line 40.

The communication unit 41 performs data communication with external devices such as the in-vehicle unit 1A and the external terminal 2 under the control of the control unit 44. The storage unit 42 is composed of various types of memory such as RAM, ROM, and non-volatile memory (including a hard disk drive, flash memory, etc.). The storage unit 42 stores programs for the server device 5 to execute predetermined processes. The storage unit 42 also includes a map DB 4. The control unit 44 includes a CPU, GPU, etc., and controls the entire server device 5. The control unit 44 also executes the programs stored in the storage unit 42 to perform processes required to display workload-related information on the external terminal 2.

In this way, even if the server device 5 executes the processing required to display workload-related information on the external terminal 2, the driving call system, as in the embodiment, allows the user of the external terminal 2 to easily recognize the driver's workload state, and enables the user to choose a convenient time to talk to the driver. In this modified example, the server device 5 is an example of an "information processing device."

As described above, the in-vehicle device 1 or the server device 5 is an information processing device used in a system that mediates communication between a driver of a target vehicle, which is a moving body, and a user of an external terminal 2 located outside the target vehicle, and has a calculation means, a detection means, and a display control means. The calculation means calculates a workload value that is an index value related to the degree of driving load (workload) of the driver of the target vehicle. The detection means detects the cause of the increased driving load when the workload value indicates that the driving load is higher than a predetermined standard. The display control means causes the external terminal 2 to display information related to the cause of the increased driving load. This enables the user of the external terminal 2 to choose a convenient timing for communicating with the driver.

In each of the above-mentioned embodiments, the program can be stored using various types of non-transitory computer readable media and supplied to a control unit, which is a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R/W, semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)).

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above-mentioned embodiments. Various modifications that a person skilled in the art can understand can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and amendments that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated into this document by reference.

1, 1A Vehicle-mounted device 2 External terminal 3 Communication network 4 Map DB
5 Server device 11, 21, 41 Communication unit 12, 22, 42 Storage unit 13 Input unit 14, 24, 44 Control unit 15, 25 Sensor group 16, 26 Display unit 17, 27 Sound output unit

Claims

An information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, comprising:
A calculation means for calculating an index value relating to a degree of a driving load of a driver of the moving body;
a detection means for detecting a cause of an increase in the driving load when the index value indicates that the driving load is higher than a predetermined standard;
a display control means for displaying information relating to the cause on a terminal device of the user;
An information processing device having the above configuration.
The information processing device according to claim 1, wherein the calculation means calculates the index value based on a current state of at least one of the driving behavior of the moving object, the driver, or the driving environment.
The information processing device according to claim 2, wherein the calculation means calculates a score that evaluates the current state of at least one of the driving behavior of the moving body, the driver, and the driving environment for each element related to the driving load, and calculates the index value based on the score.
The information processing device according to claim 3, wherein the detection means determines the causative element based on the score.
The information processing device according to any one of claims 1 to 4, wherein the display control means causes the terminal device to display information relating to the cause and information representing the magnitude of the index value.
The information processing device according to any one of claims 1 to 4, wherein the display control means causes the terminal device to display a line representing the trajectory of the driver's line of sight on an image captured from the moving body of an area outside the moving body.
The information processing device according to claim 6, wherein the display control means changes the display mode of the line depending on the index value.
The information processing device according to claim 7, wherein the display control means displays the line when the index value indicates that the driving load is higher than a predetermined standard, and hides the line when the index value indicates that the driving load is equal to or lower than the predetermined standard.
The information processing device according to claim 6, wherein the display control means highlights an object to which the driver should direct his or her gaze in the captured image.
A control method executed by an information processing device used in a system that mediates communication between a driver of a moving body and a user present outside the moving body, comprising:
A calculation step of calculating an index value related to a degree of a driving load of a driver of the moving body;
a detection step of detecting a cause of an increase in the driving load when the index value indicates that the driving load is higher than a predetermined standard;
a display control step of displaying information about the cause on a terminal device of the user;
The control method includes:
A program executed by a computer of an information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, comprising:
A calculation means for calculating an index value relating to a degree of a driving load of a driver of the moving body;
a detection means for detecting a cause of an increase in the driving load when the index value indicates that the driving load is higher than a predetermined standard;
A program that causes a computer to function as a display control means for displaying information regarding the cause on a terminal device of the user.
A storage medium storing the program according to claim 11.