JP2020154996A - Information processing system, agent system, information processing method, and program - Google Patents

Abstract

To provide an information processing system configured to improve convenience of a user using a ride-share service, an agent system, an information processing method, and a program.SOLUTION: An information processing system includes: a ride-share server device (300) which generates an operation schedule of a vehicle (M) on the basis of a ride request from a user, and provides a ride-share service using the vehicle; and an agent system (E) which observes a state of the user on the vehicle on the basis of information acquired by at least one of a mobile terminal device (D) of the user and an in-vehicle device mounted on the vehicle, and communicates with the ride-share server device when it is detected that the state of the user using the ride-share service is changed to a predetermined state.SELECTED DRAWING: Figure 1

Description

The present invention relates to information processing systems, agent systems, information processing methods, and programs.

Conventionally, a technology related to an agent function that provides information on driving support according to a request of a occupant, vehicle control, other applications, etc. while interacting with a vehicle occupant has been disclosed (see, for example, Patent Document 1). ..

Japanese Unexamined Patent Publication No. 2006-335231

By the way, as a new service in the future, a so-called ride-sharing service in which a plurality of users use the same shared vehicle has been proposed. However, in the conventional technique, consideration is not given to the case where the physical condition of the occupant who is using the ride sharing service changes, and the convenience for the user is not sufficient.

The present invention has been made in consideration of such circumstances, and provides an information processing system, an agent system, an information processing method, and a program capable of improving convenience for users of a ride sharing service. That is one of the purposes.

The information processing system, agent system, information processing method, and program according to the present invention have adopted the following configurations.
(1): One aspect of the present invention includes a ride-sharing server device that generates a vehicle operation plan based on a boarding request from a user and provides a ride-sharing service by the vehicle, and a mobile terminal device of the user. Based on the information acquired by at least one of the in-vehicle devices mounted on the vehicle, the state of the user while riding in the vehicle is observed, and the state of the user is determined while using the ride sharing service. This is an information processing system including an agent system that communicates with the ride-sharing server device when it is detected that the vehicle has changed to the above state.

(2): In the aspect of (1) above, when it is detected that the state of the user has changed to the predetermined state, the agent system needs to arrange an ambulance based on the state of the user. When the sex is judged and it is determined that the ambulance needs to be arranged, the ride sharing server device is requested to arrange the ambulance.

(3): In the embodiment (1) or (2), when the agent system detects that the state of the user has changed to the predetermined state, the ride-sharing server device informs the vehicle. Request a change of destination.

(4): In any one of the above (1) to (3), the agent system detects that the state of the user has changed to the predetermined state, and the rideshare Request the server device to contact the passengers of the vehicle.

(5): In any one of the above (1) to (4), the agent system detects that the state of the user has changed to the predetermined state, and the ride share. Request the server device to arrange an alternative vehicle to be used by the passengers of the vehicle.

(6): In any one of the above (1) to (5), the agent system detects that the state of the user has changed to the predetermined state, and the rideshare Information indicating the state of the user is transmitted to the server device.

(7): In any one of the above (1) to (6), the ride-sharing server device has detected that the state of the user has changed to the predetermined state by the agent system. In this case, the necessity of arranging an ambulance is determined based on the information received from the agent system, and if it is determined that the ambulance needs to be arranged, the ambulance is arranged.

(8): In any one of the above (1) to (7), the ride-sharing server device has detected that the state of the user has changed to the predetermined state by the agent system. In some cases, the destination of the vehicle is changed.

(9): In any one of the above (1) to (8), the ride-sharing server device has detected that the state of the user has changed to the predetermined state by the agent system. In this case, the passenger of the vehicle is contacted.

(10): In any one of the above (1) to (9), the ride-sharing server device has detected that the state of the user has changed to the predetermined state by the agent system. In this case, an alternative vehicle to be used by the passengers of the vehicle is arranged.

(11): In another aspect of the present invention, the first computer generates a vehicle operation plan based on a boarding request from the user, provides the ride sharing service by the vehicle, and the second computer uses the vehicle. Based on the information acquired by at least one of the mobile terminal device of the person and the in-vehicle device mounted on the vehicle, the state of the user while riding in the vehicle is observed, and during the use of the ride sharing service. This is an information processing method that communicates with the first computer when it is detected that the state of the user has changed to a predetermined state.

(12): In another aspect of the present invention, the first computer is made to generate a vehicle operation plan based on a boarding request from the user, the ride sharing service by the vehicle is provided, and the second computer is made to use the vehicle. Based on the information acquired by at least one of the mobile terminal device of the person and the in-vehicle device mounted on the vehicle, the state of the user while riding in the vehicle is observed, and the ride sharing service is being used. This is a program that causes communication with the first computer when it is detected that the state of the user has changed to a predetermined state.

(13): Another aspect of the present invention is based on at least one of a mobile terminal device of a user who uses the ride-sharing service of the vehicle and an in-vehicle device mounted on the vehicle, and is in the vehicle. The ride-sharing service when the state-observing unit for observing the state of the user and the state-observing unit detect that the state of the user has changed to a predetermined state while using the ride-sharing service. It is an agent system equipped with a notification unit that communicates with a ride-sharing server device that manages the vehicle.

(14): In another aspect of the present invention, the computer is attached to the vehicle based on at least one of a user's mobile terminal device that uses the vehicle ride sharing service and an in-vehicle device mounted on the vehicle. When the state of the user while riding is observed and the state observation unit detects that the state of the user has changed to a predetermined state while using the ride sharing service, the ride sharing service is provided. It is an information processing method that communicates with the ride-sharing server device to be managed.

(15): Another aspect of the present invention is based on the computer based on at least one of a mobile terminal device of a user who uses the ride sharing service of the vehicle and an in-vehicle device mounted on the vehicle. When the state of the user is observed while riding and the state observation unit detects that the state of the user has changed to a predetermined state while using the ride sharing service, the ride sharing service is provided. It is a program that communicates with the ride-sharing server device that it manages.

According to (1) to (15), it is possible to improve the convenience of the user of the ride sharing service.

It is a block diagram of the information processing system 1 of 1st Embodiment. It is a block diagram of the vehicle M with respect to traveling. It is a block diagram of the vehicle M about the agent device 100. It is a figure which shows an example of the internal structure of the vehicle M. It is a block diagram which shows the structure of the agent server 200 and a part of the structure of the agent apparatus 100. It is a block diagram of the state observation part 230. It is a figure which shows an example of the content of image IM1. It is a figure which shows an example of the contents of the medical history information 252a. It is a block diagram of the action content determination unit 232 at the time of an accident of an occupant. It is a figure which shows an example of the content of image IM2. It is a block diagram of a ride-sharing server 300. It is a figure which shows an example of the content of boarding condition information 384. It is a figure which shows an example of the contents of the operation schedule information 388. It is a block diagram of the occupant accident response part 340. It is a block diagram of the agent server 200A of the 2nd Embodiment. It is a block diagram of the ride-sharing server 300A of the 2nd Embodiment. It is a block diagram of the agent server 200B of a modification.

Hereinafter, the information processing system, the agent system, the information processing method, and the embodiment of the program of the present invention will be described with reference to the drawings. The information processing system is realized, for example, in cooperation with a ride-sharing server device (hereinafter, referred to as "ride-sharing server") and an agent system.

The ride-sharing server is a device for supporting the joint use (ride-sharing) of one or more vehicles by a plurality of users. The vehicle used for ride sharing is, for example, an autonomous driving vehicle that basically does not require a driving operation. Hereinafter, the self-driving vehicle will be described as being used for ride sharing, but a manually driven vehicle may be used. When the ride-sharing server acquires a boarding request from the user's terminal device by communication, it searches for a vehicle (vehicle that can be dispatched) that matches the boarding conditions defined in the boarding request. Communication may include both data communication and voice communication or telephone. The ride-sharing server generates an operation plan of the vehicle selected by the search and provides the ride-sharing service to the user.

Part or all of the agent system is realized by the agent device. Hereinafter, as an example of the agent device, an agent device mounted on a vehicle and having a plurality of types of agent functions will be described. The agent function is, for example, a function of providing various information based on a request (command) included in the utterance of the occupant or mediating a network service while interacting with the occupant of the vehicle. The functions, processing procedures, controls, output modes and contents of each of the plurality of types of agents may be different. In addition, some of the agent functions may have a function of controlling devices in the vehicle (for example, devices related to driving control and vehicle body control).

Agent functions include, for example, a voice recognition function that recognizes the voice of an occupant (a function that converts voice into text), a natural language processing function (a function that understands the structure and meaning of text), a dialogue management function, and a network. It is realized by using a network search function that searches for other devices or a predetermined database owned by the own device in an integrated manner. Some or all of these functions may be realized by AI (Artificial Intelligence) technology. In addition, a part of the configuration for performing these functions (particularly, the voice recognition function and the natural language processing interpretation function) is an agent server device (particularly, an agent server device capable of communicating with the vehicle-mounted communication device of the vehicle or a general-purpose communication device brought into the vehicle). It may be mounted on an external device). In the following description, it is assumed that a part of the configuration is mounted on the agent server device (hereinafter referred to as “agent server”), and the agent device and the agent server cooperate to realize the agent system. Further, a service provider (service entity) in which an agent device and an agent server cooperate to appear virtually is called an agent. A part or all of the agent system may be realized by an application program or a browser started on the user's mobile terminal device. Further, in the present specification, the expressions "user" and "occupant" are used with almost the same meaning and can be read as each other.

[overall structure]
FIG. 1 is a configuration diagram of an information processing system 1. The information processing system 1 includes, for example, one or more terminal devices D, one or more vehicles M on which the agent device 100 is mounted, and a plurality of agent servers 200-1, 200-2, 200-3, ... It includes a share server 300. The number after the hyphen at the end of the code shall be an identifier for distinguishing agents. When it is not distinguished which agent server it is, it may be simply referred to as an agent server 200. Although three agent servers 200 are shown in FIG. 1, the number of agent servers 200 may be two or four or more. Each agent server 200 is operated by a provider of agent systems different from each other. Therefore, the agents in the present invention are agents realized by different providers. Examples of providers include automobile manufacturers, network service providers, e-commerce businesses, sellers and manufacturers of mobile terminal devices (for example, terminal device D), and any entity (corporation, group, individual, etc.). ) Can be the provider of the agent system. In the present embodiment, the agent system E is realized by the cooperation of the agent device 100 and one or more agent servers 200.

The agent device 100 can communicate with the agent server 200 and the write share server 300, respectively, via the network NW. The agent device 100 interacts with the occupant of the vehicle M, transmits the voice from the occupant to the agent server 200, and presents the answer obtained from the agent server 200 to the occupant in the form of voice output or image display. The network NW includes, for example, a part or all of the Internet, a cellular network, a Wi-Fi network, a WAN (Wide Area Network), a LAN (Local Area Network), a public line, a telephone line, a wireless base station, and the like. Various web servers 400 are connected to the network NW. The various web servers 400 include, for example, a web server 400-1 of a disaster emergency information center that manages the dispatch of ambulances, and a web server 400-2 of each medical institution. The ride-sharing server 300 can communicate with the web server 400-1 of the disaster emergency information center and the web server 400-2 of each medical institution via the network NW, respectively.

The terminal device D is a terminal device used by one or more users U. Note that "used by user U" may include that a terminal device that can be used by an unspecified number of people, such as a terminal device of an Internet cafe, is temporarily used by user U. The ride-sharing server 300 acquires a boarding request of the user U who is a user of the ride-sharing service through, for example, the terminal device D, and generates an operation plan of the vehicle M based on the acquired boarding request. Based on the operation plan generated by the ride-sharing server 300, the vehicle M makes the user U board at the desired boarding point of the user U and carries the user U to the destination.

<First Embodiment>
First, the first embodiment will be described. In the first embodiment, the agent system E proactively determines the action content for protecting the user U when the physical condition of the user U while riding in the vehicle M that provides the ride sharing service changes. This is an example of In the following explanation, the user U who has a physical condition among the users U who got on the vehicle M is "user U1", and the other user U who is on the vehicle M is "passenger U2". It is called. Further, the terminal device D carried by the user U1 is referred to as "terminal device D1", and the terminal device D carried by the passenger U2 is referred to as "terminal device D2".

[1. Terminal device]
The terminal device D is, for example, a smartphone, a tablet terminal, a personal computer, or the like. In the terminal device D, an application program for using the ride sharing system, a browser, or the like is started to support the services described below. In the following description, it is assumed that the terminal device D is a mobile terminal device such as a smartphone and the application program (ride sharing application) is running. The ride-sharing application communicates with the ride-sharing server 300 according to the operation of the user U, sends a request of the user U to the ride-sharing server 300, and performs a push notification based on the information received from the ride-sharing server 300. Or

[2. vehicle]
[2.1 Vehicle configuration for driving]
The vehicle M is, for example, a vehicle having four or more wheels on which a plurality of users U can ride, but may be a motorcycle or other vehicle. FIG. 2 is a configuration diagram of the vehicle M regarding traveling. The vehicle M includes, for example, an outside world monitoring unit 12, an in-vehicle communication device 14, a navigation device 16, a recommended lane determination device 18, an automatic driving control unit 20, a driving force output device 22, a braking device 24, and steering. A device 26 is provided.

The outside world monitoring unit 12 includes, for example, a camera, a radar, a LIDAR (Light Detection and Ranging), an object recognition device that performs sensor fusion processing based on these outputs, and the like. The outside world monitoring unit 12 estimates the types of objects (particularly vehicles, pedestrians, and bicycles) existing around the vehicle M, and outputs the information of the position and speed to the automatic driving control unit 20.

The in-vehicle communication device 14 is, for example, a wireless communication module for connecting to a network NW or directly communicating with a terminal device of another vehicle or a pedestrian. The in-vehicle communication device 14 performs wireless communication based on Wi-Fi, DSRC (Dedicated Short Range Communications), Bluetooth (registered trademark), and other communication standards. As the in-vehicle communication device 14, a plurality of in-vehicle communication devices 14 may be prepared according to the application.

The navigation device 16 includes a navigation HMI (Human Machine Interface), a positioning device such as a GPS (Global Positioning System), a storage device that stores map information, and a control device (navigation controller) that performs route search and the like. .. A part or all of the microphone 32, the vehicle interior camera 34, the display / operation device 36, and the speaker unit 38, which will be described later, may be used as the navigation HMI. The navigation device 16 searches for a route (navigation route) for moving from the position of the vehicle M specified by the positioning device to the destination, and guides the vehicle M using the navigation HMI so that the vehicle M can travel along the route. Information can be output. The route search function may be provided in a navigation server accessible via the network NW. In this case, the navigation device 16 acquires a route from the navigation server and outputs guidance information. Further, in the case of the present embodiment, the route to the destination may be designated by the ride sharing server 300. The agent device 100 may be constructed based on the navigation controller of the navigation device 16, and in that case, the navigation controller and the agent device 100 are integrally configured on the hardware. The navigation device 16 outputs the route information determined or acquired by any of the above methods to the recommended lane determination device 18.

The recommended lane determination device 18 includes, for example, an MPU (Micro Processing Unit) and various storage devices. The storage device stores high-precision map information that is more detailed than the navigation map. The high-precision map information includes, for example, information such as road width, slope, curvature, and signal position for each lane. The recommended lane determination device 18 determines a preferable recommended lane for traveling along the route input from the navigation device 16 and outputs the recommended lane to the automatic driving control unit 20.

The automatic operation control unit 20 includes one or more processors such as a CPU (Central Processing Unit) and an MPU, and various storage devices. In principle, the automatic driving control unit 20 travels in the recommended lane determined by the recommended lane determining device 18, and the vehicle M is moved so as to avoid contact with an object whose position and speed have been input from the outside world monitoring unit 12. Run automatically. For example, the automatic driving control unit 20 generates a target track on which the vehicle M will travel in the future, and a part or all of the driving force output device 22, the braking device 24, and the steering device 26 so as to travel on the generated target track. To control.

The driving force output device 22 outputs a traveling driving force (torque) for the vehicle M to travel to the driving wheels. The driving force output device 22 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, and the like, and a power ECU that controls them.

The brake device 24 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to the information input from the automatic operation control unit 20 or the information input from the operation operator so that the brake torque corresponding to the braking operation is output to each wheel.

The steering device 26 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies a force to the rack and pinion mechanism to change the direction of the steering wheel. The steering ECU drives the electric motor according to the information input from the automatic operation control unit 20 or the information input from the operation operator, and changes the direction of the steering wheel.

[2.2 Vehicle configuration related to agent device]
FIG. 3 is a configuration diagram of the vehicle M relating to the agent device 100. The vehicle M includes, for example, one or more microphones 32, one or more vehicle interior cameras 34, a display / operation device 36, a speaker unit 38, a vehicle device 40, an occupant authentication device 52, and an occupant position recognition device. 54 and the agent device 100 are mounted. In addition, a general-purpose communication device 60 such as a smartphone may be brought into the vehicle interior and used as a communication device. An example of the general-purpose communication device 60 is a terminal device D carried by the user U. These devices are connected to each other by a multiplex communication line such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, or the like. The configuration shown in FIG. 3 is merely an example, and a part of the configuration may be omitted or another configuration may be added.

The microphone 32 is a sound collecting unit that collects the sound emitted in the vehicle interior. The microphone 32 may be provided at one or more representative positions in the vehicle interior, or may be provided at each of the driver's seat, the passenger's seat, and the rear seat.

The vehicle interior camera 34 is an information acquisition unit that acquires an image (or video) of the vehicle interior. The vehicle interior camera is a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera installed in the vehicle interior. The vehicle interior camera 34 may be provided at one or more representative positions in the vehicle interior, or may be provided, for example, in each of the driver's seat, the passenger seat, and the rear seat.

The display / operation device 36 is a device (or device group) capable of displaying an image and receiving an input operation. The display / operation device 36 includes, for example, a display device configured as a touch panel. The display / operation device 36 may further include a HUD (Head Up Display) or a mechanical input device. The display / operation device 36 may be shared by the agent device 100 and the navigation device 16. The display / operation device 36 may be provided at one or more representative positions in the vehicle interior, or may be provided at each of the driver's seat, the passenger's seat, and the rear seat, for example.

The speaker unit 38 includes, for example, one or more speakers (sound output units) arranged in the vehicle interior. The speaker unit 38 may be provided at one or more representative positions in the vehicle interior, or may be provided, for example, in each of the driver's seat, the passenger seat, and the rear seat.

Each of the microphone 32, the vehicle interior camera 34, the display / operation device 36, and the speaker unit 38 is an example of the "vehicle-mounted device" mounted on the vehicle M. In addition, the agent device 100 acquires the voice and image of the user U by using the microphone and the camera mounted on the terminal device D of the user U in place of or in addition to the microphone 32 and the vehicle interior camera 34. May be good. Further, the agent device 100 uses the display device and the speaker mounted on the terminal device D of the user U in place of or in addition to the display / operation device 36 and the speaker unit 38 to display an image to the user U. It may be displayed or output sound. Such an embodiment is realized by, for example, starting an application program (agent application) installed in the terminal device D and realizing a part of the agent system E, or an interactive general-purpose application program. Therefore, in the following description, each of the microphone 32, the vehicle interior camera 34, the display / operation device 36, and the speaker unit 38 may be read as “terminal device D”.

The vehicle equipment 40 includes, for example, an engine start motor, a door lock device, a door opening / closing device, a window, a window opening / closing device, a window opening / closing control device, a seat, a seat position control device, a room mirror and an angle position control device thereof. Lighting devices inside and outside the vehicle and their control devices, wipers and defoggers and their respective control devices, direction indicator lights and their control devices, air conditioning devices, vehicle information devices such as mileage and tire pressure information and fuel level information, etc. including.

The occupant authentication device 52 is a device for determining whether or not the user who intends to board the vehicle M is a legitimate user. For example, when the reservation of the vehicle M is registered by the ride sharing server 300 based on the boarding request of the user U, the terminal device D of the user U has a virtual key (virtual key) for boarding the reserved vehicle M. Key) is sent. The virtual key is information indicating that the user U is a legitimate occupant of the vehicle M. The virtual key is, for example, information different for each user U (information unique to each user U), and includes, for example, identification information (user ID) for identifying the user U. The virtual key is, for example, information that can be transmitted by wireless communication between the terminal device D and the vehicle-mounted communication device 14 of the vehicle M, and is a two-dimensional bar code displayed on the display device of the terminal device D (for example, QR (for example). It may be a registered trademark) code) or the like.

The occupant authentication device 52 determines whether or not the user who intends to board the vehicle M is a legitimate user based on the virtual key acquired from the terminal device D of the user U. When the occupant authentication device 52 determines that the user who intends to board the vehicle M is a legitimate user U, the occupant authentication device 52 releases the door lock of the vehicle M and allows the user U to board. For example, the occupant authentication device 52 transmits the identification information (user ID) included in the virtual key acquired from the terminal device D of the user U to the agent server 200 and the ride sharing server via the in-vehicle communication device 14 and the network NW. Output to 300. As a result, the agent server 200 and the ride sharing server 300 can identify (identify) the user U who has boarded the vehicle M.

The occupant position recognition device 54 is, for example, a seating sensor. The seating sensor includes a pressure sensor provided at the bottom of the seat, a tension sensor attached to the seat belt, and the like. The occupant position recognition device 54 is an example of a seating position recognition unit that recognizes the seating position of an occupant in the vehicle interior. A part or all of the occupant position recognition device 54 may be composed of the vehicle interior camera 34 and the image processing unit 113 described later. The seating position of the occupant detected by the occupant position recognition device 54 is output to the agent server 200 via the vehicle-mounted communication device 14 and the network NW. As a result, the agent server 200 can manage the user U specified based on the identification information (user ID) included in the key information in association with the seating position of the user U.

FIG. 4 is a diagram showing an example of the internal configuration of the vehicle M. A partition P is provided in the vehicle interior to partition the driver's seat, the passenger's seat, the left rear seat, and the right rear seat, for example, for the purpose of ensuring privacy. In this case, the microphone 32, the vehicle interior camera 34, the display / operation device 36, and the speaker unit 38 may be provided in each seat. In this case, the agent device 100 may make a different agent appear for each seat by using the display / operation device 36 and the speaker unit 38 individually provided for each seat. However, the partition P is not an essential component, and the space between the seats may be open as in the conventional vehicle.

[3. Agent device]
Next, returning to FIG. 3, the agent device 100 will be described. The agent device 100 includes a management unit 110, agent function units 150-1, 150-2, 150-3, and a pairing application execution unit 152. The management unit 110 includes, for example, an sound processing unit 112, an image processing unit 113, a WU (Wake Up) determination unit 114 for each agent, a display control unit 116, and a voice control unit 118. When it is not distinguished which agent function unit it is, it is simply referred to as an agent function unit 150. The three agent function units 150 are shown only as an example corresponding to the number of agent servers 200 in FIG. 1, and the number of agent function units 150 may be two or four or more. It may be. The software layout shown in FIG. 3 is simply shown for the sake of explanation, and is actually modified arbitrarily so that, for example, the management unit 110 may intervene between the agent function unit 150 and the in-vehicle communication device 14. can do.

Each component of the agent device 100 is realized by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components are hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), etc. It may be realized by (including circuits), or it may be realized by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device including a non-transient storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or a removable storage device such as a DVD or a CD-ROM. It is stored in a medium (non-transient storage medium) and may be installed by mounting the storage medium in a drive device.

The management unit 110 functions by executing a program such as an OS (Operating System) or middleware.

The sound processing unit 112 performs sound processing on the input sound so as to be in a state suitable for recognizing a wakeup word preset for each agent. The sound processed by the sound processing unit 112 is output to the agent server 200 via the vehicle-mounted communication device 14 and the network NW.

The image processing unit 113 performs image processing on the image (or video) acquired by the vehicle interior camera 34, and acquires information capable of recognizing the posture of the user U, the facial expression of the user U, and the like. The image processed by the image processing unit 113 is output to the agent server 200 via the vehicle-mounted communication device 14 and the network NW.

The WU determination unit 114 for each agent exists corresponding to each of the agent function units 150-1, 150-2, and 150-3, and recognizes a wakeup word predetermined for each agent. The WU determination unit 114 for each agent recognizes the meaning of the voice from the voice (voice stream) subjected to the acoustic processing. First, the WU determination unit 114 for each agent detects a voice section based on the amplitude and zero intersection of the voice waveform in the voice stream. The WU determination unit 114 for each agent may perform frame-by-frame speech recognition based on a mixture Gaussian mixture model (GMM) and section detection based on non-speech recognition.

Next, the WU determination unit 114 for each agent converts the voice in the detected voice section into text and converts it into character information. Then, the WU determination unit 114 for each agent determines whether or not the textual character information corresponds to the wakeup word. When it is determined that the word is a wakeup word, the WU determination unit 114 for each agent activates the corresponding agent function unit 150. The agent server 200 may be equipped with a function corresponding to the WU determination unit 114 for each agent. In this case, when the management unit 110 transmits the voice stream to which the sound processing has been performed by the sound processing unit 112 to the agent server 200 and determines that the agent server 200 is a wakeup word, the management unit 110 follows an instruction from the agent server 200. The agent function unit 150 starts. It should be noted that each agent function unit 150 may be always activated and may determine the wakeup word by itself. In this case, the management unit 110 does not need to include the WU determination unit 114 for each agent.

The agent function unit 150 makes an agent appear in cooperation with the corresponding agent server 200, and provides a service including a voice response in response to the utterance of the occupant of the vehicle M. The agent function unit 150 may include one to which the authority to control the vehicle device 40 is granted. Further, the agent function unit 150 may be one that cooperates with the general-purpose communication device 60 via the pairing application execution unit 152 and communicates with the agent server 200. For example, the agent function unit 150-1 is given the authority to control the vehicle device 40. The agent function unit 150-1 communicates with the agent server 200-1 via the vehicle-mounted communication device 14. The agent function unit 150-2 communicates with the agent server 200-2 via the vehicle-mounted communication device 14. The agent function unit 150-3 cooperates with the general-purpose communication device 60 via the pairing application execution unit 152, and communicates with the agent server 200-3. The pairing application execution unit 152 pairs with the general-purpose communication device 60 by Bluetooth, for example, and connects the agent function unit 150-3 and the general-purpose communication device 60. The agent function unit 150-3 may be connected to the general-purpose communication device 60 by wired communication using USB (Universal Serial Bus) or the like. Agent 1 is the agent that the agent function unit 150-1 and the agent server 200-1 collaborate to appear, and agent 2 is the agent that the agent function unit 150-2 and the agent server 200-2 collaborate to appear. An agent that the unit 150-3 and the agent server 200-3 collaborate to appear may be referred to as an agent 3.

The display control unit 116 causes the display device of the display / operation device 36 to display an image in response to an instruction from the agent function unit 150. The display control unit 116 generates, for example, an image of an anthropomorphic agent (hereinafter, referred to as “agent image”) that communicates with an occupant in the vehicle interior under the control of a part of the agent function unit 150. The displayed agent image is displayed on the display device of the display / operation device 36. The agent image is, for example, an image of a mode of talking to an occupant. The agent image may include, for example, a facial image such that the facial expression and the facial orientation are recognized by the viewer (occupant) at least. Further, the agent image may be an animation image.

The voice control unit 118 causes a part or all of the speakers included in the speaker unit 30 to output voice in response to an instruction from the agent function unit 150.

[4. Agent server]
[4.1 Overall configuration of agent server]
FIG. 5 is a configuration diagram showing a configuration of the agent server 200 and a part of the configuration of the agent device 100. Hereinafter, the operation of the agent function unit 150 and the like together with the configuration of the agent server 200 will be described. Here, the description of the physical communication from the agent device 100 to the network NW will be omitted.

The agent server 200 includes a communication unit 210. The communication unit 210 is a network interface such as a NIC (Network Interface Card). Further, the agent server 200 includes, for example, a voice recognition unit 220, a natural language processing unit 222, a dialogue management unit 224, a network search unit 226, a response sentence generation unit 228, a state observation unit 230, and an occupant change. It includes an action content determination unit 232 and a notification unit 234. These components are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware such as LSI, ASIC, FPGA, GPU (including circuit part; circuitry), or realized by collaboration between software and hardware. May be good. The program may be stored in advance in a storage device such as an HDD or flash memory (a storage device including a non-transient storage medium), or a removable storage medium such as a DVD or a CD-ROM (non-transient). It is stored in a sex storage medium) and may be installed by attaching the storage medium to a drive device.

The agent server 200 includes a storage unit 250. The storage unit 250 is realized by the above-mentioned various storage devices. Data and programs such as a personal profile 252, a dictionary DB (database) 254, a knowledge base DB 256, a response rule DB 258, and an occupant state pattern DB 260 are stored in the storage unit 250. The personal profile 252 includes medical history information 252a, which will be described later. The response rule DB 258 includes the occupant incident response rule 258a, which will be described later.

In the agent device 100, the agent function unit 150 transmits a voice stream or a voice stream that has undergone processing such as compression or coding to the agent server 200. Further, the agent function unit 150 transmits the image processed by the image processing unit 113 to the agent server 200.

When the agent function unit 150 recognizes a voice command capable of local processing (processing that does not go through the agent server 200), the agent function unit 150 may perform the processing requested by the voice command. The voice command that can be locally processed is a voice command that can be answered by referring to a storage unit (not shown) included in the agent device 100, or in the case of the agent function unit 150-1, the vehicle device 40 is controlled. It may be a voice command (for example, a command to turn on the air conditioner). Therefore, the agent function unit 150 may have a part of the functions provided in the agent server 200.

When the voice stream is acquired, the voice recognition unit 220 performs voice recognition and outputs textual character information, and the natural language processing unit 222 interprets the character information with reference to the dictionary DB 254. The dictionary DB 254 is associated with abstract semantic information with respect to character information. The dictionary DB 254 may include list information of synonyms and synonyms. The processing of the voice recognition unit 220 and the processing of the natural language processing unit 222 are not clearly separated in stages, and the voice recognition unit 220 corrects the recognition result in response to the processing result of the natural language processing unit 222. It may be done by influencing each other.

For example, when the natural language processing unit 222 recognizes the meanings such as "today's weather" and "how is the weather" as the recognition result, the natural language processing unit 222 generates a command replaced with the standard character information "today's weather". As a result, even if there is a character fluctuation in the voice of the request, it is possible to facilitate the dialogue according to the request. Further, the natural language processing unit 222 may recognize the meaning of character information by using artificial intelligence processing such as machine learning processing using probability, or may generate a command based on the recognition result.

The dialogue management unit 224 determines the content of the utterance to the occupant of the vehicle M based on the processing result (command) of the natural language processing unit 222 with reference to the personal profile 252, the knowledge base DB 256, and the response rule DB 258. The personal profile 252 includes the personal information of the occupants, hobbies and preferences, the history of past dialogues, etc., which are stored for each occupant (that is, associated with the user ID). The knowledge base DB 256 is information that defines the relationships between things. The response rule DB 258 is information that defines the actions (answers, device control contents, etc.) that the agent should perform in response to the command.

Further, the dialogue management unit 224 uses at least one of the feature information obtained from the voice stream and the feature information obtained from the image in the case of the vehicle M in which the occupant authentication device 52 is not provided, for example, to use the personal profile 252. The occupant may be identified by collating with. In this case, in the personal profile 252, for example, personal information is associated with voice or appearance (appearance) feature information. Voice feature information is, for example, information on speaking characteristics such as voice pitch, intonation, and rhythm (sound pitch pattern), and feature quantities based on Mel Frequency Cepstrum Coefficients. .. The voice feature information is, for example, information obtained by having the occupant utter a predetermined word or sentence at the time of initial registration of the occupant and recognizing the uttered voice. Appearance feature information is information on the amount of features such as profile, facial contour, position and shape of each part of the face such as eyes and nose, and hair color.

The dialogue management unit 224 causes the network search unit 226 to perform a search when the command requests information that can be searched via the network NW. The network search unit 226 accesses various web servers 400 via the network NW and acquires desired information.

The response sentence generation unit 228 generates a response sentence and transmits it to the agent device 100 so that the content of the utterance determined by the dialogue management unit 224 is transmitted to the occupant of the vehicle M. When the occupant is identified as an occupant registered in the personal profile 252, the response sentence generation unit 228 calls the occupant's name or generates a response sentence in a speech style similar to the occupant's speech style. May be good.

When the agent function unit 150 acquires the response sentence, the agent function unit 150 instructs the voice control unit 118 to perform voice synthesis and output the voice. Further, the agent function unit 150 instructs the display control unit 116 to display the agent image in accordance with the audio output. In this way, the agent function in which the virtually appearing agent responds to the occupant of the vehicle M is realized.

[4.2 Condition Observation Unit]
The state observing unit 230 observes the state of the user U while riding in the vehicle M, for example, based on the information acquired by the microphone 32 of the vehicle M and the vehicle interior camera 34. For example, the state observation unit 230 is based on an audio stream acquired by the microphone 32 and subjected to acoustic processing by the acoustic processing unit 112 and an image acquired by the vehicle interior camera 34 and image-processed by the image processing unit 113. , Observe the state of the user U while riding in the vehicle M. However, the state observing unit 230 may observe the state of the user U while riding in the vehicle M based on either the information acquired by the microphone 32 or the information acquired by the vehicle interior camera 34. Good. Further, the state observing unit 230 may observe the state of the user U while riding in the vehicle M based on the information acquired by the terminal device D in place of / in addition to the microphone 32 and the vehicle interior camera 34. ..

FIG. 6 is a configuration diagram of the state observation unit 230. The state observation unit 230 includes, for example, a voice observation unit 230a, an appearance observation unit 230b, a voice call unit 230c, and an incident determination unit 230d. Here, in the occupant state pattern DB 260 stored in the storage unit 250, a plurality of state patterns that model the state (state) of the user U in the vehicle interior are registered. These state patterns are, for example, typical "speech content", "voice tone", "voice intonation", "voice tremor", "voice loudness", and "body" when an abnormality occurs in the physical condition. "Position", "Facial expression", "Gestures appearing on the body and face", etc. Some specific examples are "speech such as headache, numbness, etc.", "voice is cramping", "voice is trembling", "Ritsu is not spinning", "crouching". , "I'm closing my face with both hands", "I'm holding my stomach with my hands", "I'm cramping", etc.

The voice observation unit 230a compares the voice stream obtained by the sound processing unit 112 with the state pattern related to the voice registered in the occupant state pattern DB 260, for example, at a predetermined cycle. In addition, when the voice of the normal state (when the physical condition is good) for each user U is registered in the personal profile 252, the voice observation unit 230a replaces or adds the voice obtained by the sound processing unit 112. The stream may be compared with the normal voice for each user U registered in the personal profile 252. The comparison result by the voice observation unit 230a is output to, for example, the voice-calling unit 230c, the accident determination unit 230d, and the occupant change behavior content determination unit 232.

The appearance observing unit 230b has, for example, an appearance (including gestures, facial expressions, etc.) included in the image obtained by the image processing unit 113 and a state pattern related to the appearance registered in the occupant state pattern DB 260 at a predetermined cycle. Compare with. In addition, when the appearance in the normal state (when the physical condition is good) for each user U is registered in the personal profile 252, the appearance observation unit 230b is an image obtained by the image processing unit 113 in place of or in addition to the above. The appearance included in the above may be compared with the normal appearance of each user U registered in the personal profile 252. The comparison result by the appearance observation unit 230b is output to, for example, the voice-calling unit 230c, the accident determination unit 230d, and the occupant change behavior content determination unit 232.

When it is determined that there is a high possibility that the physical condition of the user U1 has changed based on the comparison results of the voice observing unit 230a and the appearance observing unit 230b (the calculated value indicating the probability is equal to or higher than the threshold value). In some cases), by outputting a predetermined instruction to the agent device 100, the display / operation device 36 and the speaker unit 38 of the vehicle M are used to call out to the user U1 (speech to the user U1). For example, the agent image is displayed on the display device of the display / operation device 36, and the agent is virtually made to appear.

FIG. 7 is a diagram showing an example of the contents of the image IM1 displayed on the display / operation device 36 based on the instruction of the voice-calling unit 230c. For example, the agent utters the contents such as "Are you okay?" And "Are you feeling unwell?" To the user U1. In addition, the agent may call the name of the user U1. The contents of these calls (utterance contents) are registered in advance in, for example, the response rule DB258a at the time of an accident of an occupant included in the response rule DB258. The presence / absence and degree of the reaction of the user U1 to these calls are acquired through the microphone 32 and the vehicle interior camera 34, processed by the sound processing unit 112 and the image processing unit 113, and output to the incident determination unit 230d of the agent server 200. Will be done.

The incident determination unit 230d changes the physical condition of the user U1 based on the comparison result by the voice observation unit 230a, the comparison result by the appearance observation unit 230b, the reaction of the user U1 after the voice is spoken by the voice observation unit 230c, and the like. Whether or not (whether or not the state of the user U1 has changed to a predetermined state) is determined.

For example, in the incident determination unit 230d, the degree of coincidence between the voice stream obtained by the sound processing unit 112 and the voice-related state pattern (state pattern at the time of physical condition change) registered in the occupant state pattern DB 260 is equal to or higher than the threshold value. In this case, it is determined that the physical condition of the user U1 has changed. Further, the incident determination unit 230d is used when the degree of deviation between the voice stream obtained by the sound processing unit 112 and the normal voice for each user U registered in the personal profile 252 is equal to or higher than the threshold value. It is determined that the physical condition of the person U1 has changed.

Similarly, the incident determination unit 230d has an appearance (including gestures, facial expressions, etc.) included in the image obtained by the image processing unit 113, and a state pattern (physical condition abnormality) related to the appearance registered in the occupant state pattern DB 260. When the degree of agreement with the time state pattern) is equal to or greater than the threshold value, it is determined that the physical condition of the user U1 has changed. Further, when the degree of deviation between the appearance included in the image obtained by the image processing unit 113 and the normal appearance of each user U registered in the personal profile 252 is equal to or higher than the threshold value in the incident determination unit 230d. In addition, it is determined that the physical condition of the user U1 has changed.

Further, when the incident determination unit 230d makes a voice call by the voice call unit 230c, but there is no reaction of the user U1 to the voice call, or when the degree of the reaction of the user U1 is less than the threshold value, the user U1 Judge that something is wrong with your physical condition. The determination result of the incident determination unit 230d is output to, for example, the occupant accident behavior content determination unit 232 and the notification unit 234.

The abnormality determination unit 230d may refer to the medical history information 252a included in the personal profile 252 when determining an abnormality in the physical condition of the user U1. FIG. 8 is a diagram showing an example of the contents of the medical history information 252a. The medical history information 252a is information in which the chronic disease and the pre-existing illness of the user U are registered for each user U. The medical history information 252a is information registered arbitrarily by the user U.

For example, the abnormality determination unit 230d can sensitively detect an abnormality in a typical symptom (for example, convulsions or seizure) in a chronic disease or a pre-existing condition for each user U by referring to the medical history information 252a. And one or more thresholds used in the comparison by the appearance observer 230b may be individually adjusted (eg, lower or higher).

[4.3 Action content determination department when occupants change]
The behavior content determination unit 232 at the time of an occupant change (hereinafter referred to as "behavior content determination unit 232") protects the user U1 when the change determination unit 230d determines that the physical condition of the user U1 has changed. Decide what action to take to do. FIG. 9 is a configuration diagram of the action content determination unit 232. The action content determination unit 232 includes, for example, an emergency level determination unit 232a, a request questioning unit 232b, and an action selection unit 232c.

The emergency level determination unit 232a is used based on the state of the user U observed by the state observation unit 230 (for example, the comparison result by the voice observation unit 230a and the appearance observation unit 230b, the reaction to the voice by the voice observation unit 230c). It is determined whether or not the condition of the person U1 is equal to or higher than a predetermined emergency level. A given emergency level is an imminent condition in which an ambulance needs to be arranged. The predetermined emergency level is, for example, when the user U1 has a large convulsion or there is no response to the voice call by the voice call unit 230c. The determination result of the emergency level determination unit 232a is output to the action selection unit 232c.

The request-questioning unit 232b asks the user U1 to select the content of the action in order to protect the user U1 when the user U1 is conscious (for example, when there is a reaction to the voice by the voice-calling unit 230c). Perform the processing for. For example, the request questioning unit 232b outputs a predetermined instruction to the agent device 100, and then uses the display / operation device 36 and the speaker unit 38 of the vehicle M to give the user U1 an action desired by the user U1. Ask a question (speaking to user U1) to have the user select. The question is asked, for example, by displaying the agent image on the display device of the display / operation device 36 and making the agent appear virtually.

FIG. 10 is a diagram showing an example of the contents of the image IM2 displayed on the display / operation device 36 based on the instruction of the request questioning unit 232b. The agent utters to the user U1, for example, "Do you want to call an ambulance?", "Do you want to go to a nearby hospital as it is?", "Do you want to return to your home?" The reaction of the user U1 to the question by the request questioning unit 232b is acquired through the microphone 32 and the vehicle interior camera 34, processed by the sound processing unit 112 and the image processing unit 113, and output to the action selection unit 232c of the agent server 200. To.

Returning to FIG. 9, the action selection unit 232c will be described. The action selection unit 232c selects an action necessary to protect the user U1 when the change determination unit 230d detects that an abnormality has occurred in the physical condition of the user U1 while using the ride sharing service. decide.

For example, the action selection unit 232c determines the content to be requested to the ride sharing server 300 based on the determination result of the emergency level determination unit 232a and the reaction (response content) of the user U1 to the question of the request inquiry unit 232b. .. For example, the action selection unit 232c decides to arrange an ambulance when the emergency level determination unit 232a determines that the state of the user U1 is equal to or higher than a predetermined emergency level. In addition, the action selection unit 232c decides to arrange an ambulance when there is a reaction of "I want you to call an ambulance" as a reaction of the user U1 to the question by the request questioning unit 232b.

On the other hand, the action selection unit 232c wants the user U1 to go to the hospital (for example, a nearby hospital or a hospital designated by the user U1 via an agent) as it is as a reaction of the user U1 to the question by the request questioning unit 232b. If there is a reaction, it is decided to change the destination of vehicle M to a hospital. Further, when the action selection unit 232c has a reaction that the user U1 wants to go to the home of the user U1 as a reaction of the question by the request questioning unit 232b, the destination of the vehicle M is set to the home of the user U1. Decide to change. The decision content of the action selection unit 232c is output to the notification unit 234.

Returning to FIG. 5, the notification unit 234 will be described. When the incident determination unit 230d detects that the physical condition of the user U1 has changed while using the ride sharing service, the notification unit 234 communicates with the ride sharing server 300 via the network NW to share the ride. A predetermined notification is given to the server 300. The predetermined notification is, for example, a notification indicating that an abnormality has occurred in the physical condition of the user U1. In the present embodiment, the notification unit 234 further requests the ride sharing server 300 for the content of the action selected (determined) by the action selection unit 232c.

For example, the notification unit 234 requests the ride sharing server 300 to arrange an ambulance when the action selection unit 232c determines that the ambulance is to be arranged. When the action selection unit 232c determines that the destination of the vehicle M is changed to the hospital, the notification unit 234 requests the ride sharing server 300 to change the destination of the vehicle M. Similarly, when the action selection unit 232c determines that the destination of the vehicle M is changed to the home of the user U1, the notification unit 234 requests the ride sharing server 300 to change the destination of the vehicle M. To do.

In addition, the notification unit 234 transmits information indicating the state (condition and symptom) of the user U1 in which the physical condition has changed to the ride sharing server 300. The notification unit 234 links the identification information (for example, the user ID) indicating the user U1 whose physical condition has changed based on the seating position of the user U1 and the state of the user U1 whose physical condition has changed. It is attached and transmitted to the ride sharing server 300. As a result, the ride-sharing server 300 can recognize which user U's physical condition has changed.

Further, the notification unit 234 requests the ride sharing server 300 to notify the passenger U2 of the vehicle M that the physical condition of the user U1 has changed. When the action selection unit 232c determines that the action selection unit 232c arranges the ambulance, the notification unit 234 requests the ride sharing server 300 to notify the passenger U2 of the vehicle M that the ambulance will be arranged. When the action selection unit 232c determines that the destination of the vehicle M is changed, the notification unit 234 changes the destination of the vehicle M to the passenger U2 of the vehicle M with respect to the ride sharing server 300. I request that you inform me that I will arrange an alternative vehicle for passenger U2. Further, when the action selection unit 232c determines that the destination of the vehicle M is changed, the notification unit 234 requests the ride sharing server 300 to arrange an alternative vehicle to be used by the passenger U2 of the vehicle M. To do.

In addition, the notification unit 234 rides on the fact that the physical condition of the user U1 has changed, arranges an ambulance, changes the destination of the vehicle M, arranges an alternative vehicle for the passenger U2, and the like. Instead of asking the ride-sharing server 300 to notify the person U2, the physical condition of the user U1 has changed, an ambulance is arranged, the destination of the vehicle M is changed, and the passenger U2 In the agent device 100 of the vehicle M or another agent server 200 (for example, the agent servers 200-1, 200-2, 200-3), the passenger U2 is informed that an alternative vehicle will be arranged for the purpose. , You may request an agent server 200) different from the agent server 200 that recognizes that the physical condition of the user U1 has changed. In this case, the agent device 100 or the agent server 200 that has received the request causes the display / operation device 36 corresponding to the passenger U2 to display the agent image, causes a virtual agent to appear, and informs the passenger U2 of the above contents. To do.

[5. Ride share server]
[5.1 Rideshare server configuration]
FIG. 11 is a configuration diagram of the ride sharing server 300. The ride-sharing server 300 includes, for example, a communication unit 310, an acquisition unit 320, an operation management unit 330, an occupant accident response unit 340, and a storage unit 380.

The communication unit 310 is, for example, a network card for connecting to the network NW. The communication unit 310 communicates with the terminal device D and the vehicle M via the network NW. Further, the storage unit 380 is realized by an HDD (Hard Disk Drive), a flash memory, a RAM (Random Access Memory), a ROM (Read Only Memory), or the like.

The acquisition unit 320, the operation management unit 330, and the occupant accident response unit 340 are realized by, for example, a processor such as a CPU executing a program (software) stored in the storage unit 380. Further, some or all of these functional parts may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or software. And may be realized by the cooperation of hardware.

The acquisition unit 320 acquires a boarding request issued from the terminal device D of the user U via the communication unit 310 and the network NW, and stores the boarding conditions included in the boarding request as boarding condition information 384 in the storage unit 380. sign up. The user information 382 includes personal information of the service applicant A, the service user U, the service provider S, and the like. The user information 382 includes the home address of the user U.

FIG. 12 is a diagram showing an example of the contents of the boarding condition information 384. The boarding condition information 384 is a vehicle allocation flag (for example, 1) indicating a desired boarding point, a destination, a desired boarding time, and whether or not the vehicle allocation has been decided for the user ID which is the identification information of the user U registered in advance. Is the information that has been decided to be dispatched, and 0 indicates that the vehicle has not been dispatched). The content of the information other than the vehicle allocation flag is determined by the ride sharing application of the terminal device D accepting the input of the user U, and is transmitted to the ride sharing server 300 as a boarding request. Hereinafter, a series of information associated with one user ID in the boarding condition information 384 may be referred to as a record.

The operation management unit 330 refers to the boarding condition information 384, the map information 386, and the operation schedule information 388, and searches for the vehicle M that can be dispatched. The map information 386 includes facility information showing an outline of various facilities in addition to node and link information (even if such information is included in the navigation map and high-precision map of vehicle M). Good). For example, the operation management unit 330 roughly groups the records included in the boarding condition information 384 that are close to the time zone and the traveling section from the desired boarding point to the destination, and further, one vehicle M. One or more records corresponding to one or more users U that can be carried by the user U are extracted and registered in the storage unit 380 as a part of the operation schedule information 388.

FIG. 13 is a diagram showing an example of the contents of the operation schedule information 388. The operation schedule information 388 is the coordinates of the departure place, the waypoint, and the arrival place, the estimated arrival time of the vehicle M, and each way for the vehicle ID which is the identification information of the vehicle M managed by the ride sharing server 300. This is information in which the user ID of the user U who gets on the ground and the user ID of the user U who gets off the vehicle are associated with each other. The departure and arrival points are usually garages and the like. In the operation schedule information 388, information on "vacant vehicles" whose operation schedule has not yet been determined is also registered. In this case, only the coordinates of the departure place are registered for the vacant vehicle. As described above, the operation management unit 330 may collectively determine the operation schedule of one vehicle M by collecting the boarding requests from the plurality of users U, or search for the already determined operation schedule and separate it. The operation schedule may be changed by incorporating the boarding request of the user U. That is, when searching for a vehicle M that can be dispatched, the operation management unit 330 may search for a vehicle M whose operation schedule has not yet been determined, or may search for a vehicle M whose operation schedule has already been determined. You may search for the one that can include the boarding request of the user U. The operation management unit 330 transmits information on the route (waypoint) based on the operation schedule information 388 and the estimated passing time to the vehicle M at a predetermined timing. When the vehicle M is a manually driven vehicle, the operation management unit 330 notifies the terminal device of the driver of the vehicle M or the in-vehicle terminal device of the change request of the boarding request.

[5.2 Crew Accident Response Department]
When the agent system E detects that the physical condition of the user U1 has changed while using the ride-sharing service, and receives a predetermined request from the agent server 200, the occupant accident response unit 340 receives a predetermined request from the agent server 200. Perform the requested action. FIG. 14 shows a configuration diagram of the occupant accident response unit 340. The occupant accident response unit 340 has, for example, an ambulance arrangement unit 340a, a medical institution communication unit 340b, an operation plan change unit 340c, an alternative vehicle arrangement unit 340d, and a post-change operation plan guidance unit 340e.

When the agent server 200 requests the ambulance arrangement unit 340a, the ambulance arrangement unit 340a arranges the ambulance. For example, the ambulance arrangement unit 340a communicates with the web server 400-1 of the disaster emergency information center via the network NW. The ambulance arranging unit 340a indicates that there is a user U1 who needs to arrange an ambulance for the web server 400-1, the current position of the vehicle M1, and the ambulance and the vehicle M generated by the operation plan changing unit 340c described later. The web server 400-1 is notified of the merging candidate position, the state (condition and symptom) of the user U1 transmitted from the agent server 200, and the like.

When the destination of the vehicle M is changed to a hospital, the medical institution liaison unit 340b searches for a candidate hospital to which the vehicle M heads based on the medical institution information 390 registered in the storage unit 380. In the medical institution information 290, information such as the location of each medical institution, reception hours, and holidays is registered. The medical institution liaison unit 340b selects a candidate for the hospital to which the vehicle M is heading based on the current position, day of the week, time, and the like of the vehicle M. When the medical institution liaison unit 340b selects a candidate for the hospital to which the vehicle M heads, it communicates with the web server 400-2 of the selected hospital. The medical institution liaison unit 340b indicates that there is a user U1 who wishes to be accepted by the web server 400-2 of the selected hospital, the current position of the vehicle M, and the vehicle M calculated by the operation plan change unit 340c described later. Notifies the web server 400-2 of the estimated time of arrival of the user, the state (condition and symptom) of the user U1 transmitted from the agent server 200, and the like. If the web server 400-2 can respond to the acceptance status of the hospital, the medical institution liaison unit 340b may confirm whether the hospital accepts the status by communicating with the web server 400-2. ..

The operation plan change unit 340c changes the operation plan of the vehicle M. For example, when the agent server 200 requests the arrangement of an ambulance, the operation plan change unit 340c searches for a merging candidate position suitable for merging with the ambulance based on the map information 386. The merging candidate position is, for example, a place where the vehicle M and the ambulance do not interfere with the traffic even if the vehicle M stops, for example, a parking lot of a park or a public facility near the current position of the vehicle M. When the merging candidate position is determined, the operation plan changing unit 340c generates a movement plan for moving to the merging candidate position, and notifies the vehicle M of the generated action plan.

Further, when the agent server 200 requests the change of the destination of the vehicle M, the operation plan change unit 340c searches for a candidate position for disembarking the passenger U2 of the vehicle M. The disembarkation candidate position is searched based on, for example, map information 386 and affiliated facility information 392 recorded in the storage unit 380. In the affiliated facility information 392, affiliated facilities affiliated with the operator of the ride sharing service are registered. The affiliated facility is preferably a facility with seats such as a cafe. In addition, it is preferable that the affiliated facilities are provided with seats exclusively for ride-sharing service users, or that discount services are provided for ride-sharing service users. When the disembarkation candidate position is determined, the operation plan change unit 340c generates a movement plan for moving to the changed destination (for example, a hospital or home) via the disembarkation candidate position, and generates an action plan. Notify vehicle M.

The alternative vehicle arranging unit 340d searches for a new vehicle M that can be dispatched by referring to the boarding condition information 384 of the passenger U2, the map information 386, and the operation schedule information 388. The alternative vehicle arranging unit 340d determines the newly dispatchable vehicle M as the alternative vehicle. The alternative vehicle arranging unit 340d generates a movement plan for directing the newly dispatched vehicle M (alternative vehicle) to the disembarkation candidate position determined by the operation plan changing unit 340c, and uses the determined movement plan as the vehicle M (alternative vehicle). Notify the vehicle).

After the change, the driving plan guide unit 340e rides the disembarkation candidate position generated by the operation plan changing unit 340c and the estimated arrival time when the vehicle M (alternative vehicle) arranged by the alternative vehicle arranging unit 340d arrives at the disembarkation candidate position. Notify person U2. For example, the changed driving plan guide unit 340e communicates with the agent server 200, uses the display / operation device 36 of the vehicle M and the speaker unit 38 to make a virtual agent corresponding to the passenger U2 appear, and the agent causes the above. The information is notified to the passenger U2.

The information indicating which passenger U2 is using which agent is, for example, the identification information (user ID) of the passenger U2 and the agent that the passenger U2 is using most frequently in the vehicle M. Is associated with the information indicating the above and is transmitted in advance from the agent device 100 to the ride sharing server 300. As a result, the changed operation plan guide unit 340e should communicate with any of the agent servers 200-1, 200-2, and 200-3 in order to make the agent corresponding to each passenger U2 appear. You can specify whether it is good or not.

According to such a configuration, when it is detected that the state of the user U has changed to a predetermined state while using the ride sharing service, the agent system E that communicates with the ride sharing server 300 is provided. , The convenience of the ride-sharing service user U can be improved. For example, when the physical condition of the user U1 changes, the agent system E and the ride sharing server 300 cooperate with each other to allow the user U1 to receive appropriate protection. As a result, the convenience of the user U can be improved.

In the present embodiment, when the physical condition of the user U1 changes and the destination of the vehicle M is changed, a substitute vehicle is arranged for the passenger U2. According to such a configuration, it is possible to reduce the disadvantage to the passenger U2 while protecting the user U1.

<Second Embodiment>
Next, the second embodiment will be described. In the second embodiment, when the physical condition of the user U who is riding in the vehicle M that provides the ride sharing service changes, the ride sharing server 300 proactively takes action to protect the user U. This is an example of determining. The configuration other than that described below is the same as that of the first embodiment.

FIG. 15 is a configuration diagram of the agent server 200A according to the second embodiment. The agent server 200A of the present embodiment does not have an action content determination unit 232. Further, the agent server 200A of the present embodiment has a notification unit 234A instead of the notification unit 234 of the first embodiment. Other than that, the configuration of the agent server 200A is the same as that of the agent server 200 of the first embodiment.

In the present embodiment, the determination result by the incident determination unit 230d of the state observation unit 230 is output to the notification unit 234A. When the incident determination unit 230d determines that the physical condition of the user U1 has changed, the notification unit 234A gives a predetermined notification to the ride sharing server 300A. The predetermined notification is, for example, a notification indicating that an abnormality has occurred in the physical condition of the user U1. The notification unit 234A further transmits information indicating the state (condition and symptom) of the user U1 (voice stream obtained by the sound processing unit 112 and an image obtained by the image processing unit 113) to the ride sharing server 300A. ..

FIG. 16 is a configuration diagram of the ride-sharing server 300A according to the second embodiment, and for convenience of explanation, only the occupant accident response unit 340A is shown. The occupant accident response unit 340A of the present embodiment has an action content determination unit 232. Other than that, the configuration of the ride-sharing server 300A is the same as that of the ride-sharing server 300 of the first embodiment. The action content determination unit 232 of the ride-sharing server 300A (action content determination unit 232 when the occupant changes) has substantially the same configuration and function as the action content determination unit 232 of the agent server 200A of the first embodiment.

That is, in the present embodiment, when the action content determination unit 232 of the ride sharing server 300A receives a notification from the agent server 200A indicating that the physical condition of the user U1 has changed (the state of the user U1 by the agent system E). (When it is detected that the vehicle has changed to a predetermined state), arrange an ambulance based on the information indicating the state (condition and symptom) of the user U1 as in the action content determination unit 232 of the first embodiment. , The destination of the vehicle M is changed to a hospital, the destination of the vehicle M is changed to a home, and the like is selected and determined. In the present embodiment, the ambulance arrangement unit 340a, the medical institution communication unit 340b, the operation plan change unit 340c, the alternative vehicle arrangement unit 340d, and the changed operation plan guidance unit 340e of the ride share server 300A are the action contents of the ride share server 300A. Various arrangements are made based on the action content determined by the decision unit 232.

Even with such a configuration, it is possible to improve the convenience of the ride-sharing service user U as in the first embodiment.

<Modification example>
Next, modifications of the first and second embodiments will be described. In this modified example, when there is a high possibility that the physical condition of the user U1 has changed, the agent functions to prompt the passenger U2 to check the state of the user U1. It is different from the first and second embodiments. The configuration other than that described below is the same as that of the first or second embodiment. In this modification, it is assumed that, for example, there is no partition P in the vehicle interior that separates the seats, or the partition P can be moved to a retracted position that does not interfere with the passenger U2.

FIG. 17 is a configuration diagram of the agent server 200B according to the modified example. The agent server 200B of this modification has a passenger voice promotion unit 230e. When the peer-speaking promotion unit 230e determines that there is a high possibility that the physical condition of the user U1 has changed based on the comparison results of the voice observation unit 230a and the appearance observation unit 230b (the calculated value indicating the probability is calculated). (When it is equal to or more than the threshold value), by outputting a predetermined instruction to the agent device 100, guidance to the same trader U2 is performed using the display / operation device 36 and the speaker unit 38 of the vehicle M.

The guidance to the same trader U2 includes a request indicating that there is a possibility that the physical condition of the user U1 has changed and that the user U1 should perform a predetermined action. The predetermined action is an action for confirming the state of the user U1 more deeply, for example, by calling out to the user U1 or shaking the shoulder of the user U1. Guidance to the same trader U2 is performed by displaying the agent image on the display device of the display / operation device 36 corresponding to the passenger U2 and making the agent appear virtually. In this case, the agent is preferably, for example, the agent that the passenger U2 frequently uses in the vehicle M. The guidance to the same trader U2 may be performed by using the terminal device D2 carried by the same trader U2.

In this modification, the presence or absence and degree of reaction of the user U1 to a predetermined action of the passenger U2 are acquired through the microphone 32 and the vehicle interior camera 34, processed by the sound processing unit 112 and the image processing unit 113, and then processed by the agent. It is output to the incident determination unit 230d of the server 200. When the passenger U2 performs a predetermined action and there is no reaction of the user U1 to the predetermined action, or when the degree of the reaction of the user U1 is less than the threshold value, the incident determination unit 230d is used. It is determined that the physical condition of the user U1 has changed. The determination result of the incident determination unit 230d is output to the occupant accident behavior content determination unit 232 and the notification unit 234.

Even with such a configuration, it is possible to improve the convenience of the ride-sharing service user U as in the first embodiment. Further, in the present embodiment, by having the passenger U2 cooperate in the determination regarding the physical condition of the user U1, the determination regarding the physical condition of the user U1 can be performed more accurately.

In the present specification, "based on XX" means "based on at least XX", and includes the case where it is based on another element in addition to XX. Further, "based on XX" is not limited to the case where XX is directly used, but also includes the case where it is based on the case where calculation or processing is performed on XX. "XX" is an arbitrary element (for example, arbitrary information). As used herein, the term "computer" means an information processing function unit realized by one or more hardware processors such as a CPU and an MPU.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added. For example, a part or all of the functions of the agent server 200 (for example, a state detection unit 230, a occupant change action content determination unit 232, and a part or all of the notification unit 234) may be provided in the agent device 100. It may be realized by an application program installed in the terminal device D or the like. The ride-sharing server 300 may be mounted on the vehicle M as a ride-sharing management device. Further, when the vehicle M is a manually driven vehicle, the communication unit 310 may communicate with the terminal device of the driver of the vehicle M via the network NW.

1 Information processing system 32 Microphone (vehicle-mounted device)
34 In-vehicle camera (vehicle-mounted device)
36 Display / operation device 38 Speaker unit 100 Agent device 200 Agent server 230 Status observation unit 232 Crew change action content determination unit 234 Notification unit 300 Ride share server 340 Crew change response unit E Agent system M Vehicle D Terminal device (mobile terminal) apparatus)

Claims (15)

A ride-sharing server device that generates a vehicle operation plan based on a boarding request from a user and provides a ride-sharing service using the vehicle.
Use of the ride-sharing service by observing the state of the user while riding in the vehicle based on information acquired by at least one of the user's mobile terminal device and the in-vehicle device mounted on the vehicle. An agent system that communicates with the ride-sharing server device when it is detected that the user's state has changed to a predetermined state.
Information processing system equipped with. When it is detected that the state of the user has changed to the predetermined state, the agent system determines the necessity of arranging an ambulance based on the state of the user, and the ambulance needs to be arranged. If it is determined, request the ride-sharing server device to arrange an ambulance.
The information processing system according to claim 1. When it is detected that the state of the user has changed to the predetermined state, the agent system requests the ride-sharing server device to change the destination of the vehicle.
The information processing system according to claim 1 or 2. The agent system requests the ride-sharing server device to contact the passenger of the vehicle when it is detected that the state of the user has changed to the predetermined state.
The information processing system according to any one of claims 1 to 3. When it is detected that the state of the user has changed to the predetermined state, the agent system requests the ride-sharing server device to arrange an alternative vehicle to be used by a passenger of the vehicle.
The information processing system according to any one of claims 1 to 4. When it is detected that the state of the user has changed to the predetermined state, the agent system transmits information indicating the state of the user to the ride-sharing server device.
The information processing system according to any one of claims 1 to 5. When the agent system detects that the user's state has changed to the predetermined state, the ride-sharing server device determines the necessity of arranging an ambulance based on the information received from the agent system. If it is determined that the ambulance needs to be arranged, the ambulance will be arranged.
The information processing system according to any one of claims 1 to 6. The ride-sharing server device changes the destination of the vehicle when it is detected by the agent system that the state of the user has changed to the predetermined state.
The information processing system according to any one of claims 1 to 7. The ride-sharing server device contacts the passenger of the vehicle when it is detected by the agent system that the state of the user has changed to the predetermined state.
The information processing system according to any one of claims 1 to 8. The ride-sharing server device arranges an alternative vehicle to be used by a passenger of the vehicle when the agent system detects that the state of the user has changed to the predetermined state.
The information processing system according to any one of claims 1 to 9. The first computer generates a vehicle operation plan based on a boarding request from the user, and provides a ride sharing service by the vehicle.
The second computer observes the state of the user while riding in the vehicle based on the information acquired by at least one of the user's mobile terminal device and the in-vehicle device mounted on the vehicle, and the state of the user is observed. When it is detected that the state of the user has changed to a predetermined state while using the ride sharing service, communication with the first computer is performed.
Information processing method. Let the first computer generate a vehicle operation plan based on the boarding request from the user, and provide the ride sharing service by the vehicle.
The second computer is made to observe the state of the user while riding in the vehicle based on the information acquired by at least one of the user's mobile terminal device and the in-vehicle device mounted on the vehicle. When it is detected that the state of the user has changed to a predetermined state while using the ride sharing service, communication with the first computer is performed.
program. A state observing unit that observes the state of the user while riding in the vehicle based on at least one of the mobile terminal device of the user who uses the ride sharing service of the vehicle and the in-vehicle device mounted on the vehicle. ,
A notification unit that communicates with the rideshare server device that manages the rideshare service when the state observation unit detects that the user's state has changed to a predetermined state while using the rideshare service. When,
Agent system with. The computer
Based on at least one of the mobile terminal device of the user who uses the ride sharing service of the vehicle and the in-vehicle device mounted on the vehicle, the state of the user while riding in the vehicle is observed.
When the state observing unit detects that the state of the user has changed to a predetermined state while using the ride-sharing service, it communicates with the ride-sharing server device that manages the ride-sharing service.
Information processing method. On the computer
Based on at least one of the mobile terminal device of the user who uses the ride sharing service of the vehicle and the in-vehicle device mounted on the vehicle, the state of the user while riding in the vehicle is observed.
When the state observing unit detects that the state of the user has changed to a predetermined state while using the ride sharing service, the ride sharing server device that manages the ride sharing service is made to communicate with the ride sharing server device.
program.

Images