JP2022148823A - Agent device - Google Patents

Abstract

To provide an agent device configured to efficiently specify information of an object to be presented, from among a plurality of targets located around a vehicle.SOLUTION: In an agent system configured by connecting an in-vehicle unit, an agent server, and an information providing server over a network, a CPU 30A of the agent server includes: a receiving unit 200 which receives a request from a user on a vehicle; a target information acquisition unit 230 which acquires target information on a target located around the current position of the vehicle; an indoor information acquisition unit 220 which acquires indoor information, which is at least one of a captured image of a user and voice information on a speech of the user; an estimation unit 240 which estimates a line of sight or a seating position of the user using the acquired indoor information and estimates a region that the user can visually recognize; a specifying unit 240 which specifies target information included in the estimated area, as an object; and a presentation unit 260 which presents the target information relating to the specified object.SELECTED DRAWING: Figure 4

Description

TECHNICAL FIELD The present invention relates to an agent device that provides an occupant of a vehicle with information on objects located around the vehicle.

Patent Literature 1 discloses a voice dialogue system that provides target information corresponding to an utterance of a speaker.

JP 2018-054790 A

The voice interaction system of Patent Document 1 determines whether or not the user's utterance is the content of the user's utterance expected as a response, based on the result of voice recognition of the user's utterance, and determines the content of the user's utterance expected as a response. , present the information corresponding to the user's utterance. However, when the information to be presented is information on a target (POI: Point Of Interest) such as a store or building located around the vehicle, there are a plurality of targets to be presented, and only the utterance content can identify the target to be presented. It was not always possible to identify information efficiently.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an agent device capable of efficiently specifying target information to be presented from a plurality of targets located around a vehicle.

1. The agent device according to claim 1, comprising: a reception unit that receives a request from a user in a vehicle; a target object information acquisition unit that acquires target information related to targets located around the current location of the vehicle; a vehicle interior information acquisition unit that acquires vehicle interior information that is at least one of a captured image of the user and voice information uttered by the user; an estimating unit for estimating an area visible to the user; a specifying unit for specifying the target information included in the estimated area as a target; and a presentation for presenting target information related to the specified target. and

In the agent device according to claim 1, when the receiving unit receives a request for target information about the target from the user in the vehicle, the target object information acquisition unit acquires the target object located around the current position of the vehicle. Get information. In the agent device, the vehicle interior information acquisition unit acquires vehicle interior information that is at least one of a captured image of the user and voice information uttered by the user, and the estimation unit uses the vehicle interior information to make the information visible to the user. to estimate the region. The agent device specifies a target included in the visible area by the specifying unit, and the presentation unit presents the target object information as the target requested by the user.

Note that the area visible to the user is estimated from, for example, the line of sight of the user captured by a camera or the position where the user is seated. Also, the line of sight of the user is estimated using, for example, a captured image, and the position where the user is seated is estimated using audio information detected by a plurality of microphones. According to the agent device, it is possible to efficiently identify the information of the target to be presented from a plurality of target candidates such as shops and buildings located around the vehicle.

ADVANTAGE OF THE INVENTION According to this invention, the information of the object to present can be efficiently specified from the several target object located around a vehicle.

1 is a diagram showing a schematic configuration of an agent system according to this embodiment; FIG. It is a block diagram showing the hardware constitutions of the vehicle concerning this embodiment. 3 is a block diagram showing the hardware configuration of an agent server according to this embodiment; FIG. 3 is a block diagram showing the functional configuration of an agent server according to this embodiment; FIG. It is a figure which shows the content of the visible list|wrist which concerns on this embodiment. 4 is a flow chart showing the flow of processing executed in the agent system according to this embodiment;

An agent system including the agent device of the present invention will be described. The agent device is positioned around a point where the vehicle travels among the functions of the vehicle, and functions as an agent that presents to the occupant of the vehicle information related to an object of interest to the occupant of the vehicle.

[First embodiment]
As shown in FIG. 1, the agent system 10 of this embodiment includes a vehicle 12, an agent server 30 as an agent device, and an information providing server 40. FIG. The vehicle 12 includes an onboard device 20 and a plurality of ECUs (Electronic Control Units) 22 that are control devices. The vehicle-mounted device 20, the agent server 30, and the information providing server 40 are interconnected via a network N.

The information providing server 40 is a server that provides information on targets such as shops and buildings (hereinafter referred to as "target information"). In the present embodiment, among multiple pieces of target object information, information relating to an object that is located around the current location where the vehicle 12 is traveling and that the occupant of the vehicle 12 is interested in will be described as the object information. Further, the target information and target information according to the present embodiment include the name of the target, the image of the target, text, an icon indicating the target, and the position information of the target.

The vehicle-mounted device 20 has a function of acquiring communication information based on the CAN protocol transmitted from each ECU 22 and transmitting it to the agent server 30 .

A vehicle control ECU, an engine ECU, a brake ECU, a body ECU, a camera ECU, and a multimedia ECU are exemplified as the ECU 22 of the present embodiment. The vehicle-mounted device 20 and each ECU 22 are interconnected via an external bus 29 .

As shown in FIG. 2, the vehicle-mounted device 20 includes a CPU (Central Processing Unit) 20A, a ROM (Read Only Memory) 20B, a RAM (Random Access Memory) 20C, an in-vehicle communication I/F (Inter Face) 20D, wireless communication It is configured including an I/F 20E and an input/output I/F 20F. The CPU 20A, ROM 20B, RAM 20C, in-vehicle communication I/F 20D, wireless communication I/F 20E, and input/output I/F 20F are communicably connected to each other via an internal bus 20G.

The CPU 20A is a central processing unit that executes various programs and controls each section. That is, the CPU 20A reads a program from the ROM 20B and executes the program using the RAM 20C as a work area.

The ROM 20B stores various programs and various data. A control program for controlling the vehicle-mounted device 20 is stored in the ROM 20B of the present embodiment.
The RAM 20C temporarily stores programs or data as a work area.

In-vehicle communication I/F 20D is an interface for connecting with ECU 22 . The interface performs communication according to the CAN (Controller Area Network) protocol. Note that the in-vehicle communication I/F 20D may employ a communication standard based on Ethernet (registered trademark). In-vehicle communication I/F 20D is connected to external bus 29 . That is, in the external bus 29, the data transmitted/received between the onboard equipment 20 and each ECU22 is transmitted/received as a communication frame based on a CAN protocol.

The wireless communication I/F 20E is a wireless communication module for communicating with the agent server 30 and the information providing server 40. FIG. The wireless communication module uses communication standards such as 5G, LTE, Wi-Fi (registered trademark), for example. A wireless communication I/F 20E is connected to the network N.

The input/output I/F 20F is an interface for communicating with the microphone 24, the speaker 25, the monitor 26, the camera 27, and the GPS device 28 mounted on the vehicle 12. FIG. The microphone 24, speaker 25, monitor 26, camera 27, and GPS device 28 may be directly connected to the internal bus 20G.

The microphone 24 is provided on an instrument panel, a center console, a front pillar, a dashboard, or the like, and is a device that collects sounds uttered by the occupant of the vehicle 12 . In addition, in this embodiment, a form in which a plurality of microphones 24 are installed in the vehicle 12 will be described.

The speaker 25 is provided on an instrument panel, a center console, a front pillar, a dashboard, or the like, and is a device for outputting sound. Further, the speaker 25 of the present embodiment outputs the sound of presenting the target information.

The monitor 26 is a liquid crystal monitor provided on an instrument panel, dashboard, or the like of the vehicle 12 for displaying various information. Also, the monitor 26 of this embodiment displays images and texts included in the target information.

The camera 27 is an imaging device that is provided above the front window and adjacent to the rearview mirror to capture an image of an occupant riding in the vehicle 12 . A mode in which the camera 27 captures a plurality of captured images will be described.

GPS device 28 is a device that measures the current location of vehicle 12 . GPS device 28 includes an antenna (not shown) that receives signals from GPS satellites. The GPS device 28 may be connected to the vehicle-mounted device 20 via a car navigation system (not shown).

The agent server 30 grasps the control state of the vehicle 12 by acquiring the communication information of the vehicle 12 from the vehicle-mounted device 20 . Also, the agent server 30 presents target information to the passengers of the vehicle 12 .

As shown in FIG. 3, the agent server 30 includes a CPU 30A, ROM 30B, RAM 30C, storage 30D and communication I/F 30E. The CPU 30A, ROM 30B, RAM 30C, storage 30D and communication I/F 30E are communicably connected to each other via an internal bus 30G. The functions of the CPU 30A, ROM 30B, RAM 30C, and communication I/F 30E are the same as those of the CPU 20A, ROM 20B, RAM 20C, and wireless communication I/F 20E of the vehicle-mounted device 20 described above. Note that the communication I/F 30E may perform wired communication.

The storage 30D as a storage unit is configured by a HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various programs and various data. A processing program 100 and a visible list 110 are stored in the storage 30D of this embodiment. Note that the processing program 100 may be stored in the ROM 30B.

A processing program 100 is a program for controlling the agent server 30 . The visible list 110 is a list that predetermines directions that can be visually recognized by the occupant from the seating position of the occupant.

As shown in FIG. 4, in the agent server 30 of the present embodiment, the CPU 30A executes the processing program 100 to obtain a reception unit 200, an intention understanding unit 210, an in-vehicle information acquisition unit 220, and a target information acquisition unit 230. , an estimation unit 240 , an identification unit 250 , and a presentation unit 260 .

The receiving unit 200 has a function of receiving communication information from the vehicle-mounted device 20 . Here, the communication information includes the information of the voice uttered by the passenger acquired by the multiple microphones 24 (hereinafter referred to as "voice information"), the multiple captured images captured by the camera 27, and the GPS device of the vehicle 12. It contains the current location of the vehicle 12 obtained using 28. The communication information also includes the seating position of the occupant acquired by a sensor that detects the seat belt (not shown).

Specifically, the receiving unit 200 receives audio information related to the voice uttered into the microphone 24 by the passenger on board the vehicle 12, the captured image captured when the utterance was made, and the vehicle 12 when the utterance was made. and the current location of the vehicle from the vehicle-mounted device 20 to receive the communication information.

The intention understanding unit 210 has a function of analyzing and understanding the intention of the occupant's speech from the voice information. The intention understanding unit 210 analyzes the intention that the occupant is requesting the target information related to the target from the speech received by the reception unit 200 . For example, the intention understanding unit 210 analyzes the passenger's utterance "Can you tell me the information about what you see over there?" and acquires that the passenger is requesting presentation of target information.

The in-vehicle information acquisition unit 220 has a function of acquiring captured images and audio information from communication information as in-vehicle information.

The target information acquisition unit 230 has a function of acquiring target information located around the vehicle 12 . Specifically, the target object information acquisition unit 230 transmits the position information of the vehicle 12 to the information providing server 40, and acquires the target object information related to the target objects located around the current location of the vehicle 12 from the information providing server 40. do.

The estimating unit 240 identifies an occupant of the vehicle 12 who has spoken to request the target information (hereinafter referred to as a "speaker"), and estimates an area visible to the speaker. Specifically, the estimation unit 240 detects a state in which the passenger is speaking from the captured image acquired by the in-vehicle information acquisition unit 220, and estimates the speaker. The estimation unit 240 also detects the direction of the line of sight of the speaker and estimates the visible area of the speaker.

Here, the region that can be visually recognized by the speaker is a horizontal viewing angle range that can be visually recognized by the speaker, with the direction of the line of sight of the speaker as a base axis. Note that the viewing angle may be set to a predetermined value such as 30 degrees, or may be set according to the speed of the vehicle 12 .

For example, when the speaker looks at the window on the left side of the rear seat of the vehicle 12 in the traveling direction and says, "Can you tell me the information about what you see over there?" The speaker is estimated by detecting the movement of the lips when the speaker speaks. In addition, the estimation unit 240 detects the direction of the line of sight of the speaker looking at the window on the left side of the traveling direction of the rear seat of the vehicle 12, and detects the direction of the line of sight of the speaker looking at the left side of the traveling direction. Estimate the area.

The estimation unit 240 estimates the speaker using the voice information acquired by the in-vehicle information acquisition unit 220 when the speaker cannot be estimated from the captured image. Specifically, using voice information acquired from a plurality of microphones 24 , the seating position of the speaker is detected from the volume of sound captured by each microphone 24 . In addition, the estimation unit 240 estimates the visible area from the sitting position of the speaker.

Here, the visible area is predetermined from the seating position, as shown in the visible list 110 shown in FIG. 5 as an example, and the visible area is estimated from the sitting position of the speaker. In addition, when an occupant is seated next to the speaker, the direction in which the occupant is sitting may not be visible.

For example, when the speaker is seated on the left side of the rear seat of the vehicle 12 in the traveling direction and the passenger is seated on the right side, the estimation unit 240 refers to the visible list 110 in FIG. to get the direction "left, right". Furthermore, the estimating unit 240 excludes the visible direction “right” because the passenger is seated on the right side of the speaker, and determines that the visible direction for the speaker is “left”. Estimate the visible area from the direction. The occupant sitting next to the speaker may be detected from the captured image, or may be detected using a seat belt sensor (not shown).

The specifying unit 250 specifies a target included in the estimated visible area from the target object information. Specifically, the identifying unit 250 compares the visible area with the position information of the target, and determines that the target included in the visible area is the target for which the speaker has requested presentation. to identify the target. Further, when there are a plurality of specified targets, the specifying unit 250 presents the characteristics of each target to the user to specify the target target.

The presentation unit 260 has a function of presenting target information to the occupants of the vehicle 12 . Specifically, the presentation unit 260 presents target object information related to the target specified by the specifying unit 250 as the target information. Here, the target information is presented using the speaker 25 and monitor 26 mounted on the vehicle 12 .

(control flow)
The flow of processing executed in the agent system 10 of this embodiment will be described using the flowchart of FIG. The processing in the agent server 30 is realized by the CPU 30A functioning as the reception unit 200, the intention understanding unit 210, the vehicle interior information acquisition unit 220, the target information acquisition unit 230, the estimation unit 240, the identification unit 250, and the presentation unit 260. be done.

In step S<b>100 , the CPU 30</b>A acquires communication information including the captured image, audio information, and the current location of the vehicle 12 .

In step S101, the CPU 30A determines whether or not the passenger has uttered a guidance command based on the acquired voice information. If the passenger utters a guidance command (step S101: YES), the CPU 30A proceeds to step S102. On the other hand, if the passenger does not utter a guidance command (step S101: NO), the CPU 30A proceeds to step S101 and acquires communication information. Here, the guidance command is, for example, voice information requesting presentation of target information by the passenger, such as "Tell me information about what you see over there?"

In step S102, CPU 30A identifies the speaker and the sitting position of the speaker using the captured image. Also, the CPU 30A identifies the sitting position of the speaker using the voice information.

In step S103, the CPU 30A detects the line of sight of the speaker from the captured image. Further, the CPU 30A acquires the visible direction from the visible list 110 using the speaker's sitting position.

In step S<b>104 , the CPU 30</b>A acquires target object information from the information providing server 40 using the current location of the vehicle 12 .

In step S105, the CPU 30A determines whether or not the line of sight of the speaker has been detected from the captured image. If the line of sight of the speaker can be detected from the captured image (step S105: YES), the CPU 30A proceeds to step S106. On the other hand, if the line of sight of the speaker cannot be detected from the captured image (step S105: NO), the CPU 30A proceeds to step S107.

In step S106, CPU 30A estimates a visible area from the line of sight of the speaker.

In step S107, the CPU 30A estimates a visible area from the acquired visible direction.

In step S108, the CPU 30A determines whether or not there is an occupant next to the speaker. If there is an occupant next to the speaker (step S108: YES), the CPU 30A proceeds to step S109. On the other hand, if there is no passenger next to the speaker (step S108: NO), the CPU 30A proceeds to step S110.

In step S109, CPU 30A excludes the direction in which the passenger is seated from the estimated visible area.

In step S110, the CPU 30A extracts a target included in the visible area as a target using the target position information included in the target information.

In step S111, the CPU 30A determines whether or not there are a plurality of targets extracted as targets. When the CPU 30A determines that there are a plurality of extracted targets (step S111: YES), the process proceeds to step S112. On the other hand, when the CPU 30A determines that there are not a plurality of extracted targets (step S111: NO), the process proceeds to step S115.

In step S112, the CPU 30A transmits the characteristics of the extracted target to the vehicle-mounted device 20 and requests a response.

In step S<b>113 , the CPU 30</b>A acquires the speaker's response from the vehicle-mounted device 20 .

In step S114, CPU 30A analyzes the acquired response of the speaker and identifies the target requested by the speaker from the extracted targets.

In step S115, the CPU 30A transmits and presents target object information related to the specified target to the vehicle-mounted device 20 as target information.

(Summary of the first embodiment)
In the agent server 30 of this embodiment, the estimating unit 240 estimates the visible area of the speaker from the captured image and the voice information in response to a request from the passenger of the vehicle 12, and the specifying unit 250 determines the visible area. The included target information is specified as the target information desired by the speaker. In addition, the agent server 30 of the present embodiment presents the vehicle 12 with the target information specified by the presentation unit 260 , and the vehicle 12 uses the speaker 25 and the monitor 26 to output the information.

As described above, according to the present embodiment, it is possible to efficiently specify the information of the target to be presented from a plurality of targets located around the vehicle.

[remarks]
Note that the various processes executed by the CPU 20A and the CPU 30A by reading the software (program) in the above embodiment may be executed by various processors other than the CPU. The processor in this case is a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing such as an FPGA (Field-Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit) for executing specific processing. A dedicated electric circuit or the like, which is a processor having a specially designed circuit configuration, is exemplified. Further, the processing described above may be executed by one of these various processors, or by a combination of two or more processors of the same or different type (for example, multiple FPGAs and a combination of CPU and FPGA). etc.). Further, the hardware structure of these various processors is, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined.

Further, in the above embodiments, each program has been described as being pre-stored (installed) in a computer-readable non-temporary recording medium. For example, the control program for the CPU 20A is pre-stored in the ROM 20B, and the processing program 100 for the CPU 30A is pre-stored in the storage 30D. However, not limited to this, each program is recorded on non-temporary recording media such as CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), and USB (Universal Serial Bus) memory. may be provided in any form. Also, the program may be downloaded from an external device via a network.

200 reception unit 220 in-vehicle information acquisition unit 230 target object information acquisition unit 240 estimation unit 250 identification unit 260 presentation unit

Claims (1)

a reception unit that receives a request from a user on board the vehicle;
a target object information acquisition unit that acquires target object information about the target objects located around the current location of the vehicle;
an in-vehicle information acquisition unit that acquires in-vehicle information that is at least one of a captured image of the user and voice information uttered by the user;
an estimating unit that identifies a line of sight or a seating position of the user using the acquired in-vehicle information and estimates an area visible to the user;
an identifying unit that identifies the target object information included in the estimated area as a target;
a presentation unit that presents the target object information related to the identified target;
agent device.

Images