JP2021096380A - Agent system, agent system control method, and program - Google Patents

Abstract

To make it possible to obtain an accurate response even if a voice operation is based on contents of a short utterance.SOLUTION: Provided is an agent system which includes: a response display control unit which displays an image of contents in response to an operation on a display unit; an utterance content interpretation unit which interprets contents of utterance by a user; an utterance content determination unit which determines whether or not the contents of the utterance interpreted by the utterance content interpretation unit is established independently as a service request; and an agent control unit which executes control for providing a service specified based on contents of operation context information indicating an operation context according to the contents of the image displayed on the display unit corresponding to when the utterance is performed and the contents of the utterance, when the utterance content determination unit determines that the service is not independently established.SELECTED DRAWING: Figure 4

Description

The present invention relates to an agent system, a control method of the agent system, and a program.

Conventionally, when it is determined that the operation voice is non-verbal as a result of voice recognition of the operation voice input in response to a question to the user, the effectiveness of the non-language input is determined according to the situation at the time of the non-language input. There is known a navigation device that determines whether to confirm the work content, hold the work content, or execute the work content according to the result of determining the effectiveness (see, for example, Patent Document 1). ..

Japanese Unexamined Patent Publication No. 2008-46299

Regarding voice operation, it is preferable to obtain an accurate response even though the content of the short utterance is short. However, in the conventional technique, it is difficult to obtain an accurate response based on the uttered content even if a short content such as a phrase that does not hold as one sentence is uttered as a voice operation.

The present invention has been made in consideration of such circumstances, and one of the objects of the present invention is to enable an accurate response to be obtained even though it is a voice operation based on the content of a short utterance.

The agent system, the control method of the agent system, and the program according to the present invention have adopted the following configurations.
(1): The agent system according to one aspect of the present invention includes a response display control unit that displays an image of the content in response to an operation on the display unit, an utterance content interpretation unit that interprets the content of the utterance by the user, and the above. The utterance content determination unit that determines whether or not the utterance content interpreted by the utterance content interpretation unit is independently established as a service request, and the utterance content determination unit are not independently established as a service. When it is determined that, based on the content of the operation context information indicating the operation context according to the content of the image displayed on the display unit corresponding to the utterance and the content of the utterance. It is provided with an agent control unit that executes control for providing the specified service.

(2): In the agent system according to the aspect (1), when a manual operation is performed as the operation, the response display control unit displays an image of the content in response to the manual operation, and the operation. When the operation by the utterance is performed, the image of the content corresponding to the content of the utterance is displayed.

(3): In the agent system according to the aspect (1) or (2), when the agent control unit is determined by the utterance content determination unit to be independently satisfied as a service request, the utterance is said. A service required by the content of the determined utterance after maintaining the content of the operation context information indicating the operation context according to the content of the image displayed on the display unit in response to the occurrence of. Is controlled to be provided.

(4): In the agent system according to the aspect (3) above, the agent control unit maintains the content of the operation context information and provides the service required by the content of the determined utterance. After the control, if the content of the utterance interpreted by the utterance content interpretation unit is determined by the utterance content determination unit that it cannot be established as a service independently, it corresponds to the time when the utterance is made. The control for providing the service specified based on the content of the operation context information indicating the operation context according to the content of the image displayed on the display unit and the content of the utterance is executed.

(5): In the control method of the agent system according to one aspect of the present invention, the computer in the agent system displays an image of the content in response to the operation on the display unit, interprets the content of the utterance by the user, and is interpreted. It is determined whether or not the content of the utterance is independently established as a service request, and when it is determined that the content of the utterance is not independently established as a service, the utterance is made. The content of the operation context information indicating the operation context according to the content of the image displayed on the display unit is maintained, and the content of the maintained operation context information and the content of the utterance are used. Take control to provide the identified service.

(6): The program according to one aspect of the present invention causes a computer to display an image of the content in response to the operation on the display unit, interprets the content of the utterance by the user, and the interpreted content of the utterance is If it is determined whether or not the utterance is independently established as a service request, and if it is determined that the content of the utterance is not independently established as a service, the display corresponds to the time when the utterance is made. Maintains the content of operation context information that indicates the context of the operation according to the content of the image displayed in the department, and provides a service that is specified based on the content of the maintained operation context information and the content of the utterance. It is intended to execute the control for doing so.

According to (1), (5), and (6), a service request is made independently because the content of the utterance as a voice operation for the image displayed on the display unit corresponds to, for example, a part of the sentence. If it does not hold, it can be treated as if the voice operation was performed according to the content of the current utterance in the context of the operation up to the present time. As a result, it is possible to obtain an accurate response even if the content of the utterance in the voice operation is short.

According to (2), the operation for the image displayed on the display unit may be either a manual operation or a voice operation. In this case, the operation context information used when the content of the utterance is not independently established as a service request may include a history of manual operation and a history of voice operation. As a result, the occupant can perform a voice operation with a simple and short utterance content regardless of whether the previous operation is a manual operation or a voice operation.

According to (3) and (4), when the content of the current utterance is independently established as a service request, the operation context information so far is maintained without being cleared. Then, depending on the content of the utterance this time, other services different from the service to which the operation context information corresponds are provided. After the provision of other services is completed, the operation context information is maintained, so that the occupant can resume the operation from the state before the current utterance was made.

It is a figure which shows the configuration example of the agent system in this embodiment. It is a figure which shows the structure of the agent apparatus in this embodiment, and the apparatus mounted on a vehicle. It is a figure which shows the configuration of the agent server in this embodiment, and a part of the configuration of an agent apparatus. It is a flowchart which shows the example of the processing procedure which the agent system of this embodiment executes in relation to the operation to touch panel performed by an occupant. It is a sequence diagram which shows a specific example of the operation of the agent system in response to the operation procedure of an occupant with respect to a touch panel.

Hereinafter, the agent system of the present invention, the control method of the agent device, and the embodiment of the program will be described with reference to the drawings.
<Embodiment>
[About agent function]
The agent device is a device that realizes a part or all of the agent system 1 including the notification control system of the present embodiment. Hereinafter, as an example of the agent device, an agent device mounted on a vehicle (hereinafter, vehicle M) on which a occupant (an example of a user) is boarded and having an agent function will be described. For the application of the present invention, the agent device does not necessarily have to have an agent function. Further, the agent device may be a portable terminal device (general-purpose terminal) such as a smartphone, but the following description will be made on the premise of an agent device having an agent function mounted on a vehicle. The agent function is, for example, providing various information based on a request (command) included in the utterance of the occupant, controlling various devices, and mediating a network service while interacting with the occupant of the vehicle M. It is a function. When the agent device has a plurality of agent functions, the agent functions may have different functions, processing procedures, controls, and output modes / contents. In addition, some of the agent functions may have a function of controlling devices in the vehicle (for example, devices related to driving control and vehicle body control).

Agent functions include, for example, a voice recognition function that recognizes the voice of an occupant (a function that converts voice into text), a natural language processing function (a function that understands the structure and meaning of text), a dialogue management function, and a network. It is realized by using a network search function that searches for another device or a predetermined database owned by the own device in an integrated manner. Some or all of these functions may be realized by AI (Artificial Intelligence) technology. In addition, a part of the configuration for performing these functions (particularly, the voice recognition function and the natural language processing interpretation function) is an agent server capable of communicating with the in-vehicle communication device of the vehicle M or the general-purpose communication device brought into the vehicle M. It may be mounted on (external device). In the following description, it is assumed that a part of the configuration is mounted on the agent server, and the agent device and the agent server cooperate to realize the agent system. Further, a service provider (service entity) in which an agent device and an agent server cooperate to appear virtually is called an agent.

[Agent system]
FIG. 1 is a diagram showing a configuration example of an agent system 1 including an agent device 100. The agent system 1 includes, for example, an agent device 100 and one or more agent servers 200. Providers that provide the agent system 1 in the present embodiment include, for example, automobile manufacturers, network service providers, electronic commerce businesses, sellers and manufacturers of mobile terminals, and any other entity (corporation, organization, etc.). An individual, etc.) can be the provider of the agent system 1. Note that FIG. 1 has described the case where there is only one agent server 200, but the present invention is not limited to this, and the agent system 1 may include two or more agent servers 200. In this case, each agent server 200 may be provided by any entity different from each other.

The agent device 100 communicates with the agent server 200 via the network NW. The network NW includes, for example, a part or all of communication networks such as the Internet, cellular network, Wi-Fi network, WAN (Wide Area Network), LAN (Local Area Network), public line, telephone line, and wireless base station. Including. Various web servers 300 are connected to the network NW, and the agent server 200 or the agent device 100 can acquire web pages from the various web servers 300 via the network NW.

The agent device 100 interacts with the occupant of the vehicle M, transmits the voice from the occupant to the agent server 200, and presents the answer obtained from the agent server 200 to the occupant in the form of voice output or image display.

[vehicle]
FIG. 2 is a diagram showing the configuration of the agent device 100 according to the embodiment and the equipment mounted on the vehicle M. The vehicle M is equipped with, for example, one or more microphones 10, a display / operation device 20, a speaker 30, a navigation device 40, an in-vehicle communication device 50, and an agent device 100. These devices are connected to each other by a multiplex communication line such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, or the like. The configuration shown in FIG. 2 is merely an example, and a part of the configuration may be omitted or another configuration may be added.

The microphone 10 is a sound collecting unit that collects sounds emitted in the vehicle interior. The display / operation device 20 is a device (or a group of devices) capable of displaying an image and accepting an input operation. The display / operation device 20 includes, for example, a display device configured as a touch panel. The display / operation device 20 may further include a HUD (Head Up Display) or a mechanical input device. The speaker 30 includes, for example, a speaker (sound output unit) arranged in the vehicle interior. The display / operation device 20 may be shared by the agent device 100 and the navigation device 40. The speaker 30 is an example of an “audio output unit”.

The navigation device 40 includes a navigation HMI (Human Machine Interface), a positioning device such as a GPS (Global Positioning System), a storage device that stores map information, and a control device (navigation controller) that performs route search and the like. .. A part or all of the microphone 10, the display / operation device 20, and the speaker 30 may be used as the navigation HMI. The navigation device 40 searches for a route (navigation route) for moving from the position of the vehicle M specified by the positioning device to the destination input by the occupant, so that the vehicle M can travel along the route. , Navi HMI is used to output guidance information. The route search function may be provided in a navigation server accessible via the network NW. In this case, the navigation device 40 acquires a route from the navigation server and outputs guidance information.

The agent device 100 may be constructed based on the navigation controller. In this case, the navigation controller and the agent device 100 are integrally configured on the hardware. The display device of the display / operation device 20 and the navigation HMI of the navigation device 40 are examples of the “display unit”.

The in-vehicle communication device 50 is, for example, a wireless communication device that can access the network NW using a cellular network or a Wi-Fi network.

[Agent device]
The agent device 100 includes a management unit 110, an agent function unit 130, an in-vehicle communication unit 140, and a storage unit 150. The management unit 110 includes, for example, an sound processing unit 112, an agent WU (Wake Up) determination unit 114, a communication control unit 116, and an output control unit 120. The software arrangement shown in FIG. 2 is simply shown for the sake of explanation, and is actually modified arbitrarily so that, for example, the management unit 110 may intervene between the agent function unit 130 and the in-vehicle communication device 50. can do. Further, in the following, an agent caused by the agent function unit 130 and the agent server 200 to appear in cooperation with each other may be simply referred to as an “agent”.

Each component of the agent device 100 is realized, for example, by executing a program (software) by a hardware processor such as a CPU (Central Processing Unit). Some or all of these components are hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), etc. It may be realized by (including circuits), or it may be realized by the cooperation of software and hardware. The storage unit 150 may be realized by a storage device (a storage device including a non-transient storage medium) such as an HDD (Hard Disk Drive) or a flash memory, and is a removable storage medium such as a DVD or a CD-ROM. It may be realized by (non-transient storage medium), or it may be a storage medium mounted on a drive device. Further, a part or all of the storage unit 150 may be an external device such as NAS or an external storage server that can be accessed by the agent device 100. The storage unit 150 stores, for example, information such as a program executed by the agent device 100.

The management unit 110 functions by executing a program such as an OS (Operating System) or middleware.

The sound processing unit 112 of the management unit 110 receives the sound collected from the microphone 10, recognizes the wake-up word preset for each agent for the received sound, and recognizes other utterance contents. Perform sound processing to make it suitable for. The wakeup word is, for example, a word or phrase for activating the target agent. The wakeup word may start a single agent or may start a plurality of agents. The acoustic processing is, for example, noise removal by filtering such as a bandpass filter, sound amplification, and the like. Further, the sound processing unit 112 outputs the sound-processed voice to the agent WU determination unit 114 and the activated agent function unit 130.

The agent WU determination unit 114 recognizes a wakeup word predetermined for the agent. The agent WU determination unit 114 recognizes the voice uttered from the voice (voice stream) to which the sound processing has been performed. First, the agent WU determination unit 114 detects a voice section based on the amplitude and zero intersection of the voice waveform in the voice stream. The agent WU determination unit 114 may perform frame-by-frame speech recognition based on a Gaussian mixture model (GMM) and section detection based on non-speech recognition.

Next, the agent WU determination unit 114 converts the voice in the detected voice section into text and converts it into character information. Then, the agent WU determination unit 114 determines whether or not the textualized character information corresponds to the wakeup word. When it is determined that the wakeup word is used, the agent WU determination unit 114 activates the agent function unit 130 corresponding to the wakeup word. The agent server 200 may be equipped with a function corresponding to the agent WU determination unit 114. In this case, when the management unit 110 transmits the voice stream to which the sound processing has been performed by the sound processing unit 112 to the agent server 200 and determines that the agent server 200 is a wakeup word, the management unit 110 follows an instruction from the agent server 200. The agent function unit 130 is started. Further, each agent function unit 130 may be always activated and may determine the wakeup word by itself. In this case, the management unit 110 does not need to include the agent WU determination unit 114.

Further, the agent WU determination unit 114 recognizes the end word included in the spoken voice by the same procedure as the above procedure, and the agent corresponding to the end word is activated (hereinafter, , If necessary, it is referred to as "starting"), the running agent function unit is terminated (stopped). The start and end of the agent may be executed, for example, by accepting a predetermined operation from the display / operation device 20, but an example of starting and stopping by voice will be described below. Further, the activated agent may be stopped when the voice input is not received for a predetermined time or more.

The communication control unit 116 controls the agent function unit 130 so that it can be connected to the network NW. For example, the communication control unit 116 controls the connection state and the like when the agent function unit 130 communicates with an external device (for example, the agent server 200) via the network. Further, the communication control unit 116 controls such as reconnection when communication is interrupted and switching of the connection state.

The output control unit 120 provides the occupants with services and the like by causing the display unit or the speaker 30 to output information such as response contents in response to an instruction from the communication control unit 116 or the agent function unit 130 or the like. The output control unit 120 includes, for example, a display control unit 122 and a voice control unit 124.

The display control unit 122 displays an image used for notifying the occupant of the vehicle M of the response content in which the agent responds to the utterance of the occupant of the vehicle M based on the information acquired by the agent function unit 130 from the agent server 200. It is displayed on the display device of the display / operation device 20.

The voice control unit 124 transmits a voice used for notifying the occupant of the vehicle M of the response content in which the agent responds to the utterance of the occupant of the vehicle M based on the information acquired by the agent function unit 130 from the agent server 200. Output to the speaker 30.

The agent function unit 130 cooperates with the agent server 200 to provide a service including a response by voice and an image in response to an utterance of a vehicle occupant. For example, the agent function unit 130 is granted the authority to control the vehicle M or the in-vehicle device mounted on the vehicle M, and the utterance content of the vehicle M recognized by the agent server 200 by the process described later is the vehicle. In the case of a command for instructing the operation of the in-vehicle devices mounted on the M, the agent function unit 130 controls those in-vehicle devices based on the command. The in-vehicle device includes a navigation device 40. Based on the control of the communication control unit 116, the agent function unit 130 communicates with the agent server 200 by the vehicle-mounted communication unit 140 via the vehicle-mounted communication device 50.

The agent function unit 130 may be assigned the authority to control the in-vehicle device according to laws, ordinances, contracts between businesses that provide agents, and the like.

The vehicle-mounted communication unit 140 communicates via the vehicle-mounted communication device 50, for example, when the agent function unit 130 connects to the network NW. The vehicle-mounted communication unit 140 outputs the information from the agent function unit 130 to the agent server 200 and other external devices via the vehicle-mounted communication device 50. Further, the vehicle-mounted communication unit 140 outputs the information input via the vehicle-mounted communication device 50 to the agent function unit 130.

The agent function unit 130 is activated based on an activation instruction by the agent WU determination unit 114, and generates and generates a response content to a request included in the voice of the occupant's utterance via the agent server 200 in response to the occupant's utterance. The response content is output to the output control unit 120. Further, when communicating with the agent server 200, the agent function unit 130 communicates according to the connection state controlled by the communication control unit 116. Further, the agent function unit 130 may stop the agent based on the control by the agent WU determination unit 114.

[Agent server]
FIG. 3 is a diagram showing a configuration of the agent server 200 and a part of the configuration of the agent device 100 according to the embodiment. Hereinafter, the operation of the agent function unit 130 and the like will be described together with the configuration of the agent server 200. Here, the description of the physical communication from the agent device 100 to the network NW will be omitted.

The agent server 200 includes a communication unit 210. The communication unit 210 is, for example, a network interface such as a NIC (Network Interface Card). Further, the agent server 200 includes, for example, a voice recognition unit 220, a natural language processing unit 221, a dialogue management unit 222, a network search unit 223, and a response content generation unit 224. These components are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware such as LSI, ASIC, FPGA, GPU (including circuit part; circuitry), or realized by collaboration between software and hardware. May be good. The program may be stored in advance in a storage device such as an HDD or a flash memory (a storage device including a non-transient storage medium), or a removable storage medium such as a DVD or a CD-ROM (non-transient). It is stored in a sex storage medium) and may be installed by mounting the storage medium in a drive device. The combination of the voice recognition unit 220 and the natural language processing unit 221 is an example of the "utterance content interpretation unit".

Further, the agent server 200 includes a storage unit 250. The storage unit 250 is realized by the same device as the various storage devices that realize the above-mentioned storage unit 150. The storage unit 250 stores, for example, data and programs such as a dictionary DB 252, a personal profile 254, a knowledge base DB 256, and a response rule DB 258.

In the agent device 100, the agent function unit 130 transmits, for example, a voice stream input from the sound processing unit 111 or the like, or a voice stream that has undergone processing such as compression or coding to the agent server 200. When the agent function unit 130 can recognize a command (request content) capable of local processing (processing that does not go through the agent server 200), the agent function unit 130 may execute the processing requested by the command. The command capable of local processing is, for example, a command that can be responded to by referring to the storage unit 150 included in the agent device 100. More specifically, the command capable of local processing is, for example, a telephone directory associated with a matching name by searching for the name of a specific person from the telephone directory data (not shown) existing in the storage unit 150. It is a command to make a call to (call the other party). Therefore, the agent function unit 130 may have a part of the functions provided in the agent server 200.

When the voice stream is acquired, the voice recognition unit 220 performs voice recognition and outputs the textualized character information, and the natural language processing unit 221 interprets the meaning of the character information while referring to the dictionary DB 252. The dictionary DB 252 is, for example, associated with abstract semantic information with respect to character information. The dictionary DB 252 includes, for example, a functional dictionary 252A and a general-purpose dictionary 252B.

The function dictionary 252A is a dictionary for covering the functions (services) provided by the agent realized by the agent server 200 in cooperation with the agent function unit 130. For example, when the agent provides a function to control an in-vehicle air conditioner, the function dictionary 252A contains "air conditioner", "air conditioning", "turn on", "turn off", "temperature", "raise", "lower", and "lower". Words such as "inside air" and "outside air" are registered in association with word types such as verbs and objects, and abstracted meanings. In addition, the functional dictionary 252A may include inter-word link information indicating that they can be used at the same time.

The general-purpose dictionary 252B is not limited to the functions provided by the agent, but is a dictionary in which general events are associated with abstract meanings. Each of the functional dictionary 252A and the general-purpose dictionary 252B may include list information of synonyms and synonyms. The function dictionary 252A and the general-purpose dictionary 252B may be prepared corresponding to each of a plurality of languages, in which case the voice recognition unit 220 and the natural language processing unit 221 have functions corresponding to preset language settings. A dictionary 252A, a general-purpose dictionary 252B, and grammatical information (not shown) are used. The processing of the voice recognition unit 220 and the processing of the natural language processing unit 221 are not clearly separated in stages, and the voice recognition unit 220 corrects the recognition result in response to the processing result of the natural language processing unit 221. It may be done by influencing each other.

The natural language processing unit 221 acquires information on functions required for responding to a request for a service included in voice (hereinafter referred to as function necessary information) as one of semantic analysis based on the recognition result by the voice recognition unit 220. .. For example, when a text such as "open the window" or "raise the temperature of the air conditioner" instructing the control of the in-vehicle device of the vehicle M is recognized as the recognition result, the natural language processing unit 221 may use the dictionary DB252 or the like. To acquire the target device / function type called "vehicle device control". Then, the natural language processing unit 221 outputs the acquired function necessary information to the agent function unit 130. The natural language processing unit 221 acquires the determination result of whether or not the service request can be executed based on the function required information. The natural language processing unit 221 generates a command corresponding to the interpreted utterance content as being able to respond to the service request when the requested function can be executed.

The dialogue management unit 222 responds to the occupant of the vehicle M (for example, the utterance content to the occupant) while referring to the personal profile 254, the knowledge base DB 256, and the response rule DB 258 based on the command generated by the natural language processing unit 221. And the image and sound to be output from the output section). The knowledge base DB 256 is information that defines the relationships between things. The response rule DB 258 is information that defines the actions (answers, device control contents, etc.) that the agent should perform in response to the command.

Further, the dialogue management unit 222 may identify the occupant by collating with the personal profile 254 using the feature information obtained from the voice stream. In this case, for example, voice feature information is further attached to the personal profile 254. The voice feature information is, for example, information on the characteristics of how to speak such as voice pitch, intonation, and rhythm (sound pitch pattern), and the feature amount based on Mel Frequency Cepstrum Coefficients and the like. .. The voice feature information is, for example, information obtained by having the occupant utter a predetermined word or sentence at the time of initial registration of the occupant and recognizing the uttered voice.

The dialogue management unit 222 causes the network search unit 223 to perform a search when the command requests information that can be searched via the network NW. The network search unit 223 accesses an external device such as a predetermined web server 300 via the network NW and acquires desired information.

The response content generation unit 224 generates a response sentence so that the content of the utterance determined by the dialogue management unit 222 can be understood by the occupants of the vehicle M, and transmits the generated response sentence to the agent device 100. Further, the response content generation unit 224 acquires the recognition result of recognizing the occupant of the vehicle M based on the image captured by the camera interior from the agent device 100, and the occupant who made an utterance including a command based on the acquired recognition result. When is identified as an occupant registered in the personal profile 254, a response sentence may be generated that calls the occupant's name or makes the occupant's speech similar to that of the occupant.

When the agent function unit 130 acquires the response sentence, the agent function unit 130 instructs the voice control unit 124 to perform voice synthesis and output the voice. Further, the agent function unit 130 instructs the display control unit 122 to display an image or the like including a response sentence.

In the agent system 1 of the present embodiment having the above configuration, the occupant can use both manual operation and voice operation as operations on the touch panel (an example of the display unit) provided as the display / operation device 20.
The manual operation is an operation in which the occupant uses an operating body such as a finger to perform a physically provided input device or operator. As an example, the manual operation on the touch panel is an operation performed by touching the display surface (operation surface) of the touch panel with an operation body such as a finger.
The voice operation is an operation in which the agent function provided in the agent system 1 of the present embodiment is used to control the equipment of the vehicle M as various services in response to the occupant speaking.
The occupant can also perform operations that can be performed manually on the touch panel by voice operation. That is, in the agent system of the present embodiment, the occupant can perform the operation corresponding to the touch panel by both the manual operation and the voice operation.
Further, the "service" in the following description refers to a function provided in response to an operation performed not only by voice operation but also by manual operation.

An example of a processing procedure executed by the agent system 1 of the present embodiment in connection with an operation (manual operation, voice operation) on the touch panel performed by the occupant will be described with reference to the flowchart of FIG. The processing shown in the figure is performed while the agent has already been started. Further, in the description of the figure, the case where the management unit 110 is set to execute the control regarding the response to the manual operation performed on the touch panel will be given as an example.

First, in the agent device 100, the management unit 110 determines whether or not the touch panel has accepted the manual operation (step S100).
When the touch panel accepts a manual operation, the management unit 110 executes control (response control) so that the operation of the device in the vehicle M can be obtained in response to the manual operation performed on the touch panel (step S102). .. At this time, the management unit 110 (an example of the response display control unit) may execute display control on the image displayed on the touch panel so as to respond to the manual operation performed this time.

When the touch panel accepts a manual operation, the agent function unit 130 executes control of the dialogue state continuation flag (dialogue state continuation flag control) (step S104).
If the agent function unit 130 is not activated when the touch panel accepts a manual operation, the agent WU determination unit 114 activates the agent function unit 130 to execute the process of step S104. May be.
The dialogue state continuation flag is a flag indicating whether or not the agent system 1 continues the dialogue state depending on the presence or absence of the set. When the dialogue state continuation flag is turned on and the dialogue state is continued, the agent system 1 accepts a voice operation and executes control in response to the content of the utterance. On the other hand, the agent system 1 does not accept the voice operation when the dialogue state continuation flag is turned off and the dialogue state is stopped. The dialogue state continuation flag is turned on and off when a certain period of time has passed since the last operation (manual operation or voice operation).
As the dialogue state continuation flag control in step S104, the agent function unit 130 turns on the dialogue state continuation flag when the dialogue state continuation flag is in the off state. That is, the agent function unit 130 in the present embodiment turns on the dialogue state continuation flag even when a manual operation is performed, so that the agent function unit 130 can receive subsequent voice operations.
Further, when the dialogue state continuation flag is on and the provision of one service is completed by the operation on the touch panel this time, the agent function unit 130 subsequently accepts the operation corresponding to the one service. There is no need. In this case, the agent function unit 130 turns off the dialogue state continuation flag as the dialogue state continuation flag control.
In addition, if the dialogue state continuation flag is on and the provision of the service of 1 has not been completed depending on the operation on the touch panel this time, the subsequent operations of the service of the 1 are accepted. Can be done. Therefore, the agent function unit 130 in this case maintains the state in which the dialogue state continuation flag is on as the dialogue state continuation flag control.

After the processing of step S106, or when it is determined in step S104 that the dialogue state continuation flag is on, the agent function unit 130 executes control related to the operation context information (operation context information control) (step S108). ..
The operation context information is information indicating the history of operations under a series of operation procedures performed for each service. For example, when a occupant performs a POI (point of interest) search in order to obtain information on a gas station near the current position, one of the operation procedures is an instruction to execute a POI search and a search target. The flow is such as category selection, narrowing search instruction for the selected category, and selection of one gas station to be presented with information from the narrowed search result. The operation context information indicates the content of each operation in such an operation procedure. For example, when searching for a gas station like the one above, [POI search execution instruction], [Select "gas station" as the search target category], "Refine search by narrowing down condition" Akasaka area "", The contents of each operation such as "select" A store "from the refined search results" are indicated by the operation context information. Further, each operation reflected in the operation context information may include either a manual operation or a voice operation.
Further, when the manual operation on the touch panel received in step S100 this time is the first operation corresponding to a certain service (for example, in the case of POI search, the operation instructing the start of POI search), the agent The functional unit 130 may execute the following processing as the operation context information control in the step S108. That is, the agent function unit 130 newly generates the operation context information including the contents of the manual operation of the touch panel received in accordance with the step S100 this time as a history, and holds the generated operation context information. In holding the operation context information, the agent function unit 130 may store the operation context information in the storage unit 150.
Further, when the manual operation on the touch panel this time is the second and subsequent operations in one service, the agent function unit 130 describes the contents of the manual operation on the touch panel this time with respect to the operation context information already held. Update to add history.
Further, the agent function unit 130 clears the operation context information when the provision of one service is completed by the manual operation on the touch panel this time.

When the manual operation on the touch panel is not performed, for example, the agent function unit 139 determines whether or not the voice collected by the microphone 10 or the like is received by the sound processing unit 112 (step S108).
When the voice is received, the agent function unit 130 transmits the voice received by the sound processing unit 112 and subjected to the sound processing to the agent server 200. In the agent server 200, the voice recognition unit 220 converts the received voice into text by executing the voice recognition process for the received voice (step S110).
Next, the natural language processing unit 221 (an example of the utterance content interpretation unit) executes natural language processing on the textualized character information and interprets the meaning of the character information (step S112). By interpreting the meaning of step S112, it is recognized what the meaning of the utterance content of the occupant is.

Next, the natural language processing unit 221 determines whether or not the dialogue state continuation flag is currently on (step S114). At this time, the natural language processing unit 221 may inquire the agent function unit 130 of the state of the dialogue state continuation flag via communication with the agent device 100.

When the dialogue state continuation flag is on, it is currently in a state of waiting for the next operation under the provision of one service. In such a state, the dialogue state continuation flag is maintained in the ON state, and the operation context information is held by the agent function unit 130 without being cleared.
In this case, the natural language processing unit 221 (an example of the utterance content determination unit) determines whether or not the utterance content whose meaning is recognized in step S112 is independently satisfied as a service request (step S116). ..
The content of the utterance that is independently established as a service request is, for example, "Search for a gas station around Akasaka" or "Set the temperature of the air conditioner to 20 degrees". The content of the utterance will be such that it will be specified. Since this utterance content is understood to mean that a POI search is required to search for gas stations around Akasaka, the utterance content is independently established as a service request.
On the other hand, the utterance content that cannot be satisfied as a service request by itself is a phrase that is partially extracted from one sentence, for example, "around Akasaka". The content of such an utterance cannot identify what kind of service is specifically requested by itself. In order to specify the content of such an utterance, it is necessary to supplement, for example, what the operation context has been so far.
The determination in step S116 may be performed as follows. For example, when the natural language processing unit 221 refers to the dictionary DB 252 to acquire the function required information, the natural language processing unit 221 may make a determination based on whether or not the function required information can be acquired based on the recognized utterance content itself. That is, if the natural language processing unit 221 can acquire the function necessary information, the natural language processing unit 221 determines that the recognized utterance content is independently satisfied as a service request. On the other hand, if the natural language processing unit 221 cannot acquire the function necessary information, it determines that the recognized utterance content is not independently satisfied as a service request.

When the recognized utterance content is not independently established as a service request, the natural language processing unit 221 maintains the retained state without clearing the operation context information held by the agent function unit 130. To be done. Then, the natural language processing unit 221 (an example of the agent control unit) refers to the operation context information whose retention is maintained (step S118).
Next, the natural language processing unit 221 complements the meaning of the utterance content recognized in step S112 this time with the history of the operation content indicated by the operation context information referred to in step S118 (step S120). Specifically, the natural language processing unit 221 converts the utterance content whose meaning has been recognized in step S112 this time into operations (manual operation, voice operation) performed on the touch panel according to the service of 1. Treat it as the content of the utterance as a subsequent voice operation.
In complementing step S120, the natural language processing unit 221 uses, for example, the dictionary DB252 to determine whether or not the meaning of the utterance content recognized this time is the next operation in the operation context for the touch panel so far. Judgment about. When it is determined that the operation content is connected as the next operation content, the natural language processing unit 221 complements the meaning of the utterance content recognized this time. On the other hand, if it is determined that the operation content is not connected as the next operation content, the processing in the figure is omitted, but the natural language processing unit 221 responds to the error by assuming that the response to the utterance content this time is impossible. You may execute the processing.

The agent function unit 130 of the agent device 100 executes control (response control) for responding to the utterance content of the voice received in step S108 this time (step S122). At this time, the agent function unit 130 (an example of the response display control unit) receives a command generated by the natural language processing unit 221 based on the meaning recognized in step S112 and the result complemented in step S120. The agent function unit 130 executes control of the device of the vehicle M in response to the received command. At this time, the agent function unit 130 executes display control in response to the command generated in step S120 so that the display of the image on the touch panel has the content corresponding to the current voice operation.
Further, the agent function unit 130 receives the response content (dialogue content) such as voice corresponding to the command generated in step S120 from the response content generation unit 224 of the agent server 200, and outputs the received response content.

Further, the agent function unit 130 executes the operation context information control (step S124). The operation context information control in step S124 updates the already held operation context information so as to add a history of the operation contents of the voice operation performed by the utterance contents recognized this time.
When the process of step S124 is completed, the process is returned to step S100.

When it is determined that the service request is independently satisfied, the agent function unit 130 executes response control by processing an interrupt as response control for the voice utterance content received in step S108 this time (step S126). ). In this case, since it is an interrupt process, the agent function unit 130 is made to maintain the held state without clearing the operation context information corresponding to the touch panel operation so far. As a result, after the processing of step S126, the occupant continues to perform subsequent operations (manual operation or voice operation) on the touch panel on which the same image as before the voice operation corresponding to the interrupt processing this time is displayed. ) Can be restarted. Further, when resuming the operation on the touch panel after the process of step S126, the processes of steps S116 to S122 can be executed. That is, when the utterance content by the voice operation is not satisfied as a service request by itself, the agent function unit 130 can take over the operation context for the previous touch panel and appropriately execute the response control.

Further, when the dialogue state continuation flag is off, the voice operation corresponding to the reception of the voice in step S108 this time means that the operation corresponding to the new service 1 is started by the voice operation. Therefore, the agent function unit 130 executes response control for the voice operation this time (step S128). In response control in step S128, the agent function unit 130 executes control according to a command generated by the natural language processing unit 221 based on the meaning recognized in step S112 this time. At this time, when the command corresponds to a predetermined operation on the touch panel, the agent function unit 130 controls the touch panel so that the image of the content corresponding to the current voice operation is displayed.

Next, the agent function unit 130 determines whether or not the response control in step S128 this time responds to the operation on the touch panel (step S130). When responding to the touch panel operation, the agent function unit 130 turns on the dialogue state continuation flag as the dialogue state continuation flag control (step S132).
Further, as the operation context information control, the agent function unit 130 generates operation context information in which the operation content recognized by the natural language processing unit 221 is shown as a history in response to the response control in step S128 this time (step S134). .. The agent function unit 130 holds the generated operation context information. After the processing in step S134, or when it is determined that the response control in step S128 does not respond to the operation on the touch panel, the processing is returned to step S100.

If it is determined that the voice cannot be accepted, it means that neither the voice operation nor the manual operation on the touch panel has been performed. In this case, the agent function unit 130 determines whether or not a certain time has elapsed since the last operation (step S136). The final operation here is either a manual operation or a voice operation.
If a certain time has not passed since the last operation, the process is returned to step S100.

When a certain time has elapsed from the last operation, the agent function unit 130 turns off the dialogue state continuation flag as the dialogue state continuation flag control when the dialogue state continuation flag is on (step S138). .. Further, the agent function unit 130 clears the operation context information when the operation context information is currently held according to the elapse of a certain time from the last operation (step S140). If a certain period of time elapses without performing an operation on the touch panel displaying the image corresponding to a certain service by the processing of steps S138 and S140, a timeout occurs and the touch panel is, for example, of the service. It will be in a state of waiting for the operation according to the start.

[Specific example of touch panel operation in this embodiment]
A specific example of the operation of the agent system 1 in response to the operation procedure of the occupant with respect to the touch panel will be described with reference to the sequence diagram of FIG. In the following description, an example will be given in which the occupant causes the navigation device 40 to perform a POI search to search for a gas station. In the figure, an operation procedure and an operation procedure of the agent system 1 in response to the operation procedure are shown.
First, the occupant instructs the start of the POI search by manually operating the touch panel (step S200).
In the agent device 100, the management unit 110 causes the name navigation device 40 to start the POI search in response to the manual operation performed in step S200. The navigation device 40 displays an image as a category selection screen on the touch panel at the start of the POI search (step S202). The operation of step S202 is realized as follows. That is, in response to the touch panel receiving the manual operation instructing the start of the POI search, the management unit 110 activates the POI search function of the navigation device 40 by executing the process of step S102 of FIG. The navigation device 40 in which the POI search function is activated displays a category selection screen on the touch panel.

The agent function unit 130 generates operation context information in response to the operation of displaying the category selection screen in step S202 (step S204). The operation of step S204 is based on the process of step S106 in FIG.

The category selection screen displayed in step S202 is a screen on which an operation of selecting a category to be searched from among the searchable category candidates in the POI search is performed. The category searched by the occupants in this case is a gas station. Therefore, the occupant manually selected the gas station as the category to be searched by manually operating the category selection screen displayed on the touch panel (step S206).
In response to the manual operation performed in step S206, the management unit 110 instructs the navigation device 40 to execute the search for the gas station by executing the process in step S102 in FIG. In response to the instruction, the navigation device 40 executes a POI search of a gas station within a certain range based on, for example, the current position (step S208). The navigation device 40 displays a search result presentation screen showing the result of searching for the gas station on the touch panel (step S210).

The agent function unit 130 updates the operation context information by executing the process of step S106 of FIG. 4 in response to the operation of displaying the search result presentation screen in step S210 (step S212).

In the figure, an example of the contents of the operation context information D1 after the operation context information is updated in step S212 is shown. The operation context information D1 indicates that the operation according to the service as the POI search function has been performed in the order of the operation of instructing the activation of the POI search and the operation of selecting the gas station as the category.

With the search result presentation screen showing the search results for gas stations displayed, the occupant wants to narrow down the search to the gas stations around Akasaka from the gas stations presented on the search result presentation screen. I thought. Here, the occupants decided to narrow down the search for gas stations around Akasaka by voice operation instead of the conventional manual operation. Therefore, the occupant spoke "around Akasaka" as a voice operation (step S214).
It is determined that the utterance content of "around Akasaka" is not independently satisfied as a service request by the processing of steps S108 to S116 of FIG. In this case, the processes of steps S118 to S122 are subsequently executed.
That is, the content of the utterance of "around Akasaka" is treated as the next voice operation following the operation procedure indicated by the operation context information D1 held by the agent function unit 130 under the POI search service. As a result, the agent function unit 130 instructs the narrow-down search of the gas station based on the predetermined position in Akasaka in correspondence with the search result presentation screen displayed on the touch panel as the response control in step S122 of FIG. To do. That is, the agent function unit 130 in this case maintains the search result presentation screen that has been displayed so far, and then produces a result in which an operation of instructing a narrowed search is performed on the search result presentation screen. ..
In response to the above response control, the navigation device 40 executes a refined search (step S216). That is, the navigation device 40 is included in a certain area range determined in response to the narrowed search based on the predetermined position of Akasaka from the gas stations presented on the search result presentation screen displayed in step S210. Extract the gas station.
The navigation device 40 displays the narrowed-down search result presented in step S216 on the narrowed-down search result presentation screen touch panel (step S218). Further, the agent system 1 outputs the response voice to the voice operation by the utterance of "around Akasaka" from the speaker 30 by the response control in step S122 of FIG. 4 (step S220).
Further, the agent function unit 130 updates the operation context information by executing the process of step S124 of FIG. 4 in response to the operation of displaying the narrowed-down search result presentation screen in step S218 (step S222). ).

For example, in the past, when the meaning of the recognized utterance content was not independently established as a service request, it was treated as an error, for example. For this reason, when the occupant executes a POI search by gas station category and then executes a narrowing search for gas stations around Akasaka by voice operation, for example, "Search for gas stations around Akasaka". I need to speak. In other words, it is necessary to utter the content that is satisfied as a service request independently. In this case, the content that the occupant should speak is long because there are many words and phrases.
On the other hand, in the present embodiment, even if the utterance content is not satisfied as a service request by itself, it is treated as a voice operation performed under the operation context so far. As a result, the occupant can perform voice operations with a short utterance content.

In the above embodiment, the agent server 200 executes a part of the agent functions such as recognition of the meaning of the utterance content and generation of the response content in response to the voice operation. However, in the present embodiment, the function executed by the agent server 200 is also configured to be executable by the agent device 100 provided in the vehicle M so that the process shown in FIG. 4 is completed in the vehicle M. May be done.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

1 ... Agent system, 10 ... Microphone, 20 ... Display / operation device, 30 ... Speaker, 40 ... Navigation device, 50 ... In-vehicle communication device, 100 ... Agent device, 110 ... Management unit, 112 ... Sound processing unit, 114 ... Agent WU determination unit, 116 ... communication control unit, 120 ... output control unit, 122 ... display control unit, 124 ... voice control unit, 130 ... agent function unit, 140 ... in-vehicle communication unit, 150 ... storage unit, 200 ... agent server, 210 ... Communication unit, 220 ... Voice recognition unit, 221 ... Natural language processing unit, 222 ... Dialogue management unit, 224 ... Network search unit, 224 ... Response content generation unit, 250 ... Storage unit, 300 ... Web server

Claims (6)

A response display control unit that displays an image of the content in response to the operation on the display unit,
The utterance content interpretation department that interprets the content of the utterance by the user,
An utterance content determination unit that determines whether or not the utterance content interpreted by the utterance content interpretation unit is independently satisfied as a service request.
When the utterance content determination unit determines that the service cannot be established independently, the context of the operation according to the content of the image displayed on the display unit in response to the utterance. An agent system including an agent control unit that executes control for providing a service specified based on the content of the operation context information indicating the above and the content of the utterance. The response display control unit
When a manual operation is performed as the operation, an image of the content corresponding to the manual operation is displayed, and when an operation by utterance is performed as the operation, an image of the content corresponding to the content of the utterance is displayed. The agent system according to claim 1. The agent control unit
When the utterance content determination unit determines that the service request is independently established, the operation according to the content of the image displayed on the display unit corresponding to the utterance is performed. The agent system according to claim 1 or 2, wherein the operation indicating the context maintains the content of the context information and controls the content of the determined utterance to provide the required service. The agent control unit
After maintaining the content of the operation context information and controlling so that the service required by the determined utterance content is provided, the utterance content interpreted by the utterance content interpretation unit is the utterance content. When the determination unit determines that the service cannot be established independently, an operation indicating the context of the operation according to the content of the image displayed on the display unit in response to the utterance. The agent system according to claim 3, which executes control for providing a service specified based on the content of context information and the content of the utterance. The computer in the agent system
Display the image of the content in response to the operation on the display,
Interpret the content of the utterance by the user and
It is determined whether or not the interpreted content of the utterance is independently satisfied as a service request.
When it is determined that the content of the utterance is not independently established as a service, the context of the operation according to the content of the image displayed on the display unit corresponding to the time when the utterance is made is set. A control method of an agent system that executes control to provide a service specified based on the content of the operation context information shown and the content of the utterance. On the computer
Display the image of the content in response to the operation on the display,
Have the user interpret the content of the utterance
It is made to judge whether or not the interpreted content of the utterance is independently satisfied as a service request.
When it is determined that the content of the utterance is not independently established as a service, the context of the operation according to the content of the image displayed on the display unit corresponding to the time when the utterance is made is set. A program that executes control to provide a service specified based on the content of the operation context information shown and the content of the utterance.

Images