JP2022072949A - Program, method, information processing apparatus, and system - Google Patents

Abstract

To provide a program which improves convenience of a user in telecommunication.SOLUTION: A terminal apparatus 10-1 executing a program has an advice control unit 183 which recognizes a message input from any one of the terminal apparatus 10-1 or other terminal apparatus 10-2 to 10-4 in communication with the terminal apparatus 10-1 or the other terminal apparatus 10-2 to 10-4. The advice control unit determines, with respect to the recognized message, based on a result of analysis of a message from the terminal apparatus 10-1 to any of the other terminal apparatuses 10-2 to 10-4 or a message from any of the other terminal apparatuses 10-2 to 10-4 to the terminal apparatus 10-1, whether or not any support for a user 1 of the terminal apparatus 10-1 is necessary, and, if determining that any advice is necessary, generates contents of an advice for the user 1 of the terminal apparatus 10-1.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to programs, methods, information processing devices, and systems.

In remote communication between users, applications may be used that analyze messages contained in a communication session to identify a task and provide assistance in completing the identified task. Patent Document 1 discloses a communication session using a virtual assistant realized by this kind of application.

Special Table 2018-522300

In Patent Document 1, the virtual assistant determines whether the message provided by the user of the client device is intended for another participant in the communication session or is intended for the virtual assistant. For this reason, the user causes the virtual assistant to enter a message intended for the virtual assistant, such as a request or command to support the user's specific task.

However, in the specification described in Patent Document 1, since it is necessary to send a message for input to the virtual assistant, the flow of conversation between users is stopped, which may be inconvenient.

Therefore, it is an object of the present disclosure to improve the convenience of the user in remote communication.

In order to solve the above problems, the program of the present disclosure is a program for causing a computer including a processor and a memory to execute the program. The program has a step of recognizing a message input from the first party or the second party, a step of analyzing the recognized message, and a first party in the remote communication between the first party and the second party to the processor. Based on the results of the analysis of the message from the second party to the second party, or the message from the second party to the first party, a step to determine whether or not support to the first party is necessary, and when it is determined to be necessary. Based on the result of the analysis, the step of generating the advice content that gives the advice to the first party is executed.

According to the present disclosure, it is possible to improve the convenience of the user in remote communication.

It is a figure which shows the example of the whole structure of the system which concerns on 1st Embodiment. It is a block diagram which shows the structure of the terminal apparatus included in the system of this embodiment. It is a block diagram which shows the example of the functional structure of the advice control part. It is a block diagram which shows the example of the functional structure of a system when remote communication is made. It is a flowchart which shows an example of the operation of the advice control unit at the time of generating advice. It is a figure which shows the example of the remark in the remote conference held between the user 1 and the user 4. It is a schematic diagram which shows the example of the analysis result output by the analysis by the intention interpretation unit and the proper noun extraction unit. It is a schematic diagram which shows the display example of the display which displays the advice content from a virtual assistant. It is a schematic diagram which shows the display example of the display which displays the advice content from a virtual assistant. It is a block diagram which shows the example of the functional structure of a terminal apparatus. It is a schematic diagram which shows the display example of the display which displays the advice content from a virtual assistant. It is a schematic diagram which shows the display example of the display which displays the memo created by a virtual assistant. It is a block diagram which shows the example of the functional structure of the system when the advice content generated by the advice control unit is shared between terminal devices. It is a schematic diagram which shows the display example of the display which displays the participant who shares the advice content. It is a schematic diagram which shows the display example of the display of the terminal apparatus used by the participant who shares the advice content. It is a block diagram which shows the example of the functional structure of the advice control part. It is a block diagram which shows the structure of the terminal apparatus included in the system of 2nd Embodiment. It is a block diagram which shows the example of the functional structure of an auxiliary control part. It is a block diagram which shows the example of the functional structure of a system when remote communication is made. It is a flowchart which shows an example of the operation of the dialogue control part when the auxiliary control part creates the minutes. It is a flowchart which shows an example of the operation of the dialogue control part when the auxiliary control part creates a translation sentence. It is a flowchart which shows an example of the operation of the dialogue control part when the auxiliary control part is operated in a surrogate mode. It is a flowchart which shows an example of the operation of the dialogue control part when the auxiliary control part creates advice to a user. It is a schematic diagram which shows the display example of the display of the terminal apparatus when the auxiliary control part is operating. It is a schematic diagram which shows the display example of the display of the terminal apparatus when one of the conference participants is using the surrogate mode. It is a block diagram which shows the structure of the terminal apparatus included in the system of another embodiment. It is a block diagram which shows the example of the functional structure of a system when remote communication is made. It is a block diagram which shows the functional configuration of a server when the advice control part is executed in a server. It is a block diagram which shows the example of the functional structure of a system when remote communication is made. It is a block diagram which shows the functional configuration of a server when the auxiliary control part is executed in a server. It is a block diagram which shows the example of the functional structure of a system when remote communication is made. It is a block diagram which shows the functional configuration of a server when the advisory control part and the auxiliary control part are executed in a server. It is a block diagram which shows the example of the functional structure of a system when remote communication is made.

Hereinafter, embodiments will be described with reference to the drawings. It should be noted that the present embodiment described below does not unreasonably limit the content of the present disclosure described in the claims. Moreover, not all of the configurations described in the present embodiment are essential constituent requirements of the present disclosure.

<Overview>
The information processing device 10 according to the present embodiment operates a virtual assistant in remote communication between a user of the own device and a user of another device. The virtual assistant according to the present embodiment analyzes the message during the session, and gives advice or assistance based on the analyzed content.

(First Embodiment)
<1 Overall configuration>
FIG. 1 is a diagram showing an example of the overall configuration of the system 1 according to the first embodiment. As shown in FIG. 1, the system 1 includes a terminal device 10 and a server 20. The terminal device 10 and the server 20 are connected to each other so as to be able to communicate with each other via the network 80 by using a wired or wireless communication standard. The network 80 is realized by, for example, the Internet and / or a communication network provided by a communication carrier. In the illustrated example, a plurality of terminal devices 10 are included in the system 1.

Although FIG. 1 shows a case where the server 20 is one computer, the server 20 may be realized by combining a plurality of computers. Further, although FIG. 1 shows a case where the number of terminal devices 10 is one, the number of terminal devices 10 accommodated in the system 1 may be a plurality of devices.

The terminal device 10 is a terminal used by a user who uses the service provided by the server 20. For example, the terminal device 10 is a terminal used by a participant who participates in a remote conference provided by the server 20. The terminal device 10 is realized by, for example, a stationary PC (Personal Computer), a laptop PC, a head-mounted display, or the like. Further, the terminal device 10 may be a portable computer such as a smartphone or a tablet terminal.

As shown in FIG. 1, the terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The communication IF 12, the input device 13, the output device 14, the memory 15, the storage 16, and the processor 19 are connected to each other so as to be communicable with each other via, for example, a bus.

The communication IF 12 is an interface for transmitting and receiving signals because the terminal device 10 communicates with an external device. The input device 13 is an input device for receiving an input operation from the user. The input device 13 includes, for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, and the like. The output device 14 is an output device for presenting information to the user. The output device 14 includes, for example, a display, a speaker, and the like.

The memory 15 is for temporarily storing a program, data processed by the program or the like, and is realized by, for example, a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage 16 is a storage device for storing data, and is realized by, for example, a non-volatile memory such as a flash memory or an HDD (Hard Disc Drive). The processor 19 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

The server 20 is a device that manages information about a service, refers to the managed information, and provides a service. For example, the server 20 manages information about the remote conference, refers to the managed information, and provides the remote conference as a service. The server 20 is, for example, a computer connected to the network 80.

The service provided by the server 20 is not limited to the remote conference. As long as it is a service that establishes remote communication among a plurality of users, it may be other than a remote conference. Further, the service provided by the server 20 is not limited to the one that conveys a message by voice. As long as it is a service that conveys a message, it may be a service that conveys a message in text format, that is, a chat service.

As shown in FIG. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The communication IF 22, the input / output IF 23, the memory 25, the storage 26, and the processor 29 are connected to each other so as to be communicable with each other via, for example, a bus.

The communication IF 22 is an interface for transmitting and receiving signals because the server 20 communicates with an external device. The input / output IF 23 functions as an interface with an input device for receiving an input operation from the user and an output device for presenting information to the user. The memory 25 is for temporarily storing a program, data processed by the program, or the like, and is realized by, for example, a volatile memory such as a DRAM.

The storage 26 is a storage device for storing data, and is realized by, for example, a non-volatile memory such as a flash memory or an HDD. The storage 26 does not necessarily have to be realized by a single circuit. The storage 26 may be realized by, for example, a plurality of storage circuits. The processor 29 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

<1.1 Configuration of terminal equipment>
FIG. 2 is a block diagram showing a configuration of a terminal device 10 included in the system 1 of the present embodiment. As shown in FIG. 2, the terminal device 10 includes a communication unit 121, an input device 13, an output device 14, a camera 160, a storage unit 170, and a control unit 180.

The communication unit 121 performs a process for the terminal device 10 to communicate with another device. The communication unit 121 performs transmission processing on the signal generated by the control unit 180, and transmits the signal to the outside (for example, the server 20). The communication unit 121 performs reception processing on the signal received from the outside and outputs the signal to the control unit 180.

The input device 13 is a device for a user who owns the terminal device 10 to input an instruction. The input device 13 is realized by, for example, a mouse 131, a keyboard 132, a touch-sensitive device 133 to which an instruction is input by touching an operation surface, and the like. Further, the input device 13 is realized by the microphone 134. The input device 13 converts an instruction or voice input from the user into an electric signal, and outputs the electric signal to the control unit 180. The input device 13 is not limited to a physical operation device such as a mouse 131 and a keyboard 132. The input device 13 may include, for example, a receiving port that receives an electric signal input from an external input device.

The output device 14 is a device for presenting information to the user who owns the terminal device 10. The output device 14 is realized by, for example, a display 141, a speaker 142, and the like. The display 141 displays data such as images, moving images, and texts according to the control of the control unit 180. The display 141 is realized by, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or the like. The speaker 142 outputs sound according to the control of the control unit 180.

The camera 160 is a device for receiving light by a light receiving element and outputting it as a shooting signal. In the camera 160, for example, a lens that receives light is installed side by side with the display 141. As a result, the camera 160 can take a picture of the user who visually recognizes the display 141.

The storage unit 170 is realized by, for example, a memory 15 and a storage 16, and stores data and a program used by the terminal device 10. Specifically, the storage unit 170 stores, for example, the first application 171 and the second application 172. The first application 171 is an application for realizing remote communication with another user. The second application 172 is an application for operating a virtual assistant in remote communication. Further, the storage unit 170 stores dialogue history data 173 regarding the history of dialogue in remote communication.

Although FIG. 2 shows a case where the second application 172 stored in the storage unit 170 is one, the number of the second application 172 stored in the storage unit 170 may be a plurality. The second application 172 may have various types depending on the content of the advice given to the user. The user may download the second application 172 according to the purpose and switch the second application 172 to be executed as needed.

The second application 172 includes data about the first trained model, data about the second trained model, and data about the program about the second application 172. The first and second trained models are generated, for example, by machine learning. In the present embodiment, the trained model represents a model generated by having a machine learning model perform machine learning according to a model learning program.

The trained model according to the present embodiment is, for example, a synthetic function with parameters in which a plurality of functions are synthesized to perform a predetermined inference based on a signal input from a user. A parameterized composition function is defined by a combination of multiple adjustable functions and parameters. The trained model according to the present embodiment may be a synthetic function with any parameter that satisfies the above requirements.

For example, if the trained model is generated using a forward propagation multi-layered network, the parameterized composition function may be, for example, a linear relationship between layers using a weight matrix, a non-linear relationship using an activation function at each layer. (Or linear relationship), and defined as a combination of biases. Weighted matrices and biases are called multi-layered network parameters. A composite function with parameters changes its form as a function depending on how the parameters are selected. In a multi-layered network, by appropriately setting the parameters to be configured, it is possible to define a function capable of outputting a preferable result from the output layer.

As the multi-layered network according to the present embodiment, for example, a deep neural network (DNN), which is a multi-layered neural network targeted for deep learning, can be used. As the DNN, for example, a convolutional neural network (CNN) for an image may be used. Further, a multi-layer neural network using an attention mechanism (attention) called "Transformer" may be used.

The first trained model is a model used for recognizing sentence intents and slots. In the present embodiment, the intent means the type of intention of the target sentence. A slot means ancillary information necessary to process the intent indicated by the intent. In the present embodiment, the “text” is a general term for information that can be a target for recognizing an intent and a slot, and is used for convenience. Specifically, for example, the text includes a text of the user's spoken voice, a text input by the user, and the like.

The first trained model is trained using training data including a combination of sentences and correct labels. In the present embodiment, the correct answer label corresponding to a certain sentence is a label indicating the intent type, the slot type, and the slot value of the sentence. Depending on the type of intent, there may be multiple slots or none. The aspect of the labeled learning data according to the present embodiment is merely an example.

The first trained model inputs data based on input information (sentence) and outputs an intent type, a slot type, and a slot value. The data based on the input information represents, for example, data in which the target sentence is subjected to necessary processing such as preprocessing and feature amount extraction and converted into information that can be processed by the trained model.

The first trained model may be any model used for recognizing sentence intents and slots, and its function does not have to be the same as that of the first trained model according to the present embodiment. As an example, the first trained model is a model that outputs the probability of being the correct type for candidates of multiple intent types, or outputs the probability of being the correct value for candidates of values of multiple slots. May be.

The second trained model is a model used for extracting proper nouns. The second trained model is trained using the training data including the combination of the sentence and the correct answer label. In the present embodiment, the correct answer label corresponding to a certain sentence is a label indicating the definition of the classification for the named entity and the value of the named entity. The aspect of the labeled learning data according to the present embodiment is merely an example.

The second trained model inputs data based on the input information (sentence), classifies the named entity, and outputs the value of the named entity. The data based on the input information represents, for example, data in which the target sentence is subjected to necessary processing such as preprocessing and feature amount extraction and converted into information that can be processed by the trained model.

The second trained model may be any model used for extracting proper nouns, and its function does not have to be the same as that of the second trained model according to the present embodiment.

The control unit 180 is realized by the processor 19 reading a program stored in the storage unit 170 and executing an instruction included in the program. The control unit 180 controls the operation of the terminal device 10. Specifically, for example, the control unit 180 exerts functions as an operation reception unit 181, a communication control unit 182, an advice control unit 183, and a display control unit 184.

The operation receiving unit 181 performs a process for receiving the user's operation input from the input device 13.

The communication control unit 182 performs a process for the terminal device 10 to communicate with another terminal device 10 participating in the remote communication according to the communication protocol. The communication control unit 182 is realized, for example, by executing the first application 171 by the processor 19. The communication control unit 182 transmits, for example, data based on the voice input from the microphone 134 to another terminal device 10 participating in the remote communication. Further, the communication control unit 182 receives the data transmitted from the other terminal device 10, converts it into voice, and outputs it from the speaker 142.

Further, the communication control unit 182 transmits, for example, text data input from a keyboard 132, a touch-sensitive device 133, or the like to another terminal device 10 participating in remote communication. Further, the communication control unit 182 receives the data transmitted from the other terminal device 10, converts it into text data, and outputs the data from the display 141.

The advice control unit 183 analyzes the message included in the communication session and performs a process of giving advice based on the analysis result to the user. The advisory control unit 183 is realized, for example, by executing the second application 172 by the processor 19. Specifically, the advice control unit 183 carries out the process shown in FIG. 3, for example.

FIG. 3 is a block diagram showing an example of the functional configuration of the advisory control unit 183. The advice control unit 183 shown in FIG. 3 includes a voice recognition unit 1831, an intention interpretation unit 1832, a proper noun extraction unit 1833, an advice determination unit 1834, an external service call unit 1835, and a response generation unit 1836.

The voice recognition unit 1831 recognizes the voice spoken by the user and converts it into text format data. As the speech recognition method, any existing method may be used. When the user inputs information in a text format, the voice recognition unit 1831 may be a text input unit.

The intention interpretation unit 1832 interprets the intention contained in the text data. Specifically, the intention interpretation unit 1832 uses, for example, the first trained model, and outputs the intent type, the slot type, and the slot value based on the input text data.

The proper noun extraction unit 1833 extracts the proper noun included in the text data. Specifically, the named entity extraction unit 1833 uses, for example, the second trained model, classifies the named entity based on the input text data, and outputs the value of the named entity. It should be noted that both the intention interpretation unit 1832 and the proper noun extraction unit 1833 may not necessarily be provided. It does not matter if either one is provided.

The advice determination unit 1834 determines whether or not to give advice to the user based on the result of analyzing the text data. Specifically, the advice determination unit 1834 refers to, for example, the dialogue history data 173, and determines whether or not advice is required for the intention interpreted by the intention interpretation unit 1832. The advice determination unit 1834 determines that advice is necessary, for example, when the interpreted intention is the content that requires the user to confirm the predetermined information.

Further, the advice determination unit 1834 determines, for example, whether or not advice is required for the proper noun extracted by the proper noun extraction unit 1833. The advice determination unit 1834 determines that advice is necessary, for example, when the proper noun is an abbreviation, when it is a technical term, or when it is not included in a general term registered in advance. The advice determination unit 1834 may determine that advice is necessary even when a predetermined proper noun is repeatedly called. In addition, the necessity of advice may be determined from past minutes, in-house documents, and general Web documents according to the number of documents containing the word. That is, for example, it may be determined that advice is necessary for a proper noun that is not in the minutes and has high importance and specialty. As a method for evaluating the importance of words contained in a sentence, an existing method such as tf-idf may be used. In this way, by determining the necessity of advice based on the importance of the word, even if the proper noun is not registered, the advice is not given to the proper noun that the person can know. Therefore, it is possible to avoid over-explanation.

In addition, the advice determination unit 1834 determines whether or not to acquire the information necessary for giving advice from an external service. The advice determination unit 1834 determines that it is necessary to acquire information from an external service, for example, when information from the outside is required in order to respond to the intention interpreted by the intention interpretation unit 1832. Further, the advice determination unit 1834 determines that it is necessary to acquire information from an external service, for example, when information from the outside is required to explain the proper noun extracted by the proper noun extraction unit 1833.

The external service calling unit 1835 acquires the information required for advice from the external service. For example, the external service caller 1835 obtains information from the external service to respond to the interpreted intent. Further, the external service calling unit 1835 acquires information for explaining the proper noun from the external service. The external service calling unit 1835 is realized by, for example, a predetermined API.

The response generation unit 1836 generates the advice content. Specifically, for example, the response generation unit 1836 executes a process corresponding to the interpreted intention and generates advice based on the execution result. Further, the response generation unit 1836 generates advice for the interpreted intention or the proper noun based on the information acquired from the external service, for example.

The display control unit 184 controls the output device 14 in order to present various information to the user. Specifically, for example, the display control unit 184 causes the display 141 to display the advice generated by the advice control unit 183.

<2. Operation>
The operation of the advice control unit 183 when remote communication is performed by a plurality of users will be described.

FIG. 4 is a block diagram showing an example of the functional configuration of the system 1 when remote communication is performed. FIG. 5 is a flowchart showing an example of the operation of the advice control unit 183 when generating advice. FIG. 4 shows a case where a remote conference is held between the terminal devices 10-1 to 10-4, and the advice control unit 183 as a virtual assistant is activated in the terminal device 10-1.

FIG. 6 is a diagram showing an example of remarks at a remote conference held between users 1 and 4. As shown in FIG. 6, first, the question M1: "What are the recent voice recognition services?" Is posed by voice from the user 1 to the users 2 to 4.

The advice control unit 183 performs voice recognition for the message M1 transmitted from the terminal device 10-1 to the terminal devices 10-2 to 10-4 by the voice recognition unit 1831 (step S11). The voice recognition unit 1831 converts the voice data for the message M1 into text data.

The advice control unit 183 performs the proper noun extraction process and the intention interpretation process in parallel for the text data. In the description of FIG. 5, first, the proper noun extraction process will be described.

The advice control unit 183 extracts the proper noun included in the text data by the proper noun extraction unit 1833 (step S12). Specifically, the named entity extraction unit 1833 uses, for example, the second trained model, classifies the named entity based on the input text data, and outputs the value of the named entity.

FIG. 7 is a schematic diagram showing an example of the analysis result output by the analysis by the intention interpretation unit 1832 and the proper noun extraction unit 1833. In the example shown in FIG. 7, the analysis result R1 represents the result of the proper noun extraction process, and the analysis result R2 represents the result of the intention interpretation process. According to the example shown in FIG. 7, the named entity extraction unit 1833 outputs the classification: “proprietary noun” and the value of the named entity: “speech recognition service” to the message M1.

The advice determination unit 1834 determines whether or not to give advice to the user 1 based on the result of analyzing the text data (step S13). Specifically, the advice determination unit 1834 determines, for example, whether or not advice is required for the proper noun extracted by the proper noun extraction unit 1833. The advice determination unit 1834 determines that advice is not required, for example, assuming that "proprietary noun = speech recognition service" is included in a general term registered in advance, not an abbreviation or a technical term (). No) in step S13, the process for the message M1 is terminated.

The advice control unit 183 interprets the intention contained in the text data by the intention interpretation unit 1832 (step S16). Specifically, the intention interpretation unit 1832 uses, for example, the first trained model, and outputs the intent type, the slot type, and the slot value based on the input text data.

According to the example shown in FIG. 7, the intention interpretation unit 1832 outputs the intent type: “search”, the slot type: “keyword”, and the slot value: “voice recognition service” to the message M1.

The advice determination unit 1834 determines whether or not to give advice to the user 1 based on the result of analyzing the text data (step S17). Specifically, the advice determination unit 1834 refers to, for example, the dialogue history data 173, and determines whether or not advice is required for the intention interpreted by the intention interpretation unit 1832. The advice determination unit 1834 determines that the intent type: "search", slot type: "keyword", slot value: "speech recognition service" is, for example, the first agenda item in a meeting, or to others. Since it is a question of, it is determined that advice is unnecessary (No in step S17), and the processing for the message M1 is terminated.

Subsequently, as shown in FIG. 6, the user 2 replies to the user 1 by voice, message M2: "GAFAM in the United States provides a voice recognition service."

The advice control unit 183 performs voice recognition for the message M2 transmitted from the terminal device 10-2 to the terminal device 10-1 by the voice recognition unit 1831 (step S11). The voice recognition unit 1831 converts the voice data for the message M2 into text data.

The advisory control unit 183 uses the second trained model, for example, to output the classification of the named entity and the value of the named entity based on the input text data by the named entity extraction unit 1833. According to the example shown in FIG. 7, the named entity extraction unit 1833 outputs the classification: “proprietary noun” and the value of the named entity: “GAFAM” to the message M2 (step S12).

The advice determination unit 1834 determines, for example, whether or not advice is required for the proper noun extracted by the proper noun extraction unit 1833 (step S13). For example, the advice determination unit 1834 determines that advice is necessary because "proprietary noun = GAFAM" is an abbreviation (Yes in step S13), and shifts the process to step S14.

In step S14, the advice determination unit 1834 determines whether or not to acquire the information necessary for giving advice from the external service. Specifically, the advice determination unit 1834 determines whether or not to acquire information from an external service, for example, depending on whether or not the extracted proper noun is included in the term registered in the storage unit 170. .. Here, for example, it is assumed that the "proprietary noun = GAFAM" is not registered in the storage unit 170, and the advice determination unit 1834 determines that the information about the "proprietary noun = GAFAM" is acquired from the external service (Yes in step S14). ).

The external service calling unit 1835 acquires the information required for advice from the external service (step S15). Specifically, for example, the external service calling unit 1835 searches for information for explaining the proper noun in the external service by inputting "proprietary noun = GAFAM" as a keyword.

The response generation unit 1836 generates advice content based on the search result (step S110). Specifically, for example, the response generation unit 1836 gives advice content A1: "GAFA is Google (registered trademark), Amazon (registered trademark)" based on the top-level search result obtained from an external service for a proper nomenclature. ), Facebook (registered trademark), Apple (registered trademark), and GAFAM refers to 5 companies including Microsoft (registered trademark). " As a result, as shown in FIG. 6, the advice content A1 of the virtual assistant for the message M2 is generated.

The advice control unit 183 outputs the intent type, the slot type, and the slot value by the intention interpretation unit 1832, for example, using the first trained model and based on the input text data (step S16). ..

According to the example shown in FIG. 7, the intention interpretation unit 1832 outputs the intent type: “search”, the slot type: “keyword”, and the slot value: “GAFAM” to the message M2.

The advice determination unit 1834 refers to, for example, the dialogue history data 173, and determines whether or not advice is required for the intention interpreted by the intention interpretation unit 1832 (step S17). The advice determination unit 1834 determines that advice is required for the intent type: "search", slot type: "keyword", and slot value: "GAFAM" (Yes in step S17), and processes in step S18. Move to.

In step S18, the advice determination unit 1834 determines whether or not to acquire the information necessary for giving advice from the external service. Specifically, the advice determination unit 1834 determines whether or not to acquire information from an external service, for example, based on the type of intent. Here, for example, since the type of intent is “search”, the advice determination unit 1834 determines that the information about “keyword = GAFAM” is acquired from the external service (Yes in step S18).

The external service calling unit 1835 acquires the information required for advice from the external service (step S15). Specifically, for example, the external service calling unit 1835 acquires information for explaining "keyword = GAFAM" from the external service. When the advice control unit 183 acquires information from the external service, the processing proceeds to step S110.

When the advice content is generated by the advice control unit 183, the display control unit 184 displays the generated advice content on the display 141. FIG. 8 is a schematic diagram showing a display example of the display 141 displaying the content of advice from the virtual assistant. In the example shown in FIG. 8, images showing users 2 to 4 attending a remote conference are displayed. In the example shown in FIG. 8, since the virtual assistant generates advice for the user 2's speech, the users 2 to 4 are displayed so that the user 1 can identify that the user 2 has spoken. Specifically, the image of the user 2 is displayed larger than the user 3 and the user 4. It should be noted that the identification display of the speaker is not limited to enlarging the image. For example, the speaker may be identified by edging the image.

Further, in the example shown in FIG. 8, the advice content A1 is displayed below the image of the user 2. The display position of the advice content A1 may be displayed anywhere as long as it is easily visible to the user 1.

According to the example shown in FIG. 6, the user 2 receives the message M2: "GAFAM in the United States provides a voice recognition service." Then, "Performance is an advanced end-to-end method of deep learning. It looks like it will go up considerably. " Based on the analysis result of this message, the advice determination unit 1834 determines that no advice is required for this message (No in step S13, No in step S17). Therefore, no advice is displayed for this message.

Similarly, in FIG. 6, user 1 remarks to users 2 to 4 the message: "GAFAM? If so, it would be easier to make automatic minutes of the meeting." Based on the analysis result of this message, the advice determination unit 1834 determines that no advice is required for this message (No in step S13, No in step S17). Therefore, no advice is displayed for this message.

Subsequently, in FIG. 6, the user 3 remarks to the users 1, 2, and 4 the message M3: "I would like to set up a teleconferencing tomorrow afternoon about that, is it possible?"

The advice control unit 183 performs voice recognition for the message M3 transmitted from the terminal device 10-3 to the terminal devices 10-1, 10-2, and 10-4 by the voice recognition unit 1831 (step S11). The voice recognition unit 1831 converts the voice data for the message M3 into text data.

The advisory control unit 183 uses the second trained model, for example, to output the classification of the named entity and the value of the named entity based on the input text data by the named entity extraction unit 1833. The named entity extraction unit 1833 outputs the classification: "date and time", the value of the named entity: "tomorrow, afternoon", etc. to the message M3 (step S12).

The advice determination unit 1834 determines, for example, whether or not advice is required for the proper noun extracted by the proper noun extraction unit 1833 (step S13). Since the proper noun has not been extracted, the advice determination unit 1834 determines that advice is unnecessary (No in step S13), and ends the process for the message M3 that has undergone the proper noun extraction.

The advice control unit 183 outputs the intent type, the slot type, and the slot value by the intention interpretation unit 1832, for example, using the first trained model and based on the input text data (step S16). ..

According to the example shown in FIG. 7, the intention interpretation unit 1832, for the message M3, has an intent type: “schedule search”, a slot type: “date and time”, a slot value: “tomorrow, afternoon”, and a slot. Type: "Participant" and slot value: "User 1" is output.

The advice determination unit 1834 refers to, for example, the dialogue history data 173, and determines whether or not advice is required for the intention interpreted by the intention interpretation unit 1832 (step S17). The advice determination unit 1834 has an intent type: "schedule search", a slot type: "date and time" and a slot value: "tomorrow, afternoon", a slot type: "participant" and a slot value: "user 1". It is determined that advice is necessary (Yes in step S17), and the process is shifted to step S18.

In step S18, the advice determination unit 1834 determines whether or not to acquire the information necessary for giving advice from the external service. Here, for example, since the type of intent is "schedule search", the advice determination unit 1834 determines that information about "date and time = tomorrow, afternoon" and "participant = user 1" is acquired from the storage unit 170. (No in step S18).

The response generation unit 1836 acquires the information required for advice from the storage unit 170 (step S19). Specifically, the response generation unit 1836 acquires an unscheduled time zone in the afternoon of tomorrow based on "date and time = tomorrow, afternoon" and "participant = user 1".

The response generation unit 1836 generates advice content based on the acquisition result (step S110). Specifically, for example, the response generation unit 1836 generates advice content A2: "The schedule for tomorrow afternoon is free" based on the information about the unscheduled time zone in tomorrow afternoon. As a result, as shown in FIG. 6, the advice content A2 of the virtual assistant for the message M3 is generated.

When the advice content A2 is generated by the advice control unit 183, the display control unit 184 displays the generated advice content A2 on the display 141.

According to the example shown in FIG. 6, when the user 1 confirms the advice content A2, the user 1 speaks to the users 2 to 4 with a message: "It is possible. How about from 1:00 pm to 2:00 pm?". Based on the analysis result of this message, the advice determination unit 1834 determines that no advice is required for this message (No in step S13, No in step S17). Therefore, no advice is displayed for this message.

Similarly, in FIG. 6, users 2 to 4 say a message: "It's good" to user 1. Based on the analysis result of this message, the advice determination unit 1834 determines that no advice is required for this message (No in step S13, No in step S17). Therefore, no advice is displayed for this message.

Subsequently, in FIG. 6, the user 1 remarks to the users 2 to 4 the message M4: "Then, the remote conference will be set from 1:00 pm to 2:00 pm tomorrow."

The advice control unit 183 performs voice recognition for the message M4 transmitted from the terminal device 10-1 to the terminal devices 10-2 to 10-4 by the voice recognition unit 1831 (step S11). The voice recognition unit 1831 converts the voice data for the message M4 into text data.

The advisory control unit 183 uses the second trained model, for example, to output the classification of the named entity and the value of the named entity based on the input text data by the named entity extraction unit 1833. The named entity extraction unit 1833 outputs the classification: "date and time", the value of the named entity: "tomorrow", etc. to the message M4 (step S12).

The advice determination unit 1834 determines, for example, whether or not advice is required for the proper noun extracted by the proper noun extraction unit 1833 (step S13). Since the proper noun has not been extracted, the advice determination unit 1834 determines that advice is unnecessary (No in step S13), and ends the process for the message M4 that has undergone the proper noun extraction.

The advice control unit 183 outputs the intent type, the slot type, and the slot value by the intention interpretation unit 1832, for example, using the first trained model and based on the input text data (step S16). ..

According to the example shown in FIG. 7, the intention interpretation unit 1832 responds to the message M4 by intent type: “schedule setting”, slot type: “date and time” and slot value: “tomorrow”, slot type: "Start time" and slot value: "1 pm", slot type: "end time" and slot value: "2:00 pm", slot type: "participant" and slot value: "user 1" Is output.

The advice determination unit 1834 refers to, for example, the dialogue history data 173, and determines whether or not advice is required for the intention interpreted by the intention interpretation unit 1832 (step S17). The advice determination unit 1834 has an intent type: "schedule setting", a slot type: "date and time" and a slot value: "tomorrow", a slot type: "start time" and a slot value: "1:00 pm". , Slot type: "End time" and Slot value: "2:00 pm", Slot type: "Participant" and Slot value: "User 1", it is determined that advice is required (step S17). Yes), the process shifts to step S18.

In step S18, the advice determination unit 1834 determines whether or not to acquire the information necessary for giving advice from the external service. Here, for example, since the type of intent is "schedule setting", the advice determination unit 1834 has "date and time = tomorrow", "start time = 1:00 pm", "end time = 2:00 pm", and "participation". It is determined that the information about "person = user 1" is acquired from the storage unit 170 (No in step S18).

The response generation unit 1836 acquires the information required for advice from the storage unit 170 (step S19). Specifically, the response generation unit 1836 is based on "date and time = tomorrow", "start time = 1:00 pm", "end time = 2:00 pm", and "participant = user 1", and tomorrow at 1:00 pm Set up a remote conference at 2:00 pm and get information that the schedule has been set.

The response generation unit 1836 generates advice content based on the acquisition result (step S110). Specifically, for example, the response generation unit 1836 gives advice based on the information that the remote conference was set from 1 pm to 2 pm tomorrow. A3: "The remote conference is held from 1 pm to 2 pm tomorrow. I have set it. " As a result, as shown in FIG. 6, the advice content A3 of the virtual assistant for the message M4 is generated.

When the advice content A3 is generated by the advice control unit 183, the display control unit 184 displays the generated advice content A3 on the display 141.

As described above, in the first embodiment, the advice control unit 183 recognizes the message input from the first party or the second party in the remote communication between the first party and the second party. The advisory control unit 183 analyzes the recognized message. The advice control unit 183 determines whether or not support for the first party is necessary based on the result of analysis of the message from the first party to the second party or the message from the second party to the first party. .. Then, when it is determined that it is necessary, the advice control unit 183 is designed to generate advice content that gives advice to the first party based on the result of the analysis. As a result, the advisory control unit 183 can create the content of advice to the first party based on the communication between the first party and the second party even if there is no message intended by the advisory control unit 183. .. Therefore, it is possible to create the content of advice to the first party without damaging the session between the first party and the second party.

Therefore, according to the information processing apparatus 10 according to the first embodiment, it is possible to improve the convenience of the user in remote communication.

In the first embodiment, the case where the virtual assistant (advice control unit 183) explains the proper noun and the case where the advice regarding the schedule is given based on the interpreted intention will be described as an example. The advice given by the advice control unit 183 according to the first embodiment is not limited to these.

The advice control unit 183 expands the range of interpretable intentions by registering a new intent. Therefore, the advice control unit 183 can give appropriate advice to the user in conversations in various fields by registering the intent corresponding to the conversation. Further, the advisory control unit 183 may realize different functions by executing the second application 172 in which different intents are registered by the processor 19. Therefore, by downloading the second application 172 corresponding to the purpose of implementation, it is possible to use the advice control unit 183 according to the application.

Specifically, for example, the advice control unit 183 may give advice such as a facilitator for facilitating a remote conference. For example, it is assumed that the minutes of the past meeting are stored in the storage unit 170 of the terminal device 10-1. The advice determination unit 1834 recognizes the topic included in the message, for example, based on the analysis result of the message issued by any of the users 1 to 4. The advice determination unit 1834 determines whether or not advice is required based on whether or not the minutes of discussing the recognized topic are stored in the storage unit 170. When the minutes of discussing the recognized topic are stored in the storage unit 170, the advice determination unit 1834 determines that advice is required.

The response generation unit 1836 acquires the information required for advice from the minutes stored in the storage unit 170. Specifically, the response generation unit 1836 searches for the minutes in which the recognized topic becomes a topic. The response generation unit 1836 generates the advice content based on the searched minutes. Specifically, for example, the response generation unit 1836 points out that the past agenda is being discussed again based on the searched minutes. Advice content: "The topic is a meeting on MM month DD day ... I came to the conclusion that. " Therefore, the advice control unit 183 can prevent the same discussion from being repeated.

Further, the advice determination unit 1834 may determine whether or not advice is necessary based on whether or not the topic changes are scarce. Whether or not the topic changes poorly is determined based on the degree of change along the time axis of the recognized topic distribution. Specifically, for example, a state in which similar discussions are repeated without a solution being presented corresponds to a state in which there is little change in the topic. If the topic changes poorly, the advice determination unit 1834 determines that advice is needed because the discussion may not have progressed.

The response generation unit 1836 generates the advice content in consideration of the possibility that the discussion has not progressed. Specifically, for example, the response generation unit 1836 generates the advice content that proposes to organize the issues: "I think it is better to organize the issues once." Therefore, the advice control unit 183 can prevent the tour from going round.

Further, for example, it is assumed that the agenda of the conference is stored in the storage unit 170 of the terminal device 10-1. The meeting agenda includes topics such as explanations of discussion points, statements of participants, today's conclusions, remaining issues, action items, and so on. The advice determination unit 1834 recognizes the topic included in the message, for example, based on the analysis result of the message issued by any of the users 1 to 4. When the remaining time of the meeting reaches a predetermined value, the advice determination unit 1834 determines whether or not advice is required based on whether or not all the topics included in the agenda have been discussed. If any of the topics included in the agenda are not discussed, the Advice and Determination Department 1834 determines that advice is needed.

Response generator 1836 generates advice based on topics not discussed in the agenda. Specifically, for example, when the action item is not discussed, the response generation unit 1836 generates the advice content: "I haven't talked about the action item, is that okay?". Therefore, the advice control unit 183 can support the operation of the output-oriented conference.

Further, for example, it is assumed that the storage unit 170 of the terminal device 10-1 stores information about the profile of the user who is participating in the conference. The advice determination unit 1834 recognizes the topic included in the message, for example, based on the analysis result of the message issued by any of the users 1 to 4. The advice determination unit 1834 grasps the proper noun included in the recognized topic, the superordinate concept of the proper noun, and the related concept. The advice determination unit 1834 determines whether or not advice is required based on whether or not a profile including the grasped proper noun, superordinate concept, and related concept is stored in the storage unit 170. When the profile including the grasped concept is stored in the storage unit 170, the advice determination unit 1834 determines that there is a participant who has knowledge about the recognized topic and needs advice.

The response generation unit 1836 acquires the information required for advice from the information regarding the profile stored in the storage unit 170. Specifically, the response generation unit 1836 searches for information on the profile including the grasped proper noun, superordinate concept, and related concept. The response generation unit 1836 generates advice content based on the acquired information. Specifically, for example, the response generation unit 1836 generates advice content: "If it is the topic, it is better to ask Mr. AAAA to talk about it." Based on the searched profile information. Since it is possible to match the topic under discussion with the participants, the advisory control unit 183 considers factors such as the temperature difference in the participants' willingness to speak, the relationship with the owner of the conference, and other factors that tend to cause bias in the statements at the conference. Eliminate, fair and appropriate facilitation according to the content will be possible.

Further, the advice control unit 183 may output the text data converted by the voice recognition unit 1831 as it is as the advice content. As a result, the contents of the remote communication will be displayed as text. By doing so, the user can understand the remote communication even when the user's hearing is weak. Further, even when the user is in an environment where it is difficult to hear the voice of the remote communication, for example, in a noisy place, the user can understand the remote communication.

Further, the advice control unit 183 may convert the text data converted by the voice recognition unit 1831 into a different language and output it. As a result, the contents of the remote communication will be displayed in different languages.

Further, in the first embodiment, an example in which the advice control unit 183 gives advice to a message input by voice has been described. However, the input message is not limited to voice. The advice control unit 183 may respond to a message input by text in the same manner.

FIG. 9 is a schematic diagram showing a display example of the display 141 displaying the content of advice from the virtual assistant. In the example shown in FIG. 9, the chat screen is displayed in the area 1411. The advice control unit 183 generates advice content for a message input in text. The display control unit 184 displays, for example, the generated advice content A1 in the area 1411 as shown in FIG.

Further, in the first embodiment, the display according to the function of the virtual assistant (advice control unit 183) is not described. However, an image that can identify the function of the advisory control unit 183 may be displayed on the display 141. For example, an image that can identify the function of the advice control unit 183 is displayed in a space related to the advice created by the advice control unit 183, specifically, the space S1 in FIG. At this time, the image displayed in the space S1 is, for example, an icon representing knowledge (for example, an icon representing a dictionary) when the advice control unit 183 explains a proper noun. Further, for example, when the advice control unit 183 gives advice regarding the schedule, the displayed image becomes an icon (for example, an icon representing a secretary) representing a person who adjusts the schedule. Further, for example, when the advice control unit 183 gives advice such as a facilitator, the displayed image becomes an icon representing the person in charge of the progress of the conference (for example, an icon representing the moderator). Further, for example, when the advice control unit 183 translates, the displayed image becomes an icon indicating that the translation is being performed (for example, an icon representing language conversion). Further, for example, when the advice control unit 183 creates the minutes, the displayed image represents an icon indicating that the contents of the meeting are recorded (for example, an icon representing a shorthand reporter or a memo creation). Icon etc.). The image that can identify the function of the advice control unit 183 can be paraphrased as an image that can identify the attribute of the advice content.

(Variation example of the first embodiment)
Although the function of the advisory control unit 183 has been described in the first embodiment, the function of the advisory control unit 183 is not limited to the above. For example, the advice control unit 183 may have a function of switching voice.

FIG. 10 is a block diagram showing an example of the functional configuration of the terminal device 10. In the example shown in FIG. 10, the terminal device 10 has a first switching unit 185 and a second switching unit 1837.

The first switching unit 185 switches on / off the mute function of the microphone 134. When the mute function is turned on, the input from the microphone 134 does not reach the communication control unit 182. On the other hand, when the mute function is turned off, the input from the microphone 134 reaches the communication control unit 182. In the example shown in FIG. 10, a state in which the mute function is turned off is shown.

The second switching unit 1837 switches whether to output the input from the microphone 134 to the communication control unit 182 or to the advisory control unit 183. The second switching unit 1837 is designed to function in front of the first switching unit 185. For example, a so-called push-to-talk method may be used in which the second switching unit 1837 is connected to the advisory control unit 183 only while a predetermined button is pressed.

When the second switching unit 1837 is connected to the advisory control unit 183, the voice input from the user is output to the advisory control unit 183. At this time, the voice is output to the advice control unit 183 by the first switching unit 185 regardless of whether the mute function is on or off. When the second switching unit 1837 is connected to the advisory control unit 183 and the mute function is turned off by the first switching unit 185, the voice input from the user is output to the communication control unit 182.

FIG. 11 is a schematic diagram showing a display example of the display 141 displaying the content of advice from the virtual assistant. In the example shown in FIG. 11, the first icon 1412 and the second icon 1413 are displayed. The first icon 1412 is an icon for operating the switching of the first switching unit 185, and when pressed, the mute function is switched on / off. The second icon 1413 is an icon for operating the switching of the second switching unit 1837, and when the icon is continuously pressed, voice input to the advice control unit 183 is possible only while the second icon 1413 is pressed.

By providing the second switching unit 1837 in this way, the advisory control unit 183 can receive the voice input from the user without leaking to other users. That is, the conversation instruction to the advice control unit 183 is not erroneously transmitted to another user. Further, by arranging the second switching unit 1837 in front of the first switching unit 185, it is possible to input a voice instruction to the virtual assistant even when the mute function is on.

As an example of using this function, for example, there is an example of creating a memo using a virtual assistant. At this time, for example, the advice control unit 183 performs voice recognition on the input voice data and generates the advice content based on the generated text data.

The user who wants to create a memo keeps pressing the second icon 1413, and inputs the content to be noted to the advice control unit 183 as voice. The advice control unit 183 converts the input voice data into text data, and generates the converted text data as the advice content. The display control unit 184 displays the generated advice content on the display 141.

FIG. 12 is a schematic diagram showing a display example of the display 141 for displaying the memo created by the virtual assistant. In the example shown in FIG. 12, a memo created by the advisory control unit 183 is displayed in the area 1414.

As described above, since the advice control unit 183 can display the voice input by the user as text in real time, the user can create a memo by voice. This eliminates the need for effort to take notes.

Further, as an example of using this function, for example, there is an example of connecting the advice control unit 183 and a desired participant.

FIG. 13 is a block diagram showing an example of the functional configuration of the system 1 when the advice content generated by the advice control unit 183 is shared between the terminal device 10-1 and the terminal device 10-2.

A desired participant, for example, a user who wants to output the advice content to the user 2, presses the second icon 1413, and while pressing the second icon 1413, utters the name of the participant who wants to connect to the advice control unit 183. The advice control unit 183 is set to send the advice content to the input participants as well.

FIG. 14 is a schematic diagram showing a display example of a display 141 displaying participants who share the content of advice. In the example shown in FIG. 14, the instruction object 1415 representing the participants sharing the advice content is arranged in the area for displaying the user 2. The nomination of the participants who share the advice content is not limited to the voice input by pressing the second icon 1413, but may be by pressing the display area of the participants.

FIG. 15 is a schematic diagram showing a display example of the display 141 of the terminal device used by the participants who share the advice content. The example shown in FIG. 15 represents a display example of the display 141 of the terminal device 10-2 operated by the user 2. In the example shown in FIG. 15, the instruction object 1416 representing the source of the advice content is arranged in the area for displaying the user 1. Further, in the example shown in FIG. 15, the advice content A1 is displayed at the lower part of the screen. Information about the sender of the advice content, for example, "[Advice content from user 1]" is displayed in the advice content A1.

In this way, since the advice control unit 183 can share the advice content with the designated participants, the user can attend the meeting while sharing the information obtained during the meeting.

Further, after connecting the advisory control unit 183 and the desired participant, the advisory control unit 183 may, for example, transmit the voice input from the setter to the shared participant. That is, after the advice partner is specified, the advice partner may be given advice by the user's own voice.

At this time, for example, if a specific group is set, information can be shared within the group. For example, the display 141 is displayed with an icon for designating the set group, and when this icon is pressed, the advice control unit 183 connects to a member of the set group. The advisory control unit 183 transmits data about the input voice to, for example, a connected member.

In this way, the advice control unit 183 can transmit the voice input from the user to the set participants. Therefore, if a group call is performed with the mute function turned on by pressing the first icon 1412, it is possible to make a call within the group during the conference.

Further, in the first embodiment, the advice control unit 183 may have a function of generating voice, for example.

FIG. 16 is a block diagram showing an example of the functional configuration of the advisory control unit 183. In the example shown in FIG. 16, the advice control unit 183 includes a voice recognition unit 1831, an intention interpretation unit 1832, a proper nomenclature extraction unit 1833, an advice determination unit 1834, an external service call unit 1835, and a response generation unit 1836, as well as a voice synthesis unit. It has 1838.

The voice synthesis unit 1838 generates voice based on the content of the advice. The advisory control unit 183 outputs the generated voice data from the speaker 142 to the user.

(Second embodiment)
In the first embodiment, an example is described in which a virtual assistant by the advice control unit 183 monitors a session between users and gives advice as needed. In the second embodiment, a case where the virtual assistant assists the user, such as participating in a session on behalf of the user, will be described.

<1 Overall configuration>
The overall configuration of the system according to the second embodiment is the same as the overall configuration of the system 1 according to the first embodiment.

<1.1 Configuration of terminal equipment>
FIG. 17 is a block diagram showing a configuration of the terminal device 10A included in the system of the present embodiment. As shown in FIG. 17, the terminal device 10A includes a communication unit 121, an input device 13, an output device 14, a camera 160, a storage unit 170A, and a control unit 180A.

The storage unit 170A is realized by, for example, a memory 15 and a storage 16, and stores data and a program used by the terminal device 10A. Specifically, the storage unit 170A stores, for example, the first application 171 and the third application 174. The third application 174 is an application for operating the virtual assistant according to the second embodiment in remote communication. Further, the storage unit 170A stores the dialogue history data 173 regarding the history of dialogue in remote communication and the minutes data 175 regarding the minutes in remote communication.

Although FIG. 17 shows a case where the third application 174 stored in the storage unit 170A is one, the number of the third application 174 stored in the storage unit 170A may be a plurality. The third application 174 may have various types depending on the content of assistance to the user. The user may download the third application 174 according to the purpose and switch the third application 174 to be executed as needed.

The third application 174 includes data about the first trained model and data about the program about the third application 174. The first trained model is generated by machine learning, for example. In the present embodiment, the trained model represents a model generated by having a machine learning model perform machine learning according to a model learning program. The third application 174 may include data related to the second trained model.

The control unit 180A is realized by the processor 19 reading a program stored in the storage unit 170A and executing an instruction included in the program. The control unit 180A controls the operation of the terminal device 10A. Specifically, for example, the control unit 180A exerts functions as an operation reception unit 181, a communication control unit 182, an auxiliary control unit 186, and a display control unit 184.

The auxiliary control unit 186 analyzes the message included in the communication session, and performs a process of assisting the user based on the analysis result. The auxiliary control unit 186 is realized, for example, by executing the third application 174 by the processor 19. Specifically, the auxiliary control unit 186 carries out the process shown in FIG. 18, for example.

FIG. 18 is a block diagram showing an example of the functional configuration of the auxiliary control unit 186. The auxiliary control unit 186 shown in FIG. 18 includes a text input unit 1861, a first intention interpretation unit 1862, a voice recognition unit 1863, a second intention interpretation unit 1864, a dialogue control unit 1865, an external service call unit 1866, and a first response generation unit. It has 1867, a second response generation unit 1868, and a voice synthesis unit 1869.

The text input unit 1861 receives an instruction input by a user who uses the auxiliary control unit 186. In the example shown in FIG. 18, it is assumed that the user inputs an instruction in a text format. When the user inputs an instruction by voice, the text input unit 1861 may be a voice recognition unit.

The first intention interpretation unit 1862 interprets the intention included in the text data input from the text input unit 1861. Specifically, the first intention interpretation unit 1862 uses, for example, the first trained model, and outputs the intent type, the slot type, and the slot value based on the input text data.

The voice recognition unit 1863 recognizes the voice output from the communication control unit 182 and converts it into text format data. As the speech recognition method, any existing method may be used. The text data converted by the voice recognition unit 1863 is stored in the storage unit 170A as the minutes data 175. When the user inputs information in a text format, the voice recognition unit 1863 may be a text input unit.

The second intention interpretation unit 1864 interprets the intention included in the text data output from the voice recognition unit 1863. Specifically, the second intention interpretation unit 1864 uses, for example, the first trained model, and outputs the intent type, the slot type, and the slot value based on the input text data.

The dialogue control unit 1865 performs processing according to an instruction from the user. Specifically, the dialogue control unit 1865 starts creating the minutes when, for example, the user instructs the user to create the minutes. When the dialogue control unit 1865 receives the intention interpreted by the second intention interpretation unit 1864 during the process of creating the minutes, the dialogue control unit 1865 outputs the received intention to the first response generation unit 1867.

Further, the dialogue control unit 1865 starts creating a translated sentence when, for example, the user instructs to create a translated sentence into a different language. When the dialogue control unit 1865 receives the intention interpreted by the second intention interpretation unit 1864 during the process of creating the translated text, the dialogue control unit 1865 outputs the received intention to the first response generation unit 1867.

Further, the dialogue control unit 1865 transitions to the proxy mode when, for example, the user instructs the user to set the proxy mode. The dialogue control unit 1865 determines whether or not a response is required for the intention interpreted by the first intention interpretation unit 1862 or the second intention interpretation unit 1864 in the surrogate mode. The dialogue control unit 1865 determines that a response is necessary, for example, when the intention interpreted by the first intention interpretation unit 1862 is to request the transmission of a message to the conference. Further, the dialogue control unit 1865 refers to, for example, the dialogue history data 173, and determines that a response is necessary when the intention interpreted by the second intention interpretation unit 1864 is to request a response from the user. .. Further, for example, when the dialogue control unit 1865 receives the intention interpreted by the second intention interpretation unit 1864 when the summary of the conference is instructed in the proxy mode, the dialogue control unit 1865 outputs the received intention to the first response generation unit 1867. do.

Further, the dialogue control unit 1865 determines, for example, whether or not to acquire information necessary for giving advice from an external service when an instruction to request advice is input from the user. The dialogue control unit 1865 determines, for example, that it is necessary to acquire information from an external service when information from the outside is required in order to respond to the intention interpreted by the first intention interpretation unit 1862. ..

The external service calling unit 1866 acquires the information required for advice from the external service. For example, the external service calling unit 1866 acquires information for responding to the intention interpreted by the first intention interpreting unit 1862 from the external service. The external service calling unit 1866 is realized by, for example, a predetermined API.

The first response generation unit 1867 executes the process requested by the user based on the intention output from the dialogue control unit 1865, and generates the response content. Specifically, for example, the first response generation unit 1867 generates the minutes of the meeting, the translation of the meeting contents, or the summary sentence of the meeting based on the intention interpreted by the second intention interpretation unit 1864. Further, the first response generation unit 1867 generates advice content based on the intention interpreted by the first intention interpretation unit 1862, for example. Further, the first response generation unit 1867 generates advice for the intention interpreted by the first intention interpretation unit 1862, for example, based on the information acquired from the external service.

The second response generation unit 1868 executes the process requested by the user based on the intention output from the dialogue control unit 1865, and generates the response content. Specifically, for example, the second response generation unit 1868 generates the response content based on the intention interpreted by the second intention interpretation unit 1864.

The voice synthesis unit 1869 generates voice based on the response content. The auxiliary control unit 186 transmits the generated voice data to another terminal device 10 via the communication control unit 182.

The display control unit 184 controls the output device 14 in order to present various information to the user. Specifically, for example, the display control unit 184 causes the display 141 to display the auxiliary contents generated by the auxiliary control unit 186.

<2. Operation>
The operation of the auxiliary control unit 186 when remote communication is performed by a plurality of users will be described.

FIG. 19 is a block diagram showing an example of the functional configuration of the system when remote communication is performed. FIG. 20 is a flowchart showing an example of the operation of the dialogue control unit 1865 when the auxiliary control unit 186 creates the minutes. FIG. 19 shows a case where a remote conference is held between the terminal devices 10A-1 to 10-4, and the auxiliary control unit 186 as a virtual assistant is activated by the terminal device 10A-1.

First, the user 1 who operates the terminal device 10A-1 operates the terminal device 10A-1 when taking minutes, and inputs, for example, "take minutes".

The text input unit 1861 accepts the "take minutes" input from the user 1. The first intention interpretation unit 1862 interprets the intention included in the text data input from the text input unit 1861. Specifically, the first intention interpretation unit 1862 uses, for example, the first trained model, and based on the input text data: "take minutes", the intent type: "minutes", the slot Type: "Process", Slot value: "Start" is output.

The dialogue control unit 1865 determines whether or not there is an instruction to create minutes from the first intention interpretation unit 1862 (step S21). Since the analysis result output from the first intention interpretation unit 1862 is the intent type: "minutes", the slot type: "processing", and the slot value: "start", the dialogue control unit 1865 is the proceedings. It is determined that there is an instruction to create a record (Yes in step S21).

When the dialogue control unit 1865 determines that the instruction to create the minutes has been given, the dialogue control unit 1865 starts creating the minutes (step S22). When the dialogue control unit 1865 starts the preparation of the minutes, the first response generation unit 1867 is notified that the preparation of the minutes has started.

Upon receiving the notification of the minutes creation, the first response generation unit 1867 creates the response content: "Starts voice recognition of the minutes." The display control unit 184 displays the created response content: "Start voice recognition of the proceedings" on the display 141.

The voice recognition unit 1863 recognizes the voice output from the communication control unit 182 and converts it into text data. The second intention interpretation unit 1864 interprets the intention included in the text data output from the voice recognition unit 1863. Specifically, the second intention interpretation unit 1864 uses, for example, the first trained model, and outputs the intent type, the slot type, and the slot value based on the input text data.

When the dialogue control unit 1865 receives the intention interpreted by the second intention interpretation unit 1864 (step S23), the dialogue control unit 1865 determines whether or not the received intention is used for creating the minutes (step S24). If it is determined that it is necessary to create the minutes (Yes in step S24), the process is shifted to step S25. When it is determined that it is not necessary to create the minutes (No in step S24), the process is shifted to step S23, and the second intention interpretation unit 1864 waits for the next intention to be output.

In step S25, the dialogue control unit 1865 outputs the intention output from the second intention interpretation unit 1864 to the first response generation unit 1867.

When the first response generation unit 1867 receives the intention analyzed by the second intention interpretation unit 1864 from the dialogue control unit 1865, the first response generation unit 1867 prepares the minutes based on the received intention. The display control unit 184 displays the created minutes on the display 141. The control unit 180 stores the created minutes in the minutes data 175.

The dialogue control unit 1865 repeats the processes of steps S23 to S25 until the user 1 inputs an instruction to stop the creation of the minutes.

FIG. 21 is a flowchart showing an example of the operation of the dialogue control unit 1865 when the auxiliary control unit 186 creates a translated sentence.

First, the user 1 who operates the terminal device 10A-1 operates the terminal device 10A-1 when creating a translated sentence, and inputs, for example, "create a translated sentence".

The text input unit 1861 accepts the "create translation" input from the user 1. The first intention interpretation unit 1862 interprets the intention included in the text data input from the text input unit 1861. Specifically, the first intention interpretation unit 1862 uses, for example, the first trained model, and based on the input text data: "create a translated sentence", the intent type: "translated sentence", the slot Type: "Process", Slot value: "Start" is output.

The dialogue control unit 1865 determines whether or not there is an instruction to create a translated sentence from the first intention interpretation unit 1862 (step S26). Since the analysis result output from the first intention interpretation unit 1862 is the intent type: "translation", the slot type: "processing", and the slot value: "start", the dialogue control unit 1865 translates. It is determined that there is an instruction to create a sentence (Yes in step S26).

When the dialogue control unit 1865 determines that there is an instruction to create a translated sentence, the dialogue control unit 1865 starts creating the translated sentence (step S27). When the dialogue control unit 1865 starts the creation of the translated sentence, the dialogue control unit 1867 notifies the first response generation unit 1867 that the creation of the translated sentence has started.

Upon receiving the notification of the translation sentence creation, the first response generation unit 1867 creates the response content: "Translation starts." The display control unit 184 displays the created response content: "Translation starts." On the display 141.

The voice recognition unit 1863 recognizes the voice output from the communication control unit 182 and converts it into text data. The second intention interpretation unit 1864 interprets the intention included in the text data output from the voice recognition unit 1863. Specifically, the second intention interpretation unit 1864 uses, for example, the first trained model, and outputs the intent type, the slot type, and the slot value based on the input text data.

When the dialogue control unit 1865 receives the intention interpreted by the second intention interpretation unit 1864 (step S28), the dialogue control unit 1865 outputs the received intention to the first response generation unit 1867 (step S29).

When the first response generation unit 1867 receives the intention analyzed by the second intention interpretation unit 1864 from the dialogue control unit 1865, the first response generation unit 1867 creates a translated sentence based on the received intention. The display control unit 184 displays the created translated text on the display 141.

The dialogue control unit 1865 repeats the processes of steps S28 and S29 until the user 1 inputs an instruction to stop the creation of the translated text.

FIG. 22 is a flowchart showing an example of the operation of the dialogue control unit 1865 when the auxiliary control unit 186 is operated in the proxy mode.

First, when the user 1 who operates the terminal device 10A-1 operates the auxiliary control unit 186 in the proxy mode, he / she operates the terminal device 10A-1 and inputs, for example, "operates as a proxy mode".

The text input unit 1861 accepts the "operate as a proxy mode" input from the user 1. The first intention interpretation unit 1862 interprets the intention included in the text data input from the text input unit 1861. Specifically, the first intention interpretation unit 1862 uses, for example, the first trained model, and based on the input text data: "operates as a surrogate mode", the intent type: "operation mode setting", slot. Type: "Operation mode", Slot value: "Substitute" is output.

The dialogue control unit 1865 determines whether or not there is an instruction to switch to the substitute mode from the first intention interpretation unit 1862 (step S210). Since the analysis result output from the first intention interpretation unit 1862 is the intent type: "operation mode setting", the slot type: "operation mode", and the slot value: "surrogate", the dialogue control unit 1865 , It is determined that there is an instruction to switch to the substitute mode (Yes in step S210).

When the dialogue control unit 1865 determines that there is an instruction to switch to the proxy mode, the dialogue control unit 1865 transitions to the proxy mode (step S211). When the dialogue control unit 1865 transitions to the surrogate mode, the dialogue control unit 1867 notifies the first response generation unit 1867 that the transition to the surrogate mode has occurred.

Upon receiving the notification of the transition to the surrogate mode, the first response generation unit 1867 creates the response content: "The surrogate mode has been set." The display control unit 184 displays the created response content: "The proxy mode has been set." On the display 141.

The voice recognition unit 1863 recognizes the voice output from the communication control unit 182 and converts it into text data. For example, the voice recognition unit 1863 converts "User 1, where is the previous minutes?", Which is input from the user 2 to the user 1, into text data. The second intention interpretation unit 1864 interprets the intention included in the text data output from the voice recognition unit 1863. Specifically, the second intention interpretation unit 1864 uses, for example, the first trained model, and the input text data: "User 1, where is the previous minutes?" Type: "Question to user 1", Slot type: "Speaker", Slot value: "User 2", Slot type: "Utterance content", Slot value: "User 1, previous minutes" Where is it? ”Is output.

When the dialogue control unit 1865 receives the intention interpreted by the second intention interpretation unit 1864 (step S212), the dialogue control unit 1865 determines whether or not it is necessary to transmit a message via the communication control unit 182 (step S213). The dialogue control unit 1865 determines that it is not necessary to transmit a message via the communication control unit 182 based on the intention interpreted by the second intention interpretation unit 1864 (No in step S213), and the response from the user 1 is received. It is determined whether or not it is required (step S214).

When it is determined that the response from the user 1 is requested based on the intention interpreted by the second intention interpretation unit 1864 (Yes in step S214), the dialogue control unit 1865 shifts the process to step S215. When it is determined that the response from the user 1 is unnecessary (No in step S214), the dialogue control unit 1865 shifts the process to step S212 and waits for the next intention to be output from the second intention interpretation unit 1864.

In step S215, the dialogue control unit 1865 has an intention output from the second intention interpretation unit 1864, an intent type: "question to user 1", a slot type: "speaker", and a slot value: "user". 2 ”, slot type:“ utterance content ”, slot value:“ User 1, where is the previous meeting minutes? ”Is output to the second response generation unit 1868.

When the second response generation unit 1868 receives the intention analyzed by the second intention interpretation unit 1864 from the dialogue control unit 1865, the response content: "User 1 is absent now, so a reply will be given later." I will. " The voice synthesis unit 1869 generates voice based on the response content. The auxiliary control unit 186 transmits the generated voice data to another terminal device 10 via the communication control unit 182.

In addition, the text input unit 1861 accepts the input from the user 1 "tell everyone" to leave the meeting in 5 minutes "". The first intention interpretation unit 1862 interprets the intention included in the text data input from the text input unit 1861. Specifically, the first intention interpretation unit 1862 uses, for example, the first trained model, and inputs based on the text data: "Tell everyone" I will leave the meeting in 5 minutes "". Outputs tent type: "Utterance request", slot type: "Utterance target", slot value: "Everyone", slot type: "Utterance content", slot value: "Leave the meeting after 5 minutes" do.

When the dialogue control unit 1865 receives the intention interpreted by the first intention interpretation unit 1862 (step S212), the dialogue control unit 1865 determines whether or not it is necessary to transmit a message via the communication control unit 182 (step S213). The dialogue control unit 1865 determines that it is necessary to transmit a message via the communication control unit 182 based on the intention interpreted by the first intention interpretation unit 1862 (Yes in step S213), and shifts the process to step S215. Let me.

In step S215, the dialogue control unit 1865 has an intention output from the first intention interpretation unit 1862, an intent type: "utterance request", a slot type: "utterance target", a slot value: "everyone", and a slot. Type: "Utterance content", slot value: "Leave the meeting after 5 minutes" is output to the second response generation unit 1868.

When the second response generation unit 1868 receives the intention analyzed by the first intention interpretation unit 1862 from the dialogue control unit 1865, the response content: "Everyone, you will leave the meeting in 5 minutes. . "Is created. The voice synthesis unit 1869 generates voice based on the response content. The voice synthesis unit 1869 may switch the voice to be generated in response to an instruction from the user. For example, the voice synthesis unit 1869 may be the voice of the assistant or the voice of the user himself according to the instruction from the user. The auxiliary control unit 186 transmits the generated voice data to another terminal device 10 via the communication control unit 182.

The dialogue control unit 1865 repeats the processes of steps S212 to S215 until the user 1 inputs an instruction to stop the proxy mode. When, for example, "cancellation of proxy mode" is input from user 1, "cancellation of proxy mode" is displayed on the display 141, and the proxy mode is canceled.

FIG. 23 is a flowchart showing an example of the operation of the dialogue control unit 1865 when the auxiliary control unit 186 creates advice for the user 1.

First, the user 1 who operates the terminal device 10A-1 operates the terminal device 10A-1 when requesting advice from the auxiliary control unit 186, for example, "Check the bus leaving Toranomon to Shibuya after 10 minutes. ".

The text input unit 1861 accepts the input from the user 1 "Check the bus leaving Toranomon to Shibuya in 10 minutes." The first intention interpretation unit 1862 interprets the intention included in the text data input from the text input unit 1861. Specifically, the first intention interpretation unit 1862 uses, for example, the first trained model, and based on the input text data: "Check the bus leaving Toranomon to Shibuya in 10 minutes." Type: "Move", Slot type: "Departure", Slot value: "Toranomon", Slot type: "Destination", Slot value: "Shibuya", Slot type: "Date", Slot Value: "10 minutes later", slot type: "means", slot value: "bus" is output.

When the dialogue control unit 1865 receives the intention interpreted by the first intention interpretation unit 1862 (step S216), the dialogue control unit 1865 determines whether or not to give advice to the user 1 based on the result of analyzing the text data (step S217). ). The dialogue control unit 1865 has an intent type: "move", a slot type: "starting point", a slot value: "Toranomon", a slot type: "destination", a slot value: "Shibuya", and a slot. Type: "Date and time", Slot value: "10 minutes later", Slot type: "Means", Slot value: "Bus", so it is determined that advice is required (Yes in step S217), and processing is performed. To step S218. When it is determined that the advice is unnecessary (No in step S217), the dialogue control unit 1865 shifts the process to step S216 and waits for the next intention to be output.

In step S218, the dialogue control unit 1865 determines whether or not to acquire the information necessary for giving advice from the external service. The dialogue control unit 1865 has an intent type: "move", a slot type: "starting point", a slot value: "Toranomon", a slot type: "destination", a slot value: "Shibuya", and a slot. Type: "Date and time", Slot value: "10 minutes later", Slot type: "Means", Slot value: "Bus", so if you get the information necessary to give advice from an external service The determination is made (Yes in step S218), and the process is shifted to step S219.

In step S219, the dialogue control unit 1865 acquires the information required for advice from the external service by the external service calling unit 1866. Specifically, for example, the external service calling unit 1866 can input "departure place = Toranomon", "destination = Shibuya", "date and time = 10 minutes later", and "means = bus" as keywords. Search the external service for information to show the bus leaving Toranomon to Shibuya in a minute. The dialogue control unit 1865 outputs the acquired information to the first response generation unit 1867.

When it is determined in step S218 that the information necessary for giving advice is not acquired from the external service (No in step S218), the dialogue control unit 1865 determines that the information is acquired from the storage unit 170A by the first response generation unit 1867. (Step S220).

In step S221, the dialogue control unit 1865 outputs the intention output from the first intention interpretation unit 1862 to the first response generation unit 1867.

When the first response generation unit 1867 receives the intention analyzed by the first intention interpretation unit 1862 from the dialogue control unit 1865, the received intention, the information acquired from the external service, or the information read from the storage unit 170A Based on the advice content: "Depart from Roppongi Dori at 12:15." The display control unit 184 displays the created advice content on the display 141.

FIG. 24 is a schematic diagram showing a display example of the display 141 of the terminal device when the auxiliary control unit 186 is operating. In the example shown in FIG. 24, for example, a display example when a plurality of third applications 174 are operating is shown. In FIG. 24, display objects 1417 to 1419 are displayed on the display 141. The display object 1417 represents a display for the conference 1. The user 1 participates in the conference 1 in the proxy mode executed by the auxiliary control unit 186. On the display object 1417, for example, a robot icon indicating that the display object 1417 is operating as a surrogate mode is displayed. Further, the display object 1417 displays a summary sentence of the conference 1 participating in the proxy mode. The robot icon can be arbitrarily changed by the operation of user 1. It may be displayed as text instead of an icon.

The display objects 1418 and 1419 represent a display for the conference 2. User 1 participates in the conference 2 in a normal state. The user 1 is set to the conference 2 so that the auxiliary control unit 186 creates a translated sentence. On the display object 1418, an icon indicating that the auxiliary control unit 186 is creating the translated text is displayed. Further, the translated text created by the auxiliary control unit 186 is displayed on the display object 1418. An image of a participant participating in the conference 2 is displayed on the display object 1419.

Further, in FIG. 24, the area 14110 is displayed on the display 141. The display control unit 184 displays the instruction input from the user 1 on the right side, and displays the response from the auxiliary control unit 186 to the input instruction on the left side. In the example shown in FIG. 24, the auxiliary control unit 186 departs from Roppongi-dori at 12:15 in response to the instruction "Check the bus leaving Toranomon to Shibuya after 10 minutes" input from user 1. Is returning the response.

The example shown in FIG. 24 shows a case where the auxiliary control unit 186 generates a translated sentence for the conference 2. It is also possible to generate the translated text for the conference 1 participating in the proxy mode. That is, a plurality of processes may be performed in parallel for the same conference.

FIG. 25 is a schematic diagram showing a display example of the display 141 of the terminal device when any of the conference participants is using the proxy mode. The example shown in FIG. 25A shows a display example of the display 141 of the terminal device 10-2 operated by the user 2. In the example shown in FIG. 25 (a), the case where the user 1 is participating in the conference in the proxy mode is shown. In FIG. 25A, the image of the user 1 is represented by, for example, a robot icon indicating that the user 1 is operating as a surrogate mode. The icon of the user 1 shown in FIG. 25A may be switched by the operation of the user 1. For example, instead of the robot icon, it may be displayed by text indicating the current situation, such as when the assistant is being handled or the person is away from the desk.

In the proxy mode, there may be a function of providing a specific service. For example, in the surrogate mode, there may be a translation function. Specifically, it is assumed that the user 3 is the machine translation service provider of the conference. In this case, for example, the user 3 sets his / her icon as the interpreter's icon from the beginning of the conference participation. Then, other conference members can recognize that the machine translation service has been introduced and the conference is being translated. As a further example, there may be a minutes creation function in the proxy mode. Specifically, it is assumed that the user 4 is the minutes creation service provider of the meeting. In this case, for example, the user 4 sets his / her icon to an icon that can be recognized as a shorthand writer. Then, other conference members can recognize that the minutes preparation service has been introduced and the minutes of the conference have been created. In this way, the visibility of the service, which is different from that of leaving the seat and sitting in the middle, can be recognized by the anthropomorphic metaphor of the participants. The example shown in FIG. 25B shows a display example of the display 141 of the terminal device 10-2 when the user 3 is the machine translation service provider and the user 4 is the minutes creation service provider.

As described above, in the second embodiment, the auxiliary control unit 186 recognizes the message input from the second party in the remote communication between the first party and the second party. Auxiliary control unit 186 analyzes the recognized message. The auxiliary control unit 186 generates information about the content of the message for the first party based on the result of the analysis. The auxiliary control unit 186 generates a response content to the message to the second party based on the result of the analysis. The auxiliary control unit 186 presents information regarding the content of the message to the first party instead of the message input from the second party. Then, the auxiliary control unit 186 presents the content of the response to the message to a second party. As a result, the auxiliary control unit 186 can present information on the content of the session to the first party while making a proxy response to the session partner. Therefore, the auxiliary control unit 186 can reduce the burden on the first party for remote communication.

Therefore, according to the information processing apparatus 10A according to the second embodiment, it is possible to improve the convenience of the user in remote communication.

In the second embodiment, an operation example of the virtual assistant (auxiliary control unit 186) has been described. The operation of the auxiliary control unit 186 according to the second embodiment is not limited to the above.

The auxiliary control unit 186 expands the range of interpretable intentions by registering a new intent. Therefore, the auxiliary control unit 186 can execute various operations by registering the intent corresponding to the desired operation. Further, the auxiliary control unit 186 may realize different functions by executing the third application 174 in which different intents are registered by the processor 19. Therefore, by downloading the third application 174 corresponding to the purpose of implementation, it is possible to use the auxiliary control unit 186 according to the application.

For example, auxiliary control unit 186 may generate a security question to access the session.

Specifically, for example, the auxiliary control unit 186 operates as follows. For example, it is assumed that a session is executed by user 1 and user 2. At this time, the auxiliary control unit 186 stores the history of the session contents executed by the user 1 and the user 2 in the storage unit 170A.

Subsequently, it is assumed that a session with the content related to the previous time is set again between the user 1 and the user 2. User 1 provides a place for a session with user 2 and accepts access from user 2.

When the auxiliary control unit 186 receives an access from the user 2, the auxiliary control unit 186 reads the history of the session contents stored in the storage unit 170A and generates a question based on the history. For example, the auxiliary control unit 186 generates a question based on the content talked with the user 1, information about the action point, and the like. Further, the auxiliary control unit 186 may generate a topic that is not included in the history as a question. The auxiliary control unit 186 generates, for example, a plurality of questions.

The auxiliary control unit 186 presents the generated question to the user 2, and receives the answer to the question from the user 2. If the user 2 can answer the question correctly, the auxiliary control unit 186 permits the user 2 to connect. The question generated by the auxiliary control unit 186 may be presented to the user 1. If the user 1 can answer the question correctly, the auxiliary control unit 186 permits the user 1 to connect.

In this way, the auxiliary control unit 186 generates a security question to prove the legitimacy of the user who has accessed the session. Also, since the content of the question is based on the information from the previous session, the user does not need to expose personal activities that touch his or her privacy. Further, since it is based on the content of the session, even a third party who lives with the user cannot participate in the session by pretending to be the user.

Auxiliary control unit 186, which generates security questions, can be used, for example, to prove the legitimacy of conference participants. For example, suppose a remote conference is held between a host user and another user. At this time, the auxiliary control unit 186 stores the minutes of the remote conference in the storage unit 170A.

Then, it is assumed that a remote conference with the same members as the previous one and the contents related to the previous one is scheduled again. The host user sets up a place for a remote conference and accepts access from other users.

When the auxiliary control unit 186 is accessed by another user, the auxiliary control unit 186 reads the minutes stored in the storage unit 170A and generates a question based on the minutes. For example, the auxiliary control unit 186 generates a question based on the name of the previous participant, the name of the person who was scheduled to attend but was absent, information on the action point, and the like. In addition, the auxiliary control unit 186 may generate a question about irregular remarks during the meeting or the behavior of the participants, which is not stored in the minutes.

The auxiliary control unit 186 presents the generated question to another user, and receives an answer to the question from the other user. If other users can answer the question correctly, allow the user who answered correctly to connect. The auxiliary control unit 186 may present the generated question to the host user and accept the answer to the question from the host user. If the host user answers the question correctly, allow the host user to connect.

In addition, the auxiliary control unit 186 may confirm the validity of the minutes together with the generation of the security question. Confirming the validity of the minutes can be rephrased as, for example, improving the accuracy of the contents of the minutes or verifying the minutes.

Specifically, for example, the auxiliary control unit 186 identifies a highly legitimate user among the users who answered the security question. The auxiliary control unit 186 generates, for example, a question to the specified user based on the content different from the minutes but similar. Auxiliary control unit 186 corrects the minutes based on the answer from the user to the question. Also, the auxiliary control unit 186 generates, for example, at least a part of the summary of the previous meeting as a question. Auxiliary control unit 186 corrects the minutes based on the answer from the user to the question.

As a result, the auxiliary control unit 186 can correct the minutes at the same time as confirming the legitimacy of the session participants.

Further, for example, the auxiliary control unit 186 may generate a proposal content based on the content of the message. The content of the proposal is, for example, a recommended product based on the content of the message.

Further, in the second embodiment, when switching the operation of the auxiliary control unit 186, an example of inputting an instruction as a sentence clearly indicating the function to be executed has been described. However, the instruction to the auxiliary control unit 186 does not have to be a sentence. The instruction to the auxiliary control unit 186 may be set with a special cant, or a keyword for reading a predetermined function of the auxiliary control unit 186 may be set in advance.

(Other embodiments)
In the first embodiment, the case where the control unit 180 has the advisory control unit 183 and the control unit 180A has the auxiliary control unit 186 has been described. The advisory control unit 183 and the auxiliary control unit 186 may be realized at the same time or by switching. FIG. 26 is a block diagram showing a configuration of a terminal device 10B included in the system of another embodiment. FIG. 27 is a block diagram showing an example of the functional configuration of the system when remote communication is performed.

The communication control unit 182 shown in FIG. 27 has a mute function. The advisory control unit 183 and the auxiliary control unit 186 also have a mute function so that the voice from the advisory control unit 183 and the auxiliary control unit 186 is not transmitted to other users when the mute function of the communication control unit 182 is turned on. ing. The mute function of the advisory control unit 183 and the auxiliary control unit 186 may be automatically turned on when the mute function of the communication control unit 182 is turned on.

In the above embodiment, the first application 171 and the second application 172 and the third application 174 are described as separate applications, but as a function of the first application 171, the second application 172 and the third application 174 are described. It may have a function.

Further, in the first embodiment, the case where the terminal device 10 executes the function of the advisory control unit 183 will be described, and in the second embodiment, the case where the terminal device 10A executes the function of the auxiliary control unit 186 will be described. bottom. However, the functions of the advisory control unit 183 and the auxiliary control unit 186 may be executed by the server 20.

FIG. 28 is a block diagram showing a functional configuration of the server 20A when the advisory control unit is executed on the server. As shown in FIG. 28, the server 20A functions as a communication unit 201, a storage unit 202, and a control unit 203.

The communication unit 201 performs a process for the server 20A to communicate with an external device.

The storage unit 202 stores data and programs used by the server 20A. For example, the storage unit 202 stores the advice control program 2021 and the dialogue history data 2022. The advice control program 2021 is a program for executing a process of displaying advice on the terminal device 10. The advisory control program 2021 corresponds to the second application 172 in the first embodiment. The dialogue history data 2022 stores the dialogue history in remote communication. The advisory control program 2021 may be a part of a communication control program (not shown) stored in the storage unit 202.

The control unit 203 is realized by the processor 29 reading the program stored in the storage unit 202 and executing the instruction included in the program. The control unit 203 controls the operation of the server 20A. Specifically, for example, the control unit 203 functions as a transmission / reception unit 2031, a storage control unit 2032, an advisory control unit 2033, and a presentation unit 2034.

The transmission / reception unit 2031 controls a process in which the server 20A transmits / receives data to / from an external device such as a terminal device 10 according to a communication protocol.

The storage control unit 2032 controls the storage of information in the storage unit 202.

The advice control unit 2033 analyzes the message included in the communication session and performs a process of giving advice based on the analysis result to the user. The advice control unit 2033, for example, generates advice for a plurality of users participating in a communication session. The intent set in the advisory control unit 2033 may be set for each user. The operation of the advisory control unit 2033 is, for example, the same as the operation of the advisory control unit 183 described in the first embodiment.

The presentation unit 2034 presents an image related to remote communication to the user. Further, the presentation unit 2034 presents the advice generated by the advice control unit 2033 to the user.

FIG. 29 is a block diagram showing an example of the functional configuration of the system when remote communication is performed. FIG. 29 shows a case where a remote conference is held between the terminal devices 10-1 to 10-4, and the advice control unit 2033 as a virtual assistant is started on the server 20A.

FIG. 30 is a block diagram showing a functional configuration of the server 20B when the auxiliary control unit is executed by the server. As shown in FIG. 30, the server 20B functions as a communication unit 201, a storage unit 202B, and a control unit 203B.

The storage unit 202B stores data and programs used by the server 20B. For example, the storage unit 202B stores the auxiliary control program 2023, the dialogue history data 2022, and the minutes data 2024. The auxiliary control program 2023 is a program for executing a process of assisting the terminal device 10. The auxiliary control program 2023 corresponds to the third application 174 in the second embodiment. Minutes data 2024 stores the minutes in remote communication. The auxiliary control program 2023 may be a part of a communication control program (not shown) stored in the storage unit 202B.

The control unit 203B is realized by the processor 29 reading the program stored in the storage unit 202B and executing the instruction included in the program. The control unit 203B controls the operation of the server 20B. Specifically, for example, the control unit 203B exerts functions as a transmission / reception unit 2031, a storage control unit 2032, an auxiliary control unit 2035, and a presentation unit 2034.

The auxiliary control unit 2035 analyzes the message included in the communication session and performs a process of assisting the user based on the analysis result. The auxiliary control unit 2035 assists, for example, a plurality of users who participate in a communication session. The intent set in the auxiliary control unit 2035 may be set for each user. The operation of the auxiliary control unit 2035 is, for example, the same as the operation of the auxiliary control unit 186 described in the second embodiment.

The presentation unit 2034 presents an image related to remote communication to the user. Further, the presentation unit 2034 presents the assistance provided by the auxiliary control unit 2035 to the user.

FIG. 31 is a block diagram showing an example of the functional configuration of the system when remote communication is performed. FIG. 31 shows a case where a remote conference is held between the terminal devices 10-1 to 10-4, and the auxiliary control unit 2035 as a virtual assistant is started on the server 20B.

The advisory control unit 2033 and the auxiliary control unit 2035 shown in FIGS. 28 to 31 may be realized at the same time or by switching. FIG. 32 is a block diagram showing a functional configuration of the server 20C when the advisory control unit 2033 and the auxiliary control unit 2035 are executed by the server. FIG. 33 is a block diagram showing an example of the functional configuration of the system when remote communication is performed.

It is not necessary that both the advisory control unit 2033 and the auxiliary control unit 2035 are executed by the server 20C. The advisory control unit 2033 or the auxiliary control unit 2035 may be executed by the terminal device 10 as the advisory control unit 183 or the auxiliary control unit 186. Specifically, the advisory control unit 2033 may be executed by the server 20C, and the auxiliary control unit 186 may be executed by the terminal device 10. Further, the advisory control unit 183 may be executed by the terminal device 10, and the auxiliary control unit 2035 may be executed by the server 20C.

Although some embodiments of the present disclosure have been described above, these embodiments can be implemented in various other embodiments, and various omissions, replacements, and modifications are made without departing from the gist of the invention. It can be performed. These embodiments and variations thereof shall be included in the scope of the invention described in the claims and the equivalent scope thereof, as well as in the scope and gist of the invention.

<Additional Notes>
The matters described in each of the above embodiments are added below.

(Appendix 1)
A program for causing a computer including a processor 19 and a memory 15 to be executed, and the program is input to the processor from a first party or a second party in remote communication between the first party and the second party. About the step of recognizing the message (step S11), the step of analyzing the recognized message (steps S12, 16), the message from the first party to the second party, or the message from the second party to the first party. Steps to determine whether or not support for the first party is necessary based on the results of the analysis (steps S13, 17), and if it is determined to be necessary, advice to give advice to the first party based on the results of the analysis. A program that executes a step (step S110) for generating contents.

(Appendix 2)
In the step of analyzing the message, in the step of interpreting the intention of the message issued by the first party or the second party, and in the step of determining the necessity of support, the first party based on the interpreted intention and the history of the message. The program described in (Appendix 1) that generates advice content based on the intent interpreted and the history of the message in the step of determining whether or not support is required and generating advice content.

(Appendix 3)
In the step of analyzing a message, the type of intent that can be interpreted can be registered in advance (Appendix 2).

(Appendix 4)
In the step of analyzing the message, the proper noun contained in the message issued from the first person or the second person is extracted, and in the step of determining the necessity of support, the first person is referred to based on the extracted proper noun. The program according to any one of (Appendix 1) to (Appendix 3), which generates advice content based on the extracted proper noun in the step of determining whether or not support is required and generating advice content.

(Appendix 5)
The program according to any one of (Appendix 1) to (Appendix 4) for generating an advice content for explaining an esoteric proper noun based on the result of analysis in the step of generating the advice content.

(Appendix 6)
The program according to any one of (Appendix 1) to (Appendix 4) for generating advice content regarding a schedule based on the result of analysis in the step of generating advice content.

(Appendix 7)
The program according to any one of (Appendix 1) to (Appendix 4) that generates advice content that assists the progress of the meeting based on the result of analysis in the step of generating advice content.

(Appendix 8)
In the step of causing the processor to execute the step (1837) of stopping the output of the message from the first party to the second party and generating the advice content, the output of the message from the first party to the second party is stopped. The program according to any one of (Appendix 1) to (Appendix 4) that generates advice content based on a message input from a first party in between.

(Appendix 9)
While the output of the message from the first party to the second party is stopped, the step of accepting the designation of one of the plurality of second parties and the step of transmitting the advice content to the designated second party. The program described in (Appendix 8).

(Appendix 10)
A program for causing a computer including a processor 19 and a memory 15 to execute, the program has a step (184) of displaying an image of a second party participating in remote communication with the first party on the processor. A program for executing a step (184) of displaying an advice content that gives advice to a first party, which is generated by analyzing a message from the second party, in association with an image of the second party.

(Appendix 11)
In the step of displaying the image of the second party, the image of the second party participating in the voice conversation with the first party is displayed, and in the step of displaying the advice content, the voice input from the second party is analyzed. The program according to (Appendix 10), which displays the generated advice content in association with the image of a second party.

(Appendix 12)
The program according to (Appendix 10) or (Appendix 11), which causes a processor to execute a step of displaying an image in which the attributes of the advice content can be identified.

(Appendix 13)
In the step of causing the processor to execute the step of displaying the icon (1413) for stopping the output of the message from the first party to the second party and displaying the advice content, the message from the first party to the second party is displayed. The program according to any one of (Appendix 10) to (Appendix 12), which displays the advice content generated by analyzing a message input from a first party while the output is stopped.

(Appendix 14)
A method executed by a computer including the processor 19 and the memory 15, in which a message input from the first party or the second party is recognized in a remote communication between the first party and the second party (step S11). ), The step of analyzing the recognized message (steps S12, 16), and the first based on the result of the analysis of the message from the first party to the second party or the message from the second party to the first party. A step of determining whether or not support for one person is necessary (steps S13 and 17), and a step of generating advice content that gives advice to the first person based on the result of analysis when it is determined to be necessary (step S110). ) And how to do it.

(Appendix 15)
An information processing device 10 including a processor 19 and a memory 15, wherein the processor recognizes a message input from a first party or a second party in remote communication between the first party and the second party. Do you need support for the first party based on the steps to analyze the recognized message and the result of the analysis of the message from the first party to the second party or the message from the second party to the first party? An information processing unit that executes a step of determining whether or not it is necessary, and a step of generating advice content that gives advice to a first party based on the result of analysis when it is determined to be necessary.

(Appendix 16)
In the remote communication between the first party and the second party, means for recognizing a message input from the first party or the second party (1831), means for analyzing the recognized message (1832, 1833), and A means (1834) for determining whether or not support for the first party is necessary based on the result of analysis of the message from the first party to the second party or the message from the second party to the first party. A system including means (1836) for generating advice content that gives advice to a first party based on the result of analysis when it is determined to be necessary.

(Appendix 17)
A program for causing a computer including a processor 19 and a memory 15 to execute, the program recognizes a message input from a second party in a remote communication between the first party and the second party. Based on the results of the analysis, step (1863), the step of analyzing the recognized message (1864), the step of generating information about the content of the message to the first party based on the result of the analysis (1867), and the step (1867). , A step of generating a response content to a message to a second party (1868), and a step of presenting information about the content of the message to the first party instead of a message input from the second party (184). A program that executes a step (182) of presenting a response content to a message to a second party.

(Appendix 18)
The program according to (Appendix 17), which generates a sentence summarizing the contents of a message based on the result of analysis in a step of generating information regarding the contents of the message.

(Appendix 19)
The program according to (Appendix 17), which generates a translated text of a message based on the result of analysis in a step of generating information regarding the content of the message.

(Appendix 20)
Have the processor perform a step in which the first party and the second party generate a question to confirm the validity of the first party or the second party based on the record of the remote communication previously participated (Appendix 17). The program described in any of (Appendix 19).

(Appendix 21)
The program according to (Appendix 20), which causes a processor to perform a step of correcting a remote communication record by using a first-party or second-party answer to a question.

(Appendix 22)
The program according to (Appendix 18) or (Appendix 19), which performs processing according to an instruction from a first party in a step of generating information regarding the content of a message.

(Appendix 23)
The first is based on the step of recognizing the message input from the first party (step S11) and the result of the analysis of the message from the first party to the second party or the message from the second party to the first party. A step of determining whether or not support for a person is necessary (steps S13 and 17), and a step of generating advice content that gives advice to a first person based on the result of analysis if it is determined to be necessary (step S110). The program according to any one of (Appendix 17) to (Appendix 22), which causes the processor to execute the step (184) of presenting the advice content to the first party.

(Appendix 24)
A program for causing a computer having a processor 19 and a memory 15 to execute the program, in which a first party and a second party participate in remote communication in the processor, and the first party receives a message from the second party. When inputting a message for the message of, based on the step of displaying the image of the second party and the message from the second party, the response is automatically generated without the input from the first party. In this case, a program that executes a step of displaying an image of a second party as a proxy image indicating that it is a proxy.

(Appendix 25)
The program according to (Appendix 24), which causes a processor to execute a step of displaying information about the content of a message, which is generated by analyzing a message from a second party, in association with a proxy image.

(Appendix 26)
The program according to (Appendix 24) or (Appendix 25), which causes a processor to execute a step of displaying an image indicating that a process of generating information regarding the content of the message is being performed based on a message from a second party.

(Appendix 27)
Have the processor execute a step of displaying the advice content that gives advice to the first party, which is generated by analyzing the message from the second party, in association with the image of the second party (Appendix 24) to (Appendix 26). The program described in any of.

(Appendix 28)
A method executed by a computer including a processor 19 and a memory 15, which is recognized as a step (1863) of recognizing a message input from a second party in remote communication between the first party and the second party. A step of analyzing the message (1864), a step of generating information about the content of the message to the first party based on the result of the analysis (1867), and a second step of responding to the message based on the result of the analysis. A step of generating a person (1868), a step of presenting information about the content of the message to the first party instead of a message input from the second party (184), and a second party responding to the message. And how to perform the steps (182) presented to.

(Appendix 29)
In the information processing apparatus 10A including the processor 19 and the memory 15, the step (1863) in which the processor recognizes a message input from the second party in the remote communication between the first party and the second party. A step of analyzing the recognized message (1864), a step of generating information about the content of the message to the first party based on the result of the analysis (1867), and a response to the message based on the result of the analysis. The step of generating the content to the second party (1868), the step of presenting the information about the content of the message to the first party instead of the message input from the second party (184), and the response content to the message. An information processing device that executes the step (182) of presenting the above to a second party.

(Appendix 30)
In the remote communication between the first party and the second party, the means for recognizing the message input from the second party (1863), the means for analyzing the recognized message (1864), and the message based on the analysis result. The means for generating information about the contents of the above (1867) and the means for generating the response contents to the message for the second party based on the result of the analysis (1868) are input from the second party. A system including means for presenting information about the content of a message to a first party (184) and means for presenting a response to the message to a second party (182) instead of the message.

1 ... System 10, 10A, 10B ... Terminal device 12 ... Communication IF
121 ... Communication unit 13 ... Input device 131 ... Mouse 132 ... Keyboard 133 ... Touch sensitive device 134 ... Microphone 14 ... Output device 141 ... Display 142 ... Speaker 15 ... Memory 16 ... Storage 160 ... Camera 170, 170A ... Storage unit 171 ... 1st application 172 ... 2nd application 173 ... Dialogue history data 174 ... 3rd application 175 ... Minute data 180, 180A ... Control unit 181 ... Operation reception unit 182 ... Communication control unit 183 ... Advice control unit 1831 ... Voice recognition unit 1832 ... Intention interpretation unit 1833 ... Specific nose extraction unit 1834 ... Advice determination unit 1835 ... External service calling unit 1836 ... Response generation unit 1837 ... Second switching unit 1838 ... Voice synthesis unit 184 ... Display control unit 185 ... First switching unit 186 Auxiliary control unit 1861 ... Text input unit 1862 ... First intention interpretation unit 1863 ... Voice recognition unit 1864 ... Second intention interpretation unit 1865 ... Dialogue control unit 1866 ... External service call unit 1867 ... First response generation unit 1868 ... Second Response generation unit 1869 ... Voice synthesis unit 19 ... Processor 20, 20A to 20C ... Server 201 ... Communication unit 202, 202B, 202C ... Storage unit 2021 ... Advice control program 2022 ... Dialogue history data 2023 ... Auxiliary control program 2024 ... Minute data 203, 203B, 203C ... Control unit 2031 ... Transmission / reception unit 2032 ... Storage control unit 2033 ... Advice control unit 2034 ... Presentation unit 2035 ... Auxiliary control unit 22 ... Communication IF
23 ... Input / output IF
25 ... Memory 26 ... Storage 29 ... Processor 80 ... Network

Claims (30)

A program for causing a computer having a processor and a memory to execute the program, wherein the program causes the processor to execute the program.
In the remote communication between the first party and the second party, the step of recognizing the message input from the first party or the second party, and
The steps to analyze the recognized message and
Based on the result of the analysis of the message from the first party to the second party or the message from the second party to the first party, it is determined whether or not support for the first party is necessary. Steps and
A program for executing a step of generating advice content that gives advice to the first party based on the result of the analysis when it is determined to be necessary. In the step of analyzing the message, the intent of the message issued by the first party or the second party is interpreted.
In the step of determining the necessity of the support, it is determined whether or not the support to the first party is necessary based on the interpreted intention and the history of the message.
The program according to claim 1, wherein in the step of generating the advice content, the advice content is generated based on the interpreted intention and the history of the message. The program according to claim 2, wherein in the step of analyzing the message, the type of intent that can be interpreted can be registered in advance. In the step of analyzing the message, the proper noun contained in the message issued from the first person or the second person is extracted.
In the step of determining the necessity of the support, it is determined whether or not the support to the first party is necessary based on the extracted proper noun.
The program according to any one of claims 1 to 3, which generates the advice content based on the extracted proper noun in the step of generating the advice content. The program according to any one of claims 1 to 4, wherein in the step of generating the advice content, the advice content for explaining the esoteric proper noun is generated based on the result of the analysis. The program according to any one of claims 1 to 4, wherein in the step of generating the advice content, the advice content regarding the schedule is generated based on the result of the analysis. The program according to any one of claims 1 to 4, wherein in the step of generating the advice content, the advice content that assists the progress of the meeting is generated based on the result of the analysis. The processor is made to execute a step of stopping the output of the message from the first party to the second party.
A claim to generate the advice content based on the message input from the first party while the output of the message from the first party to the second party is stopped in the step of generating the advice content. The program according to any one of Items 1 to 4. A step of accepting the designation of one of a plurality of existing second parties while the output of the message from the first party to the second party is stopped.
The program according to claim 8, wherein the processor executes the step of transmitting the advice content to the designated second party. A program for causing a computer having a processor and a memory to execute the program, wherein the program causes the processor to execute the program.
A step to display an image of a second party participating in remote communication with the first party,
A program for executing a step of displaying an advice content for giving advice to the first party, which is generated by analyzing a message from the second party, in association with an image of the second party. In the step of displaying the image of the second party, the image of the second party participating in the voice conversation with the first party is displayed.
The program according to claim 10, wherein in the step of displaying the advice content, the advice content generated by analyzing the voice input from the second party is displayed in association with the image of the second party. The program according to claim 10 or 11, wherein the processor performs a step of displaying an image in which the attribute of the advice content can be identified. The processor is made to perform a step of displaying an icon for stopping the output of the message from the first party to the second party.
In the step of displaying the advice content, the advice content generated by analyzing the message input from the first party while the output of the message from the first party to the second party is stopped. The program according to any one of claims 10 to 12 to be displayed. It's a way a computer with a processor and memory does it.
In the remote communication between the first party and the second party, the step of recognizing the message input from the first party or the second party, and
The steps to analyze the recognized message and
Based on the result of the analysis of the message from the first party to the second party or the message from the second party to the first party, it is determined whether or not support for the first party is necessary. Steps and
A method of performing a step of generating advice content that gives advice to the first party based on the result of the analysis when it is determined to be necessary. An information processing device equipped with a processor and a memory.
The processor
In the remote communication between the first party and the second party, the step of recognizing the message input from the first party or the second party, and
The steps to analyze the recognized message and
Based on the result of the analysis of the message from the first party to the second party or the message from the second party to the first party, it is determined whether or not support for the first party is necessary. Steps and
An information processing device that executes a step of generating advice content that gives advice to the first party based on the result of the analysis when it is determined to be necessary. A means for recognizing a message input from the first person or the second person in remote communication between the first person and the second person.
A means of analyzing the recognized message and
Based on the result of the analysis of the message from the first party to the second party or the message from the second party to the first party, it is determined whether or not support for the first party is necessary. Means and
A system provided with a means for generating advice content that gives advice to the first party based on the result of the analysis when it is determined to be necessary. A program for causing a computer having a processor and a memory to execute the program, wherein the program causes the processor to execute the program.
In the remote communication between the first party and the second party, the step of recognizing the message input from the second party and
The steps to analyze the recognized message and
A step of generating information about the content of the message for the first party based on the result of the analysis, and
Based on the result of the analysis, a step of generating a response content to the message to the second party, and
A step of presenting information about the content of the message to the first party instead of the message input from the second party.
A program for executing a step of presenting a response content to the message to the second party. The program according to claim 17, wherein in the step of generating information regarding the content of the message, a sentence summarizing the content of the message is generated based on the result of the analysis. 17. The program of claim 17, wherein in the step of generating information about the content of the message, a translation of the message is generated based on the result of the analysis. Having the processor perform a step of generating a question for the first party or the second party to confirm the validity of the first party or the second party based on the record of the remote communication in which the first party and the second party have previously participated. The program according to any one of claims 17 to 19. 20. The program of claim 20, wherein the processor is made to perform a step of correcting the record of the remote communication by using the answer of the first party or the second party to the question. The program according to claim 18 or 19, wherein in the step of generating information regarding the content of the message, processing according to an instruction from the first party is performed. The step of recognizing the message input from the first party,
Based on the result of the analysis of the message from the first party to the second party or the message from the second party to the first party, it is determined whether or not support for the first party is necessary. Steps and
If it is determined to be necessary, a step of generating advice content that gives advice to the first party based on the result of the analysis, and
The program according to any one of claims 17 to 22, which causes the processor to execute the step of presenting the advice content to the first party. A program for causing a computer having a processor and a memory to execute the program, wherein the program causes the processor to execute the program.
When the first party and the second party participate in the remote communication and the first party inputs a message for the message from the second party, the step of displaying the image of the second party and the step of displaying the image of the second party.
When a response is automatically generated based on the message from the second party without input from the first party, the image of the second party is displayed as a proxy image indicating that it is a proxy. A program that executes the steps to be performed. 24. The program of claim 24, wherein the processor performs a step of analyzing and generating a message from the second party to display information about the content of the message in association with the surrogate image. 24. The program of claim 24 or 25, wherein the processor performs a step of displaying an image indicating that a process of generating information about the content of the message is being performed based on a message from a second party. Claims 24 to 26, which cause the processor to perform a step of displaying an advice content for giving advice to the first party, which is generated by analyzing a message from the second party, in association with an image of the second party. The program described in any of. It's a way a computer with a processor and memory does it.
In the remote communication between the first party and the second party, the step of recognizing the message input from the second party and
The steps to analyze the recognized message and
A step of generating information about the content of the message for the first party based on the result of the analysis, and
Based on the result of the analysis, a step of generating a response content to the message to the second party, and
A step of presenting information about the content of the message to the first party instead of the message input from the second party.
A method of performing a step of presenting a response content to the message to the second party. An information processing device equipped with a processor and a memory.
The processor
In the remote communication between the first party and the second party, the step of recognizing the message input from the second party and
The steps to analyze the recognized message and
A step of generating information about the content of the message for the first party based on the result of the analysis, and
Based on the result of the analysis, a step of generating a response content to the message to the second party, and
A step of presenting information about the content of the message to the first party instead of the message input from the second party.
An information processing device that executes a step of presenting a response content to the message to the second party. A means for recognizing a message input from the second party in remote communication between the first party and the second party,
A means of analyzing the recognized message and
A means for generating information about the content of the message to the first party based on the result of the analysis, and
A means for generating a response content to the message to the second party based on the result of the analysis, and
A means of presenting information about the content of the message to the first party instead of the message input from the second party.
A system including means for presenting a response content to the message to the second party.

Images