JP2022056109A - Information processing device, information processing method, information processing program, and information processing system - Google Patents

Abstract

To provide an information processing device capable of supporting transfer of complicated operation contents in remote communication, an information processing method, an information processing program, and an information processing system.SOLUTION: An information processing device 100 comprises a generation unit 133 which generates auxiliary information transferred from a providing user, who is a participant into remote communication, to a listener, who is also a participant into remote communication, and for complementing operation contents to an operation object operated by the listener on the basis of information of the listener, and an output control unit 134 which outputs the auxiliary information generated by the generation unit 133 so that the auxiliary information is output to a terminal device of the listener.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing apparatus, an information processing method, an information processing program and an information processing system.

In recent years, the development of technology related to remote communication has become more and more active. In remote communication, communication is achieved by communicating the video and audio of the other party in a remote location with each other.

Japanese Unexamined Patent Publication No. 2019-61594

However, in the above-mentioned conventional technique, it may be difficult to accurately convey a complicated operation content from a transmitter to a listener as compared with face-to-face communication. There is a need for technology that can support the transmission of complex operation details in remote communication.

The information processing apparatus according to the embodiment is an operation content transmitted from a transmitter who is a participant of the remote communication to the listener based on information of a listener who is a participant of the remote communication, and is the operation content of the listener. It includes a generation unit that generates auxiliary information that supplements the operation content for the operation target operated by the user, and an output control unit that controls the auxiliary information generated by the generation unit to be output to the terminal device of the listener.

FIG. 1 is a diagram showing an example of information processing according to a comparative example. FIG. 2 is a diagram showing an example of information processing according to an embodiment. FIG. 3 is a diagram showing a configuration example of an information processing system according to an embodiment. FIG. 4 is a diagram showing a configuration example of the information processing apparatus according to the embodiment. FIG. 5 is a diagram showing an example of a participant information storage unit according to an embodiment. FIG. 6 is a diagram showing an example of information processing according to the first modification. FIG. 7 is a diagram showing an example of information processing according to the second modification. FIG. 8 is a diagram showing an example of information processing according to the third modification. FIG. 9 is a diagram showing an example of information processing according to the fourth modification. FIG. 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device.

Hereinafter, an embodiment for implementing an information processing apparatus, an information processing method, an information processing program, and an information processing system according to the present application (hereinafter, referred to as “embodiment”) will be described in detail with reference to the drawings. Note that this embodiment does not limit the information processing apparatus, information processing method, information processing program, and information processing system according to the present application. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description is omitted.

(Embodiment)
[1. Introduction]
In recent years, an increasing number of companies have introduced telework as a countermeasure against new coronavirus infectious diseases. In addition, with the introduction of telework, opportunities to use remote communication such as video conferences and web conferences are increasing more and more. For example, there is an increasing opportunity to convey the operation content of the operation target operated by the listener from the transmitter to the listener by using remote communication.

In general, it is said that it is more difficult to convey complicated operation contents in remote communication than in face-to-face communication. For example, when communicating the operation content by remote communication, the transmitter needs to convey the operation content to the listener to the listener without the actual operation target at hand, so it is possible to convey complicated operation content. It can be difficult. In addition, since the listener needs to understand the operation content transmitted from the transmitter only with linguistic information, it may be difficult to understand the complicated operation content.

Therefore, in recent years, a technique for assisting the transmission of complicated operation contents in remote communication has been developed. For example, there is a known technology that can reduce mistakes and improve transmission speed by providing 3D holographic images of transmitters in remote areas to listeners in the field (reference URL: https://dynamics.microsoft. com / en-us / mixed-reality / overview /). However, there is room for improvement in providing an online environment that is comparable to or can compete with face-to-face transmission, which is rich in visual information, auditory information, and tactile information in transmitting complicated operation contents. Specifically, in remote communication, there is a need for a mechanism that is easier to use and can absorb various cases. For example, a mechanism that optimizes the transmission content according to the attributes of the listener, a mechanism that provides the transmission content according to the listener's intention in real time, a mechanism that provides the transmission content at the timing according to the situation of the site where the listener is located, And a mechanism to provide precise operation contents is desired.

Here, an example of information processing according to a comparative example will be described with reference to FIG. FIG. 1 is a diagram showing an example of information processing according to a comparative example. In FIG. 1, the participant U11 (hereinafter, also referred to as “transmitter U11”) who is a transmitter is referred to the participants U21 and U22 (hereinafter, also referred to as “listeners U21 and U22”) who are listeners by using remote communication. On the other hand, the operation content for the device A operated by the listener is transmitted.

Further, the listeners U21 and U22 are equipped with a terminal device 10-21 and a terminal device 10-22, which are head-mounted displays, respectively, and the image G1 of the transmitter U11 is passed through the terminal device 10-21 and the terminal device 10-22, respectively. Can be seen. Further, the listeners U21 and U22 can hear the voice of the transmitter U11 through the speakers mounted on the terminal device 10-21 and the terminal device 10-22, respectively.

Further, of the two listeners, the listener U21 is an experienced person who has experience in operating the device A. Further, the listener U22 is an inexperienced person who has no experience in operating the device A, and is in a state where it is difficult to hear the sound (deafness). As shown in FIG. 1, in the comparative example, the same information is transmitted to the listeners U21 and U22 regardless of the attributes of the listener. Therefore, for the listener U22 who has deafness, it may be difficult to hear the voice of the transmitter U11, and it may be difficult to convey the operation content itself. In addition, it may be difficult for the inexperienced listener U22 to understand the transmitted operation content because unfamiliar technical terms frequently appear.

Therefore, the present invention is an operation content transmitted from a transmitter who is a participant of remote communication to a listener based on attribute information of a listener who is a participant of remote communication, and is an operation target operated by the listener. It is controlled to output auxiliary information supplementing the operation contents to the listener's terminal device. As a result, the present invention can provide auxiliary information according to the attributes of the listener in addition to the operation content transmitted by the transmitter, and thus can assist the transmission of complicated operation content in remote communication. .. Therefore, the invention of the present application can support the transmission of complicated operation contents in remote communication.

[2. An example of information processing]
Next, an example of information processing according to the embodiment will be described with reference to FIG. FIG. 2 is a diagram showing an example of information processing according to an embodiment. The information processing shown in FIG. 2 is realized by the information processing system 1 according to the embodiment. The information processing system 1 includes a terminal device 10 and an information processing device 100. In the following description, as the information processing according to the embodiment, the information processing performed by the terminal device 10 and the information processing device 100 in cooperation with each other will be described. In the present embodiment, the information processing apparatus 100 executes the information processing program according to the embodiment and performs information processing according to the embodiment by cooperating with the terminal apparatus 10.

Prior to the description of FIG. 2, the configuration of the information processing system 1 according to the embodiment will be described with reference to FIG. FIG. 3 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 3, the information processing system 1 includes a terminal device 10 and an information processing device 100. The terminal device 10 and the information processing device 100 are connected to each other via a predetermined network N so as to be communicable by wire or wirelessly. The information processing system 1 shown in FIG. 3 may include an arbitrary number of terminal devices 10 and an arbitrary number of information processing devices 100.

The terminal device 10 is an information processing device used by a participant in remote communication (hereinafter, also referred to as “participant”). The terminal device 10 is, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), a head-mounted display, or the like. In the present embodiment, the listener's terminal device 10 is assumed to be a head-mounted display. Further, it is assumed that the terminal device 10 of the transmitter is a notebook PC.

Further, an application for using the remote communication system (hereinafter, also referred to as "remote communication application") is installed in the terminal device 10. The terminal device 10 displays various images (for example, toolbars, icons, etc.) for operating the remote communication application on the screen. Further, a device having a function such as a camera, a microphone, and a speaker is mounted or connected to the terminal device 10. The terminal device 10 transmits / receives video and audio input from each device between a plurality of bases.

Further, various sensors for detecting the physical state of the participant are mounted or connected to the terminal device 10. For example, the terminal device 10 is connected to the above-mentioned sensors such as a camera and a microphone. The terminal device 10 detects sensor information indicating the physical state of the participant by various sensors. For example, the terminal device 10 detects an image of a participant by a camera as an example of sensor information. When the terminal device 10 detects the sensor information, the terminal device 10 transmits the detected sensor information to the information processing device 100.

Further, the terminal device 10 receives auxiliary information from the information processing device 100. Upon receiving the auxiliary information, the terminal device 10 outputs the auxiliary information. For example, when the auxiliary information is an image, the terminal device 10 displays the auxiliary information on the screen. Further, when the auxiliary information is voice, the terminal device 10 outputs the complementary information from the speaker.

As shown in FIG. 3, the terminal device 10 may be described separately as terminal devices 10-1 to 10-N (N is a natural number) depending on the participants who use the terminal device 10. For example, the terminal device 10-21 is a terminal device 10 used by the participant U21 shown in FIG. Further, for example, the terminal device 10-22 is a terminal device 10 used by the participant U22 shown in FIG. Further, in the following, when the terminal devices 10-1 to 10-N (N is a natural number) will be described without particular distinction, they will be referred to as the terminal device 10.

The information processing device 100 acquires listener information from the terminal device 10. Specifically, the information processing device 100 acquires the sensor information of the listener from the terminal device 10. When the information processing apparatus 100 acquires the sensor information of the listener, the information processing apparatus 100 determines whether or not the listener has difficulty in the operation related to the operation content based on the acquired sensor information of the listener. Subsequently, when the information processing apparatus 100 determines that the listener has difficulty in the operation related to the operation content, the operation content is transmitted from the transmitter to the listener based on the attribute information of the listener. Generates auxiliary information that supplements the operation content for the operation target operated by the listener. Further, the information processing device 100 controls to output the generated auxiliary information to the listener's terminal device 10. Specifically, the information processing device 100 transmits the generated auxiliary information to the listener's terminal device 10.

From here, it returns to the explanation of FIG. In FIG. 2, similarly to FIG. 1, the transmitter U11 transmits the operation contents for the device A operated by the listener to the listeners U21 and U22 by using remote communication. Here, in the example shown in FIG. 2, the information processing device 100 outputs auxiliary information supplementing the operation contents for the device A to the terminal device 10-22 of the listener U22 based on the attribute information of the listener U22. Is different.

Although the illustration is omitted, first, it is assumed that the listener U22 said, "There are many words that I do not understand and it is difficult to hear the voice." At this time, the microphone mounted on the terminal device 10-22 worn by the listener U22 detects the voice of the listener U22's speech. Subsequently, when the terminal device 10-22 detects the voice of the listener U22's speech, the terminal device 10-22 transmits the detected voice data to the information processing device 100.

The information processing device 100 acquires the voice data of the listener U22 from the terminal device 10-22. Subsequently, when the information processing apparatus 100 acquires the voice data of the listener U22, the information processing apparatus 100 determines whether or not the listener U22 has difficulty in the operation related to the operation content based on the acquired voice data of the listener U22. In FIG. 2, the information processing apparatus 100 determines that the listener U22 is having difficulty in operating the operation content.

When the information processing apparatus 100 determines that the listener U22 is having difficulty in operating the operation content, the information processing apparatus 100 is transmitted by the transmitter U11 as an example of auxiliary information based on the attribute information that the listener U22 has deafness. An image G21 including subtitles in which the operation content is written is generated. Subsequently, the information processing device 100 controls the generated image G21 to be displayed on the terminal device 10-22 of the listener U22.

As a result, the information processing apparatus 100 can visually perceive the operation content by the image G21 including the subtitles as auxiliary information even when the deafness listener U22 has difficulty in hearing the voice of the transmitter U11. To. Therefore, the information processing apparatus 100 can help the listener U22, who has deafness, to transmit the operation content.

Further, when the information processing apparatus 100 determines that the listener U22 is having difficulty in operating the operation content, the information processing apparatus 100 uses the transmitter U11 as an example of auxiliary information based on the attribute information that the listener U22 is an inexperienced person. An image G22 including a subtitle containing the meaning of a technical term appearing in the transmitted operation content is generated. Subsequently, the information processing device 100 controls the generated image G22 to be displayed on the terminal device 10-22 of the listener U22.

As a result, the information processing apparatus 100 confirms the meaning of the technical term each time by the image G22 including the subtitle as auxiliary information even when the technical term unfamiliar to the listener U22 who is an inexperienced person frequently appears. Can be made possible. Therefore, the information processing apparatus 100 can help the listener U22, who is an inexperienced person, to understand the operation contents.

[3. Information processing device configuration]
Next, the configuration of the information processing apparatus 100 according to the embodiment will be described with reference to FIG. FIG. 4 is a diagram showing a configuration example of the information processing apparatus 100 according to the embodiment. As shown in FIG. 4, the information processing apparatus 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 includes an input unit (for example, a keyboard, a mouse, etc.) that receives various operations from the administrator of the information processing device 100, and a display unit (for example, a liquid crystal display, etc.) for displaying various information. You may have.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 110 is connected to the network by wire or wirelessly, and transmits / receives information to / from, for example, the terminal device 10.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. As shown in FIG. 4, the storage unit 120 has a participant information storage unit 121 and a sensor information storage unit 122.

(Participant Information Memory Unit 121)
The participant information storage unit 121 stores various information regarding the attributes of the participants. Specifically, the participant information storage unit 121 stores the identification information (also referred to as a participant ID) that can identify the participant and the attribute information of the participant in association with each other. An example of the participant information storage unit 121 will be described with reference to FIG. FIG. 5 is a diagram showing an example of a participant information storage unit according to an embodiment.

In the example shown in FIG. 5, the participant information storage unit 121 has the "experienced person" which is the attribute information of the participant U21 indicating that he / she is an experienced person of the operation for the device A, and the participant ID "U21" of the participant U21. And are stored in association with each other. Further, the participant information storage unit 121 is an attribute information of the participant U22 indicating that he / she is an inexperienced person in the operation of the device A, and an attribute of the participant U22 indicating that the sound is difficult to hear. The information "deafness" and the participant ID "U22" of the participant U22 are stored in association with each other.

(Sensor information storage unit 122)
The sensor information storage unit 122 stores various sensor information indicating the physical state of the participant. Specifically, the sensor information storage unit 122 stores the identification information (also referred to as a participant ID) that can identify the participant and the sensor information of the participant in association with each other.

In the example shown in FIG. 2, the sensor information storage unit 122 stores the image data and voice data of the participant U11 in association with the participant ID “U11” of the participant U11. The sensor information storage unit 122 stores the image data and audio data of the participant U21 in association with the participant ID “U21” of the participant U21. Further, the sensor information storage unit 122 stores the image data and audio data of the participant U22 in association with the participant ID “U22” of the participant U22.

(Control unit 130)
Returning to the description of FIG. 4, the control unit 130 is a controller, and is stored in a storage device inside the information processing device 100 by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Various programs (corresponding to an example of an information processing program) are realized by executing the RAM as a work area. Further, the control unit 130 is a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 4, the control unit 130 includes an acquisition unit 131, a determination unit 132, a generation unit 133, and an output control unit 134, and realizes or executes the information processing operation described below. The internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 4, and may be any other configuration as long as it is configured to perform information processing described later.

(Acquisition unit 131)
The acquisition unit 131 acquires the participant's information from the terminal device 10. Specifically, the acquisition unit 131 acquires the listener's information from the terminal device 10. Further, the acquisition unit 131 acquires the sensor information of the listener from the terminal device 10. For example, the acquisition unit 131 acquires image data and audio data of the listener from the terminal device 10. Subsequently, when the acquisition unit 131 acquires the sensor information of the listener, the acquisition unit 131 stores the acquired sensor information in the sensor information storage unit 122.

Further, the acquisition unit 131 acquires the information of the transmitter from the terminal device 10. Specifically, the acquisition unit 131 acquires the sensor information of the transmitter from the terminal device 10. For example, the acquisition unit 131 acquires image data and voice data of the transmitter from the terminal device 10. Subsequently, when the acquisition unit 131 acquires the sensor information of the transmitter, the acquisition unit 131 stores the acquired sensor information in the sensor information storage unit 122.

(Judgment unit 132)
The determination unit 132 determines whether or not the listener has difficulty in operating the operation content based on the information of the listener. Specifically, the determination unit 132 determines whether or not the listener has difficulty in operating the operation content based on the sensor information of the listener acquired by the acquisition unit 131.

For example, the determination unit 132 determines whether or not the listener has difficulty in operating the operation content based on the voice data of the listener acquired by the acquisition unit 131. For example, the determination unit 132 converts the listener's voice data acquired by the acquisition unit 131 into a character string by using a known voice recognition technique. In the example shown in FIG. 2, the determination unit 132 converts the voice data of the listener U22 into a character string "many unknown words and the voice is difficult to hear."

Subsequently, the determination unit 132 morphologically analyzes the converted character string and decomposes it into words. Subsequently, the determination unit 132 determines whether or not a specific word is included in the decomposed words. For example, the determination unit 132 determines whether or not a word included in a word list (hereinafter, also referred to as a word list) that is likely to be issued when the listener has difficulty in operating the operation content is included in the decomposed words. Is determined. For example, suppose the word list contains words such as "I don't know," "I'm in trouble," and "I'm hard to hear." In the example shown in FIG. 2, the determination unit 132 determines that the decomposed words include the words "not understood" and "difficult to hear" included in the word list.

Subsequently, when it is determined that the decomposed word includes a specific word, the determination unit 132 determines that the listener has difficulty in the operation related to the operation content. On the other hand, when it is determined that the decomposed word does not include a specific word, the determination unit 132 determines that the listener has not made any difficulty in the operation related to the operation content. In the example shown in FIG. 2, the determination unit 132 determines that the decomposed words include the words "I do not understand" and "difficult to hear" which are the words included in the word list, so that the listener U22 operates the operation contents. Judge that it is causing difficulty.

(Generation unit 133)
The generation unit 133 is an operation content transmitted from the transmitter who is a participant of the remote communication to the listener based on the information of the listener who is a participant of the remote communication, and is an operation for the operation target operated by the listener. Generate supplementary information to supplement the content. Specifically, the generation unit 133 generates auxiliary information when the determination unit 132 determines that the listener is having difficulty in operation. For example, the generation unit 133 generates auxiliary information based on the attribute information regarding the attributes of the listener as an example of the information of the listener.

In the example shown in FIG. 2, when the determination unit 132 determines that the listener U22 is having difficulty in the operation related to the operation content, the generation unit 133 refers to the participant information storage unit 121 and attributes the listener U22. Get information. For example, the generation unit 133 acquires the attribute information that the listener U22 has deafness. Subsequently, when the generation unit 133 acquires the attribute information that the listener U22 has deafness, the generation unit 133 obtains the operation content transmitted by the transmitter U11 as an example of the auxiliary information based on the attribute information that the listener U22 has deafness. The image G21 including the subtitles in characters is generated. For example, the generation unit 133 refers to the sensor information storage unit 122 and acquires the voice data of the transmitter U11. Subsequently, the generation unit 133 converts the voice data of the transmitter U11 into a character string by using a known voice recognition technique. Subsequently, the generation unit 133 generates an image G21 including subtitles in which the operation content transmitted by the transmitter U11 is a character, based on the converted character string.

Further, when the determination unit 132 determines that the listener U22 is having difficulty in operating the operation content, the generation unit 133 refers to the participant information storage unit 121 and has an attribute that the listener U22 is an inexperienced person. Get information. Subsequently, when the generation unit 133 acquires the attribute information that the listener U22 is an inexperienced person, the operation transmitted by the transmitter U11 as an example of the auxiliary information based on the attribute information that the listener U22 is an inexperienced person. An image G22 including subtitles in which the meanings of technical terms appearing in the content are written is generated. For example, the generation unit 133 refers to the sensor information storage unit 122 and acquires the voice data of the transmitter U11. Subsequently, the generation unit 133 converts the voice data of the transmitter U11 into a character string by using a known voice recognition technique. Subsequently, the generation unit 133 extracts technical terms appearing in the operation content transmitted by the transmitter U11 from the converted character string. Subsequently, the generation unit 133 refers to a dictionary or the like in which the meaning of the technical term is posted, and generates an image G22 including a subtitle containing the meaning of the extracted technical term as a character.

(Output control unit 134)
The output control unit 134 controls to output the auxiliary information generated by the generation unit 133 to the terminal device 10 of the listener. Specifically, when the auxiliary information is generated by the generation unit 133, the output control unit 134 transmits the auxiliary information generated by the generation unit 133 to the listener's terminal device 10. In the example shown in FIG. 2, when the image G21 and the image G22 are generated by the generation unit 133, the output control unit 134 transmits the generated images G21 and the image G22 to the terminal device 10-22 of the listener U22.

[4. Modification example]
The information processing system 1 according to the above-described embodiment may be implemented in various different forms other than the above-described embodiment. Therefore, another embodiment of the information processing system 1 will be described below. The same parts as those in the embodiment are designated by the same reference numerals, and the description thereof will be omitted.

[4-1. First modification]
First, information processing according to the first modification will be described with reference to FIG. FIG. 6 is a diagram showing an example of information processing according to the first modification. In FIG. 6, similarly to FIG. 2, the transmitter U11 transmits the operation contents for the device A operated by the listener to the listeners U21 and U22 by using remote communication. Here, in the example shown in FIG. 6, the information processing apparatus 100 outputs auxiliary information supplementing the operation contents to the apparatus A to the terminal apparatus 10-22 of the listener U22 based on the voice information of the site where the listener U22 is located. The point is different from FIG.

Although not shown, it is assumed that the listener U22 first said, "There are some unreadable Chinese characters, so I want you to use hiragana. Also, I want you to make the characters larger." At this time, the microphone mounted on the terminal device 10-22 worn by the listener U22 detects the voice of the listener U22's speech. Subsequently, when the terminal device 10-22 detects the voice of the listener U22's speech, the terminal device 10-22 transmits the detected voice data to the information processing device 100.

The acquisition unit 131 acquires the voice data of the listener U22 from the terminal device 10-22. The determination unit 132 uses a known voice recognition technique to convert the voice data of the listener U22 acquired by the acquisition unit 131 into a character string "There are unreadable Chinese characters, so please make them hiragana. Also, make the characters larger. I want you to do it. "

Subsequently, the determination unit 132 morphologically analyzes the converted character string and decomposes it into words. Subsequently, the determination unit 132 determines whether or not the word included in the above-mentioned word list is included in the decomposed word. In the example shown in FIG. 6, the determination unit 132 determines that the decomposed words include the words "unreadable" and "want to do" included in the word list.

Subsequently, the determination unit 132 determines that the decomposed words include the words "unreadable" and "want to read" included in the word list, so that the listener U22 has difficulty in operating the operation content. Is determined to have occurred.

In the example shown in FIG. 6, when the determination unit 132 determines that the listener U22 is having difficulty in operating the operation content, the generation unit 133 includes "hiragana" included in the character string converted from the voice data of the listener U22. Based on "I want you to do it", as an example of auxiliary information, an image G31 including a subtitle in which the operation content transmitted by the transmitter U11 is described in hiragana is generated.

Specifically, the generation unit 133 analyzes that the meaning of "I want you to convert to hiragana" included in the character string obtained by converting the voice data of the listener U22 is "I want you to convert to hiragana". Subsequently, when the generation unit 133 semantically analyzes the character string converted from the voice data of the listener U22, the generation unit 133 refers to the processing list stored in the storage unit 120 and executes the processing corresponding to the result of the semantic analysis. More specifically, the storage unit 120 stores a processing list in which the character string "I want you to do XX" and the processing corresponding to "XX" are associated with each other. For example, the storage unit 120 stores a processing list in which the character string "I want you to convert to hiragana" and the processing to convert to hiragana are associated with each other. The generation unit 133 executes a process of converting the voice data of the transmitter U11 into hiragana as a process corresponding to the character string "I want you to convert to hiragana" which is the result of the semantic analysis. For example, the generation unit 133 converts the voice data of the transmitter U11 into a character string by using a known voice recognition technique. Subsequently, the generation unit 133 converts the converted character string into hiragana. Subsequently, the generation unit 133 generates an image G31 including a subtitle in which the operation content transmitted by the transmitter U11 is described in hiragana.

Further, when the determination unit 132 determines that the listener U22 is having difficulty in the operation related to the operation content, the generation unit 133 "makes the character larger" included in the character string obtained by converting the voice data of the listener U22. Based on "I want", as an example of auxiliary information, an image G31 including a subtitle in which the size of characters indicating the operation content transmitted by the transmitter U11 is enlarged is generated.

Specifically, the generation unit 133 analyzes that the meaning of "I want you to make the character larger" included in the character string converted from the voice data of the listener U22 is "I want you to increase the size of the character". .. When the generation unit 133 semantically analyzes the character string converted from the voice data of the listener U22, the generation unit 133 refers to the processing list stored in the storage unit 120 and executes the processing based on the result of the semantic analysis. For example, the storage unit 120 stores a processing list in which a character string "I want you to increase the size of a character" and a process for increasing the size of a character are associated with each other. The generation unit 133 executes a process of enlarging the size of the character indicating the operation content transmitted by the transmitter U11 as a process corresponding to the character string "I want you to increase the size of the character" which is the result of the semantic analysis. do. For example, the generation unit 133 generates an image G31 including a subtitle in which the size of the character indicating the operation content transmitted by the transmitter U11 is enlarged based on the character string obtained by converting the voice data of the transmitter U11.

When the image G31 is generated by the generation unit 133, the output control unit 134 transmits the generated image G31 to the terminal device 10-22 of the listener U22.

In FIG. 6, as an example of auxiliary information, the generation unit 133 has described an example of generating character information in which the operation content transmitted by the transmitter U11 is expressed in hiragana. However, the generation unit 133 generates character information expressed in other than hiragana. You may. For example, the generation unit 133 may generate character information written in katakana, kanji, or romaji as an example of auxiliary information.

The generation unit 133 may generate character information translated into a language other than Japanese as an example of auxiliary information. For example, the generation unit 133 may generate subtitles in which the operation content transmitted by the transmitter is translated into English based on the attribute information that the listener's mother tongue is English.

Further, in FIG. 6, as an example of auxiliary information, an example in which the generation unit 133 generates character information in which the size of characters indicating the operation contents transmitted by the transmitter U11 is enlarged has been described, but the present invention is limited to this. I can't. For example, the generation unit 133 may generate character information in which the size of characters is reduced as an example of auxiliary information.

[4-2. Second variant]
Next, the information processing according to the second modification will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of information processing according to the second modification. In FIG. 7, the transmitter U11 uses remote communication to convey to the listeners U23 and U24 located at the same site, the operation content for the device O1 operated by the listener.

The acquisition unit 131 acquires image information of the site where the listeners U21 and U22 are located. For example, the acquisition unit 131 acquires image information of the site where the listeners U21 and U22 are located from the camera C1 installed at the site where the listeners U21 and U22 are located.

The determination unit 132 determines the timing at which the operation content is transmitted from the transmitter to the listener based on the image information of the site where the listeners U21 and U22 are located. Specifically, the determination unit 132 determines whether or not the listeners U21 and U22 are in the process of confirming the operation content based on the image information of the site acquired by the acquisition unit 131.

For example, the determination unit 132 identifies a region of the device O1 included in the image information at the site by using a known object recognition technique. Further, the determination unit 132 identifies the person areas of the listeners U21 and U22 included in the image information of the site by using a known posture estimation technique. Subsequently, the determination unit 132 determines whether or not at least one of the specified listeners U21 and U22 is in contact with the device O1. When the determination unit 132 determines that at least one of the listeners U21 and U22 is in contact with the device O1, it is determined that the listeners U21 and U22 are in the process of confirming the operation content. Subsequently, when the determination unit 132 determines that the listeners U21 and U22 are in the process of confirming the operation content, the determination unit 132 determines that it is not the timing for transmitting the operation content from the transmitter to the listener.

When the determination unit 132 determines that it is not the timing for transmitting the operation content from the transmitter to the listener, the generation unit 133 generates an image prompting the user to wait for a while to transmit the next operation content. When the output control unit 134 generates an image prompting the generation unit 133 to wait for a while to transmit the next operation content, the output control unit 134 controls to display the generated image on the terminal device 10 of the transmitter.

On the other hand, when the determination unit 132 determines that neither the listener U21 nor the U22 is in contact with the device O1, the determination unit 132 determines that the listeners U21 and U22 are not in the process of confirming the operation content. Subsequently, when the determination unit 132 determines that the listeners U21 and U22 are not in the process of confirming the operation content, the determination unit 132 determines that it is the timing for transmitting the operation content from the transmitter to the listener.

When the determination unit 132 determines that it is the timing for transmitting the operation content from the transmitter to the listener, the generation unit 133 may generate an image prompting the transmission of the next operation content. When the output control unit 134 generates an image prompting the generation unit 133 to transmit the next operation content, the output control unit 134 may control to display the generated image on the terminal device 10 of the transmitter. In this way, the output control unit 134 controls to output the information regarding the timing determined by the determination unit 132 to the terminal device 10 of the transmitter.

FIG. 7 has described an example in which the determination unit 132 determines the timing at which the operation content is transmitted from the transmitter to the listener based on the image information of the site where the listeners U21 and U22 are located. It does not have to be image information. Specifically, the determination unit 132 may determine the timing at which the operation content is transmitted from the transmitter to the listener based on the voice information of the site where the listener is located. The determination unit 132 determines whether or not there is no utterance by the listeners U21 and U22 for a predetermined time or longer based on the voice information at the site. For example, when the determination unit 132 determines that there is no utterance by the listeners U21 and U22 for a predetermined time or longer, the determination unit 132 may determine that it is the timing for transmitting the operation content from the transmitter to the listener. Alternatively, the determination unit 132 determines whether or not the utterance by the listeners U21 and U22 includes a word such as "OK" based on the voice information at the site. For example, when the determination unit 132 determines that the utterances made by the listeners U21 and U22 include a word such as "OK", the determination unit 132 may determine that it is the timing for transmitting the operation content from the transmitter to the listener.

Further, the determination unit 132 may determine the timing at which the operation content is transmitted from the transmitter to the listener based on the signal information generated from the device O1 operated by the listener. For example, the determination unit 132 can acquire signal information generated only when the listener is operating the device O1 from the device O1. Subsequently, the determination unit 132 determines, based on the acquired signal information, whether or not the state in which the listener is not operating is a predetermined time or longer. For example, when the determination unit 132 determines that the state in which the operation by the listener has not been performed is for a predetermined time or longer, the determination unit 132 may determine that it is the timing for transmitting the operation content from the transmitter to the listener. Alternatively, the determination unit 132 determines whether or not the state in which the operation by the listener is being performed is equal to or longer than a predetermined time, based on the acquired signal information. For example, when the determination unit 132 determines that the state in which the operation by the listener is being performed is for a predetermined time or longer, the determination unit 132 may determine that it is not the timing for transmitting the operation content from the transmitter to the listener.

[4-3. Third variant]
Next, the information processing according to the third modification will be described with reference to FIG. FIG. 8 is a diagram showing an example of information processing according to the third modification. In FIG. 8, the transmitter U11 conveys to the listener U25 the operation content of the drone controller O25 operated by the listener U25 by using remote communication.

In FIG. 8, the acquisition unit 131 acquires an operation moving image of an operation on the controller O25 by the listener U25. For example, the acquisition unit 131 acquires an operation moving image of the operation of the listener U25 on the controller O25 from the camera C25 mounted on the terminal device 10-25 of the listener U25, which is a spectacle-type head-mounted display. The output control unit 134 controls to display the operation moving image acquired by the acquisition unit 131 on the screen of the terminal device 10 of the handset U11. As a result, the transmitter side can repeatedly play back the operation moving image acquired by the acquisition unit 131, which makes it easier for the listener side to evaluate.

Further, the acquisition unit 131 acquires evaluation information indicating the evaluation of the transmitter U11 with respect to the operation of the listener U25 from the terminal device 10 of the transmitter U11. The output control unit 134 controls to output the evaluation information acquired by the acquisition unit 131 to the terminal device 10-25 of the listener U25.

[4-4. Fourth variant]
Next, the information processing according to the fourth modification will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of information processing according to the fourth modification. In FIG. 9, similarly to FIG. 8, the transmitter U11 conveys to the listener U26 the operation content of the drone controller O26 operated by the listener U26 by using remote communication.

In FIG. 9, the generation unit 133 generates the sample moving image G42, which is auxiliary information, based on the signal information of the controller O26 used by the listener U26 to operate the drone as an example of the listener's information. Specifically, the acquisition unit 131 acquires the signal information of the controller O26 from the controller O26 of the listener U26. Subsequently, the determination unit 132 determines whether or not the listener U26 has difficulty in the operation related to the operation content based on the signal information acquired by the acquisition unit 131. For example, the determination unit 132 determines the operation currently being performed by the listener U26 based on the signal information acquired by the acquisition unit 131. Subsequently, the determination unit 132 determines how much the operation currently being performed by the listener U26 is delayed from the operation procedure based on the operation procedure prepared in advance. For example, when the determination unit 132 determines that the operation currently being performed by the listener U26 is delayed by exceeding a predetermined threshold value as compared with the operation procedure, the listener U26 has difficulty in operating the operation content. Is determined. When the determination unit 132 determines that the listener U26 is having difficulty in operating the operation content, the generation unit 133 generates a sample moving image G42 that captures the sample operation for the drone. The output control unit 134 controls to display the sample moving image G42 generated by the generation unit 133 on the terminal devices 10-26 of the listener U26. For example, the output control unit 134 controls so that the sample moving image G42 of the same size is displayed next to the moving image G41 that captures the actual operation by the listener U26.

[4-5. Other variants]
As an example of the listener's information, the generation unit 133 may generate auxiliary information based on the image information of the site where the listener is located. For example, the generation unit 133 generates a part image showing the operation part of the operation target or a direction image showing the operation direction with respect to the operation target as an example of the auxiliary information based on the image information of the site where the listener is located.

Further, the generation unit 133 may generate an enlarged image in which a predetermined area to be operated is enlarged as an example of auxiliary information based on the image information of the site where the listener is located. For example, the generation unit 133 generates an enlarged image in which the precision area to be operated is enlarged.

[5. effect〕
As described above, the information processing apparatus 100 according to the embodiment includes a generation unit 133 and an output control unit 134. The generation unit 133 is an operation content transmitted from the transmitter who is a participant of the remote communication to the listener based on the information of the listener who is a participant of the remote communication, and is an operation for the operation target operated by the listener. Generate supplementary information to supplement the content. The output control unit 134 controls to output the auxiliary information generated by the generation unit 133 to the terminal device 10 of the listener. Further, as the listener information, the generation unit 133 includes attribute information regarding the attributes of the listener, audio information of the site where the listener is located, image information of the site where the listener is located, or control used for the listener to operate the operation target. Auxiliary information is generated based on the signal information of the device.

As a result, the information processing apparatus 100 can provide auxiliary information according to the listener in addition to the operation content transmitted by the transmitter, so that it is possible to assist the transmission of complicated operation content in remote communication. .. Therefore, the invention of the present application can support the transmission of complicated operation contents in remote communication.

Further, the generation unit 133 generates, as auxiliary information, a part image showing the operation part of the operation target or a direction image showing the operation direction with respect to the operation target. The output control unit 134 displays at least one of a site image and a direction image on the screen of the listener's terminal device 10.

As a result, the information processing apparatus 100 can provide a part image showing the operation part of the operation target or a direction image showing the operation direction with respect to the operation target, in addition to the operation content transmitted by the transmitter. It can help the communication of complicated operation contents in communication.

In addition, the generation unit 133 generates an enlarged image in which a predetermined area to be operated is enlarged as auxiliary information. The output control unit 134 displays the enlarged image on the screen of the listener's terminal device 10.

As a result, the information processing apparatus 100 can enlarge and display the precision area, for example, and can assist in the transmission of the operation content related to the precision area in remote communication.

In addition, the generation unit 133 generates character information indicating the operation content or character information explaining the meaning of terms related to the operation content as auxiliary information. The output control unit 134 displays the character information on the screen of the listener's terminal device 10.

Thereby, for example, the information processing apparatus 100 makes it possible to visually perceive the operation content by the character information which is the auxiliary information even when the listener who has deafness has difficulty in hearing the voice of the transmitter. Therefore, the information processing apparatus 100 can help, for example, the transmission of the operation content to the listener who has deafness. Further, the information processing apparatus 100 makes it possible to confirm the meaning of the technical term each time by the character information which is the auxiliary information even when the technical term which is not familiar to the listener who is an inexperienced person frequently appears. be able to. Therefore, the information processing apparatus 100 can help, for example, an inexperienced listener to understand the operation content.

In addition, the generation unit 133 generates character information in which the size of the character is enlarged or reduced as auxiliary information. The output control unit 134 displays the character information whose character size is enlarged or reduced on the screen of the listener's terminal device 10.

As a result, the information processing apparatus 100 can help a listener with low visual acuity to visually perceive the operation content of the transmitter, for example, by enlarging the characters. Further, the information processing apparatus 100 can display the operation content of the transmitter on the screen so as not to disturb the listener's hand or the operation target by, for example, reducing the characters. Therefore, the information processing apparatus 100 can help the listener to visually understand the operation content.

In addition, the generation unit 133 generates character information written in hiragana, katakana, kanji, or romaji as auxiliary information. The output control unit 134 displays character information written in hiragana, katakana, kanji, or romaji on the screen of the listener's terminal device 10.

Thereby, the information processing apparatus 100 can help the listener to understand the operation content of the transmitter by the display mode of Japanese according to the listener.

In addition, the generation unit 133 generates character information translated into a language other than Japanese as auxiliary information. The output control unit 134 displays the character information translated into another language on the screen of the listener's terminal device 10.

Thereby, the information processing apparatus 100 can help the listener to understand the operation content of the transmitter in the language corresponding to the listener.

Further, the information processing apparatus 100 further includes a determination unit 132. The determination unit 132 determines whether or not the listener has difficulty in operating the operation content based on the information of the listener. The generation unit 133 generates auxiliary information when the determination unit 132 determines that the listener is having difficulty in operation.

As a result, the information processing apparatus 100 can generate auxiliary information when the listener is in trouble, and can help the listener to understand the operation content of the transmitter.

Further, the determination unit 132 determines the timing at which the operation content is transmitted from the transmitter to the listener based on the information of the listener. The output control unit 134 controls to output the information regarding the timing determined by the determination unit 132 to the terminal device 10 of the transmitter.

As a result, the information processing apparatus 100 can transmit the operation content from the transmitter to the listener at a timing that is easy for the listener to accept, so that it is possible to help the listener understand the operation content of the transmitter.

In addition, the generation unit 133 generates a sample moving image of the sample operation for the operation target as auxiliary information. The output control unit 134 controls to display the sample moving image on the screen of the listener's terminal device 10.

As a result, the information processing apparatus 100 can see the sample moving image in addition to the operation content transmitted by the transmitter, and can help the listener to understand the operation content of the transmitter.

Further, the information processing apparatus 100 further includes an acquisition unit 131. The acquisition unit 131 acquires an operation moving image that captures an operation on an operation target by the listener. The output control unit 134 controls to display the operation moving image on the screen of the terminal device 10 of the handset.

As a result, the information processing apparatus 100 can give feedback to the transmitter to what extent the operation content transmitted by the information processing apparatus 100 is transmitted to the listener.

In addition, the acquisition unit 131 acquires evaluation information indicating the evaluation of the transmitter with respect to the operation of the listener. The output control unit 134 controls to output the evaluation information to the listener's terminal device 10.

As a result, the information processing apparatus 100 can feed back the evaluation of the transmitter to the listener as to how much the operation content is correctly understood.

[6. Hardware configuration]
Further, the information processing apparatus 100 according to the above-described embodiment is realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing apparatus 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores a program executed by the CPU 1100, data used by such a program, and the like. The communication interface 1500 receives data from another device via a predetermined communication network and sends the data to the CPU 1100, and transmits the data generated by the CPU 1100 to the other device via the predetermined communication network.

The CPU 1100 controls an output device such as a display or a printer, and an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. Further, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides the program or data to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. And so on.

For example, when the computer 1000 functions as the information processing apparatus 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the function of the control unit 130 by executing the program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be acquired from another device via a predetermined communication network.

Although some of the embodiments of the present application have been described in detail with reference to the drawings, these are examples, and various modifications are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other modified forms.

[7. others〕
Further, among the processes described in the above-described embodiments and modifications, all or part of the processes described as being automatically performed can be manually performed, or are described as being manually performed. It is also possible to automatically perform all or part of the processed processing by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown in the figure.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically distributed in any unit according to various loads and usage conditions. Can be integrated and configured.

Further, the above-described embodiments and modifications can be appropriately combined as long as the processing contents do not contradict each other.

Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the generation unit can be read as a generation means or a generation circuit.

1 Information processing system 10 Terminal device 100 Information processing device 110 Communication unit 120 Storage unit 130 Control unit 131 Acquisition unit 132 Judgment unit 133 Generation unit 134 Output control unit

Claims (16)

Based on the information of the listener who is a participant of the remote communication, the operation content transmitted from the transmitter who is the participant of the remote communication to the listener, and the operation content for the operation target operated by the listener. A generator that generates supplementary information,
An output control unit that controls the auxiliary information generated by the generation unit to be output to the listener's terminal device.
Information processing device equipped with. The generator is
As the listener information, attribute information regarding the attributes of the listener, audio information at the site where the listener is located, image information at the site where the listener is located, or a control device used by the listener to operate the operation target. Generates the auxiliary information based on the signal information of
The information processing apparatus according to claim 1. The generator is
As the auxiliary information, a part image showing the operation part of the operation target or a direction image showing the operation direction with respect to the operation target is generated.
The output control unit
At least one of the site image and the direction image is displayed on the screen of the listener's terminal device.
The information processing apparatus according to claim 1 or 2. The generator is
As the auxiliary information, an enlarged image in which a predetermined area of the operation target is enlarged is generated.
The output control unit
Displaying the enlarged image on the screen of the listener's terminal device,
The information processing apparatus according to any one of claims 1 to 3. The generator is
As the auxiliary information, character information indicating the operation content or character information explaining the meaning of terms related to the operation content is generated.
The output control unit
Displaying the character information on the screen of the listener's terminal device,
The information processing apparatus according to any one of claims 1 to 4. The generator is
As the auxiliary information, the character information in which the size of the character is enlarged or reduced is generated.
The output control unit
The character information in which the size of the character is enlarged or reduced is displayed on the screen of the terminal device of the listener.
The information processing apparatus according to claim 5. The generator is
As the auxiliary information, the character information written in hiragana, katakana, kanji or romaji is generated.
The output control unit
The character information expressed in the hiragana, the katakana, the kanji, or the romaji is displayed on the screen of the terminal device of the listener.
The information processing apparatus according to claim 5 or 6. The generator is
As the auxiliary information, the character information translated into a language other than Japanese is generated.
The output control unit
The character information translated into the other language is displayed on the screen of the terminal device of the listener.
The information processing apparatus according to any one of claims 5 to 7. Further, a determination unit for determining whether or not the listener has difficulty in the operation related to the operation content based on the listener's information is provided.
The generator is
When the determination unit determines that the listener is having difficulty in the operation, the auxiliary information is generated.
The information processing apparatus according to any one of claims 1 to 8. Further, a determination unit for determining the timing at which the operation content is transmitted from the transmitter to the listener based on the listener's information is provided.
The output control unit
Controlled to output information regarding the timing determined by the determination unit to the terminal device of the transmitter.
The information processing apparatus according to any one of claims 1 to 9. The generator is
As the auxiliary information, a sample moving image of the sample operation for the operation target is generated.
The output control unit
Controlling the display of the sample moving image on the screen of the listener's terminal device,
The information processing apparatus according to any one of claims 1 to 10. Further provided with an acquisition unit for acquiring an operation moving image of an operation of the operation target by the listener.
The output control unit
Controlling the operation video to be displayed on the screen of the terminal device of the transmitter,
The information processing apparatus according to any one of claims 1 to 11. The acquisition unit
Obtaining evaluation information indicating the evaluation of the transmitter with respect to the operation of the listener,
The output control unit
Controlled to output the evaluation information to the listener's terminal device.
The information processing apparatus according to claim 12. It is an information processing method executed by a computer.
Based on the information of the listener who is a participant of the remote communication, the operation content transmitted from the transmitter who is the participant of the remote communication to the listener, and the operation content for the operation target operated by the listener. The generation process to generate supplementary information and
An output control step that controls the auxiliary information generated by the generation step to be output to the listener's terminal device, and
Information processing methods including. Based on the information of the listener who is a participant of the remote communication, the operation content transmitted from the transmitter who is the participant of the remote communication to the listener, and the operation content for the operation target operated by the listener. The generation procedure to generate supplementary information and
An output control procedure for controlling the auxiliary information generated by the generation procedure to be output to the listener's terminal device, and an output control procedure.
An information processing program characterized by having a computer execute. An information processing system that includes an information processing device and a terminal device of a participant in remote communication.
The information processing device is
Based on the information of the listener who is a participant of the remote communication, the operation content transmitted from the transmitter who is the participant of the remote communication to the listener, and the operation content for the operation target operated by the listener. A generator that generates supplementary information,
An output control unit that controls the auxiliary information generated by the generation unit to be output to the listener's terminal device.
Equipped with
The terminal device of the listener who is the participant
Outputs the auxiliary information generated by the generator.
Information processing system.

Images