JP7152454B2 - Information processing device, information processing method, information processing program, and information processing system - Google Patents

Description

The present invention relates to an information processing device, an information processing method, an information processing program, and an information processing system.

In recent years, the development of technology related to remote communication has become more and more popular. In remote communication, communication is achieved by mutual communication of images, voices, etc. of a partner at a remote location.

JP 2019-61594 A

However, with the above-described conventional technology, it may be difficult for the sender to accurately convey complicated operation details to the listener compared to face-to-face communication. There is a demand for a technology that can support the transmission of complicated operation details in remote communication.

An information processing apparatus according to an embodiment provides an operation content transmitted from a transmitter who is a participant in remote communication to the listener based on information about the listener who is a participant in the remote communication. a generation unit that generates auxiliary information that supplements the operation content for the operation target operated by the , and an output control unit that controls to output the auxiliary information generated by the generation unit to the terminal device of the listener.

FIG. 1 is a diagram illustrating an example of information processing according to a comparative example. FIG. 2 is a diagram illustrating an example of information processing according to the embodiment; FIG. 3 is a diagram illustrating a configuration example of an information processing system according to the embodiment; FIG. 4 is a diagram illustrating a configuration example of an information processing apparatus according to the embodiment; FIG. 5 is a diagram illustrating an example of a participant information storage unit according to the embodiment; FIG. 6 is a diagram illustrating an example of information processing according to the first modification. FIG. 7 is a diagram illustrating an example of information processing according to the second modification. FIG. 8 is a diagram illustrating an example of information processing according to the third modification. FIG. 9 is a diagram illustrating an example of information processing according to the fourth modification. FIG. 10 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing apparatus.

Hereinafter, modes for implementing an information processing apparatus, an information processing method, an information processing program, and an information processing system (hereinafter referred to as "embodiments") according to the present application will be described in detail with reference to the drawings. Note that the information processing apparatus, information processing method, information processing program, and information processing system according to the present application are not limited to these embodiments. Also, in each of the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

(embodiment)
[1. Introduction]
In recent years, an increasing number of companies have introduced telework as a countermeasure against the new coronavirus infection. In addition, with the introduction of telework, there are more and more opportunities to use remote communication such as video conferences and web conferences. For example, using remote communication, there are increasing opportunities for a sender to transmit to a listener the details of an operation performed on an operation target operated by the listener.

In general, it is said that it is more difficult to convey complicated operation details in remote communication than in face-to-face communication. For example, when transmitting operation details by remote communication, the transmitter needs to communicate the operation details of the operation target to the listener when the actual operation target is not at hand, so it is possible to transmit complicated operation details. It can be difficult. In addition, since the listener needs to understand the content of the operation transmitted from the sender only with verbal information, it may be difficult for the listener to understand the content of the complicated operation.

Therefore, in recent years, techniques have been developed for assisting transmission of complicated operation contents in remote communication. For example, there is a known technology that reduces mistakes and improves communication speed by providing a 3D holographic image of a remote speaker to a listener on site (Reference URL: https://dynamics.microsoft.com/). com/en-us/mixed-reality/overview/). However, there is room for improvement in providing an online environment that can compete with or compete with face-to-face communication, which is rich in visual, auditory, and tactile information, in communicating complex operational content. Specifically, in remote communication, there is a need for a mechanism that is more user-friendly and can absorb various cases. For example, a mechanism to optimize the content of communication according to the attributes of the listener, a mechanism to provide the content of communication according to the intention of the listener in real time, a mechanism to provide the content of communication at the timing according to the situation of the site where the listener is located, There is also a demand for a mechanism that provides precise operation details.

Here, an example of information processing according to a comparative example will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of information processing according to a comparative example. In FIG. 1, using remote communication, a participant U11 who is a communicator (hereinafter also referred to as “communicator U11”) communicates with participants U21 and U22 who are listeners (hereinafter also referred to as “listeners U21 and U22”). On the other hand, the content of the operation for the device A operated by the listener is transmitted.

Listeners U21 and U22 wear terminal devices 10-21 and 10-22, which are head-mounted displays, respectively. can see Listeners U21 and U22 can hear the voice of the communicator U11 through speakers installed in the terminal devices 10-21 and 10-22, respectively.

Also, of the two listeners, the listener U21 is an experienced person who has experience operating the device A. FIG. Also, the listener U22 is an inexperienced person who has no experience of operating the device A, and is in a state where it is difficult to hear sounds (hard of hearing). As shown in FIG. 1, in the comparative example, the same information is transmitted to listeners U21 and U22 regardless of the attributes of the listeners. For this reason, the hearing-impaired listener U22 may find it difficult to hear the voice of the transmitter U11, making it difficult to transmit the operation content itself. In addition, for the inexperienced listener U22, unfamiliar technical terms frequently appear, so there are cases where it is difficult for the listener U22 to understand the transmitted operation details.

Therefore, according to the present invention, based on the attribute information of the listener who is a participant in the remote communication, the operation content transmitted from the transmitter who is a participant in the remote communication to the listener is an operation target operated by the listener. control so that auxiliary information that supplements the operation content for the listener is output to the terminal device of the listener. As a result, the present invention can provide supplementary information according to the attributes of the listener in addition to the operational details transmitted by the sender, so that it is possible to assist the transmission of complicated operational details in remote communication. . Therefore, the present invention can support transmission of complicated operation contents in remote communication.

[2. Example of information processing]
Next, an example of information processing according to the embodiment will be described using FIG. FIG. 2 is a diagram illustrating an example of information processing according to the embodiment; The information processing shown in FIG. 2 is implemented by the information processing system 1 according to the embodiment. The information processing system 1 includes a terminal device 10 and an information processing device 100 . In the following description, information processing performed by the terminal device 10 and the information processing device 100 in cooperation will be described as information processing according to the embodiment. In the present embodiment, the information processing apparatus 100 executes the information processing program according to the embodiment and cooperates with the terminal device 10 to perform information processing according to the embodiment.

Prior to the description of FIG. 2, the configuration of the information processing system 1 according to the embodiment will be described using FIG. FIG. 3 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 3 , the information processing system 1 includes a terminal device 10 and an information processing device 100 . The terminal device 10 and the information processing device 100 are connected via a predetermined network N so as to be communicable by wire or wirelessly. The information processing system 1 shown in FIG. 3 may include any number of terminal devices 10 and any number of information processing apparatuses 100 .

The terminal device 10 is an information processing device used by a participant in remote communication (hereinafter also referred to as "participant"). The terminal device 10 is, for example, a smart phone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), a head-mounted display, or the like. In this embodiment, the listener's terminal device 10 is assumed to be a head-mounted display. It is also assumed that the terminal device 10 of the sender is a notebook PC.

An application for using the remote communication system (hereinafter also referred to as “remote communication application”) is installed in the terminal device 10 . The terminal device 10 displays various images (for example, toolbars, icons, etc.) for operating the remote communication application on the screen. Devices having functions such as a camera, a microphone, and a speaker are installed in or connected to the terminal device 10 . The terminal device 10 transmits and receives video and audio input from each device between a plurality of bases.

In addition, the terminal device 10 is equipped with or connected to various sensors for detecting the physical state of the participants. For example, the terminal device 10 is connected to sensors such as the above-described camera and microphone. The terminal device 10 detects sensor information indicating the physical state of the participant using various sensors. For example, the terminal device 10 detects an image of a participant using a camera as an example of sensor information. When detecting the sensor information, the terminal device 10 transmits the detected sensor information to the information processing device 100 .

The terminal device 10 also receives auxiliary information from the information processing device 100 . Upon receiving the auxiliary information, the terminal device 10 outputs the auxiliary information. For example, when the auxiliary information is an image, the terminal device 10 displays the auxiliary information on the screen. Also, when the auxiliary information is voice, the terminal device 10 outputs complementary information from the speaker.

Note that, as shown in FIG. 3, the terminal devices 10 may be classified as terminal devices 10-1 to 10-N (N is a natural number) according to the participants using the terminal devices 10. FIG. For example, the terminal device 10-21 is the terminal device 10 used by the participant U21 shown in FIG. Also, for example, the terminal device 10-22 is the terminal device 10 used by the participant U22 shown in FIG. Further, hereinafter, the terminal devices 10-1 to 10-N (N is a natural number) will be referred to as the terminal device 10 when they are not distinguished from each other.

The information processing device 100 acquires listener information from the terminal device 10 . Specifically, the information processing device 100 acquires the listener's sensor information from the terminal device 10 . After acquiring the listener's sensor information, the information processing apparatus 100 determines whether or not the listener is having difficulty performing an operation related to the operation content based on the acquired listener's sensor information. Subsequently, when the information processing apparatus 100 determines that the listener is having difficulty in performing an operation related to the operation content, the information processing apparatus 100, based on the listener's attribute information, is the operation content transmitted from the sender to the listener, Auxiliary information that supplements the operation content for the operation target operated by the listener is generated. The information processing device 100 also controls the generated auxiliary information to be output to the terminal device 10 of the listener. Specifically, the information processing device 100 transmits the generated auxiliary information to the terminal device 10 of the listener.

From here, the description returns to FIG. In FIG. 2, as in FIG. 1, the communicator U11 uses remote communication to communicate to listeners U21 and U22 the details of the operation of the device A operated by the listeners. Here, in the example shown in FIG. 2, the information processing apparatus 100 outputs auxiliary information that supplements the operation content for the device A to the terminal device 10-22 of the listener U22 based on the attribute information of the listener U22. different from

Although illustration is omitted, first, it is assumed that the listener U22 says, "There are many words I do not understand, and it is difficult to hear the voice." At this time, the microphone mounted on the terminal device 10-22 worn by the listener U22 detects the speech of the listener U22. Subsequently, when the terminal device 10-22 detects the speech of the listener U22, the terminal device 10-22 transmits the detected speech data to the information processing device 100. FIG.

The information processing device 100 acquires the voice data of the listener U22 from the terminal device 10-22. Subsequently, when acquiring the voice data of the listener U22, the information processing apparatus 100 determines whether or not the listener U22 is having difficulty performing operations related to the operation content based on the acquired voice data of the listener U22. In FIG. 2, the information processing apparatus 100 determines that the listener U22 is having difficulty in performing an operation related to the operation content.

When the information processing apparatus 100 determines that the listener U22 is having difficulty in performing an operation related to the operation content, based on the attribute information that the listener U22 is hard of hearing, an example of the auxiliary information is transmitted by the transmitter U11. An image G21 including subtitles in which the contents of the operation are made into characters is generated. Subsequently, the information processing device 100 controls to display the generated image G21 on the terminal device 10-22 of the listener U22.

As a result, the information processing apparatus 100 can visually perceive the operation content from the image G21 including subtitles, which is the auxiliary information, even when the hearing-impaired listener U22 has difficulty in hearing the voice of the transmitter U11. to Therefore, the information processing device 100 can help convey the operation content to the hearing-impaired listener U22.

Further, when the information processing apparatus 100 determines that the listener U22 is having difficulty in performing an operation related to the operation content, based on the attribute information that the listener U22 is an inexperienced person, the information processing apparatus 100 outputs, as an example of auxiliary information, An image G22 is generated that includes subtitles in which technical terms appearing in the transmitted operation content are written in characters. Subsequently, the information processing device 100 controls to display the generated image G22 on the terminal device 10-22 of the listener U22.

As a result, even when technical terms unfamiliar to the inexperienced listener U22 appear frequently, the information processing apparatus 100 can confirm the meaning of the technical terms each time using the image G22 including subtitles, which is auxiliary information. can be made possible. Therefore, the information processing device 100 can help the inexperienced listener U22 to understand the operation content.

[3. Configuration of Information Processing Device]
Next, the configuration of the information processing apparatus 100 according to the embodiment will be described using FIG. FIG. 4 is a diagram illustrating a configuration example of the information processing apparatus 100 according to the embodiment. As shown in FIG. 4, the information processing apparatus 100 has a communication section 110, a storage section 120, and a control section . The information processing apparatus 100 includes an input unit (for example, a keyboard, a mouse, etc.) for receiving various operations from an administrator of the information processing apparatus 100, and a display unit (for example, a liquid crystal display) for displaying various information. may have.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 110 is connected to a network by wire or wirelessly, and transmits and receives information to and from the terminal device 10, for example.

(storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 has a participant information storage unit 121 and a sensor information storage unit 122, as shown in FIG.

(Participant information storage unit 121)
The participant information storage unit 121 stores various types of information regarding attributes of participants. Specifically, the participant information storage unit 121 associates and stores identification information (also referred to as a participant ID) capable of identifying a participant and attribute information of the participant. An example of the participant information storage unit 121 will be described with reference to FIG. FIG. 5 is a diagram illustrating an example of a participant information storage unit according to the embodiment;

In the example shown in FIG. 5, the participant information storage unit 121 stores the attribute information "experienced" of the participant U21 indicating that the participant U21 has experience of operating the device A, and the participant ID "U21" of the participant U21. are associated with each other and stored. In addition, the participant information storage unit 121 stores the attribute information of the participant U22 indicating that the participant U22 is inexperienced in operating the device A, and the attribute information of the participant U22 indicating that the sound is difficult to hear. The information "hard of hearing" and the participant ID "U22" of the participant U22 are associated and stored.

(Sensor information storage unit 122)
The sensor information storage unit 122 stores various sensor information indicating physical states of participants. Specifically, the sensor information storage unit 122 associates and stores identification information (also referred to as a participant ID) that can identify a participant and the sensor information of the participant.

In the example shown in FIG. 2, the sensor information storage unit 122 stores the image data and audio data of the participant U11 in association with the participant ID "U11" of the participant U11. The sensor information storage unit 122 stores the image data and audio data of the participant U21 in association with the participant ID "U21" of the participant U21. Further, the sensor information storage unit 122 stores the image data and audio data of the participant U22 in association with the participant ID "U22" of the participant U22.

(control unit 130)
Returning to the description of FIG. 4, the control unit 130 is a controller, and is stored in a storage device inside the information processing apparatus 100 by, for example, a CPU (Central Processing Unit) or MPU (Micro Processing Unit). Various programs (corresponding to an example of an information processing program) are executed by using the RAM as a work area. Also, the control unit 130 is a controller, and is implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 4, the control unit 130 has an acquisition unit 131, a determination unit 132, a generation unit 133, and an output control unit 134, and implements or executes the information processing operation described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 4, and may be another configuration as long as it performs information processing to be described later.

(Acquisition unit 131)
The acquisition unit 131 acquires information on participants from the terminal device 10 . Specifically, the acquisition unit 131 acquires listener information from the terminal device 10 . In addition, the acquisition unit 131 acquires the listener's sensor information from the terminal device 10 . For example, the acquisition unit 131 acquires image data and audio data of the listener from the terminal device 10 . Subsequently, when acquiring the sensor information of the listener, the acquisition unit 131 stores the acquired sensor information in the sensor information storage unit 122 .

Also, the acquisition unit 131 acquires the information of the sender from the terminal device 10 . Specifically, the acquisition unit 131 acquires sensor information of the sender from the terminal device 10 . For example, the acquisition unit 131 acquires image data and voice data of the sender from the terminal device 10 . Subsequently, when acquiring the sensor information of the sender, the acquisition unit 131 stores the acquired sensor information in the sensor information storage unit 122 .

(Determination unit 132)
The determination unit 132 determines whether or not the listener is having difficulty performing an operation related to the operation content, based on the listener's information. Specifically, based on the listener's sensor information acquired by the acquisition unit 131, the determination unit 132 determines whether or not the listener is having difficulty performing an operation related to the operation content.

For example, the determination unit 132 determines whether or not the listener is having difficulty performing an operation related to the operation content, based on the listener's voice data acquired by the acquisition unit 131 . For example, the determination unit 132 converts the listener's voice data acquired by the acquisition unit 131 into a character string using a known voice recognition technology. In the example shown in FIG. 2, the determination unit 132 converts the speech data of the listener U22 into a character string "There are many words I do not understand, and it is difficult to hear the voice."

Subsequently, the determination unit 132 morphologically analyzes the converted character string and breaks it down into words. Subsequently, the determination unit 132 determines whether or not a specific word is included in the decomposed words. For example, the determination unit 132 determines whether words included in a word list (hereinafter, also referred to as a word list) that are likely to be uttered when the listener is having difficulty performing an operation related to the operation content are included in the decomposed words. determine whether For example, suppose that the word list includes words such as "I don't understand", "I'm having trouble", and "Hard to hear". In the example shown in FIG. 2, the determination unit 132 determines that the decomposed words include the words "I don't understand" and "Difficult to hear" included in the word list.

Subsequently, when determining that a specific word is included in the decomposed words, the determination unit 132 determines that the listener is having difficulty performing the operation related to the operation content. On the other hand, when determining that the specific word is not included in the decomposed words, the determination unit 132 determines that the listener does not have difficulty in performing the operation related to the operation content. In the example shown in FIG. 2, the determination unit 132 determines that the decomposed words include the words "I don't understand" and "Difficult to hear" that are included in the word list. It is determined that there is difficulty in

(Generating unit 133)
Based on the information of the listener who is a participant in the remote communication, the generating unit 133 generates operation contents transmitted from the transmitter who is a participant in the remote communication to the listener, which is an operation for an operation target operated by the listener. Generate auxiliary information to supplement the content. Specifically, the generation unit 133 generates the auxiliary information when the determination unit 132 determines that the listener is having difficulty with the operation. For example, the generation unit 133 generates auxiliary information based on attribute information about listener attributes as an example of listener information.

In the example shown in FIG. 2 , when the determination unit 132 determines that the listener U22 is having difficulty in performing an operation related to the operation content, the generation unit 133 refers to the participant information storage unit 121 to determine the attributes of the listener U22. Get information. For example, the generator 133 acquires attribute information indicating that the listener U22 is hard of hearing. Subsequently, when the attribute information indicating that the listener U22 is hard of hearing is acquired, the generation unit 133 generates operation details transmitted by the transmitter U11 as an example of auxiliary information based on the attribute information that the listener U22 is hard of hearing. An image G21 including subtitles in text is generated. For example, the generation unit 133 refers to the sensor information storage unit 122 and acquires voice data of the transmitter U11. Next, the generation unit 133 converts the speech data of the sender U11 into a character string using a known speech recognition technology. Next, based on the converted character string, the generation unit 133 generates an image G21 including captions in which the content of the operation transmitted by the messenger U11 is written.

In addition, when the determination unit 132 determines that the listener U22 is having difficulty in performing the operation related to the operation content, the generation unit 133 refers to the participant information storage unit 121 to determine whether the listener U22 has the attribute that the listener U22 is an inexperienced person. Get information. Next, when acquiring the attribute information that the listener U22 is an inexperienced person, the generating unit 133 generates an operation transmitted by the communicator U11 as an example of auxiliary information based on the attribute information that the listener U22 is an inexperienced person. An image G22 is generated that includes subtitles in which the meanings of technical terms appearing in the content are written in characters. For example, the generation unit 133 refers to the sensor information storage unit 122 and acquires voice data of the transmitter U11. Next, the generation unit 133 converts the speech data of the sender U11 into a character string using a known speech recognition technology. Subsequently, the generating unit 133 extracts technical terms appearing in the operation content transmitted by the transmitter U11 from the converted character string. Subsequently, the generation unit 133 refers to a dictionary or the like that lists the meanings of technical terms, and generates an image G22 including subtitles in which the meanings of the extracted technical terms are written.

(Output control unit 134)
The output control unit 134 controls to output the auxiliary information generated by the generation unit 133 to the terminal device 10 of the listener. Specifically, when the auxiliary information is generated by the generating unit 133, the output control unit 134 transmits the auxiliary information generated by the generating unit 133 to the terminal device 10 of the listener. In the example shown in FIG. 2, when the image G21 and the image G22 are generated by the generation unit 133, the output control unit 134 transmits the generated image G21 and the image G22 to the terminal device 10-22 of the listener U22.

[4. Modification]
The information processing system 1 according to the above-described embodiments may be implemented in various different forms other than the above-described embodiments. Therefore, other embodiments of the information processing system 1 will be described below. In addition, the same code|symbol is attached|subjected to the same part as embodiment, and description is abbreviate|omitted.

[4-1. First modification]
First, with reference to FIG. 6, information processing according to the first modified example will be described. FIG. 6 is a diagram illustrating an example of information processing according to the first modification. In FIG. 6, as in FIG. 2, using remote communication, the transmitter U11 communicates to listeners U21 and U22 the details of the operation of device A operated by the listeners. Here, in the example shown in FIG. 6, the information processing device 100 outputs auxiliary information supplementing the operation content of the device A to the terminal device 10-22 of the listener U22 based on the voice information of the site where the listener U22 is located. 2 differs from FIG.

Although illustration is omitted, first, it is assumed that the listener U22 says, "Since there are some kanji characters that I cannot read, I want you to write them in hiragana. Also, I want you to make the characters larger." At this time, the microphone mounted on the terminal device 10-22 worn by the listener U22 detects the speech of the listener U22. Subsequently, when the terminal device 10-22 detects the speech of the listener U22, the terminal device 10-22 transmits the detected speech data to the information processing device 100. FIG.

Acquisition unit 131 acquires voice data of listener U22 from terminal device 10-22. Using a known speech recognition technique, the determination unit 132 converts the voice data of the listener U22 acquired by the acquisition unit 131 into a character string "There are kanji characters that cannot be read, so please use hiragana. I want you to do it."

Subsequently, the determination unit 132 morphologically analyzes the converted character string and breaks it down into words. Subsequently, the determination unit 132 determines whether or not the words included in the word list described above are included in the decomposed words. In the example shown in FIG. 6, the determining unit 132 determines that the decomposed words include the words "I can't read" and "I want you to do" that are included in the word list.

Subsequently, the determination unit 132 determines that the decomposed words include the words "can't read" and "want to do", which are included in the word list. is determined to have occurred.

In the example shown in FIG. 6 , when the determination unit 132 determines that the listener U22 is having difficulty in performing an operation related to the operation content, the generation unit 133 converts the voice data of the listener U22 into the character string “Hiragana Based on "I want you to do this", an image G31 is generated as an example of the auxiliary information, which includes captions written in hiragana that describe the details of the operation transmitted by the messenger U11.

Specifically, the generating unit 133 analyzes the meaning of "I want you to use hiragana" included in the character string obtained by converting the voice data of the listener U22 as "I want you to use hiragana". Next, after semantically analyzing the character string obtained by converting the speech data of the listener U22, the generating unit 133 refers to the processing list stored in the storage unit 120 and executes processing corresponding to the result of the semantic analysis. More specifically, the storage unit 120 stores a process list in which a character string "I want you to do XX" and a process corresponding to "XX" are associated with each other. For example, the storage unit 120 stores a process list in which a character string "Please convert to hiragana" and a process for converting to hiragana are associated with each other. The generation unit 133 executes a process of converting the voice data of the sender U11 into hiragana as a process corresponding to the character string "Please convert to hiragana" which is the result of the semantic analysis. For example, the generator 133 converts voice data of the transmitter U11 into a character string using a known voice recognition technology. Next, the generation unit 133 converts the converted character string into hiragana. Subsequently, the generation unit 133 generates an image G31 including subtitles in which the content of the operation transmitted by the messenger U11 is written in hiragana.

In addition, when the determination unit 132 determines that the listener U22 is having difficulty in performing an operation related to the operation content, the generation unit 133 selects "Make the characters larger" included in the character string obtained by converting the voice data of the listener U22. Based on "I want you", an image G31 is generated as an example of the auxiliary information, which includes subtitles in which the size of the characters indicating the details of the operation transmitted by the messenger U11 is enlarged.

Specifically, the generation unit 133 analyzes the meaning of "I want you to make the characters larger" included in the character string obtained by converting the speech data of the listener U22 as "I want you to enlarge the character size". . After semantically analyzing the character string obtained by converting the voice data of the listener U22, the generation unit 133 refers to the processing list stored in the storage unit 120 and executes processing based on the result of the semantic analysis. For example, the storage unit 120 stores a processing list in which a character string “I want you to enlarge the character size” and a process for increasing the character size are associated with each other. The generation unit 133 executes a process of enlarging the size of the characters indicating the details of the operation transmitted by the transmitter U11 as a process corresponding to the character string "Please enlarge the character size" that is the result of the semantic analysis. do. For example, the generating unit 133 generates an image G31 including subtitles in which the size of the characters indicating the details of the operation transmitted by the transmitter U11 is enlarged based on the character string obtained by converting the voice data of the transmitter U11.

When the image G31 is generated by the generation unit 133, the output control unit 134 transmits the generated image G31 to the terminal device 10-22 of the listener U22.

FIG. 6 illustrates an example in which the generation unit 133 generates character information written in hiragana for the content of the operation transmitted by the messenger U11 as an example of the auxiliary information. may For example, the generation unit 133 may generate character information written in katakana, kanji, or romaji as an example of the auxiliary information.

Note that the generation unit 133 may generate character information translated into a language other than Japanese as an example of the auxiliary information. For example, the generation unit 133 may generate subtitles by translating the details of the operation transmitted by the sender into English based on the attribute information indicating that the native language of the listener is English.

In addition, in FIG. 6, an example in which the generation unit 133 generates, as an example of the auxiliary information, character information in which the size of the characters indicating the details of the operation transmitted by the messenger U11 is enlarged has been described. can't For example, the generation unit 133 may generate character information in which the character size is reduced as an example of the auxiliary information.

[4-2. Second modification]
Next, information processing according to the second modification will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of information processing according to the second modification. In FIG. 7, using remote communication, the communicator U11 communicates to listeners U23 and U24 located at the same site the details of the operation of the device O1 operated by the listeners.

Acquisition unit 131 acquires image information of the site where listeners U21 and U22 are located. For example, the acquiring unit 131 acquires image information of the site where the listeners U21 and U22 are located from the camera C1 installed at the site where the listeners U21 and U22 are located.

The determination unit 132 determines the timing of transmitting the operation content from the transmitter to the listeners based on the image information of the site where the listeners U21 and U22 are located. Specifically, the determination unit 132 determines whether or not the listeners U21 and U22 are checking the operation content based on the image information of the site acquired by the acquisition unit 131 .

For example, the determination unit 132 identifies the area of the device O1 included in the image information of the site using a known object recognition technique. The determination unit 132 also uses a known posture estimation technique to identify the person areas of the listeners U21 and U22 included in the image information of the scene. Next, determination unit 132 determines whether or not at least one of identified listeners U21 and U22 is in contact with device O1. When determining that at least one of the listeners U21 and U22 is in contact with the device O1, the determination unit 132 determines that the listeners U21 and U22 are confirming the operation details. Subsequently, when the determination unit 132 determines that the listeners U21 and U22 are in the process of confirming the operation details, it determines that it is not the time for the transmitter to transmit the operation details to the listeners.

When the determination unit 132 determines that it is not the timing for transmitting the operation content from the transmitter to the listener, the generation unit 133 generates an image prompting the listener to wait for a while before transmitting the next operation content. When the generation unit 133 generates an image prompting the user to wait for a while before transmitting the next operation content, the output control unit 134 controls the terminal device 10 of the transmitter to display the generated image.

On the other hand, when determining that neither of the listeners U21 and U22 is touching the device O1, the determination unit 132 determines that the listeners U21 and U22 are not currently confirming the operation content. Subsequently, when the determination unit 132 determines that the listeners U21 and U22 are not in the process of confirming the operation details, it determines that it is time for the transmitter to transmit the operation details to the listeners.

When the determination unit 132 determines that it is time to transmit the operation content from the transmitter to the listener, the generation unit 133 may generate an image prompting transmission of the next operation content. When the generation unit 133 generates an image prompting the transmission of the next operation content, the output control unit 134 may control to display the generated image on the terminal device 10 of the transmitter. In this way, the output control unit 134 controls to output the information about the timing determined by the determination unit 132 to the terminal device 10 of the sender.

FIG. 7 describes an example in which the determination unit 132 determines the timing of transmitting the operation content from the transmitter to the listener based on the image information of the site where the listeners U21 and U22 are located. It does not have to be image information. Specifically, the determination unit 132 may determine the timing of transmitting the operation content from the transmitter to the listener based on the voice information of the site where the listener is located. The determination unit 132 determines whether or not the listeners U21 and U22 have not spoken for a predetermined period of time or longer, based on the on-site audio information. For example, if the determination unit 132 determines that the listeners U21 and U22 have not spoken for a predetermined period of time or longer, the determination unit 132 may determine that it is time for the transmitter to transmit the operation details to the listeners. Alternatively, the determination unit 132 determines whether or not the utterances by the listeners U21 and U22 include words such as "OK" based on the voice information at the site. For example, if it is determined that the utterances by the listeners U21 and U22 include the phrase "OK", the determination unit 132 may determine that it is time for the transmitter to transmit the operation details to the listener.

Further, the determination unit 132 may determine the timing of transmitting the operation content from the transmitter to the listener based on the signal information generated from the device O1 operated by the listener. For example, the determination unit 132 can acquire signal information from the device O1 that is generated only when the listener is operating the device O1. Next, based on the acquired signal information, the determination unit 132 determines whether or not the listener has not performed any operation for a predetermined time or longer. For example, if the determination unit 132 determines that the listener has not performed an operation for a predetermined period of time or longer, the determination unit 132 may determine that it is time for the transmitter to transmit the operation content to the listener. Alternatively, the determination unit 132 determines whether or not the state in which the listener is performing an operation is longer than or equal to a predetermined time, based on the acquired signal information. For example, if it is determined that the listener is performing an operation for a predetermined period of time or more, the determination unit 132 may determine that it is not the time to transmit the operation content from the transmitter to the listener.

[4-3. Third modification]
Next, information processing according to the third modification will be described with reference to FIG. FIG. 8 is a diagram illustrating an example of information processing according to the third modification. In FIG. 8, using remote communication, the communicator U11 communicates to the listener U25 the details of the operation of the controller O25 of the drone operated by the listener U25.

In FIG. 8, the acquisition unit 131 acquires an operation video imaged of the operation of the controller O25 by the listener U25. For example, the acquisition unit 131 acquires an operation video imaged of the operation of the controller O25 by the listener U25 from the camera C25 mounted on the terminal device 10-25 of the listener U25, which is a glasses-type head-mounted display. The output control unit 134 controls to display the operation video acquired by the acquisition unit 131 on the screen of the terminal device 10 of the transmitter U11. This allows the sender to repeatedly reproduce the operation video acquired by the acquisition unit 131, making it easier for the listener to evaluate.

In addition, the acquisition unit 131 acquires evaluation information indicating the evaluation of the communicator U11 with respect to the operation of the listener U25 from the terminal device 10 of the communicator U11. The output control unit 134 controls to output the evaluation information acquired by the acquisition unit 131 to the terminal device 10-25 of the listener U25.

[4-4. Fourth modification]
Next, information processing according to the fourth modification will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of information processing according to the fourth modification. In FIG. 9, as in FIG. 8, the communicator U11 uses remote communication to communicate to the listener U26 the details of the operation of the controller O26 of the drone operated by the listener U26.

In FIG. 9, the generation unit 133 generates a sample moving image G42, which is auxiliary information, based on the signal information of the controller O26 used by the listener U26 to operate the drone, as an example of listener information. Specifically, the acquisition unit 131 acquires the signal information of the controller O26 from the controller O26 of the listener U26. Subsequently, based on the signal information acquired by the acquisition unit 131, the determination unit 132 determines whether or not the listener U26 is having difficulty performing the operation related to the operation content. For example, the determination unit 132 determines the operation currently being performed by the listener U26 based on the signal information acquired by the acquisition unit 131 . Subsequently, the determination unit 132 determines how much the operation currently performed by the listener U26 is behind the operation procedure based on the operation procedure prepared in advance. For example, if the determination unit 132 determines that the operation currently being performed by the listener U26 is delayed by more than a predetermined threshold compared to the operation procedure, the listener U26 is having difficulty performing an operation related to the operation content. I judge. When the determination unit 132 determines that the listener U26 is having difficulty in performing an operation related to the operation content, the generation unit 133 generates a sample moving image G42 in which a sample operation for the drone is captured. The output control unit 134 controls to display the sample moving image G42 generated by the generation unit 133 on the terminal device 10-26 of the listener U26. For example, the output control unit 134 performs control so that a sample moving image G42 of the same size is displayed next to the moving image G41 in which the actual operation by the listener U26 is captured.

[4-5. Other Modifications]
The generation unit 133 may generate auxiliary information based on image information of the site where the listener is located, as an example of the listener's information. For example, the generation unit 133 generates, as an example of the auxiliary information, a region image indicating the operation region of the operation target or a direction image indicating the operation direction with respect to the operation target, based on the image information of the site where the listener is located.

Further, the generation unit 133 may generate an enlarged image obtained by enlarging a predetermined area of the operation target as an example of the auxiliary information based on the image information of the site where the listener is located. For example, the generation unit 133 generates an enlarged image obtained by enlarging the precision region of the operation target.

[5. effect〕
As described above, the information processing apparatus 100 according to the embodiment includes the generator 133 and the output controller 134 . Based on the information of the listener who is a participant in the remote communication, the generating unit 133 generates operation contents transmitted from the transmitter who is a participant in the remote communication to the listener, which is an operation for an operation target operated by the listener. Generate auxiliary information to supplement the content. The output control unit 134 controls to output the auxiliary information generated by the generation unit 133 to the terminal device 10 of the listener. In addition, the generating unit 133 generates, as listener information, attribute information related to the listener's attributes, audio information of the site where the listener is located, image information of the site where the listener is located, or control information used by the listener to operate the operation target. Auxiliary information is generated based on the device signal information.

As a result, the information processing apparatus 100 can provide supplementary information corresponding to the listener in addition to the operation details transmitted by the sender, so that it is possible to assist the transmission of complicated operation details in remote communication. . Therefore, the present invention can support transmission of complicated operation contents in remote communication.

In addition, the generation unit 133 generates, as auxiliary information, a region image indicating the operation region of the operation target, or a direction image indicating the operation direction with respect to the operation target. The output control unit 134 displays at least one of the part image and the direction image on the screen of the terminal device 10 of the listener.

As a result, the information processing apparatus 100 can provide, in addition to the operation content transmitted by the transmitter, a region image indicating the operation region of the operation target or a direction image indicating the operation direction with respect to the operation target. It can help to convey complicated operation contents in communication.

In addition, the generation unit 133 generates an enlarged image obtained by enlarging a predetermined area of the operation target as auxiliary information. The output control unit 134 displays the enlarged image on the screen of the terminal device 10 of the listener.

As a result, the information processing apparatus 100 can, for example, enlarge and display the precision area, thereby helping to transmit the operation content regarding the precision area in remote communication.

In addition, the generation unit 133 generates, as auxiliary information, character information indicating the content of the operation or character information explaining the meaning of terms related to the content of the operation. The output control unit 134 displays the character information on the screen of the terminal device 10 of the listener.

As a result, the information processing apparatus 100 enables, for example, a hearing-impaired listener to visually perceive the operation content based on the character information, which is auxiliary information, even when it is difficult for the listener to hear the speaker's voice. Therefore, the information processing apparatus 100 can help convey operation details to, for example, a hearing-impaired listener. In addition, the information processing apparatus 100 makes it possible to check the meaning of technical terms each time, for example, using character information, which is auxiliary information, even when technical terms that are unfamiliar to an inexperienced listener appear frequently. be able to. Therefore, the information processing apparatus 100 can help, for example, an inexperienced listener to understand the operation content.

In addition, the generation unit 133 generates character information in which the character size is enlarged or reduced as auxiliary information. The output control unit 134 displays the character information whose character size is enlarged or reduced on the screen of the terminal device 10 of the listener.

As a result, the information processing apparatus 100 can help a listener with poor eyesight to visually perceive the operator's operation details, for example, by enlarging the characters. Further, the information processing apparatus 100 can display the operator's operation details on the screen so as not to interfere with the listener's hand or the operation target, for example, by making the characters smaller. Therefore, the information processing apparatus 100 can help the listener to visually understand the operation content.

The generation unit 133 also generates character information written in hiragana, katakana, kanji, or romaji as auxiliary information. The output control unit 134 displays character information written in hiragana, katakana, kanji, or romaji on the screen of the terminal device 10 of the listener.

As a result, the information processing apparatus 100 can help the listener to understand the details of the operator's operation by displaying Japanese according to the listener.

The generation unit 133 also generates character information translated into a language other than Japanese as auxiliary information. The output control unit 134 displays the character information translated into another language on the screen of the terminal device 10 of the listener.

As a result, the information processing apparatus 100 can help the listener to understand the details of the operator's operation in a language suitable for the listener.

Information processing apparatus 100 further includes determination unit 132 . The determination unit 132 determines whether or not the listener is having difficulty performing an operation related to the operation content, based on the listener's information. The generation unit 133 generates auxiliary information when the determination unit 132 determines that the listener is having difficulty in operating.

As a result, the information processing apparatus 100 can generate auxiliary information when the listener is in trouble, so that the listener can help the listener to understand the operator's operation contents.

Also, the determination unit 132 determines the timing of transmitting the operation content from the transmitter to the listener based on the listener's information. The output control unit 134 controls to output the information about the timing determined by the determination unit 132 to the terminal device 10 of the sender.

As a result, the information processing apparatus 100 can transmit the operation content from the transmitter to the listener at a timing that is easy for the listener to accept, thereby helping the listener to understand the transmitter's operation content.

In addition, the generation unit 133 generates, as auxiliary information, a sample moving image in which a sample operation on the operation target is captured. The output control unit 134 controls to display the sample moving image on the screen of the terminal device 10 of the listener.

As a result, the information processing apparatus 100 can view the sample moving image in addition to the details of the operation transmitted by the narrator, thereby helping the listener to understand the details of the narrator's operation.

Information processing apparatus 100 further includes acquisition unit 131 . Acquisition unit 131 acquires an operation video in which an operation on an operation target by a listener is imaged. The output control unit 134 controls to display the operation video on the screen of the terminal device 10 of the sender.

As a result, the information processing apparatus 100 can give feedback to the transmitter as to how well the details of the operation transmitted by the information processing apparatus 100 have been conveyed to the listener.

In addition, the acquisition unit 131 acquires evaluation information indicating the transmitter's evaluation of the listener's operation. The output control unit 134 controls to output the evaluation information to the terminal device 10 of the listener.

As a result, the information processing apparatus 100 can feed back the listener's evaluation of how well the operation contents are correctly understood.

[6. Hardware configuration]
Also, the information processing apparatus 100 according to the above-described embodiments is implemented by a computer 1000 configured as shown in FIG. 10, for example. FIG. 10 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing apparatus 100. As shown in FIG. Computer 1000 includes CPU 1100 , RAM 1200 , ROM 1300 , HDD 1400 , communication interface (I/F) 1500 , input/output interface (I/F) 1600 and media interface (I/F) 1700 .

The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each section. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. Communication interface 1500 receives data from another device via a predetermined communication network, sends the data to CPU 1100, and transmits data generated by CPU 1100 to another device via a predetermined communication network.

The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, through an input/output interface 1600 . CPU 1100 acquires data from an input device via input/output interface 1600 . CPU 1100 also outputs the generated data to an output device via input/output interface 1600 .

Media interface 1700 reads programs or data stored in recording medium 1800 and provides them to CPU 1100 via RAM 1200 . CPU 1100 loads such a program from recording medium 1800 onto RAM 1200 via media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. etc.

For example, when the computer 1000 functions as the information processing apparatus 100 according to the embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 130 by executing programs loaded on the RAM 1200 . CPU 1100 of computer 1000 reads these programs from recording medium 1800 and executes them, but as another example, these programs may be obtained from another device via a predetermined communication network.

As described above, some of the embodiments of the present application have been described in detail based on the drawings. It is possible to carry out the invention in other forms with modifications.

[7. others〕
Further, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or described as being performed manually. All or part of the processing can also be performed automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Also, the above-described embodiments and modifications can be appropriately combined within a range that does not contradict the processing contents.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the generating unit can be read as generating means or a generating circuit.

1 information processing system 10 terminal device 100 information processing device 110 communication unit 120 storage unit 130 control unit 131 acquisition unit 132 determination unit 133 generation unit 134 output control unit

Claims (16)

Based on the information of the listener who is a participant in the remote communication, the operation content transmitted from the transmitter who is a participant in the remote communication to the listener, and the operation content for the operation target operated by the listener. a generation unit that generates auxiliary information to be supplemented when it is determined that the listener is having difficulty in performing an operation related to the operation content ;
an output control unit that controls to output the auxiliary information generated by the generation unit to the terminal device of the listener;
Information processing device. The generating unit
Attribute information relating to attributes of the listener, audio information of the site where the listener is located, image information of the site where the listener is located, or a control device used by the listener to operate the operation target as the information of the listener. generating the auxiliary information based on the signal information of
The information processing device according to claim 1 . The generating unit
generating, as the auxiliary information, a part image indicating an operation part of the operation target or a direction image indicating an operation direction with respect to the operation target;
The output control unit is
displaying at least one of the region image and the direction image on the screen of the terminal device of the listener;
The information processing apparatus according to claim 1 or 2. The generating unit
generating an enlarged image obtained by enlarging a predetermined area of the operation target as the auxiliary information;
The output control unit is
displaying the enlarged image on the screen of the terminal device of the listener;
The information processing apparatus according to any one of claims 1 to 3. The generating unit
generating, as the auxiliary information, character information indicating the content of the operation or character information explaining the meaning of terms related to the content of the operation;
The output control unit is
displaying the character information on the screen of the terminal device of the listener;
The information processing apparatus according to any one of claims 1 to 4. The generating unit
generating the character information in which the character size is enlarged or reduced as the auxiliary information;
The output control unit is
displaying the character information with the character size enlarged or reduced on the screen of the terminal device of the listener;
The information processing device according to claim 5 . The generating unit
generating the character information written in hiragana, katakana, kanji, or romaji as the auxiliary information;
The output control unit is
Displaying the character information written in the hiragana, the katakana, the kanji, or the romaji on the screen of the terminal device of the listener;
The information processing apparatus according to claim 5 or 6. The generating unit
generating the character information translated into a language other than Japanese as the auxiliary information;
The output control unit is
displaying the character information translated into the other language on the screen of the terminal device of the listener;
The information processing apparatus according to any one of claims 5 to 7. Based on the information of the listener, further comprising a determination unit that determines whether or not the listener is having difficulty with the operation related to the operation content,
The generating unit
generating the auxiliary information when the determination unit determines that the listener is having difficulty with the operation;
The information processing apparatus according to any one of claims 1 to 8. further comprising a determination unit that determines a timing for transmitting the operation content from the transmitter to the listener based on the listener's information,
The output control unit is
Control to output information about the timing determined by the determination unit to the terminal device of the transmitter;
The information processing apparatus according to any one of claims 1 to 9. The generating unit
generating, as the auxiliary information, a sample video image of a sample operation on the operation target;
The output control unit is
Control to display the sample video on the screen of the terminal device of the listener,
The information processing apparatus according to any one of claims 1 to 10. Further comprising an acquisition unit that acquires an operation video in which the listener's operation on the operation target is captured,
The output control unit is
controlling to display the operation video on the screen of the terminal device of the sender;
The information processing apparatus according to any one of claims 1 to 11. The acquisition unit
Acquiring evaluation information indicating the reporter's evaluation of the listener's operation,
The output control unit is
controlling to output the evaluation information to the terminal device of the listener;
The information processing apparatus according to claim 12. A computer-executed information processing method comprising:
Based on the information of the listener who is a participant in the remote communication, the operation content transmitted from the transmitter who is a participant in the remote communication to the listener, and the operation content for the operation target operated by the listener. a generation step of generating auxiliary information to be supplemented when it is determined that the listener is having difficulty in performing an operation related to the operation content ;
an output control step of controlling output of the auxiliary information generated by the generating step to the terminal device of the listener;
Information processing method including. Based on the information of the listener who is a participant in the remote communication, the operation content transmitted from the transmitter who is a participant in the remote communication to the listener, and the operation content for the operation target operated by the listener. a generation procedure for generating auxiliary information to be supplemented when it is determined that the listener is having difficulty performing an operation related to the operation content ;
an output control procedure for controlling output of the auxiliary information generated by the generation procedure to the terminal device of the listener;
An information processing program characterized by causing a computer to execute An information processing system including an information processing device and a terminal device of a remote communication participant,
The information processing device is
Based on the information of the listener who is a participant in the remote communication, the operation content transmitted from the transmitter who is a participant in the remote communication to the listener, and the operation content for the operation target operated by the listener. a generation unit that generates auxiliary information to be supplemented when it is determined that the listener is having difficulty in performing an operation related to the operation content ;
an output control unit that controls to output the auxiliary information generated by the generation unit to the terminal device of the listener;
with
The terminal device of the listener who is the participant,
outputting the auxiliary information generated by the generator;
Information processing system.

Images