JP2022103490A - Information processing device, information processing method, and computer program - Google Patents

Abstract

To provide a technique to convert outgoing and incoming voice data into text, and compare the results of both voice data converted to text so that it is made possible to determine the delay and degradation of data due to network and Web-RTC and notify a user of the same.SOLUTION: An information processing device has a communication screen. The information processing device accepts voice data input, determines whether the received voice data is delayed or degraded, and notifies a user in response to a determination result.SELECTED DRAWING: Figure 4

Description

The present invention relates to voice analysis and real-time communication.

Systems for communicating between remote locations have become widespread. For example, at a construction site, workers at the site can use information processing terminals such as smartphones and tablets to supervise remote locations. It is becoming possible to work while communicating with the operator.
Communication is performed by transmitting each other's video and audio, but depending on the communication environment, delay or deterioration of the video or audio data may occur. In order to deal with this delay and deterioration, there is a technology that tries to communicate smoothly by converting voice into text. For example, Patent Document 1 and Patent Document 2.

Japanese Unexamined Patent Publication No. 2020-88818 Special Table 2016-529839 Gazette

Patent Document 1 describes a voice text conversion technique for converting a call between a calling terminal and an incoming terminal into text and displaying the converted text on the outgoing terminal and the incoming terminal to match the text of the call content. Is disclosed.
Patent Document 2 describes a technique for maintaining communication by converting voice data into text having a small data size instead of voice data when the communication channel is congested in voice communication.

However, in the invention described in Patent Document 1, only the text of the call content is matched between both the calling side and the receiving side, and also in the invention described in Patent Document 2, the communication channel is congested. It is possible to maintain communication by converting to text with a small data size instead of voice data, but to what extent the data deteriorates due to the network and Web-RTC between the people who are communicating with each other. I can't tell if I'm doing it.

By converting the voice to text, you can check the displayed voice text to see what kind of content was spoken even if the voice data is delayed, but how much the voice data is actually delayed? Cannot be grasped.
For example, when talking to a communication partner based on the situation in the video, if it is unclear how much audio data is delayed, talk to the communication partner about what point in the video. I can't tell what I'm doing, and I end up with discommunication.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a mechanism capable of recognizing a state of data delay or deterioration in communication including voice.

The present invention for solving the above-mentioned problems is an information processing apparatus having a communication screen, which comprises an input means for receiving input of voice data, a means for creating voice data received by the input means into text, and a means for creating text. It is provided with a comparison means for comparing voice text data on the transmitting side and the receiving side created by the creating means, and a determining means for determining whether or not delay and deterioration have occurred based on the comparison result by the comparing means. It is characterized by that.

According to the present invention, it is possible to recognize the situation of data delay or deterioration in communication including voice.

It is a figure which shows an example of the whole structure of the communication system in embodiment of this invention. It is a block diagram which shows an example of the hardware composition of a smartphone 101, a tablet terminal 102, a client PC 111 and the like. It is a sequence diagram which shows the outline of the voice transmission / reception of the voice communication system using the Web-RTC system. It is a flowchart which shows an example of the communication processing of a client performed between a speaker and a listener in embodiment of this invention. It is a figure which shows the flow of the process during communication in the communication process loop of FIG. It is a figure which shows the flow of the delay determination registration process in the flow of the process during communication of FIG. It is a flowchart which shows an example of the delay determination acceptance notification processing of the delay determination system in embodiment of this invention. It is a flowchart which shows an example of the delay determination process in the delay determination acceptance notification process of FIG. 7. It is a figure which shows an example of the data holding form of the delay determination system in embodiment of this invention. It is a screen composition image figure in a normal time of the mobile communication application in embodiment of this invention. It is a screen configuration image diagram of the mobile communication application in the case where the user 1 (self) is delayed in the embodiment of the present invention. It is a screen configuration image diagram of the mobile communication application in the case where the user 2 (the other party of communication) is delayed in the embodiment of the present invention. It is a screen composition image figure in a normal time of the PC communication application in embodiment of this invention. It is a screen configuration image diagram of the PC communication application in the case where the user 1 (self) is delayed in the embodiment of the present invention. It is a screen configuration image diagram of the PC communication application in the case where the user 2 (the other side of communication) is delayed in the embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 shows a mobile communication application (hereinafter referred to as a mobile application), a PC communication application (hereinafter referred to as a PC application) that communicates with each other and displays a notification based on a delay judgment by converting voice into text, and a communication system that controls communication in the cloud. It is a figure which shows an example of the whole structure of the server, the delay determination system server which performs delay determination, and the SpeechToText system server which converts voice into character text. In this embodiment, it is assumed that it is configured on the cloud, but it does not have to be configured on the cloud.
The users are the speaker side and the listener side who perform communication.

For convenience of explanation, the speaker side who executes communication will be referred to as user 1, and the listener side who receives communication on the speaker side will be referred to as user 2.
In this explanation, the speaker side and the listener side are used for convenience, but it is of course possible to exchange the positions of both parties for communication.

The user 1 who communicates can use the mobile application 100 on the smartphone 101 or the tablet terminal 102 by installing the mobile application 100 on the smartphone 101 or the tablet terminal 102.

The mobile application 100 is connected to the communication system server 121 of the cloud system 120, the delay determination system server 122, the AI / DL SpeechToText system server 123, and the PC communication application on the client PC of the user 2 via the Internet 130. It has become.

The user 2 who communicates includes the PC application 110 and the client PC 111. The PC application 110 operates on the PC by installing it on the client PC 111. The PC application is connected to the communication system server 121 of the cloud system 120, the delay determination system server 122, the AI / DL SpeechToText system server 123, and the mobile application on the smartphone 101 or tablet terminal 102 of the user 1 via the Internet 130. It has a structure of

In the above embodiment, the user 1 uses the mobile application and the user 2 uses the PC application, but in the communication, the speaker side user and the listener side user can be reversed. Is. Further, even if the combination is between mobile applications or between PC applications, the same mechanism works. Further, although an example of one-to-one communication is described, it is possible to operate by the same mechanism even in communication by a plurality of people.

The cloud system includes a communication system server 121, a delay determination system server 122, and an AI / DL SpeechToText system server 123. Each server is configured to be connected via the Internet 130.

The data 140 includes voice data 141, voice text data 142, and delay determination result data 143. Audio data is audio streaming data input from an input device. The voice text data is obtained by converting the voice data so that it can be handled as text data. The delay determination result data is data as a result of comparing voice text data and time stamps of the data and performing delay determination, and is data stored in the delay determination management table 920.

The above is an explanation of transmitting audio as communication, but it is an example of a general communication system that can exchange video together with audio.
This is the end of the description of FIG.

Next, with reference to FIG. 2, an example of a hardware configuration applicable to the client terminal according to the embodiment of the present invention will be described.

In FIG. 2, 201 is a CPU that comprehensively controls each device and controller connected to the system bus 204. Further, the ROM 202 or the external memory 211 is necessary to realize the functions executed by the BIOS (Basic Input / Output System) and the operating system (hereinafter, OS), which are the control programs of the CPU 201, and each server or each PC. Various programs described later are stored.

Reference numeral 203 denotes a RAM, which functions as a main memory, a work area, and the like of the CPU 201. The CPU 201 realizes various operations by loading a program or the like necessary for executing a process from the ROM 202 or the external memory 211 into the RAM 203 and executing the loaded program.

Further, 205 is an input controller, which controls the input from the input device 209. Examples of the input device 209 include a keyboard, a touch panel, a pointing device such as a mouse, and the like.

When the input device 209 is a touch panel, the user can give various instructions by pressing (touching with a finger or the like) the icon, the cursor, or the button displayed on the touch panel.

Further, the touch panel may be a touch panel such as a multi-touch screen that can detect a position touched by a plurality of fingers.

Reference numeral 206 denotes a video controller, which controls the display on a display such as a display 210. The display 210 refers to a display device such as a CRT or a liquid crystal display. Alternatively, it may be another display.

It should be noted that the display of the notebook personal computer integrated with the main body is also included. The external output device is not limited to the display, and may be, for example, a projector.

Further, when the device to which the hardware configuration of FIG. 2 is applied is a device capable of accepting the above-mentioned touch operation, the display 210 provides a function as an input device 209.

The 207 is a memory controller, which is connected to a hard disk (HDD) for storing boot programs, various applications, font data, user files, edit files, various data, etc., a flexible disk (FD), or a PCMCIA card slot via an adapter. It controls access to an external memory 211 such as a compact flash (registered trademark) memory.

Reference numeral 208 denotes a communication I / F controller, which connects and communicates with an external device via a network, and executes communication control processing on the network. For example, communication using TCP / IP is possible.

The CPU 201 enables display on the display 210 by, for example, executing an outline font expansion (rasterization) process on the display information area in the RAM 203. Further, the CPU 201 enables a user instruction with a mouse cursor or the like (not shown) on the display 210.

Various programs described later for realizing the present invention are recorded in the ROM 202 or the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as needed.

Further, definition files, various information, tables, etc. used when executing the above program are also stored in the external memory 211, and detailed explanations thereof will be described later.

The camera controller 212 controls video acquisition by the camera 214 and the like, and the audio controller 213 controls audio acquisition control by the microphone / speaker 215 and the like and output to the speaker.

The camera 214 is a camera built in the device to which the hardware configuration of FIG. 2 is applied. Further, the microphone / speaker 215 is a microphone and a speaker built in the device to which the hardware configuration of FIG. 2 is applied. This is the end of the description of FIG.

Next, with reference to FIG. 3, a voice transmission / reception process in a general communication system such as a Web-RTC system will be described. A brief explanation is given below.

In step S301, the utterance by the speaker is detected.

In step S302, the utterance content detected in step S301 is acquired as voice data.

In step S303, the mobile application transmits the voice data acquired by the microphone to the Web-RTC system on the cloud as Web-RTC voice data.

In step S304, the Web-RTC system on the cloud receives the voice data of the Web-RTC from the mobile application and relays the voice data of the Web-RTC to the PC application on the listener side.

In step S305, the PC application outputs the voice data received from the Web-RTC system from the output device of the PC 111 such as a speaker and an earphone.

In step S306, the listener side can hear the sound emitted by the speaker side by hearing the sound from the output device such as a speaker or earphone.

In the above series of processing, voice delay or deterioration occurs due to various factors such as bandwidth status, so that the voice input to the microphone by the speaker side is output when it is output by the output device on the listener side. It flies, and as shown in FIG. 309, the output is in the form of flying in places.

If the band is normal, at 307, the speaker points to the microphone and says, "Hello, I will work at the site in Tokyo from 14:00." Then, like 308, the speaker says "Hello, from 14:00." I will work at the site in Tokyo. "

However, if the band is poor, there will be inaudible parts or skipped parts from the speaker, such as "Hello, I will work in the talk room from 1 o'clock."
The present application is intended to enable smooth communication in the above-mentioned state.

This is the end of the description of FIG.

FIG. 4 is a diagram showing the overall flow of voice text conversion, delay judgment using voice text data 142, and notification processing of delay judgment results in the embodiment of the present invention.

In FIG. 4, the mobile application and the PC application are collectively described as a client application. Further, here, the smartphone and the PC will be collectively described as a client terminal. Hereinafter, the processing flow of FIG. 4 will be described.

In step S401, the client application receives a start instruction of the client application through the user's client terminal and activates the client application.

In step S402, the login instruction is received for the client application started in step S401, and general multi-tenant login processing is performed.

Subsequently, in step S403, the voice data 141 input from the client terminal or a voice input device such as a microphone connected to the client terminal is acquired.

In step S404, an instruction for starting communication is received from the client terminal, and communication start processing of a general communication system is performed.

In step S405, a communication start request is made to the communication system. In the communication start request, API key, connection destination ID, and video / audio data are requested as request data. The API key is text data that is a key for using the communication system as an API. The connection destination ID is an ID necessary for connecting the communication partners. It is necessary to specify the same connection destination ID between the parties who want to communicate with each other.

In step S406, the request in step S405 is approved and the communication process is started.

In step S407, when communication is started in step S406, the communication processing loop is entered.

In step S408, when the communication processing loop is entered in step S407, the process of transmitting the local voice data of the user 1 (own), the process of receiving the voice data from the user 2 (the other party) and outputting it to the user, and the delay. The three processes of receiving the notification from the judgment system are executed in parallel. The detailed processing of step S408 will be described with reference to FIG.

Subsequently, the detailed processing of step S408 will be described with reference to FIG. In the process of FIG. 5, the process of transmitting the local voice data of the user 1 itself, the process of receiving the voice data from the other party and outputting it to the user, and the process of receiving the notification from the delay determination system are performed in parallel. Execute as loop processing. The reason why each process is performed in parallel is that communication may cause a plurality of events in parallel, and real-time communication is realized by processing those events in parallel.

When the communication processing loop is entered in step S407, the video / audio data of the communication partner is received from the communication system in step S501.

In step S502, the received voice data of the communication partner is input to the delay determination registration process.

In step S503, the delay determination registration process is performed on the voice data input in step S502. In the delay judgment registration process, the transmitted / received voice data is judged as the continuity of the voice or the delimiter of the meaning as a sentence, and is processed as the voice data of the sentence delimiter. The details of the processing performed in step S503 will be described with reference to FIG.

Subsequently, the delay determination registration process in step S503 will be described with reference to FIG.
In the delay determination registration process, the continuity of the voice of the transmitted / received voice data and the delimiter of the meaning as a sentence are determined, and the process is performed to make the voice data of the sentence delimiter. This process is performed to perform the delay determination process shown in FIG.

In step S601, the voice data is divided into sentences. This is because the voice data is separated into sentences and processed in natural language. In addition, this processing is also performed in order to facilitate the text conversion processing by making the voice data into a delimited file.

In step S602, the loop processing of the speech sentence delimiter processing is started.

In step S603, the sentence-delimited voice data is transmitted to the SpeechToText system server 123. The process of step S603 is performed in order to transmit the sentence-delimited voice data to the SpeechToText system and obtain the voice text data obtained by converting the voice data into text.

At that time, in step S603, API key and voice data are requested as request data.

In step S604, the voice text data of the reception result from the SpeechToText system is transmitted to the delay determination system. The process of step S604 is performed in order to cause the delay determination system to analyze the voice data by transmitting the voice text data obtained by converting the voice data into text to the delay determination system. In step S604, the communication ID, the voice text data, the registrant, the information on the transmitting side or the receiving side, the speaker, the speaking time, and the registration time are requested as the request data.

SpeechToText is a known technique, which is a technique of obtaining the characteristics of speech data from the acquired speech data, performing language processing analysis, and transcribing the contents of the speech data as text.

When the processes of S63 and S604 are completed for all the voice data delimited by the sentence in step S601, the delay determination registration process shown in FIG. 6 is completed.

When the processing up to step S503 is completed, the processing during communication is terminated and the process proceeds to step S409.

In step S511, the audio data of the user 1 (self) input data is acquired from the video / audio device.

In step S512, voice data is input to the delay determination registration process.

In step S513, the delay determination registration process is performed on the voice data input in step S512. This is the same process as the process performed in step S503 described above.
When the processing up to step S513 is completed, the processing during communication is terminated and the process proceeds to step S409.

In step S521, reception is awaited until a delay notification arrives from the delay determination system. In the process of receiving the notification from the delay determination system in step S521, the user is always on standby in a state where the notification can be received from the delay determination system during communication.

In step S522, it is determined whether the delay notification has arrived. When the delay notification arrives (when the determination result is Yes), the process proceeds to step S523. If the delay notification has not arrived (if the determination result is No), the process returns to step S521 and waits until the notification arrives.

In step S523, when the delay notification is received, the client application displays the notification content on the screen on the client terminal and notifies the user of the delay. As a result, the user can understand how much the voice is actually delayed or deteriorated and transmitted to the other party, and communication with the other party can be smoothly performed by communicating in consideration of the delay or deterioration. can

When the processing is completed up to step S523, the processing during communication is terminated and the process proceeds to step S409.

In step S409, a logout instruction is accepted.

In step S410, the application receives an end instruction.

This is the end of the explanation of the flowchart of the process during communication in FIG.

Subsequently, the delay determination acceptance notification will be described with reference to FIG. 7. FIG. 7 describes an overall operation flow for delay determination of the delay determination system.

In step S701, the delay determination system accepts the delay determination registration request transmitted from the client application in step S604.

In step S702, the received delay determination registration content is stored in the voice text management table 900 shown in FIG. The voice text management table 900 includes a text ID 901 which is a delay determination registration content, a communication ID, a registrant 903 who is a registrant of voice text, a transmission / reception 904 which manages whether to send or receive, and a voice speaker. It is a table that manages a voice speaker 905 that manages who he / she is, a voice speaker 907 that manages the time that he / she was speaking, and a text data 908 that manages voice text data.

In step S703, it is determined whether the received delay determination registration content is the transmission side or the reception side based on the content registered in the voice text management table. If the received registration content is on the transmitting side, the process proceeds to step S704, a registration success response indicating that the registration of the delay determination is completed is transmitted to the client, and this process is terminated.

In the process of step S703, it is necessary to prepare data on both the transmitting side and the receiving side in order to perform the delay determination. Since the sender arrives first in theory, the delay determination is performed at the time of registration of the receiver, so it is determined which registration content is used. If the registered content is the receiving side, the process proceeds to step S705 and the delay determination process is performed.

The delay determination process performed in step S705 will be described with reference to FIG. As described above, since the delay determination is performed at the time of registration on the receiving side, it is performed at the timing of step S705.

In step S801, data candidates registered by the sender corresponding to the voice text data included in the registered contents are searched from the voice text management table 900. The conditions for searching are that the person with communication ID 902 and the voice speaker 905 are the same person, and the speaking time and registration time are within 60 seconds compared to the registered contents of the request data. ..

This condition is because when the communication partner repeats the same content or repeats the same content that was spoken a while ago, it is not judged that the repeated content or the repeatedly spoken content is delayed. It is adopted in. Of course, the search conditions may be changed as appropriate according to the actual operation situation.

In step S802, the data registered by the sender in the voice text management table 900 is specified by performing a full-text search on the candidate voice text data.

Specifically, the voice text data of the candidate hit as a result of the search is subjected to a full-text search by a search engine using the voice text data on the receiving side as a key, and the voice text data on the transmitting side is searched. By this process, the registered data on the transmitting side registered in advance in the voice text management table 900 is specified.

In step S803, the communication delay is determined from the difference in the speaking time between the registered data on the transmitting side and the registered data on the receiving side.

Specifically, the speech time of the registered data on the transmitting side is compared with the speech time of the receiving side included in the request from the client, and the communication delay is determined from the difference. If the difference is less than 1 second, it is judged that there is no delay, if it is within 1 to 5 seconds, it is judged that the delay is small, if it is 6 to 10 seconds or more, it is judged that there is a delay, and if it is 10 seconds or more, it is judged that there is a large delay. The result of this delay level evaluation is displayed as a pop-up on each client's terminal. For example, 1101 shown in FIG. 11, 1201 shown in FIG. 12, 1401 shown in FIG. 14, 1501 shown in FIG. 15, and the like are displayed.

In step S804, deterioration of communication is determined from the difference between the remark contents of the registered data on the transmitting side and the registered data on the receiving side.

Specifically, the voice text data of the registered data on the transmitting side and the voice text data on the receiving side included in the request from the client are compared, and the deterioration of communication is judged from the degree of matching. When the degree of agreement is 95% or more, it is judged that there is no deterioration, when it is 90 to 95%, it is judged that the deterioration is small, when it is 80 to 90%, it is judged to be deteriorated, and when it is lower than 80%, it is judged that the deterioration is large. The result of this deterioration level evaluation is displayed as a pop-up on each client's terminal. For example, 1101 shown in FIG. 11, 1201 shown in FIG. 12, 1401 shown in FIG. 14, 1501 shown in FIG. 15, and the like are displayed.

In step S805, the delay judgment and the deterioration judgment, which are the analysis results, are stored in the analysis result (result) 923 and the analysis result (deterioration) 924 of the delay judgment management storage table 920 shown in FIG. 9, respectively.

The delay judgment management table 920 manages the communication ID 921 that manages the communication ID, the voice speaker 922 that manages the voice speaker, the analysis result (delay) 923 that manages the analysis result of the delay judgment, and the analysis result of the deterioration judgment. It is a table which manages the analysis result (deterioration) 924, the update date and time 925 which manages the updated date and time, and the shooting date and time 926 which manages the shooting date and time.

In step S806, when it is determined that the delay or deterioration has occurred, the client application is notified that the delay or deterioration has occurred. The data of the same communication ID as the data in which the delay or deterioration has occurred is searched from the voice text management table 900, and a notification event including the content of the delay or deterioration is transmitted to all the registrants of the hit data.
If it is a mobile application, it is displayed as shown in FIGS. 11 and 12, and if it is a PC application, it is displayed as shown in FIGS. 14 and 15.
The above is the content of the delay determination process performed in FIG.

When the delay determination process in step S705 is completed, the process proceeds to step S706.
In step S706, a response of successful registration is sent to the client.
This completes the processing of the delay determination acceptance notification shown in FIG. 7.

Subsequently, with reference to FIG. 10, a communication screen configuration image of a normal mobile application will be described. The normal communication screen 1000 is when the camera is operated and the room is entered.

The icon 1001 indicates the number of users who are in the room using a smartphone.

The icon of 1002 indicates the number of users who are in the room by using the PC.

The icon of 1003 is a shutter button of the camera and an operation unit for instructing processing such as start of video shooting and end of video shooting.

The icon of 1004 is an icon indicating the camera mode. When the camera icon of 1004 is colored in yellow, it indicates that the image is taken in the camera mode.

The icon of 1005 is an icon indicating a video mode. If the 1005 video icon is colored yellow, it indicates that it was shot in video mode.

Reference numeral 1006 is a display that allows the user to confirm which room the user is currently in.
In this case, it can be seen that the room A2 is in the room. This is the end of the description of FIG.

With reference to FIG. 11, a screen configuration image of the communication screen of the mobile application when the user 1 (self) is delayed will be described.

When the communication is delayed or deteriorated in the user 1 (self), a pop-up such as 1101 is superimposed and displayed on the normal communication screen.

Pop-up 1101 notifies that a delay has occurred. Then, the delay level and the deterioration level are displayed. In addition, the other party of communication notifies that it is difficult to hear the voice of user 1 (self).

As a result, the user 1 (self) can instantly grasp that the delay or deterioration of the audio data and the video data is currently occurring during the communication with the other party.

In addition, the user (self) can understand how much the voice is actually delayed or deteriorated, and can smoothly communicate with the other party by communicating in consideration of the delay or deterioration. can. This is the end of the description of FIG.

Subsequently, with reference to FIG. 12, a screen configuration image of the communication screen of the mobile application when the user 2 (communication partner) is delayed will be described.

When the communication is delayed or deteriorated in the user 2 (communication partner), a pop-up such as 1201 is superimposed and displayed on the normal communication screen.

Pop-up 1201 notifies that a delay has occurred. Then, the delay level is displayed. "Small" is displayed in the pop-up of 1201.

When the other party is delayed, the deterioration of the voice is actually heard and known, so that the pop-up 1201 does not need to display the deterioration level.

In addition, a notification that the user 1 who is the communication partner of the user 2 (communication partner) may not be aware of the deterioration and a message prompting the user to carry out the conversation in consideration of the delay are displayed.

As a result, the user 2 can instantly grasp that the delay of the audio data and the video data is currently occurring between the communication with the other party.

In addition, by communicating in consideration of delay, it is possible to smoothly communicate with the other party. This is the end of the description of FIG.

Subsequently, with reference to FIG. 13, a screen configuration image of a communication screen of a normal PC application will be described. 1300 is the screen of the entire communication of the PC application.

At 1301, it is possible to select a room for communication.

The 1302 is a screen that can be enlarged and viewed by the user by selecting the screen.
Of course, the image displayed on the screen of 1302 may be switched at a fixed timing even if the user does not select it.

1303, 1304, 1305, and 1306 are areas for displaying images sent from each user participating in the room. In each area, the user name is displayed so that the video sent from the user and the video from which user can be understood.

When the user who operates on the PC selects one of the areas 1303, 1304, 1305, and 1306 on the right side of the screen displaying the image sent by the user participating in the room, the user selects it. The image displayed in the area is enlarged and displayed on the screen of 1302.

In the example of FIG. 13, since the user has selected 1305, the image projected on the area of 1305 is enlarged and displayed on 1302. When the user selects 1305, the image displayed in 1305 is moved to 1302, and a message indicates that the image is displayed. In this case, it is displayed as VIEW.

Reference numeral 1307 is an icon for selecting a camera mode. Then, when the camera mode is selected, the image displayed on the screen can be taken as a screenshot, or the picture can be taken by using the camera built in the PC or connected from the outside.

1308 is an icon for selecting a video mode. Select the video mode and the video will start shooting. It is also possible to shoot the screen itself as an image, and it is also possible to take an image including the user's face and body by using a built-in or externally connected camera. This enables communication even with gestures.

1309 is a photo list icon. You can check the photos you have taken by pressing the photo list icon.

1310 is a screen capture sharing icon. By pressing the screen capture sharing icon, the screen captured by the user can be shared with other users who participate in the room. This is the end of the description of FIG.

FIG. 14 is a communication screen of the PC application when the user 1 (self) is delayed.

In a pop-up format such as 1401, a notification that a delay has occurred and a delay level and a deterioration level are displayed to notify that the voice is difficult to hear.

As a result, it is possible to instantly grasp that the delay of the audio data and the video data is currently occurring between the communication with the other party. This is the end of the description of FIG.

FIG. 15 is a communication screen when user 2 (communication partner) is delayed.

If there is a communication partner with a delay, a notification bar such as 1500 is displayed to notify that the delay has occurred. Mouse over the notification bar 1500 to see a pop-up image like 1501.

As with the mobile app, you can actually hear the voice to see if the other party is delayed, so the deterioration level is not displayed. This is the end of the description of FIG.

Although the present invention has been described above, according to the present invention, the voice data of the calling side and the called side are converted into text, and after comparing the results of the voice data of both converted into text, data delay or deterioration due to the network or Web-RTC is performed. Can be determined and notified to the user.

As a result, in communication in which delay or deterioration occurs, users can communicate while grasping and considering each other's delay or deterioration situation, so that the user's annoyance can be reduced.

Conventionally, the user 1 (self) cannot determine whether or not his / her voice is transmitted to the other party, and even if his / her voice is transmitted, how much delay occurs when there is a delay. I couldn't tell if I was doing it. However, by notifying the user of the degree of delay or deterioration according to the present invention, the person who speaks can speak slowly, divide sentences, and so on. In addition, since it is possible to make a statement while assuming that there is a delay, it is less likely that the statement will be incurred.

Although the embodiments of the present application have been described above, the present invention can be implemented as, for example, a system, an apparatus, a method, a program, a recording medium, or the like. Specifically, it may be applied to a system composed of a plurality of devices, or may be applied to a device composed of one device.

Further, the program in the present invention is a program in which a computer can execute the processing method of the flowchart shown in FIG. 4, and the storage medium of the present invention stores a program in which the computer can execute the processing method in FIG. The program in the present invention may be a program for each processing method of each device of FIG.

As described above, a recording medium recording a program that realizes the functions of the above-described embodiment is supplied to the system or device, and the computer (or CPU or MPU) of the system or device stores the program in the recording medium. Needless to say, the object of the present invention can be achieved by reading and executing.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium on which the program is recorded constitutes the present invention.

Recording media for supplying programs include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon. A disc or the like can be used.

Further, by executing the program read by the computer, not only the function of the above-described embodiment is realized, but also the OS (operating system) or the like running on the computer is actually realized based on the instruction of the program. Needless to say, there are cases where a part or all of the processing is performed and the processing realizes the functions of the above-described embodiment.

Further, after the program read from the recording medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the function expansion board is based on the instruction of the program code. It goes without saying that there are cases where the CPU or the like provided in the function expansion unit performs a part or all of the actual processing, and the processing realizes the functions of the above-described embodiment.

Further, the present invention may be applied to a system composed of a plurality of devices or a device composed of one device. Needless to say, the present invention can also be applied when it is achieved by supplying a program to a system or an apparatus. In this case, by reading the recording medium containing the program for achieving the present invention into the system or the device, the system or the device can enjoy the effect of the present invention.

Further, by downloading and reading a program for achieving the present invention from a server, database, or the like on a network by a communication program, the system or device can enjoy the effect of the present invention. It should be noted that the present invention also includes all the configurations in which each of the above-described embodiments and modifications thereof are combined.

100 Mobile Communication Application 110 PC Communication Application 120 Cloud System 130 Internet

Claims (8)

An information processing device with a communication screen
An input means that accepts voice data input and
A means for creating text of voice data received by the input means, and a means for creating text.
A comparison means for comparing voice text data on the transmitting side and the receiving side created by the creating means, and a comparison means.
A determination means for determining whether or not delay and deterioration have occurred based on the comparison result of the comparison means.
An information processing device characterized by being equipped with. The information processing device is
A notification means that gives a notification corresponding to the determination result by the determination means, and
The information processing apparatus according to claim 1, wherein the information processing apparatus is provided. The determination means according to claim 1 or 2, wherein the determination means determines whether delay and deterioration have occurred by comparing the degree of matching between the voice text data on the transmitting side and the voice text data on the receiving side. Information processing device. The information processing according to claim 1 to 3, wherein the determination means determines the degree of delay and the degree of deterioration according to the degree of coincidence between the voice text data on the transmitting side and the voice text data on the receiving side. Device. The information processing apparatus according to claim 1, wherein the notification means gives a notification corresponding to the degree of delay and the degree of deterioration determined by the determination means. The information processing apparatus according to claim 1, wherein the notification means superimposes and displays on the communication screen that a delay or deterioration has occurred. It is a control method of an information processing device having a communication screen.
The information processing device is
An input step that accepts voice data input and
The creation step of converting the voice data that received the input by the input step into text, and
A comparison step that compares the voice text data on the transmitting side and the receiving side created by the creation step, and
A determination step for determining whether or not delay and deterioration have occurred based on the comparison result in the comparison step, and a determination step.
An information processing device characterized by executing. An information processing device with a communication screen,
An input means that accepts voice data input and
A means for creating text of voice data received by the input means, and a means for creating text.
A comparison means for comparing voice text data on the transmitting side and the receiving side created by the creating means, and a comparison means.
A determination means for determining whether or not delay and deterioration have occurred based on the comparison result of the comparison means.
A program to make it work.

Images