JP2023081469A - Program, apparatus, method, and system - Google Patents

Abstract

To easily grasp a state of voice of a call partner.SOLUTION: A program for operating a terminal device 10 that comprises a processor 19 and a memory 15 and can transmit/receive voice data to/from another terminal device 10 makes a processor 29 execute a step of receiving voice data transmitted from the other terminal device 10, a step of evaluating a state of the received voice data and outputting it as a first evaluation result, a step of returning the first evaluation result to the other terminal device 10, and a step of outputting voice based on the received voice data to a user.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to programs, devices, methods and systems.

When talking with a web conference system or voice communication application, the speaker's voice may not reach the other party, or the voice may be difficult to hear. This may be caused by speech input problems on the speaking side, communication problems on the speaking side, and/or communication problems on the receiving side. When the voice is difficult to hear due to the above reasons, it is difficult for the speaker himself/herself to notice it, and he/she does not notice it until the other party points it out.

Techniques related to the above-described problem are disclosed, for example, in Patent Documents 1 and 2.

US Pat. No. 5,900,009 discloses a system and method for automatically verifying that messages received from users are understandable. As an example, the system computes an intelligibility estimate of the input speech, compares it to an intelligibility threshold, and determines that the computed intelligibility estimate is below the intelligibility threshold. If so, the user is prompted to repeat at least part of the message.

Also, US Pat. No. 6,200,003 discloses a voice control apparatus that indicates to the user when voice input is likely not to be processed by the device due to poor signal-to-noise ratio.

U.S. Pat. No. 7,660,716 U.S. Pat. No. 9,558,758

In a situation where multiple users are making a voice call, at least the state of the voice when the user who speaks is input to the terminal and the state of the voice when the voice reaches the other party are the same. Not necessarily. Therefore, a user who is talking with a plurality of people cannot easily know whether there is a problem with the voice state of the user himself or the other party.

Accordingly, the present disclosure has been made to solve the above problems, and its object is to easily grasp the state of the voice of the other party.

A program for operating a terminal device having a processor and a memory and capable of transmitting and receiving voice data to and from another terminal device. The program causes the processor to receive voice data transmitted from another terminal device, evaluate the state of the received voice data and output it as a first evaluation result, and transmit the first evaluation result to the processor. and outputting voice based on the received voice data to the user.

According to the present disclosure, it is possible to easily grasp the state of the voice of the other party.

It is a figure showing composition of the whole system of an embodiment. It is a figure which shows the functional structure of the terminal device of embodiment. It is a block diagram showing the example of functional composition of the voice judgment part of an embodiment, and a presentation control part. It is a figure which shows the functional structure of the server of embodiment. It is a figure which shows the data structure of the database of embodiment. It is a figure which shows the data structure of the database of embodiment. It is a flowchart which shows an example of the processing flow in the system of embodiment. 8 is a flow chart showing another example of the processing flow in the system of the embodiment; 8 is a flow chart showing another example of the processing flow in the system of the embodiment; It is a sequence diagram showing an example of a processing flow in the system of the embodiment. It is a schematic diagram showing an example of the screen displayed on the terminal device of the embodiment. It is a schematic diagram showing another example of the screen displayed on the terminal device of the embodiment. It is a schematic diagram showing an example of the screen displayed on the terminal device of the embodiment. It is a schematic diagram showing another example of the screen displayed on the terminal device of the embodiment. It is a schematic diagram showing another example of the screen displayed on the terminal device of the embodiment.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all the drawings for explaining the embodiments, common constituent elements are denoted by the same reference numerals, and repeated explanations are omitted. It should be noted that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Also, not all the components shown in the embodiments are essential components of the present disclosure. Each figure is a schematic diagram and is not necessarily strictly illustrated.

Also, in the following description, a "processor" is one or more processors. The at least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core.

Also, at least one processor may be a broadly defined processor such as a hardware circuit (for example, FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)) that performs part or all of the processing.

In the following explanation, the expression "xxx table" may be used to describe information that produces an output for an input. It may be a learning model such as a generated neural network. Therefore, the "xxx table" can be called "xxx information".

Also, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table. good.

Further, in the following description, the processing may be described using the term “program” as the subject. As it is used, the subject of processing may be a processor (or a device, such as a controller, having that processor).

The program may be installed in a device such as a computer, or may be, for example, in a program distribution server or a computer-readable (eg, non-temporary) recording medium. Also, in the following description, two or more programs may be implemented as one program, and one program may be implemented as two or more programs.

In the following description, identification numbers are used as identification information for various objects, but identification information of types other than identification numbers (for example, identifiers including alphabetic characters and symbols) may be employed.

In addition, in the following description, when describing the same type of elements without distinguishing between them, reference symbols (or common symbols among the reference symbols) are used, and when describing the same types of elements with different An identification number (or reference sign) may be used.

Also, in the following description, control lines and information lines indicate those considered necessary for the description, and not all control lines and information lines are necessarily indicated on the product. All configurations may be interconnected.

<0 System Overview>
In the system of the present disclosure, as an example, between terminal devices such as smartphones and tablet terminals that are capable of mutual communication using a network such as a mobile communication network, how the voice uttered by the user of the terminal device is expressed in the terminal device of the other party. In other words, the voice state of the other party can be easily grasped. For this reason, the terminal device receives voice data transmitted from another terminal device that is the other party of the call, evaluates the state of the received voice data, outputs it as a first evaluation result, and outputs this first evaluation result. to the other terminal device.

Also, in the system of the present disclosure, it is possible to easily grasp the state of the voice uttered by the user of the terminal device. For this reason, the terminal device evaluates the quality of voice data based on the voice uttered by the user, and presents the user with the third evaluation result, which is the quality evaluation result.

Furthermore, in the system of the present disclosure, the terminal device is used to evaluate the variation in voice power of the user during mutual communication, and the evaluation result of the variation is presented to the user. For this reason, a server that mediates calls between terminal devices detects the voice power of the input voice for each terminal device, calculates the average value and variance of the voice power based on the detection results, and calculates the calculated average value and variance. Based on the variance, it is determined whether or not the difference between the voice power of each speaker using a plurality of terminal devices and the average value is a predetermined value or more, and the utterances determined to have a difference of a predetermined value or more A fourth determination result is presented to the person.

<1 Configuration diagram of the entire system>
FIG. 1 is a diagram showing an example of the overall configuration of a system 1 according to an embodiment. As shown in FIG. 1, the system 1 includes a terminal device 10 and a server 20. As shown in FIG. The terminal device 10 and the server 20 are connected to communicate with each other via a network 80 using a wired or wireless communication standard (including a mobile communication standard). In the illustrated example, the system 1 includes a plurality of terminal devices 10 .

The network 80 is composed of various mobile communication systems constructed by the Internet, LAN, wireless base stations, and the like. For example, networks include 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), wireless networks (for example, Wi-Fi (registered trademark)) that can be connected to the Internet through predetermined access points, and the like. When connecting wirelessly, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), and the like. In the case of wired connection, the network includes direct connection using a USB (Universal Serial Bus) cable or the like.

Although FIG. 1 shows the case where the server 20 is one computer, the server 20 may be realized by combining a plurality of computers. Although FIG. 1 shows a case where there are three terminal devices 10, the number of terminal devices 10 accommodated in the system 1 is not limited, and may be two or four or more. .

The terminal devices 10 are terminals configured to be able to communicate with each other via the server 20 . Further, the terminal device 10 is a terminal capable of transmitting/receiving voice data to/from another terminal device 10, in other words, capable of mutual communication by voice spoken by the user of the terminal device 10. FIG. For example, the terminal device 10 is an information processing device such as a smart phone, a tablet terminal, etc., capable of mutual communication using a network such as a mobile communication network. Alternatively, the terminal device 10 may be an information processing device such as a stationary PC (Personal Computer), a laptop PC, a head-mounted display, etc., in which a predetermined voice conversation application is installed.

As shown in FIG. 1 , the terminal device 10 includes a communication IF (Interface) 12 , an input device 13 , an output device 14 , a memory 15 , a storage 16 and a processor 19 . The communication IF 12, the input device 13, the output device 14, the memory 15, the storage 16, and the processor 19 are communicably connected to each other via a bus, for example.

The communication IF 12 is an interface for transmitting and receiving signals including voice data so that the terminal device 10 can communicate with an external device, including voice communication. The input device 13 is an input device for receiving an input operation from a user. The input device 13 includes, for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, and the like. The output device 14 is an output device for presenting information to the user. The output device 14 includes, for example, a display and speakers.

The memory 15 temporarily stores programs, data processed by the programs, and the like, and is realized by, for example, a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage 16 is a storage device for storing data, and is realized by non-volatile memory such as flash memory and HDD (Hard Disc Drive). The processor 19 is hardware for executing an instruction set described in a program, and is composed of arithmetic units, registers, peripheral circuits, and the like.

The server 20 is an information processing device that manages information about services, refers to the managed information, and provides services. Furthermore, the server 20 mutually communicates between a plurality of terminal devices 10 (not only between two terminal devices 10, but also between these three if there are three terminal devices 10 as shown in FIG. 1). Provides a service that enables two-way communication by sending and receiving voice data. Server 20 is, for example, a computer connected to network 80 .

Note that the server 20 may provide a service for mutual data communication (including text-based messages) in addition to the service that enables mutual communication. Examples of such services include a service that allows access to sites on the Internet using the terminal device 10, and a service that transmits messages in text format, that is, a chat service.

As shown in FIG. 1, the server 20 includes a communication IF 22, an input/output IF 23, a memory 25, a storage 26, and a processor 29. The communication IF 22, the input/output IF 23, the memory 25, the storage 26, and the processor 29 are communicably connected to each other via a bus, for example.

The communication IF 22 is an interface for transmitting and receiving signals for the server 20 to communicate with an external device. The input/output IF 23 functions as an interface with an input device for receiving input operations from the user and an output device for presenting information to the user. The memory 25 is for temporarily storing programs, data processed by the programs, etc., and is realized by a volatile memory such as a DRAM, for example.

The storage 26 is a storage device for storing data, and is implemented by non-volatile memory such as flash memory and HDD. Storage 26 does not necessarily have to be realized by a single circuit. The storage 26 may be realized by, for example, multiple storage circuits. The processor 29 is hardware for executing an instruction set described in a program, and is composed of arithmetic units, registers, peripheral circuits, and the like.

<1.1 Configuration of terminal device>
FIG. 2 is a block diagram showing the configuration of the terminal device 10 included in the system 1 of this embodiment. As shown in FIG. 2 , the terminal device 10 includes a communication section 121 , an input device 13 , an output device 14 , a camera 160 , a storage section 170 and a control section 180 .

The communication unit 121 performs processing for the terminal device 10 to communicate with other devices. The communication unit 121 performs transmission processing on the signal generated by the control unit 180 and transmits the signal to the outside (for example, the server 20). Communication unit 121 performs reception processing on a signal received from the outside, and outputs the signal to control unit 180 .

In addition, the communication unit 121 performs processing for transmitting and receiving audio data to and from another terminal device 10 . Therefore, the communication unit 121 performs communication conforming to the standard of the mobile communication network. Primarily, the communication unit 121 communicates with the base station assigned to the area where the terminal device 10 exists, and the mobile communication system including the base station and the server 20 communicates between the terminal devices 10 Realize transmission and reception of voice data. The standard of the mobile communication network is the standard to which the services provided by the telecommunications carrier operating the mobile communication system comply with when the system 1 of the present embodiment is in operation. Examples of such standards include IMT-Advanced, which is the standard for the 4th generation mobile communication system, and IMT-2020, which is the standard for the 5th generation mobile communication system. Also, in areas where the third generation mobile communication system is still in operation, IMT-2000, which is the standard for this third generation mobile communication system, may be used. Since the mobile communication standards listed above are applicable to both voice communication and data communication, the communication unit 121 can perform both voice communication and data communication.

The input device 13 is a device for the user who owns the terminal device 10 to input instructions. The input device 13 is realized by, for example, a mouse 131, a keyboard 132, and a touch sensitive device 133 that inputs instructions by touching an operation surface. Also, the input device 13 is implemented by a microphone 134 . The input device 13 converts an instruction or voice input by the user into an electrical signal and outputs the electrical signal to the control unit 180 . Note that the input device 13 is not limited to physical operation devices such as the mouse 131 and keyboard 132 . The input device 13 may include, for example, a reception port that receives an electrical signal input from an external input device. Furthermore, depending on the specifications of the terminal device 10, the mouse 131 and keyboard 132 may be omitted. In this case, the touch-sensitive device 133 is used to input various texts and the like.

The output device 14 is a device for presenting information to the user who owns the terminal device 10 . The output device 14 is realized by, for example, a display 141, a speaker 142, and the like. The display 141 displays data such as images, moving images, and text under the control of the control unit 180 . The display 141 is implemented by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. Speaker 142 outputs sound under the control of control unit 180 .

In FIG. 2, the microphone 134 and/or the speaker 142 of the terminal device 10 include those that can be used separately from other components of the terminal device 10 by wireless communication means such as Bluetooth (registered trademark). . For example, the microphone 134 and speaker 142 are configured separately as a so-called headset.

The camera 160 is a device for receiving light with a light receiving element and outputting it as a photographing signal. The camera 160 has, for example, a lens for receiving light arranged side by side with the display 141 . This allows the camera 160 to capture an image of the user viewing the display 141 .

The storage unit 170 is realized by, for example, the memory 15 and the storage 16, and stores data and programs used by the terminal device 10. FIG. Specifically, the storage unit 170 stores an application 171, for example. The application 171 is executed by the control unit 180 to implement a function implementation unit such as an operation receiving unit 181 in the control unit 180, which will be described later. The storage unit 170 also stores evaluation results by the voice determination unit 185 , detection result data 172 storing the determination results, and an evaluation table 173 that serves as criteria for determination by the voice determination unit 185 .

Control unit 180 is implemented by processor 19 reading a program (application 171) stored in storage unit 170 and executing instructions included in the program. The control unit 180 controls operations of the terminal device 10 . Specifically, for example, the control unit 180 functions as an operation reception unit 181 , a communication control unit 182 , an audio input unit 183 , an audio output unit 184 , an audio determination unit 185 and a presentation control unit 186 .

The operation accepting unit 181 performs processing for accepting a user's operation input from the input device 13 .

The communication control unit 182 performs processing for the terminal device 10 to communicate with another terminal device 10 that is in mutual communication (including mutual communication) according to a communication protocol. The communication control unit 182 transmits, for example, voice data obtained by converting the voice input from the microphone 134 by the voice input unit 183 to the other terminal device 10 in mutual communication. The communication control unit 182 also receives audio data transmitted from another terminal device 10 , sends the audio data to the audio output unit 184 , converts the data into audio by the audio output unit 184 , and outputs the data from the speaker 142 .

Also, the communication control unit 182 transmits text data input from, for example, the keyboard 132, the touch sensitive device 133, etc. to the other terminal devices 10 that are in mutual communication. The communication control unit 182 also receives data transmitted from another terminal device 10 , converts it into text data, and outputs the text data from the display 141 .

The voice input unit 183 converts the voice uttered by the user of the terminal device 10 and input from the microphone 174 into voice data, and sends the voice data to the communication control unit 182 . The audio output unit 184 converts audio data output by the communication control unit 182 into audio, and outputs the audio from the speaker 142 .

The audio determination unit 185 performs various signal processing on the audio data output by the audio input unit 183 and the audio output unit 184, evaluates and determines the quality of the audio data, and outputs the evaluation results and determination results as detection results. Stored in data 172 . Details will be described later.

The presentation control unit 186 controls the output device 14 to present various information to the user. Specifically, for example, the presentation control unit 186 causes the display 141 to display the evaluation result by the audio determination unit 185 . Also, for example, the presentation control unit 186 converts the evaluation result into audio and sends it to the audio output unit 184 , and outputs the evaluation result as audio through the audio output unit 184 and the speaker 142 .

FIG. 3 is a block diagram showing an example of the functional configuration of the terminal device 10 including the communication control section 182, the voice determination section 185 and the presentation control section 186. As shown in FIG. The communication control section 182 has an encoding processing section 1821 , a transmitting section 1822 , a receiving section 1823 and a decoding processing section 1824 . Also, the voice determination section 185 has a voice power detection section 1851 , an SN ratio detection section 1852 , a microphone characteristic detection section 1853 , an input voice evaluation section 1854 and a received voice evaluation section 1855 . Furthermore, the presentation control unit 186 has an input voice state presenting unit 1861 , a caller receiving voice state presenting unit 1862 and a voice power state presenting unit 1863 .

The encoding processing unit 1821 compresses the audio data output from the audio input unit 183 by performing encoding processing (encoding processing) on the audio data, and sends the compressed data to the transmission unit 1822 . The encoding process complies with mobile communication standards. At this time, the encoding processing unit 1821 converts the encoded audio data into packets and assigns numbers to each packet. A series of processing by the encoding processing unit 1821 is processing known as VoLTE (Voice over Long Term Evolution) if mobile communication by the terminal device 10 is based on the LTE standard. It is also a process generally known as VoIP (Voice over Internet Protocol).

The transmission unit 1822 wirelessly transmits the audio data encoded by the encoding processing unit 1821 via an antenna (not shown) included in the transmission unit 1822 and sends the data to the transmission/reception unit 2031 of the server 20 . Wireless communication by the transmitting unit 1822 also complies with the mobile communication network standards already described.

The transmitting unit 1822 assigns a unique identifier to each terminal device 10 when wirelessly transmitting voice data. Further, the transmission unit 1822 generates the evaluation score of the received voice output by the received voice evaluation unit 1855, which is the first evaluation result, and the power detection result of the input voice detected by the voice power detection unit 1851, which is the third evaluation result. is superimposed on the voice data and transmitted to the terminal device 10 that is the other party.

Receiving section 1823 receives radio input transmitted from transmitting/receiving section 2031 of server 20 and received via an antenna (not shown) included in receiving section 1823, and transmits the received radio input to the decoding processing section. Here, the wireless input received by the receiving unit 1823 is provided with an identifier unique to the terminal device 10 of the other party of communication, which is provided by the transmitting unit 1822 of the terminal device 10 of the other party of communication. In addition, the receiving unit 1823 receives the second evaluation result, which is the evaluation score of the received voice superimposed on the wireless input received from the terminal device 10 of the other party, and transmits the second evaluation result to the other party. It is sent to the calling party reception voice state presenting unit 1862 together with an identifier unique to the terminal device 10 . Furthermore, the receiving unit 1823 determines that the voice power of the voice data uttered by the user of the terminal device 10, which is the fourth determination result sent from the server 20, is equal to the voice power of the plurality of terminal devices 10 currently engaged in a call. It receives the determination result that the average value of the voice power calculated from the above is equal to or greater than the predetermined value, and sends this fourth evaluation result to the voice power state presenting unit 1863 .

The decoding processing unit 1824 decodes the received data received by the receiving unit 1823 to audio data, and outputs the audio data to the audio output unit 184 . In addition, the decoding processing unit 1824 calculates the packet loss rate (packet loss rate) when decoding the signal sent from the terminal device 10 currently in communication, received by the receiving unit 1823, and calculates the calculated packet The loss rate is sent to received voice evaluation section 1855 . The method of calculating the packet loss rate is well known and may be specified in the communication standard, so detailed description will not be given here. At this time, the decoding processing unit 1824 adds the identifier of the terminal device 10 of the other party to the calculated packet loss rate and sends it to the received voice evaluation unit 1855 .

The voice power detection unit 1851 detects the voice power of the voice data obtained by converting the voice uttered by the user of the terminal device 10, who is the speaker, by the voice input unit 183, and outputs the detection result to the input voice evaluation unit 1854 and the transmission unit 1822. Output. At this time, the voice power detection unit 1851 quantizes the voice power detection result and outputs it as a score. The quantization method is not particularly limited, but in the voice power detection unit 1851 of this embodiment, the voice power detection result is given as a score of 0 if good, -1 if slightly bad, and -2 if bad. to output

The SN ratio detection unit 1852 detects the SN ratio of the voice data obtained by converting the voice uttered by the user of the terminal device 10 , who is the speaker, by the voice input unit 183 , and outputs the detection result to the input voice evaluation unit 1854 . At this time, the SN ratio detection unit 1852 quantizes the detection result of the SN ratio and outputs it as a score. The quantization method is not particularly limited, but in the SN ratio detection unit 1852 of the present embodiment, the SN ratio detection result is given as a score of 0 if good, -1 if slightly bad, and -2 if bad. to output The SN ratio detection unit 1852 may be implemented using a trained model that has been trained using the SN ratio as input data and the evaluation of the input SN ratio as correct output data.

The microphone characteristic detection unit 1853 detects the microphone characteristics of the voice input unit 183 (microphone 134) from the voice data obtained by converting the voice uttered by the user of the terminal device 10, who is the speaker, by the voice input unit 183, and inputs the detection result. Output to the speech evaluation unit 1854 . At this time, the microphone characteristic detection unit 1853 quantizes the microphone characteristic detection result and outputs it as a score. The quantization method is not particularly limited, but the microphone characteristic detection unit 1853 of the present embodiment gives the detection result of the microphone characteristic as a score of 0 if good, -1 if slightly bad, and -2 if bad. to output The microphone characteristic detection unit 1853 may be implemented using a trained model trained using microphone characteristics as input data and evaluation of the input microphone characteristics as correct output data.

Preferably, the voice power detection unit 1851, the SN ratio detection unit 1852, and the microphone characteristics detection unit 1853 perform the detection operation every time the user of the terminal device 10 pronounces, that is, every time voice is input to the voice input unit 183. I do. Alternatively, the voice power detection unit 1851, the SN ratio detection unit 1852, and the microphone characteristic detection unit 1853 may perform detection work in real time (that is, continuously), or may perform detection at predetermined time intervals (for example, every second). You can work.

Then, the voice power detection unit 1851 , the SN ratio detection unit 1852 and the microphone characteristics detection unit 1853 store detection results in the detection result data 172 . The detection result stored in the detection result data 172 is updated each time the detection operation by the voice power detection section 1851, the SN ratio detection section 1852, and the microphone characteristic detection section 1853 is performed. Therefore, the detection result stored in the detection result data 172 is updated each time a voice is input to the voice input unit 183 . When the voice input stops, the detection result data 172 is not updated (overwritten), so that the detection result data 172 retains the last detection value.

The voice power detection unit 1851, the SN ratio detection unit 1852, and the microphone characteristics detection unit 1853 quantize their own detection results into three types of scores of 0, -1, and -2, but the quantization level is also The quantization is not limited to these three stages, and may be quantized in two stages, or in four or more stages.

The input speech evaluation unit 1854 quantizes (scores) the detection results respectively input from the audio power detection unit 1851, the SN ratio detection unit 1852, and the microphone characteristics detection unit 1853 as evaluation values, determines an integrated score, and determines the input speech. It is output to the state presenting section 1861 and the other party's received voice state presenting section 1862 . The integrated score output from the input speech state presentation unit 1861 is the evaluation result of the quality of the speech data based on the speech uttered by the user of the terminal device 10, which is the third evaluation result.

Although there is no particular limitation on the calculation method of the integrated score by the input speech evaluation unit 1854, in the input speech evaluation unit 1854 of this embodiment, the The scores that are the quantized detection results are added, and the integrated score is determined based on the added value (sum). For example, if the sum of the scores input from the voice power detection unit 1851, the SN ratio detection unit 1852, and the microphone characteristic detection unit 1853 is 0, it is good, -1 is somewhat bad, and -2 is bad. and More specifically, the score input from the voice power detection unit 1851 is good (0), the score input from the SN ratio detection unit 1852 is good (0), and the score input from the microphone characteristics detection unit 1853 is good ( 0), the input speech evaluation unit 1854 determines that the sum of these scores is 0, so the integrated score is good (0). Also, the score input from the voice power detection unit 1851 is good (0), the score input from the SN ratio detection unit 1852 is slightly bad (-1), and the score input from the microphone characteristics detection unit 1853 is bad ( If it is -1), the input speech evaluation unit 1854 determines that the sum of these scores is -2, so the integrated score is bad (-2). Here, when the sum of the scores input from the voice power detection unit 1851, the SN ratio detection unit 1852, and the microphone characteristics detection unit 1853 is -2 or less, the input voice evaluation unit 1854 evaluates the integrated score as bad (-2 ) may be treated as As an example, if the scores input from the voice power detection unit 1851, the SN ratio detection unit 1852, and the microphone characteristics detection unit 1853 are all slightly bad (-1), the sum of these scores will be -3. The integrated score calculated by the input speech evaluation unit 1854 may be evaluated as bad (-2). Also, the quantization stage of the integrated score by the input speech evaluation unit 1854 is not limited to the three stages of 0, -1, and -2, and may be quantized in two stages or four or more stages. In addition, an integrated score calculation method other than simply taking the sum of the input scores from the audio power detection unit 1851, the SN ratio detection unit 1852, and the microphone characteristics detection unit 1853 is possible. As an example, an integrated score may be calculated using four arithmetic operations and functions based on a predetermined relational expression in which each score is added with a predetermined weight.

Received audio evaluation section 1855 calculates the evaluation score of received audio based on the packet loss rate during compound processing in decoding processing section 1824, which is output from decoding processing section 1824, and transmits this evaluation score to transmission section 1822. send to At this time, the received voice evaluation unit 1855 quantizes the calculation result of the evaluation score of the received voice and outputs it as a score. The quantization method is not particularly limited, but the received voice evaluation unit 1855 of this embodiment evaluates the received voice as a score of 0 if good, -1 if somewhat bad, and -2 if bad. output the calculation result of Furthermore, the received voice evaluation unit 1855 adds an identifier unique to the terminal device 10 that has output the voice data related to this evaluation score to the calculated evaluation score, and transmits the result to the transmission unit 1822 .

Note that the speech determination unit 185 may detect and evaluate only speech segments using a speech segment detection technique. In this case, it is possible to output an evaluation result based on the degree of overlapping of received voices. For example, when three or more people are making a voice call at the same time, there is a possibility that it was difficult for the other users to hear the content of the voice call made by the user who superimposed the utterance on the other user's utterance. You may output the evaluation result that there exists.

In addition, when the microphone 134 of the terminal device 10 of the other party is receiving a large amount of surrounding environmental noise, the voice determination unit 185 may detect this and notify the user of the terminal device 10 of the detection result. For example, a detection result may be output indicating that it may be difficult to hear the other party because the surroundings are noisy.

Furthermore, even if the voice interval detection time of the received voice from each terminal device 10 is transmitted and received between the terminal devices 10 during mutual communication, and the voice determination unit 185 outputs the evaluation result based on this information. good. For example, if there is a significant difference between the voice interval detection time of the transmitted voice on the transmitting side and the voice interval detection time of the received voice on the receiving side, the evaluation result is output indicating that the receiving side may have had difficulty hearing. You may More specifically, when the voice interval detection time of the transmitted voice on the transmitting side was 5 seconds, but the voice interval detection time of the received voice on the receiving side was 3 seconds, the receiving unit 1823 and/or the decoding It is presumed that there was some problem in the processing in the processing unit 1824 . Such problems can occur separately from packet loss. Therefore, it is significant to detect a mismatch between the voice period detection time of the transmitted voice on the transmitting side and the voice period detection time of the received voice on the receiving side.

The input speech state presentation unit 1861 displays the integrated score output from the input speech evaluation unit 1854 on the display 141 of the terminal device 10 .

In addition, the receiving voice state presentation unit 1862 calculates the score of the receiving voice state of the calling party based on the integrated score output by the input voice evaluation unit 1864 and the evaluation score of the received voice output from the receiving unit 1823. and displayed on the display 141 of the terminal device 10 . At this time, the caller's received voice state presenting unit 1862 quantizes the caller's received voice state and outputs it as a score. Although there is no particular limitation on the method of quantization, in the calling party received voice state presentation unit 1862 of the present embodiment, the integrated score output by the input voice evaluation unit 1864 is 0 if it is good, and - It is quantized as a score of 1 if it is bad, and -2 if it is bad, and the evaluation score of the received voice output from the receiving unit 1823 is also 0 if it is good, -1 if it is bad, and -2 if it is bad. If the sum of these scores is 0, it is good, -1 is somewhat bad, and -2 is bad. Here, when the sum of the integrated score output by the input voice evaluation unit 1864 and the evaluation score of the received voice output from the receiving unit 1823 is -2 or less, the caller received voice state presentation unit 1862 may be treated as bad (-2).

Preferably, the calling party received voice state presenting unit 1862 also displays the evaluation score of the received voice output from the receiving unit 1823 on the display 141 of the terminal device 10 .

The voice power state presentation unit 1863 determines whether the voice power of the voice data uttered by the user of the terminal device 10 sent from the server 20 is the voice power calculated from the voice powers of the plurality of terminal devices 10 currently engaged in a call. A determination result indicating that the average value is equal to or greater than a predetermined value is received, and the power state of the voice uttered by the user of the terminal device 10 is displayed on the display 141 of the terminal device 10 based on the determination result. At this time, the voice power state display unit 1863 does not display the voice power state on the display 141 of the terminal device 10 while there is a user who has not yet spoken among the users who are engaged in mutual communication.

Display format of the integrated score by the input voice state presentation unit 1861, the received voice evaluation score and the score of the other party's received voice state by the caller reception voice state presentation unit 1862, and the display format of the determination result by the voice power state presentation unit 1863 is not particularly limited, and examples include displaying the score itself, displaying the score on a scale, and changing the display color according to the score. If the score is good (0), that is, if the state of the voice uttered by the user of the terminal device 10 is good, or if the state of the received voice of the terminal device 10 of the other party is good, the display A display mode may be adopted in which nothing is displayed on the display 141 and a display corresponding to the score is displayed on the display 141 when the score is slightly bad (-1) or bad (-2).

Here, an identifier unique to the terminal device 10 for which the score was created (evaluated) is assigned to the received voice evaluation score. Therefore, the input voice state presenting unit 1861 and the other party's receiving voice state presenting unit 1862 use this identifier and the information of the terminal device 10, which is the other party of the call, stored in the storage unit 170 of the terminal device 10 in advance. When linking the phone book (this includes the phone number and the unique name such as the user name that identifies the caller registered by the user), and displaying the score, the icon that displays the user name will be displayed. You may perform the display accompanying. In addition, since it is obvious that the integrated score output from the input speech evaluation unit 1854 relates to the user who owns the terminal device 10, when the integrated score is displayed, the initial registration of the terminal device 10, etc. may be displayed with an icon displaying the user name registered by the user himself/herself.

In this way, when adopting the mode of displaying the icon displaying the user name related to the score, the score is good (0), that is, the state of the voice uttered by the user of the terminal device 10 is good, or , the icon is not displayed on the display 141 if the state of the received voice of the terminal device 10 of the other party is good, and the icon is displayed on the display 141 when the score is slightly poor (-1) or poor (-2). At the same time, display according to this score may be performed. In addition, when the score is slightly bad (-1) or bad (-2), the transparency of the icon is increased to make it difficult to see, or the entire icon is gradually colored in a specific color (for example, red). may

Furthermore, when the user taps (touches) an icon or a score scale, information that serves as the basis for the score may be displayed on the display 141 of the terminal device 10 . Similarly, evaluation results and the like may be displayed as text. When the evaluation result or the like is displayed in text, it is preferable to display the text on the display 141 for a certain period of time (for example, several seconds) and erase the text display when the user performs an input operation such as tapping the display 141 . A specific example of the display mode will be described later.

Furthermore, alternatively, the form of presentation by the presentation control unit 186 is not limited to the form of displaying on the display 141 of the terminal device 10 described above, but can also be a form of notifying by voice using the speaker 142 of the terminal device 10 . More specifically, for example, the presentation control unit 186 converts the evaluation result input from the speech determination unit 185 or the reception unit 1823 or a message prepared in advance based on the evaluation result into speech data by speech synthesis. However, it is also possible to present (notify) the user with this voice data. Alternatively, the presentation control unit 186 can present (notify) the user with simple sound data such as an alarm sound, buzzer sound, or chime. Sound (voice) data generated by the presentation control unit 186 is input to the audio output unit 184 and presented to the user via the audio output unit 184 and the speaker 142 .

There is no particular limitation on the form of notification by voice of the presentation control unit 186, but the determination result itself and/or information that is the basis of the determination result (score) may be notified by voice. At this time, as described above, the user who is currently engaged in voice communication is already known, so the voice notification is made together with the information on the received voice state of which user (for example, the registered name of the user registered in the telephone directory). You may Information related to the input voice information uttered by the user of the terminal device 10 may be voice-notified together with the user's own user name, or may be omitted. The method of voice notification may be appropriately selected from known means, and there is no particular limitation such as using a voice synthesis module.

Alternatively, the speech determination unit 185 may detect and evaluate only speech segments using a speech segment detection technique.
<1.2 Server configuration>

FIG. 4 is a block diagram showing the functional configuration of the server 20 included in the system 1 of this embodiment. As shown in FIG. 4 , the server 20 functions as a communication section 201 , a storage section 202 and a control section 203 .

The communication unit 201 performs processing for the server 20 to communicate with an external device.

The storage unit 202 is realized by, for example, the memory 25 and the storage 26, and stores data and programs used by the server 20. FIG. Specifically, the storage unit 202 stores an application 2021, for example. The application 2021 is executed by the control unit 203 to implement a function implementation unit such as a transmission/reception unit 2031 in the control unit 203, which will be described later. The storage unit 202 also stores calculation result data 2022 in which the calculation result by the audio power distribution calculation unit 2034 is stored.

The control unit 203 is implemented by the processor 29 reading a program (application 2021) stored in the storage unit 202 and executing instructions included in the program. The control unit 203 controls operations of the server 20 . Specifically, for example, the control unit 203 functions as a transmission/reception unit 2031 , a storage control unit 2032 , a communication control unit 2033 , an audio power distribution calculation unit 2034 and an audio power determination unit 2035 .

The transmitting/receiving unit 2031 controls processing for transmitting/receiving data between the server 20 and an external device such as the terminal device 10 according to a communication protocol.

Storage control unit 2032 controls storage of information in storage unit 202 .

The communication control unit 2033 performs processing for realizing mobile communication according to a predetermined communication protocol between the terminal devices 10 .

The voice power distribution calculation unit 2034 calculates a plurality of utterances during mutual communication based on the voice power of each speaker detected by the voice power detection unit 1851 of the plurality of terminal devices 10 currently engaged in mutual communication via the server 20. The average and variance of the voice power of the user are calculated and sent to the voice power determination section 2035 . Also, the voice power distribution calculator 2034 stores the calculation result in the calculation result data 2022 .

The voice power determination unit 2035 calculates the difference between the voice power of each speaker and the average of the voice power based on the average and variance of the voice power calculated by the voice power distribution calculation unit 2034. It determines whether or not it is equal to or greater than the value, and notifies the terminal device 10 during mutual communication of the determination result.

It is preferable that the voice power distribution calculation unit 2034 and the voice power determination unit 2035 perform the calculation operation and the determination operation of the mean and variance of the voice power at predetermined intervals, preferably at least one second or more. This is because the voice power state display (notification) in the terminal device 10 becomes unstable if the average and variance of voice power are calculated and determined at too short time intervals. Similarly, the audio power distribution calculation unit 2034 and the audio power determination unit 2035 take a moving average using past calculation results of a predetermined number of times in order to stabilize the calculation results of the average value and variance of the audio power. A smoothed calculation result may be used.

Also, when there is a change in the user during mutual communication, the voice power distribution calculation unit 2034 and the voice power determination unit 2035 redo the voice power average and variance calculation and determination operations. That is, the operation of calculating and determining the mean and variance of voice power is newly performed.

<2 Data structure>
5 and 6 are diagrams showing the data structure of the database stored in the terminal device 10. FIG. Note that FIGS. 5 and 6 are examples, and do not exclude data not described.

FIG. 5 is a diagram showing the data structure of the evaluation table 173. As shown in FIG. As shown in FIG. 5, the evaluation table 173 stores scores when the input speech evaluation unit 1854 evaluates the audio power, SN ratio, and microphone characteristics as good, somewhat bad, and bad. Also, the evaluation table 173 is a table indicating whether the final evaluation result is good, somewhat bad, or bad as a result of the input speech evaluation unit 1854 summing these scores to calculate an integrated score.

FIG. 6 is a diagram showing the data structure of the detection result data 172. As shown in FIG. As shown in FIG. 6, each record of the detection result data 172 includes, for example, the item “call ID”, the item “start time”, the item “end time”, the item “user ID”, and the item “voice power score", item "SN ratio score", item "microphone characteristic score", item "input voice score", and item "received voice score". The information stored in the detection result data 172 is updated each time the voice determination unit 185 performs a determination operation.

The item “call ID” is information for identifying individual calls made between the terminal device 10 and another terminal device 10 that is a call partner. The item "start time" is information on the time when the call specified by the item "call ID" started. The item "end time" is information on the time when the call specified by the item "call ID" ended. The item “user ID” is information for identifying the other terminal device 10 that is the other party of the call specified by the item “call ID”. Preferably, the item “user ID” is an identifier for identifying another terminal device 10 received by the receiving unit 1823 . Since three or more users can make simultaneous calls using the terminal device 10, a plurality of pieces of identification information can be stored in the item "user ID". The item “audio power score” is a score that is the result of detection by the audio power detection unit 1851 . The item “SN ratio score” is a score that is the result of detection by the SN ratio detection unit 1852 . The item “microphone characteristic score” is a score that is the result of detection by the microphone characteristic detection unit 1853 . The item “input speech score” is a score that is the result of determination by the input speech evaluation unit 1854 . The item “received voice score” is the score that is the result of evaluation by the received voice evaluation unit 1855 .

<3 Operation example>
An example of operations of the terminal device 10 and the server 20 will be described below.

FIG. 7 is a flow chart showing an example of the operation of the terminal device 10. As shown in FIG. FIG. 7 is a flow chart showing an example of the operation when the voice determination unit 185 detects and determines based on the input voice uttered by the user of the terminal device 10 and notifies the user via the presentation control unit 186. be.

Various operations shown in the flowcharts and sequence diagrams of FIGS. 7 to 9 can be executed in parallel.

In step S<b>11 , the control unit 180 of the terminal device 10 performs call communication to a specific terminal device 10 via the server 20 . Specifically, for example, the control unit 180 causes the communication control unit 182 to make a call to a specific terminal device 10 via the transmission/reception unit 2031 and the communication control unit 2033 of the server 20 . Since the calling operation by the communication control unit 182 is a known operation, no further explanation will be given. Although the flowchart of FIG. 7 shows an example in which the terminal device 10 performs a calling operation, the same operation is performed when the terminal device 10 performs a call receiving operation. That is, in the operation shown in the flowchart of FIG. 7, it is arbitrary whether the terminal device 10 performs the calling operation or the receiving operation.

Then, in step S12, the control unit 180 waits for a call to be received from the specific terminal device 10 that performed call-originating communication in step S11.

Thereafter, the control unit 180 performs a voice call with the specific terminal device 10 that received the call, that is, the specific terminal device 10 of the other party. Specifically, for example, the control unit 180 processes voice input uttered by the user of the terminal device 10 using the voice input unit 183 and converts it into voice data, and the voice data processed by the communication control unit 182. is transmitted to the specific terminal device 10 via the transmission/reception unit 2031 of the server 20, and the voice data transmitted from the specific terminal device 10 is received by the communication control unit 182 via the transmission/reception unit 2031 of the server 20. Then, the audio output unit 184 converts it into audio.

In addition, in the system 1 of the present embodiment, it is also possible for three or more users to make mutual calls. A detailed description of when to add a user for intercommunication (whether to perform an outgoing call operation or an incoming call operation) is not provided here.

In step S<b>13 , the control unit 180 accepts the input voice uttered by the user of the terminal device 10 . Specifically, for example, the control unit 180 receives an input voice uttered by the user of the terminal device 10 through the voice input unit 183 . Then, in steps S14 to S16, the control unit 180 performs detection processing for the input voice uttered by the user of the terminal device 10, which is accepted in step S13. Specifically, for example, the control unit 180 detects the sound power of the input sound with the sound power detection unit 1851, detects the SN ratio of the input sound with the SN ratio detection unit 1852, and detects the input sound with the microphone characteristics detection unit 1853. to detect microphone characteristics. All of the processes shown in steps S14 to S16 are executed in parallel.

In step S17, the control section 180 performs evaluation operation of the input speech based on the speech power and the like detected in steps S14 to S16. Specifically, for example, the control unit 180 causes the input speech evaluation unit 1854 to perform an input speech evaluation operation based on the detected speech power and the like, and calculates an integrated score. Input speech evaluation section 1854 then sends the evaluation result to presentation control section 186 .

In step S<b>18 , the control unit 180 causes the display 141 of the terminal device 10 to display the evaluation result in step S<b>16 . Specifically, for example, the control unit 180 causes the presentation control unit 186 (the input voice state presentation unit 1861 and the other party's received voice state presentation unit 1862) to control the evaluation result (calculated integrated score) evaluated by the input voice evaluation unit 1854 ), the evaluation result is displayed on the display 141 of the terminal device 10 .

In step S19, the control unit 180 determines whether there is an instruction to disconnect the call from the terminal device 10, or whether the terminal device 10 of the other party has disconnected the call. If the call is still in progress (NO in step S19), the process returns to step S13, and if it is determined that the call has been disconnected (YES in step S19), the call disconnection process is performed and the program shown in FIG. 7 ends.

FIG. 8 is a flow chart showing another example of the operation of the terminal device 10. FIG. In FIG. 8, based on the received voice from the other party with whom the user is making mutual calls, the voice determination unit 185 performs detection and determination operation, 9 is a flow chart showing an example of an operation when transmitting a determination result;

In step S<b>21 , the control unit 180 of the terminal device 10 receives a call from a specific terminal device 10 via the server 20 . Specifically, for example, the control unit 180 causes the communication control unit 182 to receive a call from a specific terminal device 10 via the transmission/reception unit 2031 and the communication control unit 2033 of the server 20 . Since the incoming call operation by the communication control unit 182 is a known operation, no further explanation will be given. Although the flowchart of FIG. 8 shows an example in which a call is received from the terminal device 10, the same operation is performed when the terminal device 10 performs a call origination operation. That is, in the operation shown in the flowchart of FIG. 8 as well, it is arbitrary whether the terminal device 10 performs the calling operation or the receiving operation.

Then, in step S22, the control unit 180 waits until a call is established with the specific terminal device 10 that performed the incoming call operation in step S21. process. After that, the control unit 180 makes a voice call with the specific terminal device 10 with which the call has been established.

In step S<b>23 , the control unit 180 receives voice data from the specific terminal device 10 that is transmitted from the specific terminal device 10 that is the other party and received by the terminal device 10 . Specifically, for example, the control unit 180 receives voice data transmitted from the specific terminal device 10 that is the communication partner and transmitted via the transmission/reception unit 2031 of the server 20 by the communication control unit 182 .

Next, in step S24, the control section 180 performs an evaluation operation of the received audio data received in step S23. Specifically, for example, the control unit 180 causes the received voice evaluation unit 1855 to perform an evaluation operation on the received voice data received in step S23, and calculates an evaluation score.

Furthermore, in step S25, the control unit 180 transmits the received voice evaluation result performed in step S24 to the other terminal device 10 that transmitted the voice. Specifically, for example, the control unit 180 causes the input voice evaluation unit 1854 to transmit the received voice evaluation result to the specific terminal device 10 that transmitted the voice via the communication control unit 182 and the transmission/reception unit 2031 of the server 20. do.

Then, in step S26, control unit 180 outputs the audio data received in step S23. Specifically, for example, the control unit 180 outputs the audio data received and decoded by the communication control unit 182 as audio through the audio output unit 184 and the speaker 142 .

FIG. 9 is a flow chart showing another example of the operation of the terminal device 10. FIG. FIG. 9 shows the operation of receiving the received voice evaluation results sent from the other terminal device 10 with which the user is making mutual calls and presenting them to the user of the terminal device 10 by the presentation control unit 186. 4 is a flowchart representing an example;

The operations of steps S31 to S33 are the same as steps S21 to S23 of FIG. Although the flowchart of FIG. 9 shows an example in which the terminal device 10 performs a call-in operation, the same operation is performed when the terminal device 10 performs a call-out operation. That is, in the operation shown in the flowchart of FIG. 9 as well, it is arbitrary whether the terminal device 10 performs the calling operation or the receiving operation.

In step S34, the control unit 180 evaluates the received voice in the specific terminal device 10 transmitted from the specific terminal device 10 that is the other party of the call (this evaluation result is the voice uttered by the user of the terminal device 10). ), which is an evaluation result of how the other terminal device 10 received it. Specifically, for example, the control unit 180 receives the evaluation result of the received voice in the specific terminal device 10 received by the communication control unit 182 (receiving unit 1823). The receiving unit 1823 then sends the evaluation result to the presentation control unit 186 .

In step S35, the control unit 180 causes the display 141 of the terminal device 10 to display the evaluation result received in step S34. Specifically, for example, the control unit 180 causes the presentation control unit 186 (calling party reception voice state presentation unit 1862) to display the evaluation result (calculated integrated score) received by the reception unit 1823. The evaluation result is displayed on the display 141 .

The operation of step S36 is the same as step S26 of FIG.

In step S37, the control unit 180 determines whether there is an instruction to disconnect the call from the terminal device 10, or whether the terminal device 10 of the other party has disconnected the call. If the call is still in progress (NO in step S37), the process returns to step S23, and if it is determined that the call has been disconnected (YES in step S37), the call disconnection process is performed and the program shown in FIG. 9 ends.

FIG. 10 is a sequence diagram showing an example of operations of the terminal device 10 and the server 20. As shown in FIG. FIG. 10 shows the voice power distribution calculation unit 2034 and the voice power determination unit of the server 20 based on the input voice uttered by the terminal device 10 when the user is making a mutual call using two or more terminal devices 10. 2035 is a flowchart showing an example of an operation when 2035 performs calculation and determination operation and notifies the user via the presentation control unit 186 of the terminal device 10; Note that although the terminal devices 10 are represented as the first terminal device 10 and the second terminal device 10 in FIG. In addition, the same operation as that shown in FIG. 10 is performed in a call operation by two or more terminal devices 10 .

In step S<b>41 , the control unit 180 of the first terminal device 10 makes a call to the server 20 to perform call communication to the second terminal device 10 . Specifically, for example, the control unit 180 causes the communication control unit 182 to call the server 20 to the second terminal device 10 . Note that the flowchart of FIG. 10 shows an example in which the first terminal device 10 performs a calling operation and the second terminal device 10 performs an incoming call operation. A similar operation is performed when performing an operation. That is, in the operation shown in the flowchart of FIG. 10 as well, it is arbitrary whether the first terminal device 10 performs the calling operation or the second terminal device 10 performs the calling operation.

In step S42, the server 20 performs a call operation to the second terminal device 10 related to the call operation received in step S41. Specifically, for example, the control unit 203 of the server 20 uses the transmission/reception unit 2031 and the communication control unit 2033 to call the second terminal device 10 . Since the calling operation by the transmission/reception unit 2031 and the communication control unit 2033 is a known operation, no further explanation will be given.

In step S43, the control unit 180 of the second terminal device 10, which is the target of the call in step S42, performs an incoming call operation for the call in step S42. Specifically, for example, the control unit 180 causes the communication control unit 182 to receive a call from the terminal device 10 . As a result, a call is established between the first terminal device 10 and the second terminal device 10 in step S44.

In steps S44 and S45, the control unit 180 of the first terminal device 10 and the second terminal device 10 converts the input voice input to each terminal device 10 into input voice data, It transmits the voice data to the device 10 and also receives the voice data transmitted from the terminal device 10 which is the destination of the call, converts it into voice and outputs it. Specifically, for example, the control unit 180 of the first terminal device 10 and the second terminal device 10 accepts an input voice by the voice input unit 183, converts it into voice data by the communication control unit 182, Also, the communication control unit 182 receives the voice data transmitted from the terminal device 10 that is the call destination and transmitted through the transmission/reception unit 2031 of the server 20, and this communication control unit 182 converts it into sound and outputs it as sound through the sound output unit 184 and the speaker 142 .

At step S<b>47 , the control unit 180 of the first terminal device detects the voice power of the voice uttered by the user of the first terminal device 10 and sends the result to the server 20 . Specifically, for example, the control unit 180 detects the voice power of the voice uttered by the user of the first terminal device 10 by the voice power detection unit 1851 of the voice determination unit 185, and sends the result to the server 20. Send out.

Similarly, in step S<b>48 , the control unit 180 of the second terminal device detects the voice power of the voice uttered by the user of the second terminal device 10 and sends the result to the server 20 . Specifically, for example, the control unit 180 detects the voice power of the voice uttered by the user of the second terminal device 10 by the voice power detection unit 1851 of the voice determination unit 185, and sends the result to the server 20. Send out.

In step S49, the server 20 detects the voice powers of the voices uttered by the users of the first terminal device 10 and the second terminal device 10, which are transmitted in steps S35 and S36. Calculate the mean and variance of Specifically, for example, the control unit 203 causes the voice power distribution calculation unit 2034 to determine these values based on the voice power detection results of the voices uttered by the users of the first terminal device 10 and the second terminal device 10. Calculate the mean and variance of the speech power.

Next, in step S50, the server 20 determines whether or not the difference between the average value of the audio power calculated in step S49 and the detected audio power value is equal to or greater than a predetermined value. Specifically, for example, the control unit 203 causes the audio power determination unit 2035 to determine the average value of the audio power calculated by the audio power distribution calculation unit 2034 in step S49. It is determined whether or not the voice power detected by the second terminal device 10 deviates from the average value by a predetermined value or more. In step S50, the audio power determination unit 2035 of the server 20 obtains the standard deviation of the detected audio power value based on the average value and variance of the audio power calculated by the audio power distribution calculation unit 2034 in step S49. It may be determined whether or not the deviation is equal to or greater than a predetermined value.

Then, in step S51, the server 20 sends a predetermined value to the first terminal device 10 and/or the second terminal device 10 that has transmitted the detected voice power value determined to have a deviation of a predetermined value or more in step S37. Notifies that there is a deviation greater than or equal to the value. Specifically, for example, the control unit 203 causes the voice power determining unit 2035, the transmitting/receiving unit 2031, and the communication control unit 2033 to transmit the voice power detection value determined to have a deviation of a predetermined value or more in step S50. terminal device 10 and/or the second terminal device 10 that there is a deviation of a predetermined value or more. In the example shown in FIG. 10, it is determined that there is a deviation of a predetermined value or more in the first terminal device 10. Therefore, the voice power determination unit 2035, the transmission/reception unit 2031, and the communication control unit 2033 are configured in the first terminal device 10 will be notified. Here, when it is determined that there is a deviation of a predetermined value or more, there is a possibility that the voice power is greater than the average value by a predetermined value or less, or less than a predetermined value. Alternatively, the first terminal device 10 and/or the second terminal device 10 is also notified that the information is smaller than the average value.

In step S52, the first terminal device 10 receives the notification made in step S51 and displays that there is a deviation based on the received notification. Specifically, for example, the control unit 180 receives a notification from the communication control unit 182, the communication control unit 182 sends this notification to the audio power state presentation unit 1863, and the audio power state presentation unit 1863 outputs the first The notification is displayed on the display 141 of the terminal device 10 of .

Note that FIG. 10 shows an example of a call by two terminal devices 10 (the first terminal device 10 and the second terminal device 10). Since the average value and variance are calculated and the voice power determination unit 2035 determines the deviation from this average value, the example shown in FIG. It is possible to achieve the effect of

<4 Screen example>
An example of a screen output from the terminal device 10 will be described below with reference to FIGS. 11 to 13. FIG.

FIG. 11 shows the state of the input voice, that is, the voice uttered by the user of the terminal device 10, and the received (received) voice, that is, the state of the voice uttered by the user of the specific terminal device 10 with which the terminal device 10 is in mutual communication. and the state of the voice received by the terminal device 10 is displayed on the display 141 of the terminal device 10. FIG. The display screen of the terminal device 10 shown in FIG. 11 is the display screen of the terminal device 10 of the user who is making a call using the terminal device 10. Hereinafter, the user using the terminal device 10 of FIG. called speaker. Also, the (multiple) other parties with whom the speaker is making mutual calls are called receivers. In FIG. 11 (and FIG. 12), the speaker's user name is Tanaka, and the receiver's user names are Sato and Yamada, respectively. In other words, in FIG. 11, three people with the user name Tanaka, the user name Sato, and the user name Yamada are making a simultaneous voice call.

As shown in FIG. 11, the display 141 of the terminal device 10 displays icons 900 to 902 for each user. In the example shown in FIG. 11 (and FIGS. 12 to 15), the icons 900 to 902 are displayed as pictograms, but the display form of the icons 900 to 902 is not particularly limited. It may be an image of each user (speaker, receiver) who is speaking, or may simply display the user names of the speaker and receiver. Below the icons 900 to 902, a bar 903 displays the score, which is the result of judging the voice state. Note that the bar is one aspect of the icon and may be called an indicator. In the example shown in FIG. 11, there are 3 bars 903 if the score is 0, ie good, 2 bars if the score is -1, ie moderately bad, and 1 bar if the score is -2, ie bad. are displayed. In addition, the color in which the bar is displayed is also changed according to the score. Although FIG. 11 is shown in black and white for the sake of illustration, as an example, bars indicated by white rectangles in FIG. 11 are green, bars indicated by hatched rectangles inside are yellow, Bars shown as black-filled rectangles are displayed in red.

In icons 900 to 902 shown in FIG. 11A, it is determined that the voice state of speaker Tanaka is good, the voice state of speaker Sato is somewhat poor, and the voice state of speaker Yamada is poor. Also, in the icons 900 to 902 shown in FIG. 11B, the voice states of speaker Tanaka, speaker Sato, and speaker Yamada are all determined to be bad.

Next, FIG. 12 is a screen showing detailed display of the voice state displayed in FIG.

When the user of the terminal device 10 taps the icon 900 on the display 141, a detailed display as shown in the upper part of FIG. 12 is displayed on the display 141. FIG. Since the icon 900 shown in FIG. 12(a) is the icon of the speaker, the detailed display displays the integrated score of the input voice and the determination result such as the input voice power that is the basis for the score.

Similarly, when the user of the terminal device 10 taps the icon 902 on the display 141, a detailed display as shown in FIG. 12(b) is displayed on the display 141. FIG. Since the icon 902 shown in FIG. 12B is the icon of the receiver, the detailed display displays the score of the received voice and the packet loss rate that is the basis for the score.

Furthermore, FIG. 13 shows the case where there is a deviation of a predetermined value or more between the voice power of the input voice uttered by the user of the terminal device 10 and the average value of the voice power of the input voice uttered by the user currently engaged in mutual communication. This is a screen displaying the determination result on the display 141 of the terminal device 10 when the determination is made. In the example shown in FIG. 13, when the voice power of the user's input voice of the terminal device 10 is less than or equal to the average value by a predetermined value, the display 141 displays a dialog 1100 describing the determination result.

Although FIG. 13 shows the determination result of the variation in the user's input voice power, it is also possible to present a solution based on the determination result (for example, speak softly, speak loudly).

<5 Effect of Embodiment>
As described in detail above, according to the system 1 of the present embodiment, the received voice evaluation unit 1855 of the voice determination unit 185 of the terminal device 10 evaluates the state of the received voice data received by the terminal device 10, Since the first evaluation result, which is the result of this evaluation, is sent back to the other terminal device 10 that is the other party of the call, the user of the other terminal device 10 can input his/her own input based on this first evaluation result. You can know how your voice sounds to the other party. Therefore, according to the system 1 of this embodiment, it is possible to easily grasp the state of the voice of the other party.

Further, according to the system 1 of the present embodiment, the quality of the input voice uttered by the user of the terminal device 10 is evaluated by the voice determination unit 185, and the third evaluation result, which is the quality evaluation result, is presented to the user. Therefore, it is possible to grasp the state of the input voice of oneself at the same time as the state of the voice of the other party.

Furthermore, according to the system 1 of the present embodiment, the user can grasp the variation in the input voice power of the user during mutual communication.

In other words, in a multi-person call system such as a web conference, when the input voice power of the call participants varies greatly, it is difficult to hear the voice of a participant who is too weak compared to the average input voice power of all call participants. A participant's speech that is louder than the average input speech power of all participants is loud and harsh.

When such variations occur, it becomes difficult to adjust the volume of the audio output on the receiving side. More specifically, increasing the volume to make the too quiet participant's voice more audible makes the too loud participant's voice louder and even more annoying. On the other hand, if the volume is lowered to make it easier to hear the voices of the participants who are too loud, the voices of the participants who are too quiet become even quieter and harder to hear. Therefore, a user uttering a voice grasps that the power of his own voice is too small or too large compared to the average voice power of all callers, and the user adjusts the power of his own uttered voice to reach the average. It is important to eliminate such variations in the input voice power by bringing them close to each other.

WO 2008/011901 discloses a technique related to variations in input voice power of a user during mutual communication. In this technique, at least one voice terminal superimposes and aggregates individual audio signals associated with a first group such that audio data provided by each receive channel is decoded into individual audio signals for the group. An audio signal is formed and the aggregated audio signal is encoded into first aggregated audio data. Further, the at least two voice terminals are associated with the second group, and the first aggregated voice data is provided to the voice terminals associated with the second group for each output of the first aggregated voice data. be done.

However, in a situation where a plurality of users are making a voice call, even if the input voice level of each user exceeds the threshold, the input voice level of each user may vary among the multiple users participating in the call. If it is on, it will lead to hearing difficulty.

Even in the above-described technique, no consideration is given to feedback on variations in the input voice levels of multiple users participating in a call.

On the other hand, according to the system 1 of the present embodiment, the variation in the input voice levels of a plurality of users, that is, the average value and variance of the detected values of the input voice power of the terminal device 10 are calculated by the voice power distribution calculation unit 2034, The voice power determination unit 2035 determines whether or not there is a deviation of a predetermined value or more from the calculated average value, and presents the fourth determination result, which is the result of this determination, to the user (speaker) of the terminal device 10. Therefore, the user presented with the fourth determination result can grasp the variation in the input voice power.

<6 Notes>
It should be noted that the above-described embodiments describe the configurations in detail in order to explain the present disclosure in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Also, part of the configuration of each embodiment can be added, deleted, or replaced with another configuration.

As an example, a configuration in which the terminal device 10 includes only the received voice evaluation section 1855 in the voice determination section 185 is also possible. A configuration having only the received voice evaluation unit 1855 and the caller received voice state presentation unit 1862 is also possible. Furthermore, a configuration in which the server 20 does not have the audio power distribution calculation unit 2034 and the audio power determination unit 2035 is also possible. In this case, the terminal device 10 does not have the audio power state presentation section 1863 .

Further, in the above-described embodiment, the quality of the input voice uttered by the user of the terminal device 10 is evaluated from the viewpoints of voice power, SN ratio, and microphone characteristics. Received voices from other terminal devices 10 are detected and evaluated by a voice power detection unit 1851, an SN ratio detection unit 1852, and a microphone characteristic detection unit 1853, and evaluated by calculating an integrated score by an input voice evaluation unit 1854. you can go Then, the quality evaluation result by the input voice evaluation unit 1854 is sent to the specific terminal device 10 of the other party via the transmission unit 1822 together with the evaluation result of the received voice evaluation unit 1855 based on the packet loss rate. An identifier for identifying the terminal device 10 is added to the quality evaluation result by the input speech evaluation section 1854 and transmitted, as with the evaluation result of the received speech evaluation section 1855 .

FIG. 14 shows a screen in which the speaker's input voice is subjected to quality evaluation in the receiver's terminal device 10, and the result is received by the speaker's terminal device 10 and displayed on its display 141. FIG.

The screen shown in FIG. 14 is similar to the screen shown in FIG. Icons 901 and 902 and a bar 903 for are displayed. Further, although the bar 903 is displayed in the screen example shown in FIG. 14, only the icons 901 and 902 may be displayed. This also applies to screen examples in FIGS. 11 and 12. FIG. Also, FIG. 14 shows an example in which the speaker icon 900 is not displayed, but the speaker icon 900 may be displayed.

Furthermore, FIG. 15, like FIG. 12, is a screen displaying the details of the voice state displayed in FIG. In FIG. 15, when the listener's icon (icon 902 in FIG. 14) is tapped, the score when the speaker's input voice is received by the receiver's terminal device 10, the packet loss rate as the basis, and further displays the integrated score of the input voice of the receiver, which is the result of the quality evaluation, and the determination results such as the input voice power, which is the basis for the score.

Here, when three or more users are making simultaneous calls, the evaluation results regarding the received voice are sent separately from the plurality of terminal devices 10 that are the parties to the call. Therefore, when presenting the evaluation result by the caller reception voice state presentation unit 1862, all the evaluation results transmitted from the terminal device 10 that is the caller may be presented, or at least one terminal may be displayed based on the evaluation result. The evaluation results transmitted from the device 10 may be selected and presented. For example, an evaluation result transmitted from a communication partner with a low received voice evaluation may be discarded. Also, as an example, receiving a bad evaluation result means that there is a person on the other end of the conversation who is difficult to hear your own conversation. Also, for the quality evaluation result, which terminal device 10 has the evaluation result may be presented together with, for example, the user name. Furthermore, evaluation results transmitted from communication partners with high evaluations of received voices may also be displayed. In addition, when there are five users in mutual communication, the user name of the user with the highest received voice evaluation or the user name of the user with the lowest received voice evaluation may be displayed. You may display the order (ranking) of.

Further, each of the above configurations, functions, processing units, processing means, and the like may be realized by hardware, for example, by designing them in an integrated circuit. The present invention can also be implemented by software program code that implements the functions of the embodiments. In this case, a computer is provided with a storage medium recording the program code, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium storing it constitute the present invention. Storage media for supplying such program codes include, for example, flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, and non-volatile memory cards. , ROM and the like are used.

Also, the program code that implements the functions described in this embodiment can be implemented in a wide range of programs or script languages, such as assembler, C/C++, perl, Shell, PHP, and Java (registered trademark).

Furthermore, by distributing the program code of the software that implements the functions of the embodiment via a network, it can be stored in storage means such as a hard disk or memory of a computer, or in a storage medium such as a CD-RW or CD-R. Alternatively, a processor provided in the computer may read and execute the program code stored in the storage means or the storage medium.

The items described in the above embodiments will be added below.

(Appendix 1)
A program (171) for operating a terminal device (10) comprising a processor (19) and a memory (15) and capable of transmitting and receiving voice data to and from another terminal device (10), the program (171) The processor (19) receives voice data transmitted from another terminal device (10) (S23), evaluates the state of the received voice data and outputs it as a first evaluation result ( S24), a step of replying the first evaluation result to another terminal device (10) (S25), and a step of outputting voice based on the received voice data to the user (S26).
(Appendix 2)
A step of receiving a second evaluation result obtained by evaluating the state of the voice data transmitted from the other terminal device (10) and received by the other terminal device (10) (S33 ) and presenting the received second evaluation result to the user (S35).
(Appendix 3)
The program according to appendix 2, further executing a step (S35) of presenting icons (900 to 902) corresponding to the second evaluation result to the user.
(Appendix 4)
3. The program according to appendix 3, wherein in the step (S35) of presenting the icons (900 to 902) to the user, the display mode of the image regarding the user of the other terminal device (10) is changed according to the second evaluation result.
(Appendix 5)
3. The program according to appendix 2, wherein in the step of presenting the second evaluation result to the user (S35), a sound is generated according to the second evaluation result.
(Appendix 6)
6. The method according to any one of appendices 2 to 5, wherein in the step of presenting the second evaluation result to the user (S35), if the second evaluation result is better than a predetermined state, the second evaluation result is not presented. program.
(Appendix 7)
The second evaluation result is attached with an identifier specifying the other terminal device (10) that received the voice data, and in the step of presenting the second evaluation result to the user (S35), the second evaluation result 7. The program according to any one of appendices 2 to 6, wherein the result is presented to the user in an identifiable manner by another terminal device (10).
(Appendix 8)
8. The program according to appendix 7, wherein in the step (S35) of presenting the second evaluation result to the user, the program is presented in association with the image of the user of the other terminal device (10).
(Appendix 9)
In the step of receiving voice data (S23), voice data encoded and converted into packets is received and decoded. 9. The program according to any one of appendices 2 to 8, wherein in the step of outputting as the first evaluation result (S24), the state of the audio data is evaluated based on the packet loss rate during decoding.
(Appendix 10)
10. The program according to appendix 9, wherein in the step of presenting the second evaluation result to the user (S35), a packet loss rate in communication with another terminal device (10) is presented to the user as the second evaluation result.
(Appendix 11)
9. The program according to any one of Appendices 2 to 8, wherein in the step of outputting as the first evaluation result (S24), speech section detection is used to evaluate the state of the speech data.
(Appendix 12)
9. The program according to any one of appendices 2 to 8, wherein in the step of outputting as the first evaluation result (S24), the state of the audio data is evaluated based on the quality of the audio data.
(Appendix 13)
13. The program according to supplementary note 12, wherein in the step of presenting the second evaluation result to the user (S35), the quality of voice data based on the voice uttered by the user is presented to the user as the second evaluation result.
(Appendix 14)
In the step of receiving a second evaluation result (S33), a plurality of second evaluation results are received from a plurality of other terminal devices (10), and in the step of presenting the second evaluation results to the user (S26) 14. The program of claim 13, which removes second evaluation results other than at least one.
(Appendix 15)
In the step of receiving voice data (S23), voice data to which a first identifier specifying the other terminal device (10) is assigned is received, and in the step of outputting as a first evaluation result (S24), the first In the step (S25) of attaching a second identifier specifying the terminal device (10) to the evaluation result of and returning the first evaluation result to the other terminal device (10), the first 15. The program according to any one of appendices 1 to 14, wherein the evaluation result of is transmitted to another terminal device (10) specified by the first identifier.
(Appendix 16)
A step (S17) of evaluating the quality of voice data based on the voice uttered by the user of the terminal device (10), and the evaluation result of the quality of the voice data based on the voice uttered by the user of the terminal device (10). 16. The program according to any one of appendices 1 to 15, further executing a step (S18) of presenting the third evaluation result to the user.
(Appendix 17)
In the step of evaluating the quality of the voice data (S17), the characteristics of the voice uttered by the user of the terminal device (10), namely the power of the voice, the signal-to-noise ratio, the microphone characteristics, or a combination of detections of at least two of these, 17. The program of clause 16, wherein the program evaluates the quality of the audio data based on.
(Appendix 18)
In the step (S17) of evaluating the quality of voice data, characteristics are detected each time voice data is input from the user of the terminal device (10), and the value of the last detected characteristics is held when the input of voice data stops. 17. The program according to Supplementary Note 17.
(Appendix 19)
19. The program according to appendix 17 or 18, wherein in the step of evaluating the quality of the audio data (S17), the quality of the audio data is evaluated by calculating a score based on the characteristic detection result.
(Appendix 20)
In the step (S18) of presenting the third evaluation result to the user of the terminal device (10), the third evaluation result is not presented unless voice data is input from the user of the terminal device (10). 20. The program according to any one of 19.
(Appendix 21)
In the step (S18) of presenting the third evaluation result to the user of the terminal device (10), the third evaluation result is the power of the voice, the SN ratio, the microphone characteristics, or a combination of at least two of these detections. 20. A program according to any one of clauses 17-19 for presentation to a user of a device (10).
(Appendix 22)
A program (2021) for operating a server (20) comprising a processor (29) and a memory (25) and for transmitting and receiving voice data between a plurality of terminal devices (10), the program (2021) the processor (29) to detect the voice power of the input voice for each terminal device (10) (S47, S48), and to calculate the average value and the variance of the voice power based on the voice power detection result. Based on the step (S49) and the calculated average value and variance, it is determined whether or not the difference between the voice power of each speaker using the plurality of terminal devices (10) and the average value is equal to or greater than a predetermined value. and a step (S51) of presenting a fourth determination result to the speaker determined to have a difference between the voice power and the average value equal to or greater than a predetermined value.
(Appendix 23)
Described in appendix 22, wherein in the step of detecting voice power (S47, S48), voice power is detected each time voice is input from the terminal device (10), and the last detected value is retained when voice input is interrupted. program.
(Appendix 24)
24. The program according to appendix 22 or 23, wherein in the step of calculating the mean value and variance of voice power (S49), the mean value and variance of voice power are calculated at intervals of seconds.
(Appendix 25)
25. The program according to any one of appendices 22 to 24, wherein in the step (S50) of determining whether or not the difference between the voice power and the average value is equal to or greater than a predetermined value, the determination is made at intervals of seconds.
(Appendix 26)
The voice data is attached with an identifier specifying the terminal device (10) that sent the voice data. 26. The program according to any one of appendices 22 to 25, for recalculating the average value and variance of voice power when detecting that the current speaker has been changed.
(Appendix 27)
In the step of presenting the fourth determination result (S51), if there is a speaker who is connected to the communication line but does not speak, the fourth determination result is not presented. program as described.
(Appendix 28)
28. The program according to any one of appendices 22 to 27, wherein in the step of presenting the fourth determination result (S51), a solution is presented together with the fourth determination result.
(Appendix 29)
A device (10) comprising a processor (19) and a memory (15) and capable of transmitting and receiving audio data to and from another device (10), wherein the processor (19) receives data transmitted from the other device (10). a step of receiving the received voice data (S23); a step of evaluating the state of the received voice data and outputting it as a first evaluation result (S24); and returning the first evaluation result to the other device (10). (S25), and a step (S26) of outputting voice based on the received voice data to the user.
(Appendix 30)
A method performed by a computer (10) comprising a processor (19) and a memory (15) and capable of transmitting and receiving audio data to and from another computer (10), wherein the processor (19) a step of receiving the transmitted voice data (S23); a step of evaluating the state of the received voice data and outputting it as a first evaluation result (S24); and returning the first evaluation result to another computer. A method for carrying out the steps of (S25) and outputting (S26) a voice based on the received voice data to a user.
(Appendix 31)
A system (1) capable of transmitting and receiving voice data between a terminal device (10) and another terminal device (10), wherein the terminal device (10) means (1823) for receiving voice data received; means (1855) for evaluating the state of the received voice data and outputting it as a first evaluation result; and sending the first evaluation result to another terminal device (10) A system (1) comprising means (1822) for replying and means (184) for outputting voice based on received voice data to a user.
(Appendix 32)
A device (20), comprising a processor (29) and a memory (25), for transmitting and receiving voice data between a plurality of terminal devices (10), wherein the processor (29) is provided for each terminal device (10) Steps of detecting the voice power of the input voice (S47, S48); calculating the mean value and variance of the voice power based on the voice power detection result (S49); and based on the calculated mean value and variance a step (S50) of determining whether or not the difference between the voice power of each speaker using a plurality of terminal devices (10) and the average value is equal to or greater than a predetermined value; a step (S51) of presenting a fourth determination result to a speaker determined to be equal to or greater than a predetermined value.
(Appendix 33)
A method performed by a computer (20) comprising a processor (29) and a memory (25) for transmitting and receiving audio data between a plurality of terminal devices (10), wherein the processor (29) comprises: (10) steps of detecting the voice power of the input voice (S47, S48); calculating the average value and variance of the voice power based on the voice power detection result (S49); and based on the variance, a step (S50) of determining whether or not the difference between the voice power of each speaker using a plurality of terminal devices (10) and the average value is equal to or greater than a predetermined value; and a step (S51) of presenting a fourth determination result to a speaker determined to have a deviation from the average value equal to or greater than a predetermined value.
(Appendix 34)
A system (1) having a plurality of terminal devices (10) and a server (20) for transmitting and receiving voice data between the plurality of terminal devices (10), wherein the server (20) comprises terminal devices ( 10) means (2031, 2033) for detecting the voice power of each input voice; means (2034) for calculating the mean value and variance of the voice power based on the voice power detection result; Means (2035) for determining whether the difference between the voice power of each speaker using a plurality of terminal devices (10) and the average value is equal to or greater than a predetermined value based on the variance; means (2031, 2033) for presenting a fourth determination result to a speaker determined to have a deviation from the value equal to or greater than a predetermined value.

DESCRIPTION OF SYMBOLS 1... System, 10... Terminal device, 20... Server, 141... Display, 142... Speaker, 170... Storage part, 171... Application, 172... Detection result data, 173... Evaluation table, 180... Control part, 182... Communication control Part 183... Audio input unit 184... Audio output unit 185... Audio determination unit 186... Presentation control unit 202... Storage unit 203... Control unit 900 to 902... Icon 903... Bar 1100... Dialog, DESCRIPTION OF SYMBOLS 1821... Encoding process part 1822... Transmission part 1823... Reception part 1824... Decoding process part 1851... Voice power detection part 1852... SN ratio detection part 1853... Microphone characteristic detection part 1854... Input voice evaluation Part 1855 Received voice evaluation unit 1861 Input voice state presentation unit 1862 Reception voice state presentation unit 1863 Voice power state presentation unit 1864 Input voice evaluation unit 2021 Application 2022 Calculation result Data 2031 Transmitting/receiving unit 2032 Storage control unit 2033 Communication control unit 2034 Sound power distribution calculation unit 2035 Sound power determination unit

Claims (34)

A program for operating a terminal device comprising a processor and a memory and capable of transmitting and receiving audio data to and from another terminal device,
The program causes the processor to:
receiving the audio data transmitted from the other terminal device;
evaluating the state of the received audio data and outputting it as a first evaluation result;
a step of returning the first evaluation result to the other terminal device;
and outputting a voice based on the received voice data to a user. a step of receiving a second evaluation result of evaluating the state of the audio data transmitted from the terminal device and received by the other terminal device, which is transmitted from the other terminal device;
The program according to claim 1, further causing a step of presenting the received second evaluation result to the user. 3. The program according to claim 2, further causing a step of presenting an icon according to the second evaluation result to the user. 4. The program according to claim 3, wherein in the step of presenting the icon to the user, a display mode of an image related to the user on the other terminal device is changed according to the second evaluation result. 3. The program according to claim 2, wherein in the step of presenting said second evaluation result to said user, a sound corresponding to said second evaluation result is generated. 6. The step of presenting the second evaluation result to the user does not present the second evaluation result if the second evaluation result is better than a predetermined condition. program described in . The second evaluation result is attached with an identifier that identifies the other terminal device that has received the audio data,
7. The program according to any one of claims 2 to 6, wherein, in the step of presenting the second evaluation result to the user, the second evaluation result is presented to the user in an identifiable manner by another terminal device. . 8. The program according to claim 7, wherein in the step of presenting the second evaluation result to the user, the second evaluation result is presented in association with an image of the user on the other terminal device. the step of receiving the audio data includes receiving the audio data that has been encoded and converted into packets and decoding the audio data;
9. The program according to any one of claims 2 to 8, wherein in the outputting as the first evaluation result, the state of the audio data is evaluated based on the packet loss rate during the decoding. 10. The program according to claim 9, wherein in the step of presenting the second evaluation result to the user, the packet loss rate in communication with the other terminal device is presented to the user as the second evaluation result. 9. The program according to any one of claims 2 to 8, wherein the step of outputting the first evaluation result utilizes speech segment detection to evaluate the state of the speech data. 9. The program according to any one of claims 2 to 8, wherein in the step of outputting the first evaluation result, the state of the audio data is evaluated based on the quality of the audio data. 13. The program according to claim 12, wherein in the step of presenting the second evaluation result to the user, the quality of the voice data based on the voice uttered by the user is presented to the user as the second evaluation result. In the step of receiving the second evaluation result, receiving evaluations of the state of the audio data based on the quality of the audio data from the plurality of other terminal devices;
14. The program according to claim 13, wherein in the step of presenting the second evaluation results to the user, the second evaluation results other than at least one are removed. In the step of receiving the voice data, the voice data to which a first identifier specifying the other terminal device is assigned is received;
In the step of outputting as the first evaluation result, attaching a second identifier specifying the terminal device to the first evaluation result,
In the step of returning the first evaluation result to the other terminal device, the first evaluation result to which the second identifier is attached is sent to the other terminal device specified by the first identifier. 15. The program according to any one of claims 1 to 14, which is transmitted as. evaluating the quality of the voice data based on the voice uttered by the user of the terminal device;
and presenting to the user a third evaluation result, which is an evaluation result of the quality of the voice data based on the voice uttered by the user of the terminal device. program described in . The step of evaluating the quality of the voice data is based on characteristics of the voice emitted by the user of the terminal device, such as power of the voice, signal-to-noise ratio, microphone characteristics, or a combination of detection of at least two of these. 17. The program according to claim 16, wherein the quality of said audio data is evaluated by using a program. In the step of evaluating the quality of the voice data, the characteristics are detected each time the voice data is input from the user of the terminal device, and the finally detected value of the characteristics is evaluated when the input of the voice data stops. 18. The program of claim 17, comprising: 19. The program according to claim 17, wherein in the step of evaluating the quality of the audio data, the quality of the audio data is evaluated by calculating a score based on the detection result of the characteristics. The step of presenting the third evaluation result to the user of the terminal device does not present the third evaluation result unless the voice data is input from the user of the terminal device. A program according to any one of paragraphs. In the step of presenting the third evaluation result to the user of the terminal device, the third evaluation result is the power of the voice, the SN ratio, the microphone characteristics, or a combination of detection of at least two of them. to the user of the terminal device. A program for operating a server that includes a processor and a memory and transmits and receives audio data between a plurality of terminal devices,
The program causes the processor to:
detecting the audio power of the input audio for each of the terminal devices;
calculating an average value and a variance of the audio power based on the detection result of the audio power;
determining whether the difference between the voice power of each speaker using the plurality of terminal devices and the average value is equal to or greater than a predetermined value, based on the calculated average value and the variance;
and presenting a fourth determination result to the speaker determined to have a difference between the voice power and the average value equal to or greater than a predetermined value. 23. The program according to claim 22, wherein in the step of detecting the voice power, the voice power is detected each time voice is input from the terminal device, and the last detected value is held when the voice input is stopped. 24. The program according to claim 22 or 23, wherein in the step of calculating the mean value and the variance of the voice power, the mean value and the variance of the voice power are calculated at intervals of seconds. 25. The program according to any one of claims 22 to 24, wherein in the step of determining whether or not the difference between said voice power and said average value is equal to or greater than a predetermined value, determination is made at intervals of seconds. The audio data is attached with an identifier that identifies the terminal device that sent the audio data,
In the step of calculating the average value and the variance of the voice power, when it is detected that the speaker who is currently speaking is changed based on the identifier, the average value and the variance of the voice power are calculated. 26. The program according to any one of claims 22 to 25, which recalculates. 27. The method according to any one of claims 22 to 26, wherein in the step of presenting the fourth determination result, the fourth determination result is not presented if there is a speaker who is connected to the communication line but does not speak. program as described. 28. The program according to any one of claims 22 to 27, wherein in the step of presenting the fourth determination result, a solution is presented together with the fourth determination result. A device comprising a processor and memory and capable of transmitting and receiving audio data to and from another device,
The processor
receiving the audio data transmitted from the other device;
evaluating the state of the received audio data and outputting it as a first evaluation result;
returning the first evaluation result to the other device;
and outputting audio based on the received audio data to a user. A computer-implemented method comprising a processor and memory and capable of transmitting and receiving audio data to and from another computer, comprising:
The processor
receiving the audio data transmitted from the other computer;
evaluating the state of the received audio data and outputting it as a first evaluation result;
returning the first evaluation result to the other computer;
and outputting audio to a user based on the received audio data. A system capable of transmitting and receiving audio data between a terminal device and another terminal device,
The terminal device
means for receiving the audio data transmitted from the other terminal device;
means for evaluating the state of the received audio data and outputting it as a first evaluation result;
means for returning the first evaluation result to the other terminal device;
and means for outputting to a user voice based on the received voice data. A device for transmitting and receiving voice data between a plurality of terminal devices, comprising a processor and a memory,
The processor
detecting the audio power of the input audio for each of the terminal devices;
calculating an average value and a variance of the audio power based on the detection result of the audio power;
determining whether the difference between the voice power of each speaker using the plurality of terminal devices and the average value is equal to or greater than a predetermined value, based on the calculated average value and the variance;
and presenting a fourth determination result to the speaker determined to have a difference between the voice power and the average value equal to or greater than a predetermined value. 1. A computer-implemented method, comprising a processor and memory, for transmitting and receiving audio data between a plurality of terminal devices, the method comprising:
The processor
detecting the audio power of the input audio for each of the terminal devices;
calculating an average value and a variance of the audio power based on the detection result of the audio power;
determining whether the difference between the voice power of each speaker using the plurality of terminal devices and the average value is equal to or greater than a predetermined value, based on the calculated average value and the variance;
and presenting a fourth determination result to the speaker determined to have a difference between the voice power and the average value equal to or greater than a predetermined value. A system having a plurality of terminal devices and a server for transmitting and receiving voice data between the plurality of terminal devices,
The server is
means for detecting audio power of input audio for each terminal device;
means for calculating an average value and variance of the audio power based on the detection result of the audio power;
means for determining whether a difference between the voice power of each speaker using the plurality of terminal devices and the average value is equal to or greater than a predetermined value, based on the calculated average value and the variance;
and means for presenting a fourth determination result to the speaker determined to have a difference between the voice power and the average value equal to or greater than a predetermined value.

Images