KR102344618B1 - Apparatus for quality evaluation of voice recognition, and control method thereof - Google Patents

Abstract

The present invention relates to a voice recognition quality evaluation apparatus for evaluating voice recognition quality in comprehensive consideration of support functions of voice communication devices equipped with voice recognition technology in an IPTV service environment, and to a method of operating the same.

Description

Speech recognition quality evaluation device and its operation method

The present invention relates to a method for evaluating voice recognition quality in comprehensive consideration of the support functions of voice communication devices equipped with voice recognition technology in an IPTV service environment.

Recently, programs of various contents can be provided through the Internet, and an IPTV (internet protocol television) service, which is an interactive broadcasting service in which a communication service and a broadcasting service are combined, is widely distributed.

The IPTV service provides various contents such as broadcast contents, Internet contents, and VoD (Video on Demand), for example, and the user can view or use the desired service through the UI (User Interface) screen provided by the IPTV. have.

In general, a set-top box is required to receive an IPTV service.

In particular, in the case of recent set-top boxes or speakers linked to set-top boxes, voice recognition technology called AI (Artificial Intelligence) service is applied to recognize user control commands through conversation with users and respond to the recognized user control commands. It has reached the stage of performing the corresponding action.

Here, voice recognition technology is one of the representative Human Machine Interface (HMI) technologies, and its application is expanding day by day. .

However, in the case of voice recognition technology, various time variability such as a speaker, a surrounding environment, a voice recognition device, etc. exist, and it is difficult to guarantee voice recognition quality due to these factors.

Therefore, in order to improve user satisfaction in an IPTV service environment employing voice recognition technology, it is necessary to periodically evaluate and manage the voice recognition quality of voice communication devices equipped with voice recognition technology.

Accordingly, the present invention intends to propose a method for evaluating the quality of a voice communication device applied to a voice recognition technology in an IPTV service environment.

The present invention was created in view of the above circumstances, and an object of the present invention is to evaluate the voice recognition quality by comprehensively considering the support function of a voice communication device equipped with a voice recognition technology in an IPTV service environment. .

A voice recognition quality evaluation apparatus according to an embodiment of the present invention for achieving the above object is a comprehensive quality evaluation method including a basic quality evaluation method for evaluating the recognition performance of a test voice signal according to a support function of a voice communication device an evaluation method determining unit to determine and measuring a time required until a response operation for each supported function of the voice communication device is detected in the process of performing the basic quality evaluation method, and calculating at least one of the required time and the execution result of the basic quality evaluation method. and a quality evaluation unit for evaluating the quality of the voice communication device according to the comprehensive quality evaluation method based on the quality evaluation method.

Specifically, the quality evaluation unit, when the feedback utterance signal from the voice communication device is recognized with respect to the test voice signal generated based on a test script predefined according to the basic quality evaluation method, the feedback utterance signal The quality of the voice communication device may be evaluated based on the similarity between the recognized result of the feedback text and the response script matching the test script.

Specifically, the test voice signal is output through a single speaker in the form of synthesizing a noise signal according to a preset signal-to-noise ratio (SNR) with respect to a basic voice signal matching the test script, or the basic voice signal and the The noise signal may be output through each speaker.

Specifically, the quality evaluator may include a call, which is a time from the end of uttering the call command for calling the voice communication device before the utterance of the test voice signal to the point at which a reaction operation of the voice communication device to the call command is detected. It is possible to measure at least one of a required time for recognition and a required time for command processing, which is a time from the end of uttering the test command in which the test voice signal is uttered to the time when a reaction operation of the voice communication device is detected in response to the test command. .

Specifically, the time required for call recognition is the time from when the LED operation of the voice communication device is detected to the call command according to the support function of the voice communication device, and the time when a response sound of the voice communication device is detected. It may be measured as a time to at least one of the time points.

Specifically, the command processing time required is the timing at which the feedback utterance of the voice communication device starts in response to the test command according to the support function of the voice communication device, and the feedback utterance from the UI (User Interface) of the voice communication device. It may be measured as a time from when the related image signal is output to at least one time point.

In order to achieve the above object, a method of operating a voice recognition quality evaluation apparatus according to an embodiment of the present invention for achieving the above object includes a basic quality evaluation method for evaluating the recognition performance of a test voice signal according to a support function of a voice communication device. an evaluation method determining step of determining a quality evaluation method; and measuring a time required until a response operation for each supported function of the voice communication device is detected in the process of performing the basic quality evaluation method, and calculating at least one of the required time and the execution result of the basic quality evaluation method. and a quality evaluation step of evaluating the quality of the voice communication device based on the comprehensive quality evaluation method.

Specifically, in the quality evaluation step, when the feedback utterance signal from the voice communication device is recognized with respect to the test voice signal generated based on a test script predefined according to the basic quality evaluation method, the feedback utterance signal The quality of the voice communication device may be evaluated based on the similarity between the feedback text, which is a result of recognizing , and the response script matching the test script.

Specifically, the test voice signal is output through a single speaker in the form of synthesizing a noise signal according to a preset signal-to-noise ratio (SNR) with respect to a basic voice signal matching the test script, or the basic voice signal and the The noise signal may be output through each speaker.

Specifically, in the quality evaluation step, the time from the end of uttering the call command for calling the voice communication device before the utterance of the test voice signal to the point at which a reaction operation of the voice communication device to the call command is detected. It is possible to measure at least one of a call recognition time and a command processing time, which is a time from the end of the test command utterance in which the test voice signal is uttered to the time when a reaction operation of the voice communication device is detected in response to the test command. have.

Specifically, the time required for call recognition is at least among a time when an LED operation of the voice communication device is detected and a response sound of the voice communication device is detected in response to the call command according to the support function of the voice communication device. It can be measured as the time to a point in time.

Specifically, the command processing time required is the time when the feedback utterance of the voice communication device is started in response to the test command or the feedback utterance from the UI (User Interface) of the voice communication device according to the support function of the voice communication device. It may be measured as the time until the time at which the related image signal is output.

Therefore, according to the voice recognition quality evaluation apparatus and the operating method thereof according to the present invention, it is possible to evaluate and manage the voice recognition quality by comprehensively considering the support function of the voice communication device equipped with the voice recognition technology in the IPTV service environment, It is possible to greatly improve user satisfaction in IPTV services that employ voice recognition technology.

1 is an exemplary diagram for explaining an IPTV service environment according to an embodiment of the present invention.
2 is a schematic configuration diagram of a voice recognition quality evaluation apparatus according to an embodiment of the present invention.
3 and 4 are exemplary diagrams for explaining a test scenario according to an embodiment of the present invention.
5 is an exemplary diagram for explaining a required time measured for a voice communication device according to an embodiment of the present invention;
6 is a flowchart for explaining a method of operating a voice recognition quality evaluation apparatus according to an embodiment of the present invention.

Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

1 schematically shows an IPTV service environment according to an embodiment of the present invention.

The IPTV service environment according to an embodiment of the present invention includes a voice communication device 100 equipped with voice recognition technology.

The voice communication device 100 may be, for example, an AI speaker for interworking with a set-top box installed in the house to receive an IPTV service, or an AI set-top box that functions both as a set-top box and an AI speaker. .

The voice communication device 100 may recognize a user control command from a conversation with the user through voice recognition technology and perform an operation corresponding to the recognized user control command.

Meanwhile, in order to recognize a user control command from a conversation with the user and perform an accurate operation corresponding to the recognized user control command, it is necessary to ensure reliability and an objective recognition rate.

However, in the case of voice recognition technology, various time variability such as a speaker, a surrounding environment, and a voice recognition device exist, and it is difficult to guarantee voice recognition quality due to these factors.

Therefore, in order to improve user satisfaction in the IPTV service environment employing the voice recognition technology, it is necessary to periodically evaluate and manage the voice recognition quality of the voice communication device 100 equipped with the voice recognition technology.

Accordingly, in an embodiment of the present invention, as a configuration for evaluating the quality of a voice communication device equipped with a voice recognition technology in an IPTV service environment, the voice recognition quality evaluation apparatus 200 is further included.

The voice recognition quality evaluation device 200 can evaluate the voice recognition quality by comprehensively considering the support functions of the voice communication device 100, and in the following, in the configuration of the voice recognition quality evaluation device 200 for realizing this will be described in more detail.

2 shows a schematic configuration of a voice recognition quality evaluation apparatus 200 according to an embodiment of the present invention.

As shown in Fig. 2, the voice recognition quality evaluation apparatus 200 according to an embodiment of the present invention includes an evaluation method determining unit 210 that determines a quality evaluation method, and a voice command unit generating a test voice signal ( 220), a spoken speech recognition unit 230 for recognizing a feedback speech signal (voice signal), and a quality evaluation unit 260 for evaluating quality.

In addition, in one embodiment of the present invention, the configuration of the LED motion detection unit 240 and the image processing unit 250 may be further included according to the supported function of the voice communication device 100 for which quality evaluation is performed.

On the other hand, the voice recognition quality evaluation apparatus 200 according to an embodiment of the present invention controls each component in the voice recognition quality evaluation apparatus 200 that is operated for quality evaluation of the voice communication device 100 in addition to the above-described configuration. It may further include a control unit 270 as a separate component for the.

All or at least a part of the configuration of the voice recognition quality evaluation apparatus 200 may be implemented in the form of a hardware module or a software module, or may be implemented in a form in which a hardware module and a software module are combined.

Here, the software module may be understood as, for example, a command executed by a processor for controlling operations in the voice recognition quality evaluation device 200, and these commands are mounted in a memory in the voice recognition quality evaluation device 200. It may have the form of an application.

As described above, the voice recognition quality evaluation apparatus 200 according to an embodiment of the present invention can evaluate the voice recognition quality by comprehensively considering the support functions of the voice communication device 100 through the above-described configuration. Each configuration in the voice recognition quality evaluation apparatus 200 for realization will be described in more detail.

The evaluation method determining unit 210 performs a function of determining a quality evaluation method for the voice communication device 100 .

More specifically, the evaluation method determining unit 210 determines a comprehensive quality evaluation method for evaluating the quality of the voice communication device 100 according to the supported function of the voice communication device 100 .

Here, in the case of the support function of the voice communication device 100, whether the target device equipped with the voice recognition technology is an AI speaker that supports only an interface based on a voice signal or an AI set-top that supports both an interface based on a voice signal and a video signal It may vary depending on whether the box is recognized.

On the other hand, as described above, the comprehensive quality evaluation method determined according to the supported function of the voice communication device 100 may be determined to basically include a basic quality evaluation method for evaluating the recognition performance of the voice communication device 100 for the test voice signal. have.

The quality evaluation process of the voice communication device 100 according to the basic quality evaluation method is as follows.

The voice command unit 220 may generate a test voice signal based on a predefined test script, and output the generated test voice signal through the speaker 221 for utterance.

Here, the test script refers to a scenario in the form of text for evaluating the quality of the voice communication device 100, and is stored in a separate database (DB) in a form in which a plurality of test scripts and a response script for each of the test scripts are matched. It can be stored and managed.

For example, a test script may include "Raise the room temperature", "Play a fun entertainment program", etc., and the corresponding response script can include "How many degrees should I set the room temperature?", "*** Pro * *Would you like to watch the episode again?" and the like.

In this case, it goes without saying that the response script may include a script composed of several text types for one test script.

In summary, the speech command unit 220 receives any one test script selected from among a plurality of pre-stored test scripts, and converts the input test script into speech through a Text To Speech (TTS) module to perform the test. It can generate a voice signal.

Here, the voice command unit 220 may employ a random selection method to select a plurality of test scripts or, alternatively, may employ a designated selection method according to a preset order or pattern.

Of course, in addition to using the test script, the voice command unit 220 may generate a test voice signal using a source sound source and output the generated test voice signal through the speaker 101 for utterance.

To this end, the voice command unit 220 records a human voice or generates a source sound source through SSML (Speech Synthesis Markup Language)-based TTS technology, and uses deep learning technology (GAN/VAE) to select the source sound source. It is possible to generate a test voice signal composed of a plurality of tones and intonations by modulation.

Meanwhile, in generating the test voice signal, the voice command unit 220 may generate a test voice signal by synthesizing noise and a background sound with a signal-to-noise ratio (SNR) to a source sound source, and alternatively, a background noise speaker 222 . The noise and background sound may be output separately from the speaker 221 for utterance, and the voice command unit 220 may generate a test voice signal by synthesizing sound effects including echo and delay.

That is, the test voice signal is output to the speaker 221 for utterance in the form of synthesizing a noise signal according to a preset signal-to-noise ratio (SNR) with respect to a basic voice signal matching the test script, or the basic voice signal is a speaker for utterance 221 , and the noise signal may be output through a background noise speaker 222 that is a separate speaker from the speaker 221 for speech.

Here, the signal-to-noise ratio (SNR) may be calculated, for example, through [Equation 1] below.

[Formula 1]

A _signal and A _noise represent the magnitude or intensity of a voice and noise, respectively, and in this embodiment, for example, a root mean square (RMS) of an amplitude value may be defined as the intensity of each sound.

Here, in the case of the mean square root of the amplitude value, considering the case where the amplitude value of the voice comes out as a negative value, the square root of the last averaged value is calculated by averaging the value after squaring the amplitude value and adding the squared value. method can be obtained.

Furthermore, after the test voice signal is output, the spoken voice recognition unit 230 receives a feedback utterance signal (voice signal) from the voice conversation device 100 for the output test voice signal and recognizes it through the voice recognition engine. have.

In this case, the spoken speech recognition unit 230 may generate a feedback text by converting the recognition result of the feedback speech signal into text through a text processing module (not shown).

On the other hand, the spoken voice recognition unit 230 that receives the feedback utterance signal (voice signal) from the voice communication device 100 and recognizes it through the voice recognition engine, the voice communication device 100 supports only the voice signal-based interface It can be understood as a configuration considering the case of an AI speaker.

If the voice communication device 100 according to an embodiment of the present invention is an AI set-top box supporting both an audio signal and an image signal-based interface, the feedback text may be generated through the image processing unit 250 . .

That is, the image processing unit 250 recognizes the text from the HDMI (High Definition Multimedia Interface) image signal output from the voice communication device 100 using OCR (Optical Character Recognition) after the test voice signal is output. You can create feedback text.

On the other hand, when the feedback text is generated as described above, the quality evaluation unit 260 evaluates the quality of the voice communication device 100 based on the similarity between the feedback text and the test script used to generate the test voice signal and the matching response script. .

In this case, when the similarity measurement result value between the feedback text and the response script is greater than or equal to a preset similarity measurement reference value, the quality evaluation unit 260 may evaluate the quality of the quality evaluation unit 260 as excellent.

For example, FIG. 3 exemplarily shows a test scenario when the voice communication device 100 according to an embodiment of the present invention is an AI speaker.

Referring to FIG. 3 , when the voice recognition quality evaluation device 200 performs a voice utterance such as “turn on the heating in the living room”, the AI speaker may respond such as “at what degree?” The quality evaluation device 200 may perform voice recognition for the response of the AI speaker, and may perform a dialog recognition function through a text processing module.

Then, the voice recognition quality evaluation device 200 may perform a voice utterance like "25 degrees", and the AI speaker may respond like "I turned on the living room heating and set it to 25 degrees".

In this case, the voice recognition quality evaluation apparatus 200 may automatically perform a test targeting the AI speaker by performing voice recognition on the response of the AI speaker.

As another example, FIG. 4 exemplarily shows a test scenario when the voice communication device 100 according to an embodiment of the present invention is an AI set-top box.

Referring to FIG. 4 , when the voice recognition quality evaluation device 200 performs a voice utterance such as “Show me Infinite Challenge”, the AI set-top box performs a status check through image analysis and at the same time outputs a status screen to the IPTV. After that, when the video analysis is completed, the AI set-top box can output the result screen searched for through video analysis on the IPTV.

Subsequently, the voice recognition quality evaluation apparatus 200 may perform a voice utterance such as "play the first off" after confirming the search result through image analysis.

In this case, the AI set-top box can check the status through video analysis and output the status screen to IPTV at the same time. can be printed on

The description of the quality evaluation process of the voice communication device 100 defined by the basic quality evaluation method according to an embodiment of the present invention is finished, and the description of the comprehensive quality evaluation method is continued.

The quality evaluation unit 260 performs a function of evaluating the quality according to the overall quality evaluation method determined for the voice communication device 100 .

More specifically, the quality evaluation unit 260 measures and measures the required time until a reaction operation for each supported function of the voice communication device is detected in the process of performing the basic quality evaluation method for the voice communication device 100 . The quality of the voice communication device 100 according to the comprehensive quality evaluation method is evaluated using at least one of the required time and the result of performing the basic quality evaluation method.

At this time, as shown in FIG. 5 , the quality evaluation unit 260 responds to the call command from the time when the call command is uttered when the voice communication device 100 is called before the utterance of the test voice signal according to the basic quality evaluation method as shown in FIG. It is possible to measure the time required for call recognition, which is the time until the time when the reaction operation of the chatting device 100 is detected.

That is, the time required for call recognition can be measured as the time until the time when the voice communication device 100 is an AI speaker, the response sound of the AI speaker to the call command (eg, tee-dick, beep, etc.) is detected, When the conversation device 100 is an AI set-top box, it may be measured as the time until the time when the LED operation of the AI set-top box is detected in response to a call command.

Here, the LED operation of the AI set-top box may be detected by the LED operation detection unit 240 operating in conjunction with the observation camera 241 .

In addition, as shown in FIG. 5 exemplified above, the quality evaluation unit 260 responds to the test command from the time when the test voice signal is uttered (outputted) according to the basic quality evaluation method to the test command. It is possible to measure the time required for command processing, which is the time until the point at which the reaction action of ' is detected.

That is, the command processing time required may be measured as the time until the start of feedback utterance of the AI speaker for the test command when the voice communication device 100 is an AI speaker, and the voice communication device 100 is an AI set-top In the case of the box, it may be measured as the time from the UI (User Interface) of the AI set-top box to the UI reaction time, which is the time when the video signal related to the feedback utterance is output to the test command.

Here, the confirmation of the UI response time of the AI set-top box may be performed by the operation of the image processing unit 250 for detecting the HDMI image signal output from the AI set-top box through the HDMI capture board 251 .

As a result, when the quality evaluation unit 260 measures the time required for call recognition and the command processing time for the voice communication device 100 in the process of performing the basic quality evaluation method for the voice communication device 100, it is determined, respectively. By comparing with the reference value of , the quality based on the required time for the voice communication device 100 can be evaluated, and further, by reflecting the quality evaluation result based on the required time to the quality evaluation result according to the basic quality evaluation method, A quality evaluation result according to the overall quality evaluation method determined for the voice communication device 100 may be derived.

As described above, according to the configuration of the voice recognition quality evaluation apparatus 200 according to an embodiment of the present invention, the voice recognition quality is comprehensively considered in the IPTV service environment with the support function of the voice communication device equipped with the voice recognition technology. can be evaluated and managed, greatly improving user satisfaction in IPTV services employing voice recognition technology.

Hereinafter, an operation method of the voice recognition quality evaluation apparatus 200 according to an embodiment of the present invention will be described with reference to FIG. 6 .

First, the evaluation method determining unit 210 determines a comprehensive quality evaluation method for evaluating the quality of the voice communication device 100 according to the supported function of the voice communication device 100 (S10).

Here, in the case of the support function of the voice communication device 100, whether the target device equipped with the voice recognition technology is an AI speaker that supports only an interface based on a voice signal or an AI set-top that supports both an interface based on a voice signal and a video signal It may vary depending on whether the box is recognized.

On the other hand, as described above, the comprehensive quality evaluation method determined according to the supported function of the voice communication device 100 may be determined to basically include a basic quality evaluation method for evaluating the recognition performance of the voice communication device 100 for the test voice signal. have.

Next, when the quality evaluation unit 260 determines the overall quality evaluation method for the voice communication device 100 , the call command is uttered to call the voice communication device 100 before the test voice signal is uttered according to the basic quality evaluation method. The time required for call recognition, which is the time from the end time point to the time point at which a reaction operation of the voice communication device 100 is detected in response to the call command, is measured (S20).

At this time, the quality evaluation unit 260, as shown in FIG. 5 exemplified above, from the time when the call command that called the voice communication device 100 is called before the utterance of the test voice signal according to the basic quality evaluation method, from the time of uttering the corresponding call command. It is possible to measure the time required for call recognition, which is the time until the time when the reaction operation of the voice communication device 100 is detected.

That is, the time required for call recognition can be measured as the time until the time when the voice communication device 100 is an AI speaker, the response sound of the AI speaker to the call command (eg, tee-dick, beep, etc.) is detected, When the conversation device 100 is an AI set-top box, it may be measured as the time until the time when the LED operation of the AI set-top box is detected in response to a call command.

Here, the LED operation of the AI set-top box may be detected by the LED operation detection unit 240 operating in conjunction with the observation camera 241 .

Furthermore, when the measurement of the required time for call recognition for the voice communication device 100 is completed, the voice command unit 220 generates a test voice signal based on a predefined test script, and transmits the generated test voice signal. It is output through the speaker 221 for speech (S30).

Here, the test script refers to a scenario in the form of text for evaluating the quality of the voice communication device 100, and is stored in a separate database (DB) in a form in which a plurality of test scripts and a response script for each of the test scripts are matched. It can be stored and managed.

For example, a test script may include "Raise the room temperature", "Play a fun entertainment program", etc., and the corresponding response script can include "How many degrees should I set the room temperature?", "*** Pro * *Would you like to watch the episode again?" and the like.

In this case, it goes without saying that the response script may include a script composed of several text types for one test script.

In summary, the speech command unit 220 receives any one test script selected from among a plurality of pre-stored test scripts, and converts the input test script into speech through a Text To Speech (TTS) module to perform the test. It can generate a voice signal.

Here, the voice command unit 220 may employ a random selection method to select a plurality of test scripts or, alternatively, may employ a designated selection method according to a preset order or pattern.

Of course, in addition to using the test script, the voice command unit 220 may generate a test voice signal using a source sound source and output the generated test voice signal through the speaker 101 for utterance.

To this end, the voice command unit 220 records a human voice or generates a source sound source through SSML (Speech Synthesis Markup Language)-based TTS technology, and uses deep learning technology (GAN/VAE) to select the source sound source. It is possible to generate a test voice signal composed of a plurality of tones and intonations by modulation.

Meanwhile, in generating the test voice signal, the voice command unit 220 may generate a test voice signal by synthesizing noise and a background sound with a signal-to-noise ratio (SNR) to a source sound source, and alternatively, a background noise speaker 222 . The noise and background sound may be output separately from the speaker 221 for utterance, and the voice command unit 220 may generate a test voice signal by synthesizing sound effects including echo and delay.

That is, the test voice signal is output to the speaker 221 for utterance in the form of synthesizing a noise signal according to a preset signal-to-noise ratio (SNR) with respect to a basic voice signal matching the test script, or the basic voice signal is a speaker for utterance 221 , and the noise signal may be output through a background noise speaker 222 that is a separate speaker from the speaker 221 for speech.

Then, as shown in FIG. 5 exemplified above, the quality evaluation unit 260 responds to the test command from the time when the test voice signal is uttered (outputted) according to the basic quality evaluation method to the test command. ), the time required for the command processing, that is, the time until the reaction operation is detected (S40).

That is, the command processing time required may be measured as the time until the start of feedback utterance of the AI speaker for the test command when the voice communication device 100 is an AI speaker, and the voice communication device 100 is an AI set-top In the case of the box, it may be measured as the time from the UI (User Interface) of the AI set-top box to the UI reaction time, which is the time when the video signal related to the feedback utterance is output to the test command.

Here, the confirmation of the UI response time of the AI set-top box may be performed by the operation of the image processing unit 250 for detecting the HDMI image signal output from the AI set-top box through the HDMI capture board 251 .

Then, after the test voice signal is output, the spoken voice recognition unit 230 receives a feedback utterance signal (voice signal) from the voice conversation device 100 with respect to the output test voice signal and generates a recognized feedback text. do (S50).

In this case, the spoken speech recognition unit 230 may generate a feedback text by converting the recognition result of the feedback speech signal into text through a text processing module (not shown).

On the other hand, the spoken voice recognition unit 230 that receives the feedback utterance signal (voice signal) from the voice communication device 100 and recognizes it through the voice recognition engine, the voice communication device 100 supports only the voice signal-based interface It can be understood as a configuration considering the case of an AI speaker.

If the voice communication device 100 according to an embodiment of the present invention is an AI set-top box supporting both an audio signal and an image signal-based interface, the feedback text may be generated through the image processing unit 250 . .

That is, the image processing unit 250 recognizes the text from the HDMI (High Definition Multimedia Interface) image signal output from the voice communication device 100 using OCR (Optical Character Recognition) after the test voice signal is output. You can create feedback text.

Furthermore, when the feedback text is generated in this way, the quality evaluation unit 260 evaluates the quality of the voice communication device 100 based on the similarity between the feedback text and the test script used to generate the test voice signal and the matching response script ( S60).

In this case, when the similarity measurement result value between the feedback text and the response script is greater than or equal to a preset similarity measurement reference value, the quality evaluation unit 260 may evaluate the quality of the quality evaluation unit 260 as excellent.

Thereafter, the quality evaluation unit 260 sets the time required for call recognition and the command processing time measured for the voice communication device 100 in the process of performing the basic quality evaluation method for the voice communication device 100 as each criterion. By comparing the value with the value, it is possible to evaluate the quality based on the required time for the voice communication device 100, and further, by reflecting the quality evaluation result based on the required time to the quality evaluation result according to the basic quality evaluation method, voice conversation A quality evaluation result according to the overall quality evaluation method determined for the device 100 may be derived (S70-S80).

As described above, according to the operation method of the voice recognition quality evaluation apparatus 200 according to an embodiment of the present invention, voice recognition is comprehensively considered in the IPTV service environment with the support function of the voice communication device equipped with the voice recognition technology. Since quality can be evaluated and managed, user satisfaction in IPTV services employing voice recognition technology can be greatly improved.

On the other hand, the functional operations and implementations of the subject matter described in this specification are implemented as digital electronic circuits, computer software, firmware, or hardware including the structures disclosed in this specification and structural equivalents thereof, or at least one of them It can be implemented by combining. Implementations of the subject matter described herein are one or more computer program products, ie one or more modules of computer program instructions encoded on a tangible program storage medium for controlling the operation or execution by the control system. can be implemented.

The computer readable medium may be a machine readable storage device, a machine readable storage substrate, a memory device, a composition of matter that affects a machine readable radio wave signal, or a combination of one or more thereof.

As used herein, the term “system” or “device” encompasses all devices, devices and machines for controlling data, including, for example, a programmable processor, a computer, or multiple processors or computers. The control system may include, in addition to hardware, code forming an execution environment for a computer program upon request, such as, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. .

A computer program (also known as a program, software, software application, script or code) may be written in any form of a programming language, including compiled or interpreted language or a priori or procedural language, as a stand-alone program or module; It can be deployed in any form, including components, subroutines, or other units suitable for use in a computer environment. A computer program does not necessarily correspond to a file in a file system. A program may be in a single file provided to the requested program, or in multiple interacting files (eg, files that store one or more modules, subprograms, or portions of code), or portions of files that hold other programs or data. (eg, one or more scripts stored within a markup language document). The computer program may be deployed to be executed on a single computer or multiple computers located at one site or distributed over a plurality of sites and interconnected by a communication network.

On the other hand, computer-readable media suitable for storing computer program instructions and data are, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices, such as magnetic disks such as internal hard disks or external disks, magneto-optical disks and CDs. -Can include all types of non-volatile memory, media and memory devices, including ROM and DVD-ROM disks. The processor and memory may be supplemented by, or integrated into, special purpose logic circuitry.

An implementation of the subject matter described herein may include a backend component, such as a data server, or a middleware component, such as an application server, such as a web browser or graphical user that allows a user to interact with an implementation of the subject matter described herein, for example. It may be implemented in a front-end component, such as a client computer having an interface, or in a computing system including any combination of one or more of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as, for example, a communication network.

While this specification contains numerous specific implementation details, they should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. should be understood Likewise, certain features that are described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments, either individually or in any suitable subcombination. Furthermore, although features operate in a particular combination and may be initially depicted as claimed as such, one or more features from a claimed combination may in some cases be excluded from the combination, the claimed combination being a sub-combination. or a variant of a sub-combination.

Also, although operations are depicted in the drawings in a specific order in this specification, it is not to be understood that such operations must be performed in the specific order or sequential order shown or that all illustrated operations must be performed in order to achieve desirable results. Can not be done. In certain cases, multitasking and parallel processing may be advantageous. Further, the separation of the various system components of the above-described embodiments should not be construed as requiring such separation in all embodiments, and the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that

As such, this specification is not intended to limit the invention to the specific terms presented. Accordingly, although the present invention has been described in detail with reference to the above-described examples, those skilled in the art can make modifications, changes, and modifications to the examples without departing from the scope of the present invention. The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

According to the voice recognition quality evaluation apparatus and the method of operation thereof according to an embodiment of the present invention, the voice recognition quality can be evaluated by comprehensively considering the support functions of the voice communication device equipped with the voice recognition technology in the IPTV service environment. , it is an invention with industrial applicability because the possibility of marketing or business of the applied device, not just the use of the related technology, is sufficient as it goes beyond the limits of the existing technology, and it can be clearly implemented in reality.

100: voice communication device
200: voice recognition quality evaluation device
210: evaluation method determination unit 220: voice command unit
230: spoken voice recognition unit 240: LED motion detection unit
250: image processing unit 260: quality evaluation unit

Claims (12)

an evaluation method determining unit that determines a comprehensive quality evaluation method for evaluating the recognition performance of a test voice signal according to a supported function of the voice communication device; and
In the process of performing the basic quality evaluation method accompanying the comprehensive quality evaluation method, the required time until a reaction operation for each supported function of the voice communication device is detected is measured, and the required time and the basic quality evaluation method are measured. a quality evaluation unit for evaluating the quality of the voice communication device according to the comprehensive quality evaluation method based on at least one of the performance results;
The quality evaluation unit,
When the feedback utterance signal from the voice communication device is recognized with respect to the test voice signal generated based on a pre-defined test script according to the basic quality evaluation method, the feedback text resulting from recognizing the feedback utterance signal and the Evaluating the quality of the voice communication device based on the similarity between the test script and the matching response script,
The test voice signal is
A noise signal according to a preset signal-to-noise ratio (SNR) is synthesized for a basic voice signal matching the test script and output through one speaker, or the basic voice signal and the noise signal are transmitted through each speaker A voice recognition quality evaluation device, characterized in that it has an output form. delete delete The method of claim 1,
The quality evaluation unit,
The time required for call recognition, which is the time from the end of uttering the call command for calling the voice communication device prior to the utterance of the test voice signal to the point at which a reaction operation of the voice communication device to the call command is detected, and the test voice A voice recognition quality evaluation apparatus according to claim 1, wherein at least one of command processing time required is a time from a time when a signal is uttered and a test command is uttered to a point in time when a reaction operation of the voice communication device is detected in response to the test command. 5. The method of claim 4,
The time required for call recognition is
According to the support function of the voice communication device, the time until at least one of a time point at which an LED operation of the voice communication device is detected in response to the call command and a time point at which a response sound of the voice communication device is detected. Voice recognition quality evaluation device, characterized in that measured. 5. The method of claim 4,
The command processing time required is,
At least one of a time point at which the feedback utterance of the voice communication device is started in response to the test command according to the support function of the voice communication device and a time point at which a video signal related to the feedback utterance is output from a user interface (UI) of the voice communication device Voice recognition quality evaluation device, characterized in that measured as the time to the point of time. an evaluation method determination step of determining a comprehensive quality evaluation method for evaluating the recognition performance of the test voice signal according to the support function of the voice communication device; and
In the process of performing the basic quality evaluation method accompanying the comprehensive quality evaluation method, the required time until a reaction operation for each supported function of the voice communication device is detected is measured, and the required time and the basic quality evaluation method are measured. a quality evaluation step of evaluating the quality of the voice communication device according to the comprehensive quality evaluation method based on at least one of the performance results;
The quality evaluation step is,
When the feedback utterance signal from the voice communication device is recognized with respect to the test voice signal generated based on a pre-defined test script according to the basic quality evaluation method, the feedback text, which is a result of recognizing the feedback utterance signal, and the Evaluating the quality of the voice communication device based on the similarity between the test script and the matching response script,
The test voice signal is
A noise signal according to a preset signal-to-noise ratio (SNR) is synthesized for the basic voice signal matching the test script and output through one speaker, or the basic voice signal and the noise signal are transmitted through each speaker A method of operating a voice recognition quality evaluation device, characterized in that it has an output form. delete delete 8. The method of claim 7,
The quality evaluation step is
The time required for call recognition, which is the time from the end of uttering the call command for calling the voice communication device prior to the utterance of the test voice signal to the point at which a reaction operation of the voice communication device to the call command is detected, and the test voice A voice recognition quality evaluation apparatus, characterized in that at least one of command processing required time, which is a time from a time when a signal is uttered, when a test command is uttered to a point in time when a reaction operation of the voice communication device is detected in response to the test command, is measured. how it works. 11. The method of claim 10,
The time required for call recognition is
According to the support function of the voice communication device, it is measured as the time until at least one of a time when an LED operation of the voice communication device is detected in response to the call command and a time when a response sound of the voice communication device is detected. A method of operating a voice recognition quality evaluation device, characterized in that it. 11. The method of claim 10,
The command processing time required is,
According to the support function of the voice communication device, the time until the time when the feedback utterance of the voice communication device is started in response to the test command or the video signal related to the feedback utterance is output from the UI (User Interface) of the voice communication device Method of operation of a voice recognition quality evaluation device, characterized in that measured as.

Images