JP2022111564A - Information processing device, program, and information processing method - Google Patents

Abstract

To provide an information processing device, a program, and an information processing method capable of evaluating voice recognition accuracy based on a plurality of states of voice output.SOLUTION: An information processing device includes: a voice output control unit that controls a voice output from a voice output device, based on voice output conditions; an acquisition unit that acquires recognition result text information indicating the recognition result of the voice, from a voice recognition device that recognizes the voice output from the voice output device; a calculation unit for calculating recognition accuracy of the recognition result based on test text information and the recognition result text information, referring to a text storage unit that stores the test text information indicating a content of test voice included in the voice; and an output unit for outputting the calculated recognition accuracy and the conditions in association with each other. The voice output control unit changes the conditions based on the recognition accuracy calculated by the calculation unit, and controls an output of the voice from the voice output device based on the changed conditions.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing device, a program, and an information processing method.

Conventionally, there are techniques for evaluating speech recognition accuracy. For example, in the speech recognition accuracy estimation device disclosed in Patent Document 1 below, as an evaluation of the result of speech recognition, input speech is recognized and recognized based on a word alignment network that indicates the probability of being correct. Estimate the speech recognition accuracy for each word.

JP 2018-25717 A

By the way, there are various situations in which the speech recognition device is used (hereinafter also referred to as "usage situation"), such as creating minutes in a meeting and device control at home. In addition, it is conceivable that the state of output of speech to be recognized differs depending on the usage scene, such as where the speaker is and how he or she speaks. However, in the above-mentioned patent document, there is a problem that speech recognition accuracy cannot be evaluated based on the plurality of states of speech output.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an information processing apparatus, a program, and an information processing method capable of evaluating speech recognition accuracy based on a plurality of states of speech output.

An information processing apparatus according to an aspect of the present invention includes a voice output control unit that controls voice output from a voice output device based on a voice output condition, and a voice recognition unit that recognizes voice output from the voice output device. With reference to an acquisition unit for acquiring recognition result text information indicating the speech recognition result from the device and a text storage unit for storing test text information indicating the content of the test speech included in the speech, the test text information and a calculation unit for calculating the recognition accuracy of the recognition result based on the recognition result text information; The condition is changed based on the recognition accuracy calculated by , and the voice output from the voice output device is controlled based on the changed condition.

A program according to an aspect of the present invention provides a computer with an audio output control function for controlling output of audio from an audio output device based on audio output conditions, and audio obtained by recognizing the audio output from the audio output device. With reference to an acquisition function for acquiring recognition result text information indicating the speech recognition result from the recognition device and a text storage unit for storing test text information indicating the content of the test speech included in the speech, the test text information is obtained. and recognition result text information, and an output function for outputting the calculated recognition accuracy and condition in association with each other. The condition is changed based on the recognition accuracy calculated by the calculation function, and the output of the sound from the sound output device is controlled based on the changed condition.

In an information processing method according to an aspect of the present invention, a computer controls speech output from a speech output device based on speech output conditions, and a speech recognition device recognizes speech output from the speech output device. acquires recognition result text information indicating the speech recognition result, refers to the text storage unit that stores test text information indicating the content of the test speech included in the speech, and stores the test text information and the recognition result text information. Based on , calculate the recognition accuracy of the recognition result, output the calculated recognition accuracy and condition in correspondence, change the condition based on the calculated recognition accuracy, and output voice based on the changed condition Controls audio output from the device.

According to the present invention, it is possible to provide an information processing apparatus, a program, and an information processing method capable of evaluating speech recognition accuracy based on a plurality of states of speech output.

It is a figure for demonstrating the system configuration example of the evaluation system which concerns on this embodiment. BRIEF DESCRIPTION OF THE DRAWINGS It is a figure for demonstrating the outline|summary of the evaluation system which concerns on this embodiment. It is a figure showing an example of functional composition of a control device concerning this embodiment. It is a table showing an example of output conditions including a plurality of parameters of the evaluation system according to the present embodiment. It is a figure which shows the example of a screen of the evaluation system which concerns on this embodiment. It is a figure which shows the example of a screen of the evaluation system which concerns on this embodiment. It is a figure which shows an example of the relationship between the recognition accuracy of the evaluation system which concerns on this embodiment, and the volume of a sound output device. It is a figure which shows an example of the relationship of the recognition accuracy of the evaluation system which concerns on this embodiment, and the distance with a voice output device. It is a figure which shows the operation example of the control apparatus which concerns on this embodiment. It is a figure which shows an example of the hardware constitutions of the control apparatus which concerns on this embodiment.

A preferred embodiment of the present invention (hereinafter referred to as "this embodiment") will be described with reference to the accompanying drawings. It should be noted that, in each figure, the same reference numerals have the same or similar configurations.

In the present embodiment, "part", "means", "apparatus", and "system" do not simply mean physical means; It also includes the case where the functions possessed by are realized by software. Also, the function of one "unit", "means", "apparatus" or "system" may be realized by two or more physical means or devices. Also, the functions of two or more "parts", "means", "apparatus", and "system" may be realized by one physical means or apparatus.

<1. System configuration>
A system configuration example of an evaluation system 1 according to the present embodiment will be described with reference to FIG. The evaluation system 1 is a system for users such as developers and customers to evaluate the speech recognition accuracy of speech recognition devices. The evaluation system 1 can be used, for example, in the development of a speech recognition device, to evaluate the accuracy of the current product and the next product of the same model to check whether the recognition accuracy has been improved. The evaluation system 1 can also be used, for example, to evaluate and compare the recognition accuracy of the company's speech recognition device and the recognition accuracy of another company's speech recognition device.

As shown in FIG. 1, the evaluation system 1 includes a control device 100, a speech recognition device 200 to be evaluated, and speech output devices 300a to 300c for outputting speech. The evaluation system 1 may also be connected to the speech recognition system 500 via the network N, for example. Note that the audio output devices 300a to 300c are also collectively referred to as the "audio output device 300" when there is no particular need to distinguish between them.

The network N is composed of a wireless network and a wired network. Examples of networks include mobile phone networks, PHS (Personal Handy-phone System) networks, wireless LAN (Local Area Network), 3G (3rd Generation), LTE (Long Term Evolution), 4G (4th Generation), 5G ( 5th Generation), WiMax (registered trademark), infrared communication, Bluetooth (registered trademark), wired LAN, telephone line, power line network, and IEEE1394-compliant network.

The control device 100 is an information processing device that controls audio output from the audio output device 300 . Also, the control device 100 may be connected to the audio output device 300 by wire (eg, USB, HDMI (registered trademark), plug-and-jack, etc.) or wirelessly (eg, Wifi, Bluetooth, etc.). Also, the control device 100 acquires recognition result text information from the speech recognition device 200 . In order to simplify the description, an example in which the control device 100 is a single terminal device will be described, but the control device 100 is not intended to be limited to this. The control device 100 may be configured, for example, by distributing the installed functions to a plurality of terminal devices.

The “recognition result text information” is information indicating the recognition result of the voice output from the voice output device 300 in text (character string).

The speech recognition device 200 is a device to be evaluated by the evaluation system 1 . The speech recognition device 200 is, for example, an information processing device capable of communicating with the control device 100 and the speech recognition system 500 . The speech recognition device 200 acquires the voice of a speaker, converts the acquired voice into text by voice recognition, and records the text.

The speech recognition device 200 may be, for example, a so-called smart speaker that responds to acquired speech through dialogue. As another example, the speech recognition device 200 may be a general-purpose tablet terminal, smartphone, or the like. The speech recognition device 200 may be used as the speech recognition device 200 by, for example, installing a dedicated program in a general-purpose tablet terminal and executing this program.

The audio output device 300 is a device that includes an amplifier circuit and a speaker, and outputs audio. The audio output device 300 outputs audio under the control of the control device 100 . The audio output device 300 may be, for example, a speaker device, or an information processing device such as a smart phone or a laptop terminal capable of communicating with the control device 100 . One or a plurality of audio output devices 300 may be provided. Audio data of audio output by the control device 100 to the audio output device 300 (hereinafter also simply referred to as “audio data”) may be linked from the control device 100 at the time of control, or may be stored in the storage unit of the audio output device 300. (not shown) or an external storage device.

The sound that the control device 100 causes the sound output device 300 to output includes a test sound for evaluating recognition accuracy. In other words, the evaluation system 1 evaluates how well the speech recognition apparatus 200 has recognized the content of the test speech. Also, the output voice may include noise in addition to the test voice, for example. Also, the test voice may include a first voice and a second voice. The second sound is a sound that differs from the first sound in length, volume, frequency, and the like.

The speech recognition system 500 is a system capable of communicating with the speech recognition device 200 . The speech recognition system 500 receives speech data acquired from the speech recognition device 200 and recognizes the speech based on the received data.

<2. System Overview>
An overview of the evaluation system 1 will be described with reference to FIG. In this example, a plurality of speech output devices 300 are installed around a speech recognition device 200 to be evaluated, and a test is performed to evaluate the recognition accuracy of speech output from the plurality of speech output devices 300. explain. In addition, when conducting a test with the evaluation system 1, the speech recognition device 200 and the speech recognition device 200 should be placed in a room for evaluation (in this example, a measurement room) so that speech from the outside does not enter the speech recognition device 200 as much as possible. An output device 300 may be installed.

(1) As shown in FIG. 2, first, the control device 100 instructs the audio output device 300 to output audio based on audio output conditions (hereinafter also referred to as "output conditions").

(2) The audio output device 300 outputs audio based on this instruction.

The “sound output condition (output condition)” is, for example, a condition as to how the sound is to be output from the sound output device 300 . The audio output condition may include, for example, a plurality of parameters relating to audio output (hereinafter also simply referred to as "plurality of parameters" or "parameters"). Details of the parameters will be described later.

(3) The speech recognition device 200 acquires the speech output from the speech output device 300 and recognizes the acquired speech. This speech recognition processing may be executed by the function of the speech recognition device 200 itself, or may be partially or wholly delegated to the speech recognition system 500 . The speech recognition device 200 records the result of speech recognition as recognition result text information.

(4) The control device 100 acquires the recorded recognition result text information via the API of the speech recognition device 200 .

(5) The control device 100 calculates the recognition accuracy of the recognition result based on the test text information and the recognition result text information. Here, the “test text information” is information indicating the content of the test voice included in the voice to be output by the voice output device 300 by text.

(6) The control device 100 associates the recognition accuracy calculated in (5) above with the output conditions applied in (1) above, and outputs them to a screen, a file, or the like.

(7) The control device 100 feeds back the recognition accuracy calculated in (5) above, and instructs the voice output device 300 to output voice again. Specifically, the control device 100 changes the output condition based on this recognition accuracy, and instructs the voice output device 300 to output voice based on the changed output condition.

In the evaluation system 1, the above (1) to (7) may be repeated a predetermined number of times. Specifically, the speech output by the speech output device 300 and the speech recognition by the speech recognition device 200 for this speech output may be performed multiple times. Then, the control device 100 evaluates the recognition accuracy of the speech recognition device 200 while changing the output condition each time when it is executed a plurality of times.

According to the above configuration, the control device 100 can create a plurality of voice output states by changing the output conditions when evaluating the voice recognition accuracy. Therefore, according to the above configuration, the control device 100 can allow the user to evaluate the speech recognition accuracy based on a plurality of states of speech output. Therefore, the control device 100 can make the user evaluate the speech recognition accuracy in an environment assuming various usage situations of the speech recognition device 200 by creating various speech output states.

<3. Functional configuration>
A functional configuration of the control device 100 according to the present embodiment will be described with reference to FIG. As shown in FIG. 3 , the control device 100 includes a control section 110 , a storage section 130 and a communication section 140 .

The control unit 110 includes an audio output control unit 111 , an acquisition unit 112 , a calculation unit 113 , an output unit 114 and a generation unit 115 .

<Audio output controller>
The audio output control unit 111 controls the output of audio from the audio output device 300 based on the output conditions. For example, when the output condition is "to output audio data A", the audio output control unit 111 refers to the storage unit 130 that stores the audio data A, and sends the audio data A to the audio output device 300 via the communication unit 140. Audio data A may be transmitted. For example, the audio output control unit 111 outputs audio data whose output condition is “volume 〇〇 [db], audio length 〇〇 [s], playback speed is 1×, yy/mm/dd h1h1:m1m1 and h1h1: m2m2 twice”, acquires audio data that meets the output condition from the storage unit 130, connects to the audio output device 300 so as to be communicable, and outputs the acquired audio data and audio. Instructions (including parameters included in the output conditions (in this example, timing of output, number of times of output, and playback speed)) may be transmitted to the audio output device 300 .

The output condition may be stored in the storage unit 130 as condition information, for example. Moreover, the output conditions may be stored as condition information by combining a plurality of output conditions as a test scenario.

The audio data may include, for example, test audio data of test audio and noise data that emits noise. The test speech data and the noise data may be, for example, different individual data, or may be mixed. Also, the test audio data may include first test audio data of the first test audio and second test audio data of the second test audio different in length from the first test audio.

The voice output control section 111 changes the output condition based on the recognition accuracy calculated by the calculation section 113 . Then, the audio output control unit 111 controls the output of audio from the audio output device 300 based on the changed output condition.

According to the above configuration, the voice output control unit 111 can create a plurality of voice output states by changing the output conditions when evaluating the voice recognition accuracy. Therefore, according to the above configuration, the voice output control unit 111 can allow the user to evaluate the voice recognition accuracy based on a plurality of states of voice output. Therefore, the voice output control unit 111 can make the user evaluate the voice recognition accuracy in environments assuming various usage scenes of the voice recognition apparatus 200 by creating various voice output states.

For example, when the voice output by the voice output device 300 and the voice recognition by the voice recognition device 200 for this voice output are executed a plurality of times, the voice output control unit 111 performs a plurality of One or more of the parameters used may be changed to control audio output.

According to the above configuration, the audio output control unit 111 can change the parameters to be used among the plurality of parameters related to audio output, thereby giving variation to each output condition. can. Therefore, according to the above configuration, the voice output control unit 111 can allow the user to evaluate the voice recognition accuracy in environments with variations.

The plurality of parameters include, for example, information for specifying audio data of audio to be output to the audio output device 300, sound source localization position (distance and direction from each audio output device 300 with respect to the audio recognition device 200) ), speech speed (including the speed at which the speaker reads and speaks the content of the test speech, as well as the playback speed of the speech), the volume of the speech, the length of the speech, information on the speaker of the speech (for example, , gender, age, information for specifying whether it is a human voice or a synthesized voice, etc.), timing of output, or number of times of output). The "information for specifying the audio data" is, for example, the file name of the file storing the audio data and information (such as URL) for accessing this file. A plurality of parameters may be combined for each output condition, as shown in FIG. 4, for example.

The parameters may include device selection information, for example. Here, the "device selection information" is information for selecting a device to turn on the output from among the plurality of audio output devices 300 for each of a plurality of times.

The parameters may include, for example, device specific information. Here, the "device designation information" is information that designates which of the test sound and noise is to be output for each sound output device 300 in each of a plurality of times.

The parameters may include, for example, the length of the audio output by the audio output device 300 .

For example, if the speech data is data obtained by mixing test speech data and noise data, the parameters may be parameters relating to this mixing (hereinafter also referred to as "mixing parameters").

The parameter may be, for example, the difference between the loudness of the test speech and the loudness of the noise. The parameter may also be, for example, the difference between the volume of the first test sound and the volume of the second test sound.

The audio output control unit 111 may control the audio output by changing the value of each parameter, for example. The audio output control unit 111 may control the audio output by, for example, changing the value of each parameter for each preset value (for example, increasing the volume by 5 [db] sequentially). . The audio output control unit 111 may change the value of each parameter for each threshold of the threshold information or the reference value of the reference value information, for example.

The audio output control unit 111 may control the audio output by switching ON/OFF of the audio output of each of the plurality of audio output devices 300 each time, for example, further based on the device selection information.

According to the above configuration, the voice output control unit 111 can vary the sound source localization by switching on/off the voice output of each of the plurality of voice output devices 300 to evaluate the recognition accuracy.

For example, the audio output control unit 111 may control the audio output by switching the audio to be output to each of the plurality of audio output devices 300 between the test audio and the noise each time, further based on the device designation information. .

For example, it is conceivable that noise such as ambient noise affects the recognition accuracy of the speech recognition apparatus 200 depending on the usage scene, such as outdoors or in commercial facilities. According to the above configuration, the speech output control section 111 can evaluate the recognition accuracy in consideration of the influence of such noise. Therefore, according to the above configuration, the voice output control unit 111 can perform evaluations assuming various usage scenes including scenes with a lot of noise.

For example, the audio output control unit 111 may control the audio output by switching the audio output from each audio output device 300 between the first test audio and the second test audio multiple times.

For example, in a conversation in a conference room, the speech recognition apparatus 200 may be used in situations where a plurality of voices are uttered from different sound sources, such as when a plurality of speakers speak from their respective seats. According to the above configuration, the voice output control unit 111 can perform control so that voice is output under an output condition suitable for such a usage scene.

<Acquisition part>
The acquisition unit 112 acquires recognition result text information indicating the recognition result of the speech output from the speech output device 300 from the speech recognition device 200 that has recognized the speech. The acquisition mode of the acquisition unit 112 may be any mode. For example, a text file of the recognition result text information transmitted from the voice output device 300 may be received. As another example, the acquiring unit 112 acquires the recognition result text information by remotely accessing the speech recognition device 200 to acquire a text file of the recognition result text information. The implemented API may be instructed to refer to the recognition result text information, and the recognition result text information may be acquired as a result.

<Calculation unit>
The calculation unit 113 refers to the text storage unit 131 that stores the test text information, and calculates the recognition accuracy of the recognition result based on this test text information and the recognition result text information acquired by the acquisition unit 112. do. The calculation unit 113 stores the calculated recognition accuracy in the storage unit 130 as accuracy information. For example, the calculation unit 113 may associate the calculated recognition accuracy with the applied output condition and store them as accuracy information in the storage unit 130 . For example, the calculation unit 113 may calculate the degree of matching between the text of the test text information and the text of the recognition result text information as the recognition accuracy. For example, when the text of the test text information and the text of the recognition result text information match 80%, the calculation unit 113 may calculate the recognition accuracy as "0.8" or "80%."

For example, the calculation unit 113 may specify the threshold values or reference values for each of the plurality of parameters based on the combination of the output condition applied in a predetermined learning period and the recognition accuracy calculated under the output condition. The calculation unit 113 stores the identified threshold in the threshold storage unit 132 as threshold information. Further, the calculation unit 113 may store the specified reference value in the storage unit 130 as reference value information, for example.

<Volume threshold or reference value>
For example, when specifying the volume threshold, the calculation unit 113 inputs the volume of the voice in a predetermined learning period and the recognition accuracy of the corresponding voice as learning data, thereby obtaining the volume and the recognition accuracy as shown in FIG. A first pattern model may be constructed. For example, the calculation unit 113 may construct the first pattern model using statistical processing based on regression analysis, with volume as an explanatory variable (feature amount) and recognition accuracy as an objective variable (feature amount). The calculation unit 113 may calculate the recognition accuracy by inputting the sound volume to the constructed first pattern model. For example, the calculation unit 113 sets the possible range of sound volume for each usage scene to a plurality of recognition accuracy values (in this example, sufficient recognition accuracy (0.9) and acceptable recognition accuracy (0.7)). ) to classify into a plurality of levels (“high: volume at which sufficient recognition accuracy can be obtained” and “middle: volume at which acceptable recognition accuracy can be obtained”). The calculation unit 113 may specify, for example, the upper limit value and/or the lower limit value of one of the two divided ranges (R1 or R2) as the volume threshold. Further, the calculation unit 113 may specify, for example, the volume value (d1) that maximizes the recognition accuracy in the first pattern model as the volume reference value.

<Distance threshold or reference value>
For example, the calculation unit 113 inputs the distance between the speech recognition device 200 and the speech output device 300 in a predetermined learning period and the corresponding recognition accuracy as learning data, thereby calculating the distance between the speech recognition device 200 and the speech output device 300 as shown in FIG. A second pattern model of distance and recognition accuracy may be constructed. For example, the calculation unit 113 may construct the second pattern model using statistical processing based on regression analysis using the distance from the audio output device 300 as an explanatory variable (feature amount) and the recognition accuracy as an objective variable (feature amount). good. The calculation unit 113 may input the distance from the voice output device 300 to the constructed second pattern model to calculate the recognition accuracy. The calculation unit 113, for example, divides the possible range of distance from the speaker into a plurality of stages based on a plurality of recognition accuracy values, as in the case of the volume. The calculation unit 113 may specify, for example, the upper limit value and/or the lower limit value of one of the two divided ranges (R3 or R4) as the threshold value of the distance from the audio output device 300 . Further, the calculation unit 113 may specify, for example, the volume value (d2) that maximizes the recognition accuracy in the second pattern model as the distance reference value.

<Frequency threshold or reference value>
The calculation unit 113 may calculate the recognition accuracy, for example, based on the frequency of the sound output by the sound output device 300 during a predetermined learning period. The calculation unit 113 may, for example, calculate a statistical value (average value or median value) of the frequency and specify the calculated statistical value as the reference value of the frequency. Further, the calculation unit 113 may, for example, calculate the frequency band and specify the upper limit value or the lower limit value of the calculated frequency band as the frequency threshold.

For example, the calculation unit 113 may extract, as a feature quantity, the power (or sound pressure level) of at least one of a consonant or a high frequency range equal to or higher than a predetermined threshold, which is included in speech of speech data in a predetermined learning period. good. The "power" here is the so-called acoustic power, which indicates the weight (power) of each frequency in the frequency analysis of sound, and is different from the loudness and strength (volume) of sound sensed by human hearing. The power is the strength of a consonant or a high-frequency voice above a predetermined threshold. The calculation unit 113 may calculate the recognition accuracy based on the extracted feature amount. For example, the calculation unit 113 weights the recognition accuracy calculated above according to the power of the consonant, and uses the plurality of weighted recognition accuracies to determine two levels (“high” and “medium”) in the same manner as the above volume example. can be divided into

For example, the calculation unit 113 may plot the sound pressure level and frequency of the sound on a graph in which the vertical axis is the sound pressure level and the horizontal axis is the frequency. The calculation unit 113 divides the possible range of the plotted data into two recognition accuracy areas (“high” and “medium”) in the same manner as the volume example. The calculation unit 113 may calculate the recognition accuracy depending on, for example, to which of two areas the sound pressure level and frequency of the voice belong.

For example, the calculation unit 113 calculates at least one of (a) the volume, (b) the distance from the speaker, (c) the frequency, and (d) the power of a consonant or a high frequency range equal to or higher than a predetermined threshold, as described above. Composite recognition accuracy (hereinafter, also referred to as “composite recognition accuracy”) may be calculated based on the combination of recognition accuracies calculated by . The calculation unit 113 may specify the threshold value or reference value of each parameter based on the composite recognition accuracy. For example, the calculation unit 113 may specify the value of each parameter when the composite recognition accuracy is maximized as the threshold value or the reference value.

The calculation unit 113 may, for example, calculate a weighted average of the recognition accuracies of (a) to (e), and calculate the calculated weighted average as the composite recognition accuracy. For example, the calculation unit 113 may set the importance of the above (a) and (b) higher than the other (c) to (e) in this weighted average. For example, the calculation unit 113 may weight each recognition accuracy by applying a coefficient proportional to the degree of importance. Specifically, the calculation unit 113 may calculate the composite recognition accuracy using the following formula.

Combined recognition accuracy = (α x recognition accuracy of (E) above + β x recognition accuracy of (A) above + θ x recognition accuracy of (B) above + δ・recognition accuracy of (C) above) / (α + β + θ + δ)

"α" is the weighting factor for (E) above, "β" is the weighting factor for (A) above, that is, the volume, and "θ" is the weighting factor for (B) above, that is, the distance, “δ” is the above (c), that is, the frequency weighting factor. β and θ may be larger than α and δ depending on the set importance.

The calculation unit 113 may calculate, for example, a degree of correlation (correlation coefficient) indicating the degree of correlation between at least two or more parameters among a plurality of parameters. For example, the calculation unit 113 divides the covariance of the volume and the recognition accuracy by the product of the standard deviation of the volume and the standard deviation of the recognition accuracy as the degree of correlation between the volume and the recognition accuracy, thereby obtaining the correlation between the volume and the recognition accuracy. You can calculate the number. As another example, the calculation unit 113 uses a multivariate analysis technique (multiple regression analysis, logistic regression, etc.) to calculate the correlation coefficient between the two or more parameters as the degree of correlation between the two or more parameters. can be calculated.

The calculation unit 113 may calculate, for example, the reliability of the recognition result. For example, the calculation unit 113 may calculate the reliability of each word included in the recognition result, and aggregate the calculated reliability of each word to calculate the recognition accuracy of the recognition result.

The word-by-word confidence may, for example, have a predetermined range of values (eg, a range of 0.0 to 1.0). Within this predetermined range of values, the closer the number is to 1.0, ie, the upper limit, the more confidence per word indicates that there are relatively few other competing candidates with scores similar to that word. On the other hand, within this predetermined range of values, the closer the numerical value is to 0.0, i.e. the lower bound, the more confidence per word indicates that there are relatively more other competing candidates with scores similar to that word. . That is, the closer the numerical value is to the upper limit within a predetermined range, the more reliable the recognition result is output because there are no other candidates close to the top candidate in the recognition result. It can be said that

There are several methods for calculating the degree of reliability of a word. Journal, Vol.43, No.10, pp3078-3086).

<Output part>
The output unit 114 refers to the storage unit 130 that stores accuracy information, and outputs the recognition accuracy calculated by the calculation unit 113 and the output condition in association with each other. The output unit 114 may output the correspondence in any manner. For example, the correspondence may be output to a screen, a csv file, or a tabular file.

Here, an example of the output of the output unit 114 will be described with reference to FIGS. 5 and 6. FIG.

As shown in FIG. 5, the output unit 114 may output (display) a recognition accuracy evaluation screen A1 to a user terminal (not shown) for each speech recognition device 200 to be evaluated. The recognition accuracy evaluation screen A1 includes, for each output condition applied to the speech recognition apparatus 200 to be evaluated, information for identifying the output condition (No in this example), details of the output condition, and recognition corresponding to the output condition. Accuracy and an execution button for outputting audio according to output conditions are included. When the execution button is pressed by the user, the control device 100 instructs the audio output device 300 to output audio.

As shown in FIG. 6, the output unit 114 sets the recognition accuracy evaluation screen A2 for each speech recognition device to be evaluated and for the usage scene (in this example, “creation of minutes in a conference room (small scale)”). ) may be output to the user terminal. On the recognition accuracy evaluation screen A2, as the output conditions applied to the speech recognition apparatus 200 to be evaluated and the usage scene, those that do not exceed the threshold of the parameters of the usage scene are extracted and output. The recognition accuracy evaluation screen A2 includes, for each extracted output condition, information for identifying the output condition (No in this example), details of the output condition, recognition accuracy corresponding to the output condition, and speech based on the output condition. and an execution button for outputting. When the execution button is pressed by the user, the control device 100 instructs the audio output device 300 to output audio.

For example, the output unit 114 refers to the threshold storage unit 132 that stores the threshold information, and based on the threshold information, for each usage screen, the recognition accuracy is equal to or higher than a predetermined accuracy, and the associated parameter is Those that do not exceed the threshold may be extracted and output. In addition, the output unit 114 may extract and output, for example, the parameters associated with the recognition accuracy that do not exceed the threshold for each usage screen based on the threshold information. A “use scene” is a scene in which the speech recognition device 200 is used.

The “threshold information” is information indicating thresholds for each of a plurality of parameters for each usage scene. The threshold indicated by the threshold information may be, for example, a preset value (fixed value) or a value specified by the calculator 113 .

In general, increasing the volume tends to improve the recognition accuracy, but depending on the usage scene, there are situations where a high volume is not appropriate or acceptable, such as in a meeting. According to the above configuration, the output unit 114 can extract only parameters within an appropriate or permissible range and output them to the user according to the situation of use. Therefore, according to the above configuration, the user can efficiently evaluate each use scene that the user wants to evaluate.

The output unit 114 refers to, for example, a reference value storage unit (not shown) that stores reference value information, and based on the reference value information, the recognition accuracy is equal to or higher than a predetermined accuracy, and Those parameters whose associated parameters are within a predetermined range from the reference value may be extracted and output. Further, the output unit 114 may extract and output, for example, the parameters associated with the recognition accuracy within a predetermined range from the reference value for each usage screen based on the reference value information.

“Reference value information” is information indicating reference values for each of a plurality of parameters for each usage scene. The threshold indicated in the reference value information may be, for example, a preset value (fixed value) or a value specified by the calculator 113 .

In general, increasing the volume tends to improve the recognition accuracy, but depending on the usage scene, there are situations where a high volume is not appropriate or acceptable, such as in a meeting. According to the above configuration, the output unit 114 can thus extract each parameter based on the reference value and output it to the user according to the situation of use. Therefore, according to the above configuration, the user can efficiently evaluate each use scene that the user wants to evaluate.

For example, when outputting the output condition, the output unit 114 may rearrange (sort) the plurality of parameters included in the output condition in descending order of correlation with the recognition accuracy and output them. In addition, for example, when outputting the output condition, the output unit 114 outputs the parameter with the highest recognition accuracy and correlation among the plurality of parameters included in the output condition in a identifiable manner (for example, coordinated display). good too. According to such a configuration, the output unit 114 can allow the user to grasp and evaluate which parameter affected the recognition accuracy.

<Generation unit>
The generation unit 115 may generate speech data including the test speech based on the test text information using speech synthesis processing.

The generator 115 may, for example, process the first test audio data and the second test audio data into stereo audio data. Here, "stereo audio data" is audio data for stereophonically reproducing the first test audio and the second test audio. For example, the generation unit 115 may localize the sound image of the first test sound data or the second test sound data as preprocessing for stereo sound data processing. For example, the processing unit 122 may localize the sound image of the first test sound at the position of the virtual sound source for each speaker (channel) with respect to the first test sound data. The position of the virtual sound source may be set according to the sound source localization position of the output condition, for example.

<Memory section>
The storage unit 130 stores audio data, reference value information, and accuracy information. The storage unit 130 may store the above various information and data using a database management system (DBMS), or may store the above information using a file system. When using a DBMS, a table may be provided for each of the above information, and the information may be managed by associating the tables. The storage unit 130 also includes a text storage unit 131 and a threshold storage unit 132 .

The text storage unit 131 stores test text information and recognition result text information. The threshold storage unit 132 stores threshold information.

<Communication part>
The communication unit 140 transmits and receives various information and data such as voice data and text information to and from the voice recognition device 200 and the voice output device 300 via a wired network or a wireless network.

<4. Operation example>
An operation example of the control device 100 will be described with reference to FIG. Note that the order of processing in the operation example of FIG. 8 shown below is an example, and may be changed as appropriate. In this example, as a test scenario, speech output and speech recognition are executed a plurality of times based on the 1st to Nth (N is a natural number) output conditions, and the recognition accuracy of the speech recognition device 200 is calculated each time. An example will be explained.

As shown in FIG. 8, the audio output control unit 111 of the control device 100 controls audio output from the audio output device 300 based on the Nth output condition (S10). Next, the acquisition unit 112 acquires recognition result text information indicating the recognition result of the speech from the speech recognition device 200 that has recognized the speech output from the speech output device 300 (S11).

Next, the calculation unit 113 refers to the text storage unit 131 that stores the test text information, and calculates the recognition accuracy of the recognition result based on the test text information and the recognition result text information (S12). Next, the output unit 114 outputs the calculated recognition accuracy and the N-th output condition in association with each other (S13).

Next, if the test scenario has the (N+1)th output condition (Yes in S14), the audio output control unit 111 increments N (S15). The flow returns to before step S10.

Next, if there is no N+1th output condition in the test scenario (No in S14), the flow ends.

<5. Hardware configuration>
An example of a hardware configuration when the control device 100 described above is implemented by a computer 800 will be described with reference to FIG. The function of each device can also be implemented by being divided into a plurality of devices.

As shown in FIG. 9, a computer 800 includes a processor 801, a memory 803, a storage device 805, an input I/F section 807, a data I/F section 809, a communication I/F section 811, and a display device. 813 included.

Processor 801 controls various processes in computer 800 by executing programs stored in memory 803 . For example, each functional unit included in the control unit 110 of the control device 100 can be realized by the processor 801 executing a program temporarily stored in the memory 803 .

The memory 803 is a storage medium such as a RAM (Random Access Memory). The memory 803 temporarily stores program codes of programs executed by the processor 801 and data necessary for executing the programs.

The storage device 805 is a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. A storage device 805 stores an operating system and various programs for realizing the above configurations. In addition, the storage device 805 can also store a table for registering voice data, text information (test text information and recognition result text information), threshold information, etc., and a DB for managing this table. Such programs and data are referred to by the processor 801 by being loaded into the memory 803 as necessary.

An input I/F unit 807 is a device for receiving input from the user. Specific examples of the input I/F unit 807 include a keyboard, mouse, touch panel, various sensors, wearable devices, and the like. The input I/F unit 807 may be connected to the computer 800 via an interface such as USB (Universal Serial Bus).

A data I/F unit 809 is a device for inputting data from outside the computer 800 . A specific example of the data I/F unit 809 is a drive device for reading data stored in various storage media. Data I/F unit 809 may be provided outside computer 800 . In that case, the data I/F unit 809 is connected to the computer 800 via an interface such as USB.

The communication I/F unit 811 is a device for performing data communication with a device external to the computer 800 via the Internet N by wire or wirelessly. Communication I/F unit 811 may be provided outside computer 800 . In that case, the communication I/F unit 811 is connected to the computer 800 via an interface such as USB.

The display device 813 is a device for displaying various information. Specific examples of the display device 813 include a liquid crystal display, an organic EL (Electro-Luminescence) display, and a wearable device display. The display device 813 may be provided outside the computer 800 . In that case, the display device 813 is connected to the computer 800 via, for example, a display cable. Further, when a touch panel is adopted as the input I/F section 807 , the display device 813 can be integrated with the input I/F section 807 .

In addition, this embodiment is an illustration for demonstrating this invention, and is not the meaning which limits this invention only to the embodiment. Also, the present invention can be modified in various ways without departing from the gist thereof. Furthermore, those skilled in the art can adopt embodiments in which each element described above is replaced with equivalents, and such embodiments are also included in the scope of the present invention.

[Modification]
Although the present invention has been described based on the above embodiments, the following cases are also included in the present invention.

[Modification 1]
At least a part of each configuration in the control device 100 according to the above-described embodiment may be provided in a program dedicated to the evaluation system 1 installed in the speech recognition device 200 . For example, this program may be provided with each functional unit of the control unit 110 of the control device 100, and the evaluation system 1 may be configured with the speech recognition device 200 and the speech output device 300 without separately providing the control device 100. .

[Modification 2]
Regarding the text storage unit and the threshold storage unit according to the above-described embodiment, the example provided in the storage unit 130 of the control device 100 has been described, but the text storage unit and the threshold storage unit are not limited to this. For example, the text storage unit and the threshold storage unit according to the above embodiments may be provided in the speech recognition device 200 or may be provided in an external system device.

[Modification 3]
Although not shown in the above embodiment, the audio output device 300 may be arranged manually, or the control device 100 and the audio output device 300 may be arranged by linking with an arm or the like. In the evaluation system 1, the audio output device 300 may be arranged by the audio output control unit 111 controlling (for example, rotating or extending/contracting) this arm based on the applied output conditions.

DESCRIPTION OF SYMBOLS 1... Evaluation system 100... Control apparatus 110... Control part 111... Voice output control part 112... Acquisition part 113... Calculation part 114... Output part 115... Generation part 130... Storage part 140... Communication Unit 200 Speech recognition device 300 Speech output device 500 Speech recognition system 800 Computer 801 Processor 803 Memory 805 Storage device 807 Input I/F unit 809 Data I/ F unit, 811... Communication I/F unit, 813... Display device, 817... Voice input device, 819... Voice output device.

Claims (10)

an audio output control unit that controls audio output from an audio output device based on audio output conditions;
an acquisition unit that acquires recognition result text information indicating a recognition result of the speech from the speech recognition device that recognized the speech output from the speech output device;
By referring to a text storage unit storing test text information indicating the content of the test speech included in the speech, the recognition accuracy of the recognition result is calculated based on the test text information and the recognition result text information. a calculation unit for
an output unit that associates and outputs the calculated recognition accuracy and the condition,
The voice output control unit changes the condition based on the recognition accuracy calculated by the calculation unit, and controls the output of voice from the voice output device based on the changed condition.
Information processing equipment. The speech output by the speech output device and the speech recognition by the speech recognition device for the speech output are performed multiple times,
the condition includes a plurality of parameters related to the audio output;
The audio output control unit controls the audio output by changing one or more parameters used among the plurality of parameters at each of the plurality of times.
The information processing device according to claim 1 . The output unit outputs the recognition accuracy of each of the speech recognitions performed a plurality of times in association with one or more parameters used to control the speech output in each of the times.
The information processing apparatus according to claim 2 . The output unit refers to a threshold storage unit that stores threshold information indicating thresholds for each of the plurality of parameters for each usage scene in which the speech recognition device is used, and further based on the threshold information, for each usage screen. and extracting and outputting those whose recognition accuracy is equal to or higher than a predetermined accuracy and whose associated parameters do not exceed the threshold;
The information processing apparatus according to claim 3. The plurality of parameters include at least any two of information related to the sound source localization position of the sound to be output from the sound output device, the speech rate of the sound, the volume of the sound, or the speaker of the sound,
The audio output control unit controls the audio output by changing the value of the parameter.
The information processing apparatus according to any one of claims 2 to 4. A plurality of said audio output devices exist,
The parameter includes device selection information for selecting a device whose output is to be turned on among the plurality of audio output devices at each of the times, and the audio output control unit further controls the plurality of audio output devices based on the device selection information. switching on/off the audio output of each audio output device at each time to control the audio output;
The information processing apparatus according to any one of claims 2 to 5. The audio data of the audio to be output by the audio output device includes test audio data of the test audio and noise data that emits noise,
The parameter includes device designation information that designates which of the test sound and the noise is to be output from each of the sound output devices at each of the times. and controlling the sound output by switching the sound to be output to each of the plurality of sound output devices between the test sound and the noise at each time,
The information processing device according to claim 6 . The parameter includes the length of the audio to be output to the audio output device,
The test audio data includes first test audio data of a first test audio and second test audio data of a second test audio different in length from the first test audio,
The audio output control unit controls the audio output by switching the audio to be output to each of the audio output devices between the first test audio and the second test audio each time.
The information processing apparatus according to claim 7. to the computer,
an audio output control function for controlling audio output from an audio output device based on audio output conditions;
an acquisition function for acquiring recognition result text information indicating a recognition result of the speech from the speech recognition device that recognized the speech output from the speech output device;
By referring to a text storage unit storing test text information indicating the content of the test speech included in the speech, the recognition accuracy of the recognition result is calculated based on the test text information and the recognition result text information. a calculation function to
realizing an output function that associates and outputs the calculated recognition accuracy and the condition,
The voice output control function changes the condition based on the recognition accuracy calculated by the calculation function, and controls the output of voice from the voice output device based on the changed condition.
program. the computer
controlling audio output from an audio output device based on audio output conditions,
Acquiring recognition result text information indicating a recognition result of the speech from the speech recognition device that recognized the speech output from the speech output device;
By referring to a text storage unit storing test text information indicating the content of the test speech included in the speech, the recognition accuracy of the recognition result is calculated based on the test text information and the recognition result text information. death,
outputting the calculated recognition accuracy and the condition in association with each other;
changing the condition based on the calculated recognition accuracy;
controlling audio output from the audio output device based on the changed condition;
Information processing methods.

Images