JP7341260B2 - Semi-automatic refinement for speech recognition - speech data extraction and transcription data generation method - Google Patents

Description

The present invention relates to a semi-automatic purification-speech data extraction and transcription data generation method for speech recognition.

A large amount of audio data files and corresponding transcribed text are required for model training of a speech recognizer, but manual transcription requires a lot of time and money.

According to the prior art, there has been an attempt to introduce an automatable part in the transcription process of audio data, as in the Republic of Korea Patent Publication No. 10-2083938, but this has been done in order to generate a refined corpus. The problem is that it takes a lot of effort and time, and the model is retrained with errors in the transcription process.

The present invention was proposed in order to solve the above-mentioned problems. In order to improve the recognition rate of a speech recognizer, the present invention can purify speech data and its transcription data necessary for model learning at low cost and in a short time. The purpose of the present invention is to provide a method that allows semi-automatic methods to be used within the system.

A semi-automatic refining-speech data extraction and transcription data generation method for speech recognition according to an embodiment of the present invention includes the steps of slicing speech data and constructing a refined corpus, and utilizing the refined corpus for speech recognition. The method includes a step of performing machine model learning, and a step of semi-automatically extracting purified audio data and generating transcription data.

According to the present invention, it is possible to reduce the working time and cost for generating a refined corpus using a semi-automatic refined speech data extraction algorithm.

In addition, the voice recognition rate has been improved and subtitles can be generated almost automatically, which has the effect of reducing the cost of subtitle generation for videos.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

The model learning process of a speech recognizer using speech data and transcribed text according to the prior art is shown. 4 illustrates a process of audio data slicing and construction of a refined corpus according to an embodiment of the present invention. 1 illustrates a semi-automatic refinement-audio data/transcribed text extraction machine according to an embodiment of the present invention; 1 shows a structure for training an STT model of a speech recognizer with data obtained by a semi-automatic refining-speech data/transcription text extractor according to an embodiment of the present invention; This is a summary of the recognition rate and the amount of audio data that can be extracted by automation when the reference values are set to 50%, 60%, and 70% for 2200 hours of original raw audio data. An example of extracting and utilizing audio data from a video with subtitles according to an embodiment of the present invention will be described. 3 shows an audio slicing process using subtitles, a seed model generation process, and a semi-automatic refined audio data extraction algorithm development process using the seed model according to an embodiment of the present invention.

The above-mentioned and other objects, advantages and features of the present invention, as well as methods of achieving them, will become clearer with reference to the embodiments described below in detail in conjunction with the accompanying drawings.

However, the present invention is not limited to the embodiments disclosed below, and can be realized in various forms different from each other, and the following embodiments are merely described by those of ordinary skill in the technical field to which the present invention pertains. The scope of the present invention is defined by the claims.

Meanwhile, the terms used in this specification are for describing embodiments and are not intended to limit the present invention. As used herein, the singular term also includes the plural term unless the context specifically indicates otherwise. As used in the specification, "comprises" and/or "comprising" refer to the terms "comprises" and/or "comprising" in which the referenced component, step, act, and/or element is present in one or more other components, steps, acts, and/or elements. /or does not exclude the presence or addition of elements.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following, for the understanding of those skilled in the art, the background in which the present invention was proposed will first be described, and embodiments of the present invention will be described.

In order to develop a speech-to-text (STT) device, a large amount of speech data and its transcribed text are required.

Since the speech data and transcribed text are used to train and build a speech recognizer (STT) model, the performance of the speech recognizer is greatly influenced by the accuracy of the transcribed text.

FIG. 1 shows a model learning process of a speech recognizer using speech data and transcribed text according to the prior art.

The recognition rate of real-time speech recognizers is around 85% (the recognition rate decreases when the terms used in the service domain are not common), but recognition performance of 95% or higher is required for stable service. required.

A large amount of audio data files and corresponding transcriptions are required for model learning of machine learning or deep learning-based speech recognition machines, but transcription text generation work by a stenographer, that is, manual transcription work is required. There are problems that require a lot of time and cost.

In order to improve speech recognition performance, refined answer sheets (speech data, transcribed text) are required.

The process of transcribing audio data is performed by humans (stenographers), but there are factors that reduce the performance of model learning (particularly deep learning) in this process.

This is because the transcribed text generated by the stenographer contains information that is not suitable for model learning.

There are parts that are not in the original audio file but were added by the stenographer (insert), parts that were in the original audio file but were removed by the stenographer (delete), and parts that were changed (update), so learn these. When used as data, answer sheets (voice data, transcribed text) with errors must be learned, resulting in a problem that the speech recognition rate decreases.

In order to create refined answer sheets, when examining shorthand, we listen to the original file of audio data while searching for and correcting incorrect parts of the text created by the stenographer. There are problems that require time and money.

In addition, the stenographer's transcription method (input pattern, e.g., numerical notation, nonverbal expression, speaker voice overlap, etc.) is different, and the text transcribed by the stenographer is in a form suitable for model learning of the speech recognizer. However, it is difficult to check all of them in practice.

According to research, to obtain 90% speech recognition performance, a minimum of 2000 hours of refined corpus must be trained.

In other words, at least 2,000 hours of answer sheets (audio data, transcription data) are required, but in order to create a refined corpus of 1 hour, it takes an average of 3 hours of time investment and 100,000 won or more of work. There is a cost involved, and the time and cost involved in inspecting the transcription text is also incurred.

According to the prior art, Korean Patent Publication No. 10-2083938 utilizes an approach to automatically transcribe voice data based on a base voice recognition model stored in a memory.

That is, we propose a method in which a secondary speech recognition model is generated by performing primary training on a speech recognition model based on the results of comparison between primary automatic transcription data and audio data.

However, this also has the problem that it requires a lot of effort and time to create a refined corpus by listening directly to the original audio file, searching for and correcting the mistakes in the text created by the stenographer. , there is a problem that the model may be retrained with errors in the transcription process.

The present invention was proposed in order to solve the above-mentioned problems, and by introducing an automatable part in the process of transcribing audio data, refined audio data and its transcribed text can be produced at low cost. We propose a method to extract a refined corpus within a fast period.

According to an embodiment of the present invention, purification-speech data is selected and extracted in a semi-automatic manner (limited to parts that can be automated) from the speech data necessary for the development of a speech recognition system, and transcription data thereof is generated. .

Assuming you have audio data and text data that has been previously transcribed, or audio data and text data that has been transcribed by a stenographer. Compare and refine the transcribed text using a character string matching algorithm - classify the voice data, determine the corresponding transcribed text data, and use this to train the model of the speech recognizer (STT) conduct.

According to an embodiment of the present invention, it is possible to improve the recognition rate of a speech recognizer (STT) by utilizing a refined corpus, and for this purpose, the refined speech data and its transcription necessary for model learning are Securing data in a semi-automatic manner at low cost and within a short time.

A semi-automatic refining-speech data extraction and transcription data generation method for speech recognition according to an embodiment of the present invention includes the steps of slicing speech data and constructing a refined corpus, and utilizing the refined corpus for speech recognition. The method includes a step of performing machine model learning, and a step of semi-automatically extracting purified audio data and generating transcription data.

FIG. 2 illustrates the process of audio data slicing and refined corpus construction according to an embodiment of the present invention.

If the running time of the voice data file becomes long, the recognition rate of the voice recognizer (STT) may decrease, so a voice data preprocessing operation is performed to divide the original voice data file into t seconds or less.

At this time, it is preferable that the time t is set to around 30 seconds.

The step of performing model learning of a speech recognizer using a manually refined corpus according to an embodiment of the present invention involves creating manually refined corpus data using speech data and transcribed text.

In order to obtain 90% speech recognition performance, a refined corpus of at least 2,000 hours must be trained, and a speech recognizer (STT) model is trained using about 250 hours of corpus data.

The steps of semi-automatically extracting refined audio data and generating transcribed data according to embodiments of the present invention are performed by a semi-automatic refined audio data/transcribed text extractor 310 shown in FIG.

In a situation where the STT model of the speech recognizer 311 has been trained through the process described above, the speech data (speech file U _i ) that requires refinement and extraction of speech data is input to the speech recognizer 311 and decoded. The transcribed text (decoded character string W _i ) is obtained.

The audio data (audio file U _i ) is refined by the stenographer into the transcribed text (transcribed character string V _i ) and the decoded transcribed text (decoded character string W _i ) into the audio/character string extractor 312 input, and extracts purified voice data and character strings (transcribed text) using a character string matching algorithm.

A character string matching algorithm according to an embodiment of the present invention will be described below.

In the character string matching algorithm according to the embodiment of the present invention, when the transcribed character string V _i is A and the decoded character string W _i is B, the similarity S (A, B) of the two character strings is set to a reference value (Threshold). )E or higher, an identifier indicating that the data is refined data is transferred to the voice data classifier 314 and the text data classifier 315, and the text data classifier 315 selects the appropriate character string A or B. Alternatively, the modified character string R is also transferred.

Step 1
When the transcription character string A and the decoding character string B are input to the character string matching discriminator, it is checked whether the similarity value of the two character strings is greater than or equal to the reference value, and the similarity value is determined to be a character string that is greater than or equal to the reference value. Proceed with string matching only for those.

Step 2-1
The transcription character string A and the decoding character string B are separated into word (/phrase) units, and a character string B ^* is generated by removing blank spaces from B.

文字列Ｂ^＊を生成する理由は、Ａにある単語により、Ｂでの検索を有利にするためである。
文字列Ａに対して空欄を除去した文字列Ａ^＊を生成し、同様の方法を行うことが可能である。 For example, a character string is generated as shown in Table 1 below.

The reason for generating the character string B ^* is to make the search in B more advantageous due to the word in A.
It is possible to generate a character string A ^* by removing blank spaces from the character string A and perform a similar method.

According to an embodiment of the invention, the transformation is performed in a way such as from words to syllables.

Step 2-2
A search is made to see if the first word of the transcribed character string A exists in the character string B ^* .

Step 2-2-1
At this time, if there is a matching word, the common word part of A and B ^* is replaced with a specific character (eg, tilde (~)).

Other characters besides the tilde (~) may be used, but characters that can appear within a string are not used as specific characters.

If there is no matching word, in the case of Korean, since Korean is composed of a stem and a particle, start with the first word of A and reduce it one character at a time from the end of the string. Search to see if there is a word that matches (same) ^* .

If the word that matches in the reduction process is only one character, it is meaningless, so this process is continued only up to two characters.

According to the embodiment of the present invention, languages that have the same linguistic characteristics as Korean (same word order, etc.) can be extended in a similar manner.

The above embodiment will be explained using Tables 2 and 3 as examples.

In the above example, the word "shijaku" matches in character strings A and B, so it is replaced with "~~".

Step 2-2-2
If the matching word in the above step is one character or less, calculate the similarity value (word similarity value) between the first word of A and the phrase of B ^* , and if the similarity value is greater than the reference value, step 2-3 If the similarity value is less than the reference value, step 2-4 is performed.

The "shi" portions of character strings A and B match, but this is a case of a one-character match.

If the reference value is 60 or more, the similarity value for portions A and B is 6/7 (85% or more), so proceed to step 2-3.

Step 2-3
While increasing the number of characters one by one from the front of the first word of A, match to see if the same character exists in the word (referred to as b) whose similarity value calculated in step 2-2-1 is higher than the reference value. conduct.

At this time, in order to find a group having the maximum number of the same characters, matching of the same characters as b is performed continuously.

Since a single character has no meaning, matching is performed only when two or more characters match consecutively.

The above embodiment will be explained using Tables 6 and 7 as examples.

If matching is successful, the common word part of A and B is replaced with a tilde (~).

Step 2-4
Steps 2-1 to 2-3 described above are repeated for words appearing after the first word of A.

Step 3
Using the number of remaining word groups of the transcribed character string A created by the stenographer and the decoded character string B generated by the speech recognizer, a final result is created by the process described later.

Step 3-1
If the number of word groups remaining in the transcribed character string A and the decoded character string B is the same, the transcribed character string is finally selected.

Since the strings are created manually by the stenographer, the transcription accuracy for the audio data file is generally 95% or higher, with the remaining 5% being arbitrary insertions, deletions, or changes made by the stenographer. This includes typos and typos.

On the other hand, in the case of decoded character strings, the performance is affected by the performance of the speech recognizer, but it is about 90% in a quiet environment.

Depending on the purpose, you can select the opponent's text or a slightly modified text.

The above-mentioned embodiments will be explained using Tables 8 and 9 as examples.

Since the number of remaining word groups is the same, the text of the transcription string (``Chigumbut che 356 fe kyongchalchonmunfahensa kefesigur kohenhagessumnida'') is selected.

Since the number of remaining word groups is the same, the text of the transcription string (``Cheonjue Tara 1 Chorman Choechanghayo Chushigi Paramnida'') is selected.

Step 3-2
If a word group remains in only one of the transcribed character string A and decoded character string B, the sentence in the remaining group is selected.

The reason for this is that in the transcription string, the stenographer transcribes only once portions of what the speaker has said that are repeated or ambiguous.

Based on the decoding results, it may be possible to partially decode an audio file.

The above embodiment will be explained using Tables 10 and 11 as examples.

Since the text remains only in the decoding result, select the text in the decoding result ("Tayanhan kiob mi kinyomsaobul jumbi hago issumnida").

Since only the text remains in the transcription string, select the text resulting from the transcription ("Amchorok y andero uigyolhe chushigil paramyo").

Step 3-3
When a word group remains at the end of a sentence only for decoding character string B, transcription character string A is unconditionally selected.

This is because if there is some noise at the end of the original audio file, the noise will be reflected as text in the decoding result.

The above embodiment will be explained using Table 12 as an example.

Since the text remains only in the last part of the decoding result, select the text of the transcription string (``Uri 23 Dae Kyongcharwiwonfenun Newe Hageku Yolobunwi'').

Step 3-4
If the number of remaining word groups differs between the transcribed character string A and the decoded character string B, the one with the larger number of groups is selected.

The reason is that the group with a larger number is created by reflecting the information of the audio data to the maximum extent.

The above embodiment will be explained using Table 13 as an example.

After the transcription is matched, the number of remaining groups is greater than the decoding result, so the text of the transcription (``Jeonmwiwonhoewi Kimhanpyo Wiwon Naoshoso 7 Gongae Taehayo Simsabogo Mi Chaeansolmyeon Hae Chushigi Paramnida'') is selected.

The above embodiment will be explained using Table 14 as an example.

After the decoding results are matched, the number of remaining groups is greater than the transcription result, so the transcription text (``Creso Ohiryo Chayuhwa Jeongdol 89 % Iha Jeongdoro Nachu Myeongseo Lado Chogi Tagyeol Hessmyeon Wanghwa Haneun Gosi Yogi Koitsun Dol Ipchan Imnida'' ).

Step 3-5
If a word group remains only at the end of the transcription character string A and the decoding character string B, the one with the longer number of characters is selected.

The reason is that if word groups remain in both transcription string A and decoding string B, they are not text against noise, so the one that reflects the information of the audio data to the maximum has a text with a longer number of characters. That's the way to go.

However, if the number of characters is the same, the transcription created by the stenographer is selected.

The above embodiment will be explained using Table 15 as an example.

After the transcription is matched, the number of remaining characters is greater than the decoding result, so the text of the transcription (``Jugahago songhaesajeongsawi wim chohaneul ganghwahayossumnida'') is selected.

The above embodiment will be explained using Table 16 as an example.

After the transcription is matched, the number of remaining characters is the same as the decoding result, so the text of the transcription (``Chinanghae Uri Geunminun Taehamminguk Yeoksawi'') is selected.

Hereinafter, with reference to FIG. 4, a structure for training the STT model of the speech recognizer using data obtained by the semi-automatic refinement-speech data/transcription text extraction machine will be described.

As a result, the "refined speech data and its transcribed text" obtained as a result of character string matching are utilized for STT model learning to improve the performance of the speech recognizer.

Using a character string matching algorithm, if the similarity value between two sentences is set to a standard value of 60%, approximately 70% of the entire original audio file can be automatically extracted (the speech recognition rate in this case is 90% or more).

FIG. 5 summarizes the recognition rate and the amount of audio data that can be extracted by automation when the reference values are set to 50%, 60%, and 70% for 2200 hours of original raw audio data.

Through experiments, it was found that the recognition rate for the automated extraction size was the best when the reference value was set to 60%.

It should be possible to carry out separate experiments in other service domains and estimate and use general reference values for those domains.

According to the embodiments of the present invention, it is possible to extract and utilize audio data from videos with subtitles, and it is also possible to apply and utilize video data with subtitles.

Referring to Figure 6, in the case of a video with subtitles, it is possible to extract some video sections and refine and utilize them to improve the performance of the speech recognizer, without separately performing audio transcription again. It is possible.

According to embodiments of the present invention, it is possible to utilize it as a video (including audio data) subtitle generator.

Even if a video and text are prepared, it is necessary to divide the text into sections of 10 seconds or less for learning purposes and synchronize this with the video, which is usually difficult from the beginning. Perform transcription work assuming that there is no text.

However, according to the embodiment of the present invention described above, it is possible to perform transcription work relatively easily.

Texts prepared separately by stenographers contain information that is not suitable for learning.

There are added parts that are not in the original audio file (insert), parts that are also in the audio file but have been removed by the stenographer (delete), and parts that have been changed (update), so when using it as learning data. , poor quality training data will be used, resulting in a serious decrease in recognition rate.

It takes a lot of effort and time to create a refined corpus by listening directly to the original audio files, searching for and correcting the text created by the stenographer and making mistakes.

Referring to FIG. 7, it is possible to quickly construct a large amount of purified corpus data through semi-automatic purification-extraction of audio data/transcribed text according to an embodiment of the present invention.

In the first step, a subtitle file is generated by text extraction and segmentation operations that match the video.

After extracting time data using a video and slicing the video using the sentence time based on the utterance time of each sentence, a wav audio file is extracted.

Thereafter, subtitles are automatically extracted and generated for portions that match the shortened WAV audio files.

In the second step, the wav audio file and text created in the first step are used to manually create refined corpus data, and the refined corpus is used to generate a seed model.

In the third step, the original raw audio file is decoded from the seed data model.

Only original raw audio files with a matching rate (similarity rate) of 60% or more are extracted using a semi-automatic refined audio data extraction algorithm based on the transcribed text and the decoded text results.

According to the conventional technology, the transcription process of audio data is performed by a person (stenographer), but in this process, the transcription text generated by the stenographer contains information that is not suitable for model learning, and the transcription method of the stenographer is The text transcribed by the stenographer may not be in a suitable form for the speech recognizer's model training due to differences in input patterns (e.g., numerical notation, non-verbal expressions, speaker voice overlap, etc.).

Since a speech recognizer (STT) model is trained and constructed using speech data and transcribed text, refined answer sheets (speech data and transcribed text) are required to improve speech recognition performance.

According to research, achieving 90% speech recognition performance is possible only by training a refined corpus for at least 2000 hours, but currently, when manually proceeding to create a refined corpus, However, it costs more than 100,000 won per hour.

Therefore, it would cost more than 100 million won to collect a 1,000-hour refined corpus.

Using the semi-automatic refined speech data extraction algorithm according to embodiments of the present invention, a 1000-hour refined corpus with the same recognition rate can be created in a faster time frame at only about one-fourth the cost.

Therefore, the working time and cost for generating a refined corpus using the semi-automatic refined speech data extraction algorithm according to the embodiment of the present invention can be greatly reduced.

When video (including audio data) is provided, that is, in the case of lectures, conferences, etc., it is rare that even subtitles are provided.

The reason is that the recognition rate of real-time speech recognizers remains at about 85%.

Providing subtitles in real time requires a recognition rate of 96% or higher and requires the help of a stenographer, which poses the problem of high costs.

In this situation, in order to record the minutes of some important meetings, a separate text is created later (non-real time).

When supporting ``watch again'' services at local councils, etc., they often support videos with subtitles, but this utilizes technology that mixes the video with the text that has already been created.

At this time, voice recognition technology is important as well as technology to synchronize text with the video scene, but it takes a lot of time and high cost to manually generate subtitles for a single video.

When a speech recognizer (STT) trained using refined speech data and text according to an embodiment of the present invention is used, the speech recognition rate is improved, most subtitles can be automatically generated, and development costs can be reduced. It has a great effect.

Meanwhile, the semi-automatic purification-audio data extraction and transcription data generation method for speech recognition according to an embodiment of the present invention may be implemented in a computer system or recorded on a recording medium. A computer system can include at least one processor, memory, user input devices, a data communication bus, user output devices, and storage. Each of the aforementioned components communicates data via a data communication bus.

The computer system can further include a network interface coupled to the network. The processor may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in memory and/or storage.

Memory and storage may include various forms of volatile or non-volatile storage media. For example, memory can include ROM and RAM.

Therefore, the semi-automatic purification-speech data extraction and transcription data generation method for speech recognition according to embodiments of the present invention can be implemented in a computer-executable manner. Semi-automatic refining for speech recognition according to embodiments of the present invention - When the method for extracting speech data and generating transcription data is performed on a computer device, the computer-readable command word is converted into semi-automatic refining for speech recognition according to the present invention - speech. Data extraction and transcription data generation methods can be performed.

Meanwhile, the above-described semi-automatic refining, speech data extraction and transcription data generation method for speech recognition according to the present invention can be implemented as a computer-readable code on a computer-readable recording medium. Computer-readable storage media include all types of storage media that store data that can be read by a computer system. For example, there may be a ROM (Read Only Memory), a RAM (Random Access Memory), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, etc. The computer-readable recording medium may also be stored and executed as readable code in a distributed manner over computer systems linked by a computer communication network.

Claims (12)

A method performed by a computer, the method comprising:
(a) slicing the audio data and constructing a refined corpus;
(b) performing model learning for a speech recognizer using the refined corpus;
(c) semi-automatically refining--extracting audio data and generating transcription data ;
The step (c) is
Perform string matching on the transcription string and decoding string to find groups of words with the maximum number of identical characters,
Selecting the transcription character string or the decoding character string by considering the number of word groups remaining in the transcription character string and the decoding character string.
A semi-automatic purification-speech data extraction and transcription data generation method for speech recognition. Semi-automatic purification-audio data extraction and transcription data generation for speech recognition according to claim 1, wherein the step (a) performs pre-processing to divide the original audio data file into a preset time or less. Method. According to claim 1, in the step (c), a similarity value between the transcribed character string and the decoded character string is checked, and character string matching is performed only for character strings whose similarity value is equal to or higher than a reference value. Semi-automatic refinement for speech recognition as described - A method for speech data extraction and transcription data generation. The step (c) separates the transcribed character string and the decoded character string into units of words or phrases, and generates a character string from which blank spaces are removed from the decoded character string. Semi-automatic purification for speech recognition as described in 3.-Speech data extraction and transcription data generation method. The step (c) searches for the same word as the first word of the transcribed string in the decoded string with blank spaces removed, and replaces the common word portion with a specific character. The semi-automatic purification-speech data extraction and transcription data generation method for speech recognition according to claim 4. The voice according to claim 5, wherein the step (c) calculates a similarity value between the first word of the transcribed character string and the phrase of the character string with blanks removed from the decoded character string. Semi-automatic refinement for recognition - methods for speech data extraction and transcription data generation. In step (c), the first word of the transcription string is incremented one character at a time from the beginning, and matching is performed to see if the same character exists in a word with a similarity value higher than a reference value. , Semi-automatic purification-speech data extraction and transcription data generation method for speech recognition as claimed in claim 6. The speech recognition method according to claim 1 , wherein the step (c) is to finally select the transcribed string when the number of word groups remaining in the transcribed string and the decoded string are the same. Semi-automatic refining-audio data extraction and transcription data generation method for. According to claim 1 , in the step (c), when a word group remains in only one of the transcribed character string and the decoded character string, a sentence of the remaining group is selected. Semi-automatic refinement for speech recognition as described - A method for speech data extraction and transcription data generation. Semi-automatic refinement for speech recognition-speech according to claim 1 , wherein the step (c) selects the transcribed string when word groups remain only for the decoded string. Data extraction and transcription data generation methods. According to claim 1 , in the step (c), when the number of remaining word groups is different for the transcribed character string and the decoded character string, the one with a larger number of groups is selected. Semi-automatic refinement for speech recognition - speech data extraction and transcription data generation method. According to claim 1 , in step (c), when word groups remain only in the last parts of the transcribed character string and the decoded character string, the one with the longer number of characters is selected. Semi-automatic refinement for speech recognition - speech data extraction and transcription data generation method.

Images