JP2022042460A - Voice recognition model learning method and system utilizing enhanced consistency normalization - Google Patents

Abstract

To provide a voice recognition model learning method and a system that utilize enhanced consistency normalization.SOLUTION: A voice recognition model learning method includes the steps of: receiving multiple voice samples without assigned labels; extracting a first set of voice samples for human labeling from the multiple voice samples using a voice recognition model; receiving a first set of labels corresponding to the first set of voice samples; extracting a second set of voice samples for machine labeling from the multiple voice samples using the voice recognition model; determining a second set of labels corresponding to the second set of voice samples using the voice recognition model; enhancing the second set of voice samples; and performing semi-supervised learning to update the voice recognition model based on the first set of voice samples, the first set of labels; the enhanced second set of voice samples, and the second set of labels.SELECTED DRAWING: Figure 6

Description

Application for application of Article 30, Paragraph 2 of the Patent Act has been applied. On June 19, 2nd year of Reiwa, "Efficient Active Learning for Authentic Speech Recognition Via Augmented Consistency Registration" was released on the website.

The present disclosure relates to speech recognition model learning methods and systems, specifically to efficient gradual speech recognition model learning methods and systems utilizing enhanced consistency normalization.

With the rapid development of artificial intelligence technology and IoT (Internet of Things) technology, an artificial intelligence speaker equipped with an intelligent individual or virtual secretary (Intelligent Personal Assistant) that provides users with a specific service that responds to the user's voice request. , Terminals such as smartphones are widely used. Such an intelligent personal assistant recognizes a user's voice command by using artificial intelligence voice recognition technology, and provides a service corresponding to the voice command. For example, artificial intelligence speakers provide services such as running specific applications, providing weather information, and providing information through internet searches, not to mention the ability to make phone calls through user voice commands. can do.

In order to improve the quality of such speech recognition services, it is necessary to keep updating the speech recognition model using a large amount of learning data. The conventional technique has a problem that it is costly because the human annotator must directly determine the correct label for a large number of speech samples in order to learn the speech recognition model.

Korean Published Patent Gazette No. 10-2016-0032536

The present disclosure provides a speech recognition model learning method, a computer program and a device (system) for solving the above-mentioned problems.

The present disclosure may be embodied in a variety of ways including methods, devices (systems) or computer programs.

According to one embodiment of the present disclosure, the speech recognition model learning method performed by at least one processor is a step of receiving a plurality of unlabeled speech samples, and a plurality of speech samples utilizing the speech recognition model. The stage of extracting the first set of speech samples for human labeling from, the step of receiving the first set of speech samples and the corresponding first set of labels, and multiple speeches using a speech recognition model. The stage of extracting the second set of voice samples for machine labeling from the sample, the stage of determining the second set of voice samples and the corresponding second set of labels using the voice recognition model, the second. Semi-supervised learning based on the stage of augmenting a set of audio samples and the first set of audio samples, the first set of labels, the enhanced second set of audio samples and the second set of labels. Includes the steps of performing supervised learning) to update the speech recognition model.

A computer program for executing a speech recognition model learning method according to an embodiment of the present disclosure on a computer is provided.

The speech recognition model learning system according to an embodiment of the present disclosure is coupled to a communication module, a memory, and a memory, and at least one configured to execute at least one computer-readable program contained in the memory. Includes processor. At least one program receives multiple unlabeled speech samples, utilizes a speech recognition model to extract the first set of speech samples for human labeling from the multiple speech samples, and the first set of speech. Receive the first set of labels corresponding to the sample, use the speech recognition model to extract the second set of speech samples for machine labeling from multiple speech samples, and use the speech recognition model to extract the second set. The audio sample and the corresponding second set of labels are determined, the second set of audio samples is augmented, the first set of audio samples, the first set of labels, the enhanced second set of audio samples, and the first. Includes command words for performing semi-supervised learning and updating the speech recognition model based on two sets of labels.

In the various embodiments of the present disclosure, the number of speech samples that humans must directly transcrib into a text sequence to learn a speech recognition model is reduced, reducing costs and reducing the performance of the speech recognition model. Can be avoided. Specifically, the character level error rate (charcter-level error rate; CER) is increased by about 0.26% p (that is, the performance is deteriorated) while the labeling cost is reduced by about 2/3, and the labeling cost is reduced by about 6. CER can be increased by about 1.08% p while saving about / 7.

In various examples of the present disclosure, an uncertainty score can be calculated in consideration of the binding probability of a text sequence to a speech sample, and a sample useful for learning a speech recognition model based on the uncertainty score (information sample). Can be extracted.

In the various embodiments of the present disclosure, the speech sample can be augmented without damaging the linguistic information contained in the speech sample, and such enhancement of the speech sample improves the efficiency of speech recognition model learning. Can be made to. In addition, the enhanced speech sample can be used to improve the resilience of the speech recognition model.

The embodiments of the present disclosure are described with reference to the accompanying drawings described below, wherein similar reference numbers indicate similar elements, but are not limited thereto.
It is a drawing which shows the example which a user has a service provided from a user terminal through a voice command. FIG. 5 is a schematic diagram showing a configuration in which an information processing system is connected so as to be able to communicate with a plurality of user terminals in order to provide a voice recognition service according to an embodiment of the present disclosure and learn a voice recognition model. It is a block diagram which shows the internal structure of the user terminal and the information processing system which concerns on one Example of this disclosure. It is a figure which shows the example which constructs the HLS database (DB) and MLS DB through the labeling work for the audio sample which concerns on one Example of this disclosure. It is a flowchart which shows the initial speech recognition model generation method which concerns on one Example of this disclosure. It is a flowchart which shows the gradual speech recognition model learning method which concerns on one Example of this disclosure. It is a drawing which shows the example of the speech sample for generating, updating, and testing the speech recognition model which concerns on one Example of this disclosure. It is a graph which shows the performance difference of the speech recognition model by the method of extracting the speech sample for human labeling. It is a graph which shows the difference in the performance of the speech recognition model by the speech sample enhancement method of this disclosure. It is a graph which shows the relationship between the learning cycle and the performance of a speech recognition model when the speech recognition model is updated many times by one Example of this disclosure.

Hereinafter, the specific contents for the implementation of the present disclosure will be described in detail with reference to the attached drawings. However, if the following description may unnecessarily obscure the gist of the present disclosure, specific description of well-known functions and configurations will be omitted.

In the attached drawings, the same or corresponding components are given the same reference numerals. Further, in the following description of the embodiment, it may be omitted to describe the same or corresponding components in duplicate. However, even if the techniques relating to the components are omitted, it is not intended that such components are not included in the embodiment.

The advantages and features of the disclosed examples, and how to achieve them, will be clarified with reference to the examples described below with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be embodied in various forms different from each other, except that the present embodiment completes the present disclosure and the present disclosure is a normal engineer. It is provided only to fully inform the scope of the invention.

The terms used herein will be briefly described, and the disclosed examples will be specifically described. The terminology used herein has been selected from the most widely used terms available today, taking into account the functionality of this disclosure, but this is the intent or case of a technician engaged in the relevant field, a new technique. It may change due to the appearance of. In addition, there are some terms arbitrarily selected by the applicant in specific cases, and in this case, the meaning will be described in detail in the description of the applicable invention. Therefore, the terms used in this disclosure should be defined based on the meaning of the terms and the general content of the present disclosure, not just the names of the terms.

The singular representation herein includes multiple representations unless the context explicitly specifies that it is singular. Also, multiple expressions include a singular expression unless the context explicitly specifies that they are plural. If, in the entire specification, a part contains a component, this may include other components rather than excluding the other components, unless otherwise stated. means.

Also, as used herein, the term "module" or "unit" means a software or hardware component, and the "module" or "unit" performs a role. However, "module" or "unit" is not limited to software or hardware. A "module" or "unit" may be configured to be on a storage medium that can be addressed (addressed), or may be configured to play one or more processors. Therefore, as an example, a "module" or "unit" can be a component such as a software component, an object-oriented software component, a class component, a task component, and a process, function, attribute, procedure, subroutine, or program code. It can include at least one of a segment, driver, firmware, microcode, circuit, data, database, data structure, table, array or variable. Components and "Modules" or "Units" The functionality provided within can be combined into a smaller number of components and "Modules" or "Units", or with additional components and "Modules" or "Units". Can be further separated into.

According to one embodiment of the present disclosure, a "module" or "unit" can be embodied in a processor and memory. "Processor" should be broadly interpreted to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrols, state machines, and the like. In some environments, "processor" may refer to an application specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), and the like. A "processor" is, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors coupled to a DSP core, or any other combination of such configurations. It may also refer to the same combination of processing devices as. Also, "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. "Memory" is any access memory (RAM), read-only memory (ROM), non-volatile arbitrary access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM). Also referred to as various types of processor readable media such as electrically erasable Programmable ROM (EEPROM), flash memory, magnetic or optical data storage devices, registers and the like. If the processor can read information from the memory and record the information in the read memory, the memory is said to be in electronic communication with the processor. The memory integrated in the processor is in electronic communication with the processor.

In the present disclosure, the "speech recognition model" may refer to a model that outputs text data corresponding to the linguistic information included in the input voice when speech data is input. That is, the speech recognition model can embody STT (Speech-to-Text) technology. In one embodiment of the present disclosure, the speech recognition model may correspond to an artificial neural network model generated or updated by performing supervised learning, unsupervised learning or semi-supervised learning using learning data. For example, the speech recognition model can be a Listen, Atend and Spell (LAS) -based E2E-ASR (End-to-End Automatic Speech Recognition) model.

In the present disclosure, a "speech sample" may refer to user speech data collected for learning, updating, and testing a speech recognition model. The audio sample can be a preprocessed version of the collected data and processed into a certain format. For example, audio samples are extracted from collected user speech data using a humming window with a window length of 200 ms (window-length) and a stride length of 100 ms (stride-length). A spectrogram can be included.

In the present disclosure, "label" may refer to a text sequence corresponding to an audio sample. For example, the label may be a transcription of the linguistic information and linguistic meaning contained in the speech sample into text. The label can include a pseudo label output when the speech sample is input to the speech recognition model and a correct label transcribed by the human annotator to the speech sample.

FIG. 1 is a drawing showing an example in which a user 110 is provided with a service from a user terminal 120 through a voice command. In one embodiment, the user terminal 120 can receive a voice command from the user 110 through an input device such as a microphone. In this case, the user terminal 120 can recognize the received voice command by using the voice recognition model, and can provide the user 110 with the information and / or the service corresponding to the recognized voice command. As shown in the figure, when the user 110 utters a voice command "Tell me the weather today", the user terminal 120 may automatically recognize the voice command and output today's weather forecast through a speaker or the like. can.

The user terminal 120 may be any device configured to recognize the voice command spoken by the user 110 and provide services / information corresponding to the voice command. For example, the user terminal 120 can provide services such as a voice search service, an artificial intelligence (AI) secretary service, a map navigation service, and a set-top box control service. Although the user terminal 120 is illustrated as an artificial intelligence speaker in FIG. 1, the user terminal 120 is not limited to this, and may be any device capable of recognizing a voice command and providing a service corresponding to the voice command.

In order to recognize the voice command of the user 110, the user terminal 120 can use the voice recognition model generated through machine learning or the like. Such speech recognition models can be updated through iterative / gradual learning to improve the accuracy of speech recognition. Speech recognition model performance can be maximized by using as many Human Labeled Samples (HLS) as possible, where a person listens to the speech sample and directly generates the correct label, but due to the limitations of labeling costs, the HLS A speech recognition model learning method using only labels is practically difficult. In particular, the task of labeling speech samples, that is, the task of listening to and transcribing speech samples by humans, requires much higher costs than the task of labeling images, so human labeling costs are minimized and speech recognition performance is maximized. A machine learning method that can be realized is required.

In one embodiment, semi-supervised learning (SSL) and active learning (AL) are stitched together to minimize HLS, and learning is performed using an unlabeled voice sample. Consistency regularization (CR) techniques can be used to further improve efficiency. Specifically, n voice samples with the highest uncertainty score (that is, the lowest reliability of the speech recognition model) are extracted from the unlabeled speech sample pool to perform human labeling work. Thereby, a plurality of HLS can be prepared. Here, n is a natural number and can be determined according to the budget for human labeling costs. Also, among the voice samples remaining in the voice sample pool to which no label is assigned, the voice samples whose uncertainty score is less than the predetermined critical value (that is, the reliability of the speech recognition model exceeds the critical value) are selected. Multiple machine-labeled samples (MLS) can be prepared by extracting, performing machine labeling operations, and augmenting audio samples. The speech recognition model can then be learned / updated using both HLS and MLS. Here, MLS can play a role of assisting HLS in learning / updating the speech recognition model.

FIG. 2 is linked so that the information processing system 230 can communicate with a plurality of user terminals 210_1, 210_2, 210_3 in order to provide a voice recognition service according to an embodiment of the present disclosure and learn a voice recognition model. It is a schematic diagram which shows the structure made. The information processing system 230 can include a system capable of providing a speech recognition infrastructure service through the network 220 and / or a system capable of learning a speech recognition model. In one embodiment, the information processing system 230 is a computer-executable program (eg, a downloadable application) and one or more server devices capable of storing, providing, and executing data related to a voice recognition infrastructure service or voice recognition model learning. And / or a database, or one or more distributed computing devices and / or distributed databases of a cloud computing service infrastructure. The voice recognition infrastructure service provided by the information processing system 230 can be provided to the user through a voice search application, an artificial intelligence secretary application, etc. installed in each of the plurality of user terminals 210_1, 210_2, 210_3. For example, the information processing system 230 can provide information corresponding to a voice command input from a user or perform a corresponding process through a voice search application, an artificial intelligence secretary application, or the like. Additionally, the information processing system 230 can collect speech samples from a plurality of user terminals 210_1, 210_2, 210_3 in order to learn / update the speech recognition model.

The plurality of user terminals 210_1, 210_2, 210_3 can communicate with the information processing system 230 through the network 220. The network 220 may be configured to allow communication between a plurality of user terminals 210_1, 210_2, 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, an Ethernet, a wired home network (Power Line Communication), a telephone line communication device, a wired network such as RS-serial communication, a mobile communication network, a WLAN (Wireless LAN), or Wi-Fi. It may consist of wireless networks or combinations thereof such as (Registered Trademarks), Bluetooth® and ZigBee®. The communication method is not limited, and not only the communication method utilizing the communication network (for example, mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that can be included in the network 220, but also the user terminal 210_1 , 210_2, 210_3 may also include short-range radio communication.

In FIG. 2, a mobile phone terminal 210_1, a tablet terminal 210_2, and a PC terminal 210_3 are illustrated as examples of user terminals, but the present invention is not limited to this, and the user terminals 210_1, 210_2, 210_3 can perform wired and / or wireless communication. It can be any computing device in which voice infrastructure service applications, search applications, web browser applications, etc. can be installed and performed. For example, user terminals include AI speakers, smartphones, mobile phones, navigation systems, computers, laptop computers, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, and game consoles. , Wearable devices, IoT (internet of things) devices, VR (visual reality) devices, AR (augmented reality) devices, set top boxes and the like can be included. 2 It may be configured to communicate with the information processing system 230 through 220.

FIG. 3 is a block diagram showing an internal configuration of a user terminal 210 and an information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 can execute a voice infrastructure service application or the like, and can refer to any computing device capable of wired / wireless communication. For example, the mobile phone terminal 210_1, the tablet terminal 210_2, and the PC terminal 210_3 in FIG. Etc. can be included. As illustrated, the user terminal 210 may include a memory 312, a processor 314, a communication module 316 and an input / output interface 318. Similarly, the information processing system 230 may include a memory 332, a processor 334, a communication module 336 and an input / output interface 338. As illustrated in FIG. 3, the user terminal 210 and the information processing system 230 may be configured to be able to communicate information and / or data through the network 220 using their respective communication modules 316 and 336. Further, the input / output device 320 may be configured to input information and / or data to the user terminal 210 through the input / output interface 318 and output information and / or data generated from the user terminal 210. ..

The memory 312, 332 can include any non-transient computer-readable recording medium. According to one embodiment, the memory 312, 332 has a large non-erasable memory such as a RAM (random access memory), a ROM (read only memory), a disk drive, an SSD (solid state drive), a flash memory (flash memory), and the like. A capacity storage device (permanent mass memory device) can be included. As another example, a non-extinguishing large-capacity storage device such as a ROM, SSD, flash memory, disk drive, etc. is a separate permanent storage device that is separated from the memory, and is a user terminal 210 or an information processing system 230. Can be included in. Further, the memory 312 and 332 may store an operating system and at least one program code (for example, a code for a voice infrastructure service application installed and driven in the user terminal 210).

Such software components may be loaded from a computer-readable recording medium separate from the memory 312, 332. Such a separate computer-readable recording medium can include such a recording medium that can be directly coupled to the user terminal 210 and the information processing system 230, such as a floppy drive, disk, tape, DVD /. It can include computer-readable recording media such as CD-ROM drives and memory cards. As another example, software components may be loaded into memory 312, 332 through a communication module that is not a computer-readable recording medium. For example, at least one program may be loaded into memory 312, 332 based on a computer program installed by a file provided by a file distribution system that distributes a developer or application installation file.

Processors 314 and 334 may be configured to process instructions in a computer program by performing basic arithmetic, logic and input / output operations. Instructions may be provided by processors 314, 334 by memory 312, 332 or communication modules 316, 336. For example, the processor 314, 334 may be configured to carry out an instruction received by a program code stored in a recording device such as memory 312, 332.

The communication modules 316 and 336 can provide a configuration or function for the user terminal 210 and the information system 230 to communicate with each other through the network 220, and the user terminal 210 and / or the information system 230 may be another user. It is possible to provide a configuration or function for communicating with a terminal or another system (for example, a separate cloud system). As an example, the request or data (for example, data corresponding to the user's voice instruction) generated by the processor 314 of the user terminal 210 by the program code stored in the recording device such as the memory 312 is the communication module 316. Can be transmitted to the information processing system 230 through the network 220 by the control of. On the contrary, the control signal or instruction provided by the control of the processor 334 of the information processing system 230 can be received by the user terminal 210 through the communication module 316 of the user terminal 210 via the communication module 336 and the network 220. For example, the user terminal 210 can receive information related to a voice command from the information processing system 230 through the communication module 316.

The input / output interface 318 may be a means for an interface with the input / output device 320. As an example, an input device is a device such as a camera, keyboard, microphone, mouse including an audio sensor and / or an image sensor, and an output device is a device such as a display, a speaker, a haptic feedback device (haptic feedback device), and the like. Can be included. In another example, the input / output interface 318 may be a means for an interface with a device such as a touch screen that integrates configurations or functions for performing inputs and outputs. For example, when the processor 314 of the user terminal 210 processes the instruction of the computer program loaded in the memory 312, it is configured by using the information and / or the data provided by the information processing system 230 and other user terminals. The service screen and the like can be displayed on the display through the input / output interface 318. Although the input / output device 320 is shown in FIG. 3 so as not to be included in the user terminal 210, the input / output device 320 is not limited to this, and may be configured by the user terminal 210 and one device. Further, the input / output interface 338 of the information processing system 230 is a means for connecting to the information processing system 230 or for an interface with an input or output device (not shown) that can be included in the information processing system 230. possible. In FIG. 3, the input / output interfaces 318 and 338 are shown as elements separately configured from the processors 314 and 334, but the input / output interfaces 318 and 338 are configured to be included in the processors 314 and 334. Can be done.

The user terminal 210 and the information processing system 230 can include more components than those in FIG. However, it is not necessary to clearly illustrate many prior art components. According to one embodiment, the user terminal 210 may be embodied to include at least a portion of the input / output devices 320 described above. In addition, the user terminal 210 can further include other components such as a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, a database, and the like. For example, when the user terminal 210 is a smartphone, it can include components generally included in the smartphone, for example, an acceleration sensor, a gyro sensor, a camera module, various physical buttons, and a button using a touch panel. Various components such as, input / output ports, and accelerometers for vibration can be further embodied in the user terminal 210.

According to one embodiment, the processor 314 of the user terminal 210 may be configured to operate an application or the like that provides a voice infrastructure service. At this time, the code associated with the application and / or the program may be loaded into the memory 312 of the user terminal 210. While the application and / or program is running, the processor 314 of the user terminal 210 receives information and / or data provided by the input / output device 320 through the input / output interface 318 and from the information processing system 230 through the communication module 316. Information and / or data can be received, and the received information and / or data can be processed and stored in memory 312. Further, such information and / or data can be provided to the information processing system 230 through the communication module 316.

While a program for a voice infrastructure service application or the like is running, the processor 314 inputs through an input device such as a camera, microphone including a touch screen, keyboard, audio sensor and / or image sensor coupled with an input / output interface 318. The text, image, video, audio, etc. selected or selected can be received, and the received text, image, video and / or audio, etc. can be stored in the memory 312, or the information processing system through the communication module 316 and the network 220. Can be provided for 230. In one embodiment, the processor 314 can provide the information processing system 230 through the network 220 and the communication module 316 with the voice instruction related data input by the user on the voice infrastructure service application through the input device. The processor 334 of the information processing system 230 may be configured to manage, process and / or store information and / or data received from a plurality of user terminals and / or a plurality of external systems. In one embodiment, the information processing system 230 can provide the user terminal 210 with information corresponding to the voice instruction-related data received from the user terminal 210. In addition, the information processing system 230 can collect unlabeled audio samples from the user terminal 210.

FIG. 4 is a drawing showing an example of constructing an HLS database (DB) 460 and an MLS DB 470 through labeling work for an audio sample 410 according to an embodiment of the present disclosure. The processor of the information processing system can collect the voice sample 410 to which the label is not assigned from the user terminal. The collected voice sample 410 may be stored in the voice sample DB 420 to which no label is assigned. Since performing human labeling on all collected speech samples is costly, the processor utilizes the speech recognition model 440 to provide speech samples from the speech sample 410 to perform human labeling. Can be extracted.

The processor can use an uncertainty-based AL (uncertainty-based AL) to select an informative sample of the speech samples 410 that is useful for learning the speech recognition model 440. Specifically, the processor can extract voice samples 422 for human labeling based on the uncertainty score of each voice sample. In one embodiment, the processor utilizes a pre-generated speech recognition model 440 to calculate the uncertainty score of the speech sample in the unlabeled speech sample DB 420 and has the highest uncertainty score n. An audio sample 422 can be extracted. Here, n is a natural number and can be determined according to the budget for human labeling costs.

In one embodiment, the speech sample uncertainty score can indicate the length-normalized joint probability of the text sequence output by the speech recognition model 440. For example, the uncertainty score and the reliability score of the voice sample can be calculated by using the following mathematical formulas (1) to (3).

here,

は音声サンプルＤＢ４２０内の音声サンプルを示し、

Shows the audio sample in the audio sample DB 420,

は音声認識モデル４４０により出力されるテキストシーケンス（すなわち、最も可能性が高いデコーディングされたテキスト）を示し、

Represents the text sequence output by the speech recognition model 440 (ie, the most likely decoded text).

は出力テキストシーケンスの結合確率を示し、Ｌは出力テキストシーケンスの長さを示し、

Indicates the join probability of the output text sequence, L indicates the length of the output text sequence,

は長さ正規化されたログ結合確率を示し、ＮＰは音声サンプルの不確実性スコアを示し、δは音声サンプルの信頼度スコアを示すことができる。前記で確認できるように、長いテキストに対する結合確率が過小評価（ｕｎｄｅｒｅｓｔｉｍａｔｉｎｇ）されることを防止するために、プロセッサは出力テキストの長さに基づいて結合確率を正規化することができる。一実施例において、音声サンプルの不確実性スコアは音声認識モデル４４０が音声サンプルの出力テキストシーケンスをデコーディングする間、音声認識モデル４４０のデコーダの部分で算出され得る。

Can indicate the length-normalized log binding probability, NP can indicate the uncertainty score of the voice sample, and δ can indicate the confidence score of the voice sample. As can be seen above, the processor can normalize the join probabilities based on the length of the output text in order to prevent the join probabilities for long texts from being underestimated. In one embodiment, the speech sample uncertainty score can be calculated in the decoder portion of the speech recognition model 440 while the speech recognition model 440 decodes the output text sequence of the speech sample.

N voice samples 422 with the highest uncertainty score (lowest confidence score) may be provided to the Human annotator 430 for human labeling. The human annotator 430 can listen to the transmitted n voice samples 422 and generate the correct label 432. The correct label 432 can be a text sequence transcribed from the speech contained within the audio sample. The processor can store n voice samples 422 with high uncertainty and n corresponding correct labels 432 in HLS DB460 as HLS (Human Labeled Sample). At this time, one HLS may be composed of a pair of a voice sample and a correct label.

Additionally, the processor can extract the audio sample 424 for machine labeling from the unlabeled audio sample DB 420. When preparing the MLS using a speech sample with high uncertainty (ie, a sample with low reliability of the speech recognition model 440), the MLS provides false information to the speech recognition model 440 to improve the speech recognition model performance. On the contrary, it may decrease. Therefore, the processor uses at least one voice sample having a predetermined uncertainty score below the critical value (reliability score above the critical value) among the remaining voice samples in the voice sample DB 420 for machine labeling. It can be extracted as a voice sample 424 with low uncertainty.

A speech sample 424 with low uncertainty may be provided to the speech recognition model 440 for machine labeling. The speech recognition model 440 can predict a pseudo label 442 corresponding to each of the transmitted speech samples 424. The pseudo-label can be a text sequence that is output when the speech sample is input to the speech recognition model 440.

Pseudo-labels are not only less informative than HLS, but can also be noisy, so if you process MLS in the same way as HLS, it may not be useful for learning / updating the speech recognition model 440, or it may give incorrect information. It can be provided and impair the performance of the speech recognition model 440. To prevent this, a voice sample 424 with low uncertainty may be provided to the data augmentation unit 450. The data enhancement unit 450 can augment the transmitted audio sample 424 to generate an enhanced audio sample 452. Enhancing the audio sample can mean adding distortion, noise, etc. to the audio sample. Unlike image sample enhancement, the linguistic information contained in the audio sample is very vulnerable to distortion, noise, etc., so the linguistic information in the audio sample can be easily damaged by distortion, noise, etc. Therefore, the audio sample augmentation process must be carefully designed so that the linguistic meaning within the audio sample does not change as distortion, noise, etc. are added.

According to one embodiment, the data enhancement unit 450 can perform pitch shifting on the audio sample 424. Alternatively, the data augmentation unit 450 can perform time scaling on the audio sample 424. Alternatively, the data augmentation unit 450 can add additive white Gaussian noise to the audio sample 424. The processor can store the enhanced audio sample 452 and the corresponding pseudo-label 442 in the MLS DB 470 as an MLS (Machine Labeled Sample). At this time, one MLS may consist of a pair of enhanced audio samples and pseudo-labels.

The processor can update the speech recognition model 440 using the HLS in the HLS DB460 and the MLS in the MLS DB470. According to one embodiment, the processor is semi-supervised learning based on a speech sample-correct label pair stored in HLS DB460 and an enhanced speech sample-pseudo-label pair stored in MLS DB470. Can be performed to update the speech recognition model 440. By updating the speech recognition model 440 using all of HLS and MLS, the robustness of the speech recognition model 440 can be improved.

According to one embodiment, the processor uses the speech recognition model 440 to minimize the difference between the speech sample 422 predicted by the speech recognition model 440 and the corresponding output data, and the correct label 432 of the speech sample 422. Can be updated. For example, the difference between the speech sample 422 predicted by the speech recognition model 440 and the corresponding output data and the correct label 432 can be calculated by a standard cross-entropy loss function as described below. ..

here

は指導損失（ｓｕｐｅｒｖｉｓｅｄ ｌｏｓｓ：教師あり損失）を示し、Ｂはミニバッチ（ｍｉｎｉ－ｂａｔｃｈ）の大きさを示し、Ｌ_ｎはｎ番目のＨＬＳサンプルの長さを示し、

Indicates supervised loss, B indicates the size of the mini-batch, L _n indicates the length of the nth HLS sample,

は正解ラベル４３２を示し、

Indicates the correct label 432,

は音声認識モデル４４０により予測された出力データ（すなわち、音声認識モデル４４０により予測される音声サンプル４２２と対応するテキストシーケンス）を示し、Ｈはクロス－エントロピー（ｃｒｏｓｓ－ｅｎｔｒｏｐｙ）を示す。

Indicates the output data predicted by the speech recognition model 440 (that is, the text sequence corresponding to the speech sample 422 predicted by the speech recognition model 440), and H indicates cross-entropy.

The processor also updates the speech recognition model 440 to minimize the difference between the enhanced speech sample 452 predicted by the speech recognition model 440 and the corresponding output data, and the pseudo-label 442 of the speech sample 424. can do. For example, the difference between the enhanced speech sample 452 predicted by the speech recognition model 440 and the corresponding output data and the pseudo-label 442 of the speech sample 424 can be calculated by a standard cross-entropy loss function such as:

は非指導損失（ｕｎｓｕｐｅｒｖｉｓｅｄ ｌｏｓｓ：教師なし損失）を示し、Ｂはミニバッチ（ｍｉｎｉ－ｂａｔｃｈ）の大きさを示し、Ｌ_ｎはｎ番目のＭＬＳサンプルの長さを示し、Ａは増強関数を示し、

Indicates unsupervised loss, B indicates the size of the mini-batch, L _n indicates the length of the nth MLS sample, A indicates the augmentation function,

は音声認識モデル４４０により予測される増強された音声サンプル４５２と対応する出力データ（すなわち、音声認識モデル４４０により予測される増強された音声サンプル４５２と対応するテキストシーケンス）を示し、

Shows the output data corresponding to the enhanced speech sample 452 predicted by the speech recognition model 440 (ie, the text sequence corresponding to the enhanced speech sample 452 predicted by the speech recognition model 440).

は疑似ラベル４４２を示し、Ｈはクロス－エントロピー（ｃｒｏｓｓ－ｅｎｔｒｏｐｙ）を示す。

Indicates a pseudo label 442, and H indicates cross-entropy.

Total loss used to update speech recognition model 440

は指導損失

Is a loss of guidance

と非指導損失

And non-leading loss

を統合して、下記の数式（６）のように定義され得る。

Can be integrated and defined as the following formula (6).

Here, λ can indicate the coefficient value of the non-leading loss. For example, λ can be a constant value between 0 and 1. λ can be used to weight the teaching loss using the reliable sample HLS in the process of performing semi-supervised learning to update the speech recognition model 440. Processor total loss

が最小化するように半教師あり学習を遂行することができる。

Semi-supervised learning can be performed to minimize.

In one embodiment, the processor stores new HLS and MLS in HLS DB460 and MLS DB470 according to the flow described above each time a certain amount of voice sample 410 is added to the unlabeled voice sample DB420. The process of updating the speech recognition model 440 using the HLS in the HLS DB460 and the MLS in the MLS DB470 can be repeated.

FIG. 5 is a flowchart showing an initial speech recognition model generation method 500 according to an embodiment of the present disclosure. In one embodiment, the method 500 for generating an initial speech recognition model can be performed by a processor (eg, at least one processor in an information processing system). As illustrated, method 500 of generating an initial speech recognition model can be initiated by the processor receiving a plurality of unlabeled speech samples (S510). The processor can then receive the correct label for each of the plurality of unlabeled audio samples from the human annotator (S520).

The processor can then generate an initial speech recognition model based on the pair of speech samples received in step (S510) and correct labels received in step (S520) (S530). That is, the processor can generate an initial speech recognition model by performing supervised learning of an artificial neural network model using HLS. Here, one HLS may consist of a pair of audio sample and correct label.

FIG. 6 is a flowchart showing a gradual speech recognition model learning method 600 according to an embodiment of the present disclosure. In one embodiment, the method 600 for learning a speech recognition model can be performed by a processor (eg, at least one processor in an information processing system). As illustrated, method 600 for learning a speech recognition model can be initiated by the processor receiving a plurality of unlabeled speech samples (S610). The plurality of voice samples may be voice samples collected from the user terminal while providing the voice recognition service.

In response to receiving a plurality of speech samples, the processor can utilize the speech recognition model to extract a first set of speech samples for human labeling from the plurality of speech samples (S620). In one embodiment, the processor utilizes a speech recognition model to calculate the uncertainty score for each of the speech samples, and a predetermined number of speech samples with the highest uncertainty score among the speech samples. Can be extracted as the first set of audio samples. Here, the uncertainty score can indicate the length-normalized join probability of the text sequence output by the speech recognition model.

The processor can then receive the first set of audio samples and the corresponding first set of labels (S630). Here, the labels in the first set can be human-generated correct labels. The processor can store the first set of audio samples and the first set of labels in HLS.

The processor can also use the speech recognition model to extract a second set of speech samples for machine labeling from a plurality of speech samples (S640). In one embodiment, the processor can extract at least one audio sample having a predetermined critical value or less uncertainty score from a plurality of audio samples as a second set of audio samples. The number of audio samples in the first set for human labeling may be less than the number of audio samples in the second set for machine labeling.

The processor can then utilize the speech recognition model to determine the second set of speech samples and the corresponding second set of labels (S650). Here, the label of the second set can be a pseudo label predicted by the speech recognition model.

The processor can also augment the second set of audio samples (S660). In one embodiment, the processor can perform pitch shifting on a second set of audio samples. In another embodiment, the processor can perform time scaling on a second set of audio samples. In yet another embodiment, the processor can add additive white Gaussian noise to the second set of audio samples. The processor can store the enhanced second set of audio samples and the second set of labels in MLS.

The processor then performs semi-supervised learning based on the first set of speech samples, the first set of labels, the enhanced second set of speech samples and the second set of labels to update the speech recognition model. Can be done (S670). In one embodiment, the processor recognizes speech so that the difference between the first set of speech samples predicted by the speech recognition model, the corresponding first set of output data, and the labels of the first set is minimized. You can update the model. In addition, the processor minimizes the difference between the enhanced second set of speech samples predicted by the speech recognition model, the corresponding second set of output data, and the second set of labels. The speech recognition model can be updated. Here, the difference between the first set of output data and the first set of labels, and the difference between the second set of output data and the second set of labels can be calculated by the standard cross-entropy loss function. .. As shown, the processor can progressively learn / update the speech recognition model by iteratively performing S610-S670.

FIG. 7 is a drawing illustrating examples of speech samples 710, 720, 730 for generating, updating, and testing speech recognition models according to an embodiment of the present disclosure. The processor of the information processing system can receive voice samples 710, 720, and 730 from the user terminal. The received audio sample can be classified into an initial audio sample 710, a subsequent audio sample 720, and a test audio sample 730. In one embodiment, the processor utilizes a humming window with a window length of 200 ms and a stride-length of 100 ms, and a spectrogram from the received audio sample. (Spectrogram) can be extracted.

The processor can use the initial speech sample 710 to generate an initial speech recognition model. In one embodiment, the processor can generate an initial speech recognition model by using the initial speech sample 710 to perform the initial speech recognition model generation method described above in FIG. The processor can then utilize the subsequent speech sample 720 to update the speech recognition model. In one embodiment, the processor can update the speech recognition model by using the subsequent speech sample 720 to perform the speech recognition model learning method described above in FIG. For example, the processor may divide the subsequent speech sample 720 into a plurality of sections (for example, 30 sections) and perform the speech recognition model update many times (for example, 30 times) using the speech sample in each section. can.

After the speech recognition model has been generated and updated, the processor can utilize the test speech sample 730 to test the speech recognition model performance. In one embodiment, the processor could test the speech recognition model performance by inputting each of the test speech samples 730 into the updated speech recognition model and comparing the output data with the correct label generated by the human annotator. can. The performance of the speech recognition model can be evaluated by the Character-level Error Rate (CER). Here, the CER can be determined based on the character difference between the output data and the correct label.

In one embodiment, the number of initial audio samples 710 may be less than the number of subsequent audio samples 720. For example, the initial audio sample 710 can include 110 hours of audio sample, the subsequent audio sample 720 can include 386 hours of audio sample, and the test audio sample 730 can include 56 hours of audio sample. Further, the initial audio sample 710 may be an audio sample collected before the subsequent audio sample 720, and the subsequent audio sample 720 may be an audio sample collected before the test audio sample 730. With such a configuration, the performance of the speech recognition model learning method according to the embodiment of the present disclosure can be evaluated so as to be similar to the actual situation. The performance evaluation of the speech recognition model learning method according to the embodiment of the present disclosure carried out in such an environment will be described below with reference to FIGS. 8 to 10. In the performance evaluation, the voice sample whose reliability score (δ value of the formula (3)) exceeds the critical value (τ = 0.9) was extracted and machine labeling was performed. In addition, the non-teaching loss coefficient value (λ) was used as 1 in order to emphasize the influence of MLS in speech recognition model learning.

FIG. 8 is a graph showing the performance difference of the speech recognition model by the method of extracting the speech sample for human labeling. As mentioned above, speech samples for performing human labeling can be extracted from speech samples that have not been assigned a label to train / update the speech recognition model. In the graph, "NP" indicates the case where the uncertainty score of the voice sample is calculated by using the above-mentioned mathematical formulas (1) and (2). In the graph, "RND" indicates a case where voice samples that perform human labeling are randomly selected. In the graph, "Loss" and "CER" indicate the case where the uncertainty score is calculated by a method other than the formulas (1) and (2).

In order to evaluate the usefulness of speech samples that perform human labeling extracted by each criterion for speech recognition model learning, multiple speech samples are arranged according to the criteria described above and divided into five speech sample sets. can do. For example, a total of 386.5 hours of audio samples can be aligned according to their respective criteria and divided into 5 audio sample sets of 77.3 hours. Here, "set1 / 5" is a set containing the sample with the highest uncertainty (that is, a sample useful for learning a speech recognition model), and "set5 / 5" is the sample with the lowest uncertainty (that is, the sample useful for learning a speech recognition model). A set that includes a sample) that is not useful for speech recognition model learning. After that, each voice sample set can be used to prepare an HLS, and the prepared HLS can be used to perform supervised learning to generate a voice recognition model. The performance of the generated speech recognition model can be indicated by CER (%). Here, the lower the CER (%), the better the performance of the speech recognition model.

As shown, "NP", "Loss", and "CER" each have the smallest CER (%) value at "set1 / 5" and the smallest "NP" at "set1 / 5" (%). ) Has a value. Further, in "NP", it can be confirmed that the CER (%) increases almost monotonically as the voice sample set having a low uncertainty score (that is, a high reliability score) is used. On the other hand, in "Loss" and "CER", unlike "NP", it can be confirmed that an unexpected change form appears for the CER (%) value for each voice sample set. This is because when calculating the uncertainty score of a speech sample using the "Loss" or "CER" method, the correct label and speech recognition are not taken into account the binding probabilities between text sequences predicted through the speech recognition model. This is to determine the uncertainty score by measuring the difference between the labels predicted by the model. Therefore, in order to extract the voice sample that performs human labeling, it is another criterion to calculate the uncertainty score of the voice sample by using the NP value of the voice sample (formulas (1) and (2) described above). It is more accurate and effective to calculate the uncertainty score in.

FIG. 9 is a graph showing the difference in performance of the speech recognition model by the speech sample enhancement method of the present disclosure. In the graph, "NoCR" indicates the case where data enhancement is not performed, "CR-P" indicates the case where pitch shifting is performed for the audio sample as data enhancement, and "CR-A" indicates the case where data enhancement is performed. The case where additive white Gaussian noise is added to the voice sample is shown, and “CR-S” shows the case where time scaling is performed for the voice sample as data enhancement. For example, "CR-P" indicates that the pitch of the audio sample is shifted by 2.5 steps (one step is one octave divided into eight), and "CR-A" indicates that the audio sample is SNR (Signal-). "to-Noise Ratio") indicates the addition of additive white Gaussian noise of 5 or less, and "CR-S" indicates that the reproduction speed of the audio sample is time-scaled 1.5 times faster.

In FIG. 9, the x-axis indicates the amount of HLS (ie, the time of the audio sample), and the xy of "(LUxy)" on the x-axis indicates the ratio of HLS (x) to MLS (y). For example, in the case of 38.6h (LU19), a case where semi-supervised learning is performed based on 38.6 hours of HLS and 9 times the amount of MLS of HLS to advance the speech recognition model update is shown. The graph of FIG. 9 can be analyzed with Table 1 below. Table 1 shows the performance (CER (%)) of the speech recognition model updated under the conditions corresponding to each row and each column. Here, it can be evaluated that the lower the CER (%), the better the performance of the speech recognition model.

As can be confirmed in Table 1, the performance of the speech recognition model generated through supervised learning using only 386 hours of HLS is CER = 10.89%, which is the best. Further, as can be confirmed in Table 1 and FIG. 9, it can be confirmed that the performance of the speech recognition model becomes worse as the amount of HLS decreases and the amount of MLS increases. In addition, except for LU16, the CER (%) of "NoCR" is higher than the CER (%) of "Supervised learning", so MLS including the unenhanced speech sample is rather used for learning the speech recognition model. It can be confirmed that it has a negative effect. In particular, in the performance evaluation, a speech sample that performs machine labeling based on a relatively low reliability score critical value (τ = 0.9) is extracted, and a high non-supervised loss coefficient value (λ = 1) is set. Since the speech recognition model was semi-supervised, the negative effects of inaccurate pseudo-labels in the MLS on the speech recognition model are well shown.

In Tables 1 and 9, except for "CR-S" of LU12 and LU14, when the audio sample enhanced from "NoCR" was used in each row ("CR-S", "CR-A", "CR". It can be confirmed that the CER (%) of −P ”) is low. In addition, since "CR-P" has the lowest CER (%) among "CR-S", "CR-A", and "CR-P", pitch shifting is performed for the voice sample as data enhancement. If so, it can be confirmed that the performance of the speech recognition model is the best.

On the other hand, when the number of HLS used for speech recognition model learning is small (for example, LU16 or LU19), it can be confirmed that the effect of enhancing the speech sample contained in MLS is more remarkable. For example, the CER (%) when using the LU19-enhanced audio sample is reduced by 1.26% p and 1.60% p, respectively, as compared with "Supervised learning" and "NoCR". On the other hand, when the speech recognition model is learned / updated using a sufficient amount of HLS (for example, LU12), the effect of enhancing the speech sample seems to be slight, but this is the learning effect of HLS on the speech recognition model. Is large enough.

FIG. 10 is a graph showing the relationship between the learning cycle and the performance of the speech recognition model when the speech recognition model is updated many times according to the embodiment of the present disclosure. The graph of FIG. 10 updates the speech recognition model up to the 30th time, and shows the CER (%) of the speech recognition model updated each time. It can be confirmed that the CER (%) of "NoCR" is larger than the CER (%) of "CR-S", "CR-A", and "CR-P" for each of LU12 and LU16. That is, "NoCR" causes a deterioration in the performance of the speech recognition model due to an inaccurate pseudo label (that is, an inaccurate MLS). According to the embodiments of the present disclosure, non-guidance loss

が音声認識モデルがよく分からない音声サンプルに高い信頼度スコアを付与することを制約するため、前述した問題を緩和して機械学習でＭＬＳを活用して優秀な音声認識モデル性能を提供することができる。

However, it is possible to alleviate the above-mentioned problems and utilize MLS in machine learning to provide excellent speech recognition model performance because it restricts giving a high reliability score to speech samples for which the speech recognition model is not well understood. can.

In conclusion, when the speech recognition model is trained / updated according to the examples of the present disclosure, the labeling cost can be reduced by about 2/3 only by increasing the CER by 0.26% p (that is, the performance deterioration). The labeling cost can be reduced by about 6/7 only by increasing the CER by 08% p. Therefore, it is possible to significantly reduce the human labeling cost for updating the speech recognition model while minimizing the performance degradation of the speech recognition model (for example, the performance degradation due to inaccurate MLS).

The speech recognition model learning method described above may be provided by a computer program stored on a computer-readable recording medium for execution on a computer. The medium may be one that continues to store programs that can be run on a computer, or that is temporarily stored for execution or download. Further, the medium may be a variety of recording means or storage means in the form of a single piece or a combination of several pieces of hardware, but the medium is not limited to a medium directly connected to a certain computer system and exists in a distributed manner on a network. It may be a thing. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic-optical media such as flographic discs. There may be one that is configured to store program commands, including media), and ROM, RAM, flash memory, and the like. Further, as an example of other media, a recording medium or a storage medium managed by an app store that distributes applications, a site that supplies or distributes various other software, a server, or the like can be mentioned.

The methods, actions or techniques of the present disclosure may be embodied by a variety of means. For example, such techniques may be embodied in hardware, firmware, software, or a combination thereof. The usual technicians have shown that the various exemplary logical blocks, modules, circuits and algorithmic steps described in conjunction with the present disclosure may be embodied in electronic hardware, computer software, or a combination thereof. Will be understandable. In order to articulate such mutual substitutions of hardware and software, various exemplary components, blocks, modules, circuits and stages have been generally described above in terms of their functionality. Whether such functionality is embodied in hardware or software depends on the design requirements imposed on the particular application and the overall system. The usual technician may embody the features described in various ways for each particular application, but such embodying shall not be construed as departing from the scope of this disclosure.

In the realization of hardware, the processing units used to perform the technique are one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). , Field programmable gate arrays (FPGAs), processors, controls, microprocessors, microprocessors, electronic devices, and other electronic units designed to perform the functions described in the present disclosure. , Computer, or a combination thereof.

Accordingly, the various exemplary logic blocks, modules and circuits described in conjunction with this disclosure include general purpose processors, DSPs, ASICs, FPGAs and other programmable logic devices, discrete gates and transistor logic, discrete hardware components. , Or any combination, such as those designed to perform the functions described herein, may be embodied or performed. The general purpose processor may be a microprocessor, but as an alternative, the processor may be any conventional processor, controller, microprocessor, or state machine. Processors are also embodied in combinations of computing devices, such as DSPs and microprocessors, multiple microprocessors, one or more microprocessors associated with a DSP core, or any other combination of configurations. May be good.

In the realization of firmware and / or software, the techniques are random access memory (RAM), read-only memory (read-only memory; ROM), non-volatile RAM (non-volatile random memory; NVRAM), PROM. (Prommable read-only memory), EPROM (erasable program-only memory), EEPROM (electrically remember-only memory), flash memory, compact disk (computer-readable storage such as computer or storage, such as CD), magnetic. It may be embodied by an instruction stored on the medium. Instructions may be executed by one or more processors, and the processors may be made to perform certain aspects of the functions described in the present disclosure.

Although the embodiments described above have been described as taking advantage of aspects of the subject currently disclosed in one or more stand-alone computer systems, the present disclosure is not limited to this, but for networks and distributed computing environments. It may be embodied in conjunction with any computing environment such as. Further, in the present disclosure, aspects of the subject may be embodied in multiple processing chips and devices, and storage may be similarly affected across multiple devices. Such devices may include PCs, network servers and portable devices.

Although the present disclosure has been described herein in connection with some embodiments, various variations and variations and variations within the scope of the present disclosure that are understandable by ordinary technicians in the art to which the invention of the present disclosure belongs. Changes can be made. It should also be understood that such modifications and alterations fall within the scope of the claims attached herein.

110: User 120, 210: User terminal 220: Network 230: Information processing system

Claims (16)

In a speech recognition model learning method performed by at least one processor
A stage of receiving multiple voice samples to which no label is assigned, and a stage of extracting a first set of voice samples for human labeling from the plurality of voice samples using a voice recognition model. The stage of receiving the label of the first set corresponding to the voice sample of the first set, and the voice sample of the second set for machine labeling from the plurality of voice samples using the voice recognition model. , The stage of determining the label of the second set corresponding to the second set of voice samples using the voice recognition model, and the stage of augmenting the second set of voice samples. And semi-supervised learning is performed based on the first set of audio samples, the first set of labels, the enhanced second set of audio samples, and the second set of labels. A speech recognition model learning method including a step of updating the speech recognition model. The step of enhancing the second set of audio samples is
The speech recognition model learning method according to claim 1, further comprising a step of performing pitch shifting on the second set of speech samples. The step of enhancing the second set of audio samples is
The speech recognition model learning method according to claim 1, further comprising a step of performing time scaling on the second set of speech samples. The step of enhancing the second set of audio samples is
The speech recognition model learning method according to claim 1, further comprising a step of adding additive white Gaussian noise to the second set of speech samples. The step of extracting the first set of voice samples for human labeling from the plurality of voice samples using the voice recognition model is
A step of calculating an uncertainty score for each of the plurality of speech samples using the speech recognition model, and a predetermined number of speech samples having the highest uncertainty score. The voice recognition model learning method according to any one of claims 1 to 4, wherein the voice sample is extracted as the first set of voice samples. The step of extracting a second set of voice samples for machine labeling from the plurality of voice samples using the voice recognition model is
The speech recognition model according to claim 5, wherein at least one speech sample having an uncertainty score equal to or less than a predetermined critical value among the plurality of speech samples is extracted as the second set of speech samples. Learning method. The speech recognition model learning method according to claim 5, wherein the uncertainty score indicates the length-normalized joint probability of the text sequence output by the speech recognition model. The stage of updating the speech recognition model is
The speech recognition model is updated to minimize the difference between the first set of speech samples predicted by the speech recognition model, the corresponding first set of output data, and the labels of the first set. The speech recognition model learning method according to any one of claims 1 to 7, which comprises the stage of performing. The stage of updating the speech recognition model is
The speech recognition is such that the difference between the enhanced second set of speech samples and the corresponding second set of output data and the labels of the second set as predicted by the speech recognition model is minimized. The speech recognition model learning method according to claim 8, further comprising the step of updating the model. The difference between the first set of output data and the first set of labels, and the difference between the second set of output data and the second set of labels, is the standard cross-entropy loss function (standard cross). -The speech recognition model learning method according to claim 9, which is calculated by (entropy loss function). The speech recognition model learning method according to any one of claims 1 to 10, wherein the speech recognition model is an artificial neural network model generated by performing supervised learning. The speech recognition model according to any one of claims 1 to 11, wherein the number of voice samples in the first set for human labeling is smaller than the number of voice samples in the second set for machine labeling. Learning method. The speech recognition model learning method according to any one of claims 1 to 12, wherein the label of the first set is a correct answer label generated by a person. The speech recognition model learning method according to any one of claims 1 to 13, wherein the label of the second set is a pseudo label predicted by the speech recognition model. A computer program for executing the speech recognition model learning method according to any one of claims 1 to 14 on a computer. It is a speech recognition model learning system.
Communication module and
Includes memory and at least one processor concatenated with said memory and configured to execute at least one computer-readable program contained in said memory.
The at least one program
Receive multiple unlabeled audio samples and
Using the speech recognition model, the first set of speech samples for human labeling is extracted from the plurality of speech samples.
Upon receiving the first set of audio samples and the corresponding first set of labels,
Using the speech recognition model, a second set of speech samples for machine labeling is extracted from the plurality of speech samples.
Using the speech recognition model, the second set of speech samples and the corresponding second set of labels are determined.
The second set of audio samples was enhanced to
Semi-supervised learning is performed based on the first set of speech samples, the first set of labels, the enhanced second set of speech samples, and the second set of labels to update the speech recognition model. A speech recognition model learning system that includes command words for.

Images