KR20210081166A - Spoken language identification apparatus and method in multilingual environment - Google Patents

Abstract

A method for a computing device operated by at least one processor to identify a language comprises the steps of: dividing an audio stream into one or more speech chunks; extracting a plurality of acoustic features from each speech chunk; generating learning data by tagging the audio stream language with the extracted acoustic features from each speech chunk; and learning a language identification model for estimating the language of the audio stream based on the learning data. The speech chunks are obtained by extracting a speech band section of a certain length or more from the audio stream. It is possible to identify the language of a speaker based on an artificial neural network using the features of the speech chunks.

Description

Apparatus and method for language identification in a multilingual speech environment {SPOKEN LANGUAGE IDENTIFICATION APPARATUS AND METHOD IN MULTILINGUAL ENVIRONMENT}

The present invention relates to a technology for identifying a language in a multilingual speech environment.

With the development of voice recognition technology, an intelligent personal assistant or smart speaker equipped with a microphone and a speaker is widely distributed. These electronic devices use a speech dialog system, which generates and outputs a voice response corresponding to the input voice in the order of speech recognition - natural language understanding - dialogue operation - natural language generation - speech synthesis. do.

Recently, the performance of speech recognition and speech synthesis has been greatly improved by applying deep learning to the voice conversation system used in these electronic devices. In addition, research to apply deep learning to natural language understanding and generation is being actively conducted.

On the other hand, the voice conversation system is not limited to a single language, and has started to support multiple languages. Even in countries that use one official language, multilingual natural language interface has become an essential skill in accordance with the trend of internationalization in modern society. For example, in a hotel room where guests of various nationalities stay, the customer can request necessary items through a smart speaker, play music, or receive a concierge service. Alternatively, even if the mother tongues of parents and children are different in one household, there is a trend that the smart speaker can be used in two languages.

In the multilingual situation, in the voice conversation system, a speech language identification (hereinafter referred to as 'LID') process is added to the configuration described above. That is, language identification - speech recognition - natural language understanding - dialogue operation - natural language generation - speech synthesis may be performed, or language identification and speech recognition may be performed in parallel.

A voice conversation system supporting multiple languages should include voice conversation subsystems that respectively process the supported languages. Since there is only one channel through which a user's utterance is inputted in the voice conversation system, the subsystems of the multilingual voice conversation system receive voices through the same input channel. Since subsystems supporting each language exist individually, even if the subsystem processes the input voice without a language identification process at the front stage, a problem arises in that the language must be determined from the output. Therefore, rather than parallel processing of all subsystems, identifying the language at the front is an efficient way to use the system's resources.

In a multilingual environment, before the technology for self-identifying a language in a voice conversation system was applied, a user designated a language using an input device such as a remote control, a smartphone, or a voice speech. For example, it is possible to switch to English by saying “change to English mode” to an electronic device set in Korean, and if the user is not familiar with the preset language, the user can switch the language by speaking in the target language. For example, it is a case of saying “I want to speak Korean” to an electronic device set in English.

In recent years, as a technology for automatically identifying the language of an inputted voice has been developed, there is no need for a user to manually designate an utterance language. As such language identification technology, audio signal processing and statistical techniques have been traditionally used, but a deep learning technique is being introduced.

In general, deep learning LID technology divides input utterances into frame units (for example, 10 ms), extracts acoustic features, and classifies multi-class as a model trained according to the number of languages supported by the voice conversation system. classification) is performed. For example, a system supporting bilingualism can be classified into two classes (Korean, English) or three classes (Korean, English, other languages). In this case, the acoustic characteristics are applied in units of frames regardless of the amount of data or the contents of speech.

The minimum processing unit of the audio signal input to the smart speaker is a frame, and when receiving a frame streamed by the smart speaker, the voice conversation system extracts and processes the feature to reduce the amount of information. At this time, the number of times to extract features from the streamed frame is frequent, and since most of the speech corpus data for learning is composed of sentence units, there are disadvantages in that it is necessary to wait until the sentence is finished for language identification accuracy.

As such, the LID technology applied to the multilingual voice conversation system faces several challenging environments. First of all, compared to data such as phone call data used to learn the existing LID technology, the length of voice input through the smart speaker is short. In addition, in the case of commercial systems, user experience (UX) must also be considered, so it is necessary to reduce the delay time from when a user speaks to a response. Therefore, LID, which is one stage of the multilingual voice conversation system, needs a method to reduce the delay time perceived by the user.

An object to be solved is to provide an apparatus and method for identifying a speaker's language based on an artificial neural network using the characteristics of a speech mass.

Another object of the present invention is to provide a method of extracting features of an input audio stream by dividing the input audio stream into speech chunks having a variable length.

A method for a computing device operated by at least one processor according to an embodiment to identify a language, comprising: dividing an audio stream into one or more speech chunks; extracting a plurality of acoustic features from each speech chunk; generating learning data by tagging the language of the audio stream to acoustic features extracted from the chunk, and learning a language identification model for estimating the language of the audio stream based on the learning data, wherein the The audio chunk is obtained by extracting a speech band section of a certain length or longer from the audio stream.

The language identification model may include Recurrent Neural Networks that give different weights to each of the acoustic features extracted from each of the speech chunks according to a learning result.

The plurality of acoustic features may include a Mel Frequency Cepstral Coefficient (MFCC).

The extracting may include extracting a first acoustic feature and a second acoustic feature for each audio chunk constituting the audio stream, arranging the extracted first acoustic features to generate a first feature vector, and arranging the second acoustic features to generate a second feature vector.

The learning of the language identification model may include learning different language identification models using the first feature vector and the second feature vector.

Partitioning a new audio stream into one or more speech chunks, inputting only a speech chunk including acoustic information among speech chunks of the new audio stream into the language identification model, and predicting language of the new audio stream with the language identification model It may further include the step of estimating at least one or more.

The input may include inputting only some speech chunks corresponding to the audio stream first input to the computing device in chronological order among the speech chunks divided in the new audio stream into the language identification model.

The estimating may include outputting a probability value for each prediction language together when there are a plurality of prediction languages.

A computing device according to an embodiment, comprising a memory and at least one processor executing instructions of a program loaded into the memory, wherein the program is a voice section corresponding to a preset voice band in an audio stream , and when the length of the extracted speech section exceeds a preset reference length, generating an audio stream of the speech section as a speech chunk, extracting a plurality of acoustic features from each speech chunk, the extracted and inputting acoustic features into a learned language identification model, and obtaining at least one predictive language of the audio stream from the language identification model.

The extracting may include extracting the plurality of acoustic features from some of the audio chunks corresponding to the audio stream first inputted to the computing device in chronological order among the respective audio chunks.

The inputting may include inputting different recurrent neural networks according to the extracted acoustic features.

The obtaining may include inputting the audio stream into an arbitrary speech recognition model, obtaining a speech recognition result from the speech recognition model, and determining a language of the audio stream from among the predicted languages using the speech recognition result may include the step of

According to the present invention, it is possible to quickly identify the language of the spoken voice and to stop other systems by operating only the voice recognition subsystem corresponding to the identified language, thereby saving the computational resources of the voice conversation system.

In addition, according to the present invention, it is possible to identify a language by using a portion of a voice input first, without waiting for the entire input utterance of the user, so that the processing speed of the system can be faster than the existing method.

1 is a configuration diagram of a language identification apparatus and its surrounding environment according to an embodiment.
2 is a block diagram of an existing language identification device.
3 is a block diagram of a language identification apparatus according to an embodiment.
4 is a flowchart of a method of dividing an audio stream into speech chunks according to an embodiment.
5 is an exemplary diagram illustrating a waveform of a sound mass according to an embodiment.
6 is an explanatory diagram of a method of generating learning data using features extracted from a voice mass according to an exemplary embodiment.
7 is an explanatory diagram of a method of generating learning data using features extracted from a voice mass according to another embodiment.
8 is an exemplary diagram of a method for identifying a language according to an embodiment.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as “…unit”, “…group”, and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software. have.

1 is a configuration diagram of a language identification apparatus and its surrounding environment according to an embodiment.

Referring to FIG. 1 , the user's voice is processed into an audio stream in units of frames through the electronic device 100 , and the language identification apparatus 200 divides the audio stream into voice chunks and extracts features of the voice chunks to obtain an input vector , and through the language identification model 231 using deep learning, it is possible to specify the user's language.

The electronic device 100 may be connected to the language identification device 200 by wire or wirelessly, and includes a microphone 110 that receives a sound uttered by a user, an ADC 120 that converts the uttered voice into a digital signal, and a voice signal. and a preprocessor 130 for amplifying Signal-to-Noise Ratio (SNR).

The user's speech input through the microphone 110 is converted into a digital signal through the analog-to-digital converter (ADC) 120 .

ADC 120 converts the input analog signal into a digital signal by setting a specific sampling rate and a specific bit depth, for example, the sampling rate to 16 kHz and the bit depth to 16 bits. have.

The pre-processing unit 130 enhances a signal among digital signals and suppresses noise. Signal processing may be performed in a time domain or a frequency domain, and noise cancellation, echo cancellation, voice activity detection, and the like may be applied.

2 is a block diagram of an existing language identification device.

Referring to FIG. 2 , in the conventional language identification apparatus 20 , an audio stream input in real time means frames of audio samples sequentially input from a user. Visualizing the entire sample of the waveform uttering one sentence is the same as skipping the frame-length window by the hop length from the beginning to the end of the input signal. At this time, in the attached frames, the front frame and the rear frame overlap as much as an overlap section. For example, if the hop length is set to 2 ms in a frame of 10 ms length, the overlap period with the next frame is 8 ms. Assuming that the total speech time including the silence section at the beginning of the speech and the silence section at the end of the speech is 0.91 seconds, if the frames are divided into 2 ms units except for 10 ms from 0.91 seconds, the total number of frames is 450 it's a dog

In this case, since the existing device 20 needs to extract features for each frame for 450 frames, system resources may be wasted. In addition, a delay in system processing time may occur because it has to wait for the end of a sentence or end point detection to identify a language.

For each frame divided in this way, an acoustic feature is extracted by the feature extraction unit 21 . The method of extracting the acoustic features by the feature extraction unit 21 may use a result of research on the human auditory organ. Acoustic features, prosodic features, phonemic features, etc. can be applied, and in some cases, additional lexical features and syntactic features based on the understanding of natural language can utilize Representative sound level characteristics include a Mel-Frequency Cepstral Coefficient (MFCC) and a Shifted Delta Cepstral Coefficient (SDC).

Based on the extracted features, the model learning unit 22 learns the language identification model 23 . For example, a Gaussian Mixture Model (GMM) using N Gaussian components may be generated using MFCC or SDC. Alternatively, it may be implemented as a machine learning or deep learning algorithm in which a user inputs data and features and learns a model using corresponding labels.

Thereafter, the language identification unit 24 determines the spoken language of the audio stream input in units of frames by using the learned language identification model 23 .

That is, the existing language identification apparatus 20 extracts features in units of frames. The greatest feature of the invention proposed in this specification is to extract acoustic features in units of sound chunks, which are units larger than frames. Hereinafter, the language identification apparatus 200 and the operation method proposed in the present specification will be described.

3 is a block diagram of a language identification apparatus according to an embodiment.

Referring to FIG. 3 , the language identification apparatus 200 includes a speech chunk division unit 210 for dividing an audio stream into speech chunks, and a feature extraction unit 220 for extracting features of the speech chunk and generating an input vector 410 . , the model learning unit 230 for learning the language identification model 231 using the input vector 410 and the language identification model 231 determine the language of the input utterance according to the criteria generated, and determine the probability of the language and a language identification unit 240 to output.

For the sake of explanation, the speech chunk dividing unit 210 , the feature extracting unit 220 , the model learning unit 230 , and the language identification unit 240 are named and called, but these are computing devices operated by at least one processor. Here, the speech chunk dividing unit 210 , the feature extracting unit 220 , the model learning unit 230 , and the language identification unit 240 may be implemented in one computing device or distributed in separate computing devices. When distributed in a separate computing device, the speech chunk dividing unit 210 , the feature extracting unit 220 , the model learning unit 230 , and the language identification unit 240 may communicate with each other through a communication interface. The computing device may be any device capable of executing a software program written to carry out the present invention, and may be, for example, a server, a laptop computer, or the like.

Each of the feature extraction unit 220 , the model learning unit 230 , and the language identification unit 240 may be a single deep learning model or may be implemented as a plurality of deep learning models. Also, the language identification model 231 may be a single deep learning model or may be implemented as a plurality of deep learning models. The language identification apparatus 200 may be one deep learning model or may be implemented as a plurality of deep learning models. Accordingly, one or a plurality of deep learning models corresponding to the above-described configurations may be implemented by one or a plurality of computing devices.

The speech chunk dividing unit 210 forms a speech chunk according to an algorithm for determining whether a frame input as an audio stream is speech. Specifically, it will be explained through an example in which there is one voice mass. When it is determined that the first frame to the nth frame is not a voice, the m frames from the n+1th frame are voices, and the n+m+2th frame to the last frame is not a voice, the speech chunk dividing unit 210 ) may generate a speech mass by gathering m frames determined to be speech.

On the other hand, when there are a plurality of sections determined to be speech, a plurality of speech chunks are generated. The speech chunk dividing unit 210 transmits speech chunks composed of continuous frames determined to be speech to the next stage, the feature extraction unit 220 .

The feature extraction unit 220 generates an input vector 410 for an input sentence by extracting acoustic features of each speech mass. The feature extraction unit 220 may extract various features on a specific time and frequency by a user or a developer. As another method, the feature extraction unit 220 may automatically extract an arbitrary feature in the learning process according to the structure of the artificial neural network. The acoustic features extracted for each chunk by the feature extraction unit 220 are generated as an input vector 410 and used as learning data.

The model learning unit 230 learns the language identification model 231 by using the input vector 410 and the labeling result of determining the corresponding language. In the present invention, since data is divided into speech chunks, the labeling result may be different for each speech chunk.

In the present specification, the language identification model 231 is implemented as a recurrent neural network (RNN), but a deep learning algorithm implementing the model is not necessarily limited thereto.

The language identification unit 240 may determine the language of the user's utterance by using the learned language identification model 231 and may output a probability value of the determined language together. When deciding a language, various methods can be used based on the decision theory.

As an example, one language having the highest calculated probability value may be selected. As another example, a language whose probability value exceeds a predetermined reference value may be output, and in this case, the language determined by the language identification unit 240 may be plural.

As another example, a process in which the language identification unit 240 determines a language using the language identification model 231 for the same input signal and a voice recognition process in the voice conversation subsystem of each language may be performed in parallel. . In this case, the language identification unit 240 may determine the language using another classifier that inputs a probability value calculated using the language identification model 231 and a specific value such as reliability extracted from the speech recognition result of each language as input. . That is, it is possible to simultaneously perform language identification and voice recognition on a user's utterance, and determine the language of the corresponding utterance in consideration of a result of using the learned model and a result of voice recognition.

In addition, when language identification and voice recognition are performed in parallel, if the judgment of the language identification unit 240 is faster than the voice recognition process, the voice recognition process of an unidentified language or a language with a remarkably low probability of being identified is stopped, so that the system It can prevent wastage of resources.

Meanwhile, the result output by the language identification unit 240 and the method of determining the language may vary depending on the usage plan, policy, strategy, etc. of the language identification apparatus 200 and are not limited thereto.

4 is a flowchart of a method of dividing an audio stream into speech chunks according to an embodiment.

Referring to FIG. 4 , the audio chunk dividing unit 210 resets the frame count and the buffer ( S101 ). The frame count is used to measure the length of the input audio stream, and the buffer serves to temporarily store the frame determined as voice.

The audio chunk dividing unit 210 collects an audio stream in units of frames ( S102 ).

Thereafter, the speech chunk dividing unit 210 attenuates a high-frequency component in the input audio stream by using a Gaussian filter (S103). In addition, if the Gaussian filter is used, a valley between a voice signal and a non-voice signal on the graph is deepened, so it can be helpful in determining whether the input audio stream is voice.

The speech chunk dividing unit 210 checks whether the collected current frame is speech ( S104 ). As one example of a method of confirming whether it is a voice, a voice activity detection (Voice Activity Detection, hereinafter referred to as 'VAD') technology may be used. By applying the VAD technology, it is possible to determine whether the input signal section includes a human voice through time and frequency analysis of the voice signal.

Hereinafter, the VAD technology will be described in detail. The VAD technology is a basic technology widely used in most speech processing fields such as speech coding, speech recognition, and speech enhancement. Technically, it is largely divided into Energy Thresholding, Detection with More Features, statistical techniques using energy and voice features, and machine learning or deep learning techniques that classify voice/non-voice. It can be distinguished, and is known as the VAD module of G.729 Annex B, WebRTC (Real-Time Communication) VAD, etc. The VAD technology may also be called Speech Activity Detection or Speech Detection.

Meanwhile, as another example, speech presence probability (hereinafter referred to as 'SPP') may be utilized to determine whether the collected current frame is speech. Hereinafter, the SPP will be described in detail in relation to the VAD technology.

SPP estimation is a technique closely related to VAD, and refers to the result of determining whether an output value of VAD is voice/non-voice through SPP estimation. That is, the SPP estimation result is a probability between 0 and 1, and the VAD result indicates true/false whether the result is negative. Therefore, a VAD result can be derived by setting a threshold value in the SPP estimation result. For example, the SPP may be calculated using the energy of the frame of the voice signal, and if it is higher than a preset threshold, the frame may be determined as voice. Meanwhile, the threshold value may vary according to the number of words included in the sentence in which the number of divided speech chunks is input.

Returning to FIG. 4 , when the received current frame is determined to be a voice, the voice chunk dividing unit 210 stores the current frame in a buffer and increases the frame count by 1 ( S105 ). Then, back to step S102 again to collect a new frame.

When it is determined that the input current frame is not a voice, the voice chunk dividing unit 210 determines whether a previous frame is a voice (S106). The technique described in step S104 may be used to determine whether the previous frame is voiced.

If it is determined in step S106 that the previous frame is not a voice, it is determined that a meaningful voice section is over. The audio chunk dividing unit 210 resets the frame count for counting the number of frames stored in the buffer (S107), and returns to step S102 to collect frames.

If the previous frame is a voice in step S106, the voice chunk dividing unit 210 determines that a meaningful voice section from the current frame is over, and prepares to generate a voice chunk by storing a buffer. First, the speech chunk dividing unit 210 checks whether the frame count exceeds a reference value (S108). Since the frame count means the length of the frame stored in the buffer, the audio chunk is generated only when the content temporarily stored in the buffer is larger than the preset minimum audio chunk size. This is to prevent the occurrence of a large number of meaningless short-length sound chunks. Meanwhile, the reference value is a value that is set by the administrator or the language identification device 200 and can be changed.

When the frame count exceeds the reference value, the audio chunk dividing unit 210 stores the frames stored in the buffer as one audio chunk (S109).

5 is an exemplary diagram illustrating a waveform of a sound mass according to an embodiment.

Referring to FIG. 5 , a situation in which the user has uttered the sentence "Check me out." and the voice is streamed in units of frames will be described as an example. At this time, when the utterances of different users are input for the same sentence, the waveform of the input signal varies depending on factors such as the user's utterance volume and habit, so the waveform of the user's voice may be different even if the sentence is the same, , the length of the generated speech chunks may be different from each other.

Reference numeral 310 denotes a waveform of the user's voice, the start of the waveform may be a point at which a call word is recognized, and the end point may be a point obtained using an end point detection (EPD) algorithm.

Reference numeral 320 is a picture in which a range corresponding to each word is divided in the audio waveform, and it can be seen that each word and a lump of sound are not distinguished. The third mass from the left on the waveform is a sound waveform that starts at ‘k’ in ‘Check’, includes ‘me’, and ends in ‘ou-’ in ‘out’.

Therefore, it is not easy to find out exactly where ‘k’ and ‘me’ are connected, or where ‘me’ ends and ‘ou-’ begins when looking only at the waveform.

Speech segmentation technology is used to accurately separate an audio stream representing a waveform connected to words as in 310 in word units. Speech segmentation technology is a technology for segmenting boundaries between words, syllables, and phonemes, and is used as a base technology for speech analysis problems such as speech recognition and speech synthesis.

Reference numeral 330 denotes a sound mass by cutting out a waveform based on a point having no voice information. It can be seen from the waveform that 331 represents ‘Chec-’, 332 represents ‘-k me ou-’, and 333 represents ‘-t’.

Reference numeral 340 denotes a result of the speech chunk dividing unit 210 dividing the user's utterance. Since 343 does not reach the reference length in step S106 of FIG. 4 , the speech chunk dividing unit 210 does not treat it as a valid speech chunk. Therefore, only 341, 342 and 344 are generated as valid negative chunks.

Comparing with the ideal case 330, it can be seen that the '-u-' of 342 is removed. However, when 341, 342, 344 are reproduced in succession, they sound very similar to the original sentences, and they do not interfere with human language discrimination and even understanding of the content. Accordingly, the information loss is not large and the amount of computation may be slightly lowered by the speech chunk dividing unit 210 at the same time.

Meanwhile, in the case of a different speaker for the same sentence, the length or number of generated speech chunks may vary due to differences in the speaker's voice volume, pronunciation characteristics, and the like. For example, when two speakers pronounce the sentence “Please call”, the voice mass may be “Please” or “Please c-” depending on the speaker’s utterance method.

6 is an explanatory diagram of a method of generating learning data according to an embodiment.

Referring to FIG. 6 , the learning data of the language identification model 231 is an input vector 410 in which acoustic features are extracted from each of the speech chunks 341 , 342 , 344 constituting a sentence and arranged according to the sentence order. Meanwhile, since the non-voice section 343 is not divided into speech chunks according to the criteria of FIG. 4 , features are extracted only for the three speech chunks 341 , 342 , and 344 .

The feature extraction unit 220 extracts features from each chunk to generate one input vector 410 for the input signal, thereby greatly reducing the dimension of the input signal.

For example, acoustic features such as a Mel Frequency Cepstral Coefficient (MFCC) may be extracted for each speech mass. In order to calculate the MFCC, the input sound is divided into a certain section, and a power spectrum is calculated for each section. A bank of Mel filters is then applied to the power spectrum, and the energies of each filter are summed. Then, a discrete cosine transform (DCT) is taken on log values of all filter bank energies, and some coefficients are used. Since MFCC is a known technology, a detailed description thereof will be omitted.

Meanwhile, the method of the feature extracting unit 220 to extract the acoustic features from each voice mass and the extracted acoustic features are not limited thereto. The feature extraction unit 220 may extract various features in time and frequency, such as various sounds applicable to language identification, such as a duration time of a sound, a pitch, and the like, a prosody, a phoneme, and a lexical feature. In addition, an effective wavelet scattering technique can be applied to time series data analysis.

The feature extraction unit 220 may vectorize the features of each extracted speech mass according to the order of the spoken sentences. The value of the input vector 410 may vary depending on the order in which the speech chunks are arranged, but the order may not be important because the axes forming the vector change as the order in which the input vector 410 is arranged changes. In the present invention, for the convenience of interpreting the results, it will be described that the audio chunks are vectorized according to the order in which they are received.

The feature extraction unit 220 transmits the generated input vector 410 to the model learning unit 230 .

Thereafter, the model learning unit 230 learns by inputting the generated input vector 410 and a label of the corresponding language into the language identification model 231 . The label for training the language identification model 231 in a supervised learning method means a signal (Supervised Signal) to guide the training result, a map signal, a target signal, a correct answer (Answer), a tag (Tag) may be called.

On the other hand, the language identification model 231 may be implemented as a deep learning model of various structures. For example, a multi-layer bidirectional long-short term memory (LSTM), which is a type of recurrent neural network (RNN), and a multi-layer perceptron A structure in which (Multi-Layer Perceptron, MLP) is connected may be used. Hereinafter, a general description of artificial neural networks is added.

Commonly used artificial neural networks for language identification include Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN).

MLP, also called FFNN (Feed-Forward Neural Network), FCN (Fully Connected Network), or DN (Dense Network), maps input data to a problem space to classify clusters formed by input vectors. can be used In the language identification problem, N languages may be divided into N classes.

In the classification process, a nonlinear hyperplane formed by the weights of the MLP is weighted to serve as a decision boundary 520 so that one cluster can be classified into one class. is to learn There are various methods of learning the weights, and an algorithm of the Backpropagation Algorithm series may be used as an example.

That is, the language identification process using the artificial neural network makes the learning data labeled with N languages form clusters in the problem space, learns the decision boundary 520 so that these clusters can be divided into N classes, and the decision A class is determined based on the boundary 520 .

In the learning process, the optimal hyperparameter is selected so that the performance of the artificial neural network can be improved so that a statistically correct decision can be made. Hyperparameter refers to a variable that is set in advance when training an arbitrary model, the method of encoding the label for data, the structure of the artificial neural network and the internal activation function, loss function ( Loss Function), learning algorithm-related coefficients, decision criteria, and performance metrics.

As another artificial neural network structure, CNN additionally has a convolution network that extracts features at the bottom of the FCN. CNNs have strengths in their ability to process information about complex spaces because the convolutional network automatically learns the features required when FCN classifies data.

As another artificial neural network structure, RNN can be used. Unlike in MLP, where input data passes through all nodes once and is output, RNN has a cyclic structure in which the connection between units is such that the result of the hidden layer enters the same input again.

That is, the RNN stores the results calculated so far in an internal memory (Hidden State), so it is a model suitable for processing data having spatiotemporal characteristics such as handwriting recognition and voice recognition. As an example of RNN, Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) are widely used, but it is not limited to one structure and any algorithm can be used as long as there is feedback on the weights that can be learned. can be applied. In the structure forming the RNN, one unit such as LSTM and GRU can be used as a single layer, or by applying a stacked architecture, or by making the data flow unidirectional or bidirectional, of course. to be.

In addition, since the RNN can consider the continuous flow of speech chunks, that is, the characteristics of previously entered speech chunks, it is advantageous for analyzing a specific speech chunk including information useful for language identification in an input sentence when learning the language identification model 231 . .

On the other hand, when multiple languages are included in one sentence, it is necessary to consider the before and after of the speech mass. For example, a case in which the sentence "Please check in" is input and divided into two voice masses "Check in" and "Please do" will be described. It's hard to tell if it's Korean "Check in" or English "Check in" just from the first chunk. If there is a second mass, "Do it", it can be determined that the corresponding sentence is Korean.

Returning to FIG. 6 again, the model learning unit 230 receives a vector composed of features extracted from a speech mass as an input. Referring to FIG. 6 , a vector summing all features extracted from three speech chunks is input to the model learning unit 230 .

The model learning unit 230 may process an input vector composed of the extracted features of the speech masses collected so far in order. The feature extracted for the first speech mass 341 is treated as an input vector, and when the feature 412 for the second speech mass 342 is extracted, the previously extracted feature 411 and the new feature 412 are connected as input. A vector is formed, and when a feature 413 for the last speech chunk 344 is extracted, all three features 411 , 412 , and 413 are connected to form an input vector. Therefore, the vector generated at the end is the same as the entire input vector 410 of FIG. 6 .

Meanwhile, the model learning unit 230 may not form an input vector in the entire input sentence. That is, an input vector can be formed only with the first speech chunk 341 , and an input vector can be formed only with the two speech chunks 341 and 342 . When only a portion of the speech mass is formed as an input vector and subsequent steps are performed, faster processing is possible compared to processing the entire speech as an input. In this case, how many of the previously formed speech chunks are used to form the input vector may be determined by the manager.

As another method, the model learning unit 230 may use only n speech chunks, not the whole, like the n-gram method of natural language processing. As in the unigram method, an input vector is formed only with the current speech chunk, or as in the bi-gram method, an input vector is formed with two speech chunks of the current and previous speech. According to this, an input vector is formed with the first speech chunk 341, the next input vector is formed with the first and second speech chunks 341 and 342, and the input vector is input with the second and third speech chunks 342 and 344. vector can be formed.

On the other hand, unlike the case of extracting one type of feature, there may be a plurality of types of acoustic features extracted from a voice mass, which will be described below with reference to FIG. 7 .

7 is an explanatory diagram of a method of generating learning data using features extracted from a voice mass according to another embodiment.

Referring to FIG. 7 , when the voice chunk dividing unit 210 generates an input audio stream into three voice chunks 341 , 342 , 344 , the feature extraction unit 220 extracts two features for each voice chunk By extraction, two vectors 410 and 410' can be generated. In addition, as many separate networks (not shown) as the number of extracted acoustic features may be used. The two networks can each learn the input vectors 410 and 410', each accumulating the features of the speech chunks that are sequentially input. Two independent networks can independently learn the latent meaning of each feature extracted from three speech chunks 341 , 342 , 344 . Thereafter, the model learning unit 230 forms a latent vector by successively connecting vectors output from the two networks, and the latent vector may be input to a classifier such as MLP, which is a network of the next stage.

8 is an exemplary diagram of a method for identifying a language according to an embodiment.

Referring to FIG. 8 , the language identification model 231 receives the input vector 410 generated by the feature extraction unit 220 of FIG. 4 as an input, and outputs probability values for N languages. Also, the learning result of the language identification model 231 may be expressed as a decision boundary 520 generated in the vector space 510 .

That is, the model learning unit 230 forms a vector space 510 of the input vectors 410 and forms a decision boundary 520 on the vector space 510 to classify the space corresponding to each language.

The method by which the model learning unit 230 forms the non-linear decision boundary 520 may be statistical, machine learning, or deep learning, and a support vector machine (SVM) may be used as an example of a representative machine learning technique. have. Since the operation method of the SVM is widely known, a detailed description thereof will be omitted. In addition, MLP, CNN, RNN, etc. can be used as examples of representative deep learning techniques, and the operation method is as described above.

After the decision boundary 520 is learned, when a new input vector is input, the corresponding vector may be determined as language 1 according to the decision boundary 520 as shown in FIG. 8 . Also, the language identification unit 240 may output a probability value that the new input vector corresponds to language 1.

According to the present invention, it is possible to quickly identify the language of the spoken voice and to stop other systems by operating only the voice recognition subsystem corresponding to the identified language, thereby saving the computational resources of the voice conversation system.

In addition, according to the present invention, it is possible to identify a language by using a portion of a voice input first, without waiting for the entire input utterance of the user, so that the processing speed of the system can be faster than the existing method.

The embodiment of the present invention described above is not implemented only through the apparatus and method, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improved forms of the present invention are also provided by those skilled in the art using the basic concept of the present invention as defined in the following claims. is within the scope of the right.

Claims (12)

A method for a computing device operated by at least one processor to identify a language, comprising:
segmenting the audio stream into one or more speech chunks;
extracting a plurality of acoustic features from each speech mass;
generating learning data by tagging the language of the audio stream to acoustic features extracted from each speech mass; and
Learning a language identification model for estimating the language of the audio stream based on the learning data,
The speech chunk is a speech band section of a certain length or longer is extracted from the audio stream. In claim 1,
The language identification model is
A language identification method comprising: Recurrent Neural Networks for giving different weights to each of the acoustic features extracted from each of the speech chunks according to a learning result. In claim 1,
The plurality of acoustic features are
A language identification method, including a Mel Frequency Cepstral Coefficient (MFCC). In claim 1,
The extraction step is
extracting a first acoustic characteristic and a second acoustic characteristic for each audio chunk constituting the audio stream; and
arranging the extracted first acoustic features to generate a first feature vector, and arranging the extracted second acoustic features to generate a second feature vector
A method for language identification, comprising: In claim 4,
Learning the language identification model comprises:
and learning different language identification models using the first feature vector and the second feature vector. in paragraph 1
splitting the new audio stream into one or more speech chunks;
inputting only the speech mass including sound information among the speech masses of the new audio stream into the language identification model; and
estimating at least one prediction language of the new audio stream with the language identification model;
Further comprising a, language identification method. In claim 6,
The input step is
The language identification method of inputting only a part of the speech chunks corresponding to the audio stream first input to the computing device in chronological order among the divided speech chunks in the new audio stream into the language identification model. In claim 6,
The estimating step is
When there are a plurality of prediction languages, the method for identifying a language is to output probability values for each prediction language together. A computing device comprising:
memory, and
at least one processor executing instructions of a program loaded into the memory;
the program is
extracting a voice section corresponding to a preset voice band from the audio stream, and when the length of the extracted voice section exceeds a preset reference length, generating an audio stream of the voice section as a voice mass;
extracting a plurality of acoustic features from each speech mass;
inputting the extracted acoustic features into a learned language identification model, and
obtaining at least one prediction language of the audio stream from the language identification model
A computing device comprising instructions written to execute In claim 9,
The extraction step is
and extracting the plurality of acoustic features from some of the audio chunks corresponding to the audio stream first inputted to the computing device in chronological order among the respective speech chunks. In claim 9,
The input step is
A computing device, which is input to different Recurrent Neural Networks according to the extracted acoustic features. In claim 9,
The obtaining step is
inputting the audio stream into an arbitrary speech recognition model and obtaining a speech recognition result from the speech recognition model; and
determining a language of the audio stream from among the prediction languages by using the speech recognition result
Computing device comprising a.

Images