KR102576754B1 - Deep learning-based speech improvement conversion device, system control method, and computer program - Google Patents

Abstract

The present invention provides an electronic device including a memory in which an artificial intelligence model for generating a target voice is stored, and a processor connected to the memory, wherein the processor includes audio data of a first voice of a user uttering a text. The artificial intelligence model is trained based on the audio data of the normal voice uttering the text and when the second voice of the user is input, the audio data of the input second voice is input to the artificial intelligence model, and the output of the artificial intelligence model A target voice obtained by converting the user's second voice may be obtained based on .

Description

Deep learning-based dysarthria voice improvement conversion device, system control method, and computer program

The present invention relates to a deep learning-based dysarthria voice improvement conversion technology for outputting a target voice, which is a normal voice, in which the patient's voice is converted as soon as the patient's voice uttering a text is input, wherein the target voice is the patient's voice. Depending on the selection, various voices such as the voice of the person, a male voice, a female voice, a middle-aged voice, and a specific person's voice may be converted and output.

In the case of patients with dysarthria, it is difficult to communicate in daily life, and in this case, rehabilitation can be performed through evaluation, but due to time constraints, the problem is that it is not helpful in situations where communication is needed right away. exist.

In this regard, the technology for evaluating dysarthria, such as Patent Document 1, stops at just performing the evaluation, and performing a language ability test and filling out a questionnaire according to a complicated procedure is inefficient due to time constraints.

On the other hand, the matters described as the background art above are only for improving understanding of the background of the present invention, and should not be taken as an admission that they correspond to prior art already known to those skilled in the art. will be.

Registered Patent Publication No. 10-1921890, 2018.11.20.

The problem to be solved by the present invention is to provide a deep learning-based dysarthria voice improvement conversion technology that outputs the voice of a patient with dysarthria as a normal voice in real-time communication.

Another object of the present invention is a deep learning-based dysarthria voice improvement conversion technology that enables immediate rehabilitation training for dysarthria improvement by outputting and providing a target voice, which is a normal voice, from a voice input by a patient. is to provide

In addition, the present invention is a technology researched from 2021.06.01 to 2021.12.31 with voice data for dysarthria (task identification number 21002131305088212100101BS) of the Korea Institute for Intelligence and Information Society Promotion, AI learning data construction project.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

An electronic device according to an aspect of the present invention for solving the above problems includes a memory in which an artificial intelligence model for generating a target voice is stored, and a processor connected to the memory.

Here, the processor trains an artificial intelligence model based on audio data of a user's first voice uttering a text and audio data of a normal voice uttering a text, and when the user's second voice is input, the input second voice is input. Audio data of the two voices may be input to the artificial intelligence model, and a target voice obtained by converting the second voice of the user may be obtained based on the output of the artificial intelligence model.

In addition, the artificial intelligence model compares a plurality of first feature vectors extracted from the audio data of the first voice with a plurality of second feature vectors extracted from the audio data of the second voice, and the speech target of the second voice The text may be identified, and a target voice obtained by converting the second voice to be matched with the identified text may be obtained.

In this case, the processor may obtain a frequency band of audio data for the first voice as user standard data, and obtain a target voice obtained by converting the second voice of the user based on the frequency band of the user standard data.

In addition, the processor determines the pronunciation accuracy of the second voice for the text that is the speech target of the second voice, outputs the second voice as a target voice when the pronunciation accuracy is greater than or equal to a threshold value, and outputs the second voice as a target voice when the pronunciation accuracy is less than the threshold value. Based on the output of the intelligence model, a target voice obtained by converting the user's second voice may be obtained.

Preferably, the processor obtains first subtext and second subtext constituting text that is an utterance target of the second voice, and matches each of the first subtext and the second subtext among the audio data of the second voice. Identifies an audio segment.

Then, the processor identifies pronunciation accuracy of the first audio section with respect to the first subtext and pronunciation accuracy of the second audio section with respect to the second subtext, the pronunciation accuracy with respect to the first subtext is less than a threshold value, and If the pronunciation accuracy of the 2 subtexts is equal to or greater than the threshold value, a target voice for the first subtext is obtained based on an output of the artificial intelligence model to which the first subtext is input.

Accordingly, the processor may output a converted voice of the second voice based on audio data including the target voice of the first subtext and the second audio section.

To this end, the processor obtains a user's voice for each of a plurality of texts, performs voice recognition, and matches and stores pronunciation accuracy for each of a plurality of texts.

On the other hand, the processor, among the plurality of texts, when the pronunciation accuracy of the user's newly input voice, which utters the first text whose pronunciation accuracy is less than the threshold value, is greater than or equal to the threshold value, the user's voice that utters the first text, If the first test voice is acquired a certain number of times, and the average pronunciation accuracy of the first test voice obtained a certain number of times is greater than or equal to a threshold value, the first test voice is matched to the first text based on the average pronunciation accuracy of the first test voice. Pronunciation accuracy can be updated.

On the other hand, the processor, among the plurality of texts, when the pronunciation accuracy of the user's newly input voice, which utters the second text with previously matched pronunciation accuracy equal to or greater than the threshold value, is less than the threshold value, the user's voice that utters the second text, If the second test voice is acquired a certain number of times and the average pronunciation accuracy of the test voice obtained a certain number of times is less than the threshold value, the pronunciation accuracy matched to the second text is based on the average pronunciation accuracy of the second test voice. can be updated.

In addition, the processor performs noise filtering on the audio data, sections the filtered audio data into a plurality of sections, overlaps adjacent sections by a predetermined time, and frames the audio data from the framed audio data. It may be characterized by extracting an acoustic feature vector and a plurality of MFCC (Mel Frequency Cepstral Coefficients) feature vectors.

At this time, the processor obtains at least one first MFCC feature vector by applying the MFCC technique to the framed audio data, and obtains at least one second MFCC feature vector by differentiating a value of the at least one first MFCC feature vector. and differentiating the value of the at least one second MFCC feature vector to obtain at least one third MFCC feature vector, wherein the plurality of MFCC feature vectors include at least one first MFCC feature vector and at least one second MFCC feature vector, and at least one third MFCC feature vector.

On the other hand, in the method for controlling a deep learning-based dysarthria voice improvement conversion system by a server according to another aspect of the present invention, obtaining audio data of a first voice of a user uttering a text from a user terminal; Acquiring audio data of a normal voice uttering text, training an artificial intelligence model to output a target voice based on the audio data of the first voice and the audio data of the normal voice, training the user's second voice from the user terminal. When the voice is received, inputting audio data of the received second voice into an artificial intelligence model, acquiring a target voice converted to the second voice from the artificial intelligence model, and converting the second voice through a user terminal. and outputting the target voice.

At this time, the step of acquiring the target voice converted from the second voice includes a plurality of first feature vectors extracted from the audio data of the first voice and a plurality of second feature vectors extracted from the audio data of the second voice. Comparing, identifying a text that is an utterance target of the second voice, and obtaining a target voice converted from the second voice to match the identified text, wherein the plurality of feature vectors are extracted from one audio data. It may include a plurality of acoustic feature vectors and a plurality of MFCC (Mel Frequency Cepstral Coefficients) feature vectors.

Preferably, the step of training the artificial intelligence model to output the target voice includes acquiring a frequency band of audio data for the first voice as user standard data, and based on the frequency band of the user standard data, the user's control 2 may include obtaining a target voice converted from the voice.

On the other hand, the present invention includes a computer program stored in a computer-readable recording medium to be combined with a computer, which is hardware, to perform the method according to another aspect of the present invention.

Other specific details of the invention are included in the detailed description and drawings.

According to the deep learning-based voice improvement conversion technology based on dysarthria of the present invention, the text uttered by the user is identified only by the user's voice input, and the normal voice for the text is output and provided as a target voice, so that patients with dysarthria can be improved in real time. To ensure that communication is accurate and fast.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a configuration diagram of an electronic device according to an embodiment of the present invention.
2 is a basic flowchart of voice improvement conversion based on deep learning according to an embodiment of the present invention.
3 and 4 are various system configuration diagrams according to an embodiment of the present invention.
5 is an operation flow diagram of a system according to an embodiment of the present invention.
6 is a diagram showing results of a voice enhancement conversion experiment according to an embodiment of the present invention.
7 is an exemplary view of real-life use through a user terminal according to this embodiment of the present invention.
8 is a server configuration diagram according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

The term "unit" or "module" used in the specification means a hardware component such as software, FPGA or ASIC, and "unit" or "module" performs certain roles. However, "unit" or "module" is not meant to be limited to software or hardware. A “unit” or “module” may be configured to reside in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, a “unit” or “module” may refer to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and "units" or "modules" may be combined into smaller numbers of components and "units" or "modules" or may be combined into additional components and "units" or "modules". can be further separated.

The spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe a component's correlation with other components. Spatially relative terms should be understood as including different orientations of elements in use or operation in addition to the orientations shown in the drawings. For example, if you flip a component that is shown in a drawing, a component described as "below" or "beneath" another component will be placed "above" the other component. can Thus, the exemplary term “below” may include directions of both below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

In this specification, a computer means any kind of hardware device including at least one processor, and may be understood as encompassing a software configuration operating in a corresponding hardware device according to an embodiment. For example, a computer may be understood as including a smartphone, a tablet PC, a desktop computer, a laptop computer, and user clients and applications running on each device, but is not limited thereto.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of an electronic device according to an embodiment of the present invention.

As shown, the electronic device 100 according to an aspect of the present invention includes a memory 110 in which an artificial intelligence model 111 for generating a target voice is stored, and a processor 120 connected to the memory 110. includes

In one embodiment, the electronic device 100 includes a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, and a desktop PC. ), laptop PC, netbook computer, workstation, server, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, camera, or wearable device It may include at least one of (wearable device) and artificial intelligence speaker (AI speaker).

2 is a basic flowchart of voice improvement conversion based on deep learning according to an embodiment of the present invention.

The processor 120 performs deep learning through the artificial intelligence model 111 learned for one user based on the audio data of the first voice of one user uttering one text and the audio data of normal voice uttering text. Based dysarthria speech improvement conversion can be performed.

Specifically, as shown in FIG. 2, when a user's second voice is input, audio data (digitized sound source extracted from the voice) of the input second voice is obtained (S210), and the user's voice is obtained. It is input to the artificial intelligence model 111 learned for , and based on the output of the artificial intelligence model 111, a target voice obtained by converting the second voice of the user is obtained (S220) and provided to the user (S230). .

To this end, the artificial intelligence model 111 compares a plurality of first feature vectors extracted from the audio data of the first voice with a plurality of second feature vectors extracted from the audio data of the second voice, A text that is a speech target of the voice is identified, and a target voice obtained by converting the second voice to be matched with the identified text is obtained.

In this case, values of a plurality of feature vectors of the user's voice (first voice) may be stored for each text. In this case, at least one first speech recognition model may be trained using a plurality of feature vectors stored for each text. As a result, the first speech recognition model is trained customized for the user (eg, dysarthria patient), and can recognize text uttered by the user.

On the other hand, when the voice of the user with dysarthria is input, the processor 120 identifies text according to the voice, and for the identified text, the processor 120 uses pre-stored voice acting or TTS (Text-To-Speech) technology. A normal voice such as a voice can be output as a target voice.

Specifically, in acquiring the target voice, the processor 120 obtains a frequency band of audio data for the first voice as user standard data, and based on the frequency band of the user standard data, the user's second voice is converted. A target voice can be obtained.

At this time, the processor 120 may acquire a plurality of target voices for one text, and each of the plurality of target voices has a different frequency band set, so that different tones are output.

Accordingly, the plurality of target voices may be configured according to a male tone, a female tone, an elderly tone, a child's tone, a specific person's tone, and a user tone based on user standard data.

On the other hand, dysarthria refers to a condition in which it is difficult to speak due to abnormalities in the vocal organs due to disorders and / or diseases caused by acquired or congenital factors. , laryngeal disorders, speech disorders, and Parkinson's disease.

3 and 4 are various system configuration diagrams according to an embodiment of the present invention, and FIG. 5 is an operational flowchart of the system according to an embodiment of the present invention.

As shown in FIG. 3 , the electronic device 100 of the present invention includes a voice input/output unit 130 in performing deep learning-based dysarthria visualization and rehabilitation, including a microphone ( The user's audio data may be obtained from 130a, and the target voice converted from the second voice may be output through the speaker 130b of the voice input/output unit 130.

In this case, the electronic device 100 may include the display 140 to obtain a user command for selecting a UI for a target voice tone desired by the user.

In addition, as shown in FIG. 4, the electronic device 100 of the present invention obtains audio data through the user terminal 200 by communicating with the user terminal 200, including the communication unit 150, and 2 Outputs the target voice converted from the voice.

At this time, as shown in FIG. 5 , a user command for selecting a UI for a desired tone of a target voice may be obtained through the user terminal 200 .

At this time, the user terminal is a smartphone, a tablet personal computer (tablet PC), a mobile phone (mobile phone), a video phone, an e-book reader, a desktop PC (desktop PC), a laptop PC (laptop) PC), netbook computer, workstation, server, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, camera, or wearable device, artificial It may include at least one of an AI speaker.

As shown in FIG. 5, in obtaining a plurality of feature vectors, the obtained audio data is subjected to noise filtering by the processor 120, and the processor 120 converts the filtered audio data into Sectioning into a plurality of sections, framing by overlapping adjacent sections by a certain amount of time, extracting a plurality of acoustic feature vectors and a plurality of MFCC (Mel Frequency Cepstral Coefficients) feature vectors from the framed audio data, , the plurality of feature vectors include a plurality of acoustic feature vectors and a plurality of MFCC feature vectors.

At this time, the plurality of acoustic feature vectors are at least among meanF0Hz, stdevF0Hz, meanF1Hz, stdevF1Hz, meanF2Hz, stdevF2Hz, HNR, localjitter, localabsolutejitter, rapjitter, ppq5jitter, ddpjitter, localShimmer, localdbShimmer, apq3Shimmer, apq5Shimmer, apq11Shimmer, ddaShimmer, pitch_max, and pich_min. can be extracted by processor 120 from framed audio data, including one.

At this time, the plurality of MFCC feature vectors include at least one first MFCC feature vector, at least one second MFCC feature vector, and at least one third MFCC feature vector.

The first MFCC feature vector is extracted as a maximum of 13 vectors by applying the MFCC technique to the framed audio data by the processor 120.

The second MFCC feature vector is extracted by differentiating the value of at least one first MFCC feature vector by the processor 120, and the third MFCC feature vector is extracted by the processor 120 at least one second MFCC It can be extracted by differentiating the value of the feature vector.

Accordingly, the processor 120 has a maximum of 20 acoustic feature vectors and a maximum of 78 MFCC feature vectors (for each frame, the maximum number of first MFCC feature vectors is 13, the maximum number due to one-time differentiation is 13, After obtaining the maximum number of 　13, 　total 13*3=39, which is the maximum number due to 3 derivatives, and then calculating the average and standard deviation of each of them for all frames, use them as a feature vector. Up to 98 feature vectors including 3*2=78 MFCC feature vectors can be extracted from audio data.

The processor 120 classifies the collected audio data of the user's voice into training data and prediction data at a certain ratio (eg, 8 for learning: 2 for prediction), and trains the artificial intelligence model 111 through the training data. , It is possible to determine the accuracy of the artificial intelligence model 111 based on the output classification result by inputting prediction data to the learned artificial intelligence model 111.

Accordingly, learning may be performed with the accuracy of the artificial intelligence model 111 for converting the voice with dysarthria into the target voice as a target of 99% or more.

Meanwhile, the artificial intelligence model 111 may include a target speech generation model.

The target speech generation model includes an encoder that receives a source speech signal that is a user speech signal, text that is an utterance target of a user speech book, and a target speech signal for a target speech converted into a user speech.

In addition, the target speech generation model is an attention-based spectrogram decoder and an automatic speech recognition (ASR) decoder (additionally used decoder, and the learning result of the target speech generation model through the process of pre-layering the text of the output target speech signal). , attention-based LSTM), which uses an attention mechanism between the encoder and decoder to allow the model to learn the relationship between the text and the spectrogram.

Accordingly, the target speech generation model converts the spectrogram output from the spectrogram decoder through a speech synthesizer based on the Griffin-Lim algorithm (GLA, a technology for restoring a linear spectrogram into speech by inferring lost phase information), A voice signal can be generated.

Meanwhile, in the modeling algorithm, an end-to-end sequence-to-sequence model structure may be applied. In this case, the input data is the user's voice signal, and the output data is a target voice signal converted from the voice signal.

In the learning process, data for each of a source voice signal that is a user voice signal, a text that is an utterance target of a user voice, and a target voice signal for a target voice converted into a user voice is required.

As an example of application of the end-to-end sequence-to-sequence model structure, one encoder, one decoder, and one vocoder constituting the neural network are applied, and at this time, the decoder is an attention-based decoder.

According to this, the encoder generates a voice signal as a hidden feature vector for each frame, the decoder outputs a spectrogram of the voice signal based on the data output from the encoder, and the vocoder generates a wave in the time domain based on the output of the decoder. signal can be output.

Specifically, the encoder may perform the following process in receiving a voice signal sampled in units of 16 kHz from the processor 120 and outputting a 512-dimentional encoder represntation.

The audio signal sampled at 16 kHz has 80 dimensional log-mel spectrogram features over a range of 125-7600 Hz, 2 convolutional layers with ReLU activations (downsampling by 4), one bidirectional convolutional LSTM layers convolving only in the frequency domain, And 512-dimentional encoder represntation, which is output data, can be output by sequentially passing through three bidirectional LSTM layers.

Then, the decoder receives the output of the encoder and passes it through an autoregressive RNN that predicts the spectrogram one frame at a time, thereby outputting a target spectrogram, that is, a target speech signal. It is a diagram of a result of a voice improvement conversion experiment according to an embodiment.

In response to input of a patient's voice uttering a text (eg, phone, zoo, car, peanut, mother, ear, tweet, hospital, etc.) to the processor 120, a similar pronunciation or a target voice with the same pronunciation through google TTS It was possible to obtain, and it was proved that the train loss decreased according to the number of learning times of the artificial intelligence model 111 by the processor 120.

7 is an exemplary view of real life use through the user terminal 200 according to this embodiment of the present invention.

As an example, when the patient's guardian requests a sentence list according to the situation through the AI speaker, the sentence list according to the situation is generated from the AI speaker.

At this time, the AI speaker may apply the target voice having the patient's tone to the sentence list through the artificial intelligence model 111 learned for the patient's first voice and transmit it to the patient's terminal.

When the patient communicates according to the situation, the patient's terminal can output the target voice at high speed without a separate conversion operation based on the patient's second voice and the sentence list.

Meanwhile, as an embodiment, the processor 120 determines the pronunciation accuracy of the second voice for the text that is the speech target of the second voice, and when the pronunciation accuracy is greater than or equal to a threshold value, outputs the second voice as the target voice, and pronounces the accuracy of the second voice. When is less than the threshold value, a target voice obtained by converting the user's second voice may be obtained based on the output of the artificial intelligence model 111 .

For example, the processor 120 may assign a high pronunciation accuracy to a text that can be communicated without converting a target voice, and assign a low pronunciation accuracy to a text that does not, with respect to voices of a plurality of texts of one user. . For example, pronunciation accuracy may be identified as low as the difference between the resultant text obtained according to the speech recognition result and the text intended to be uttered by the user increases. Here, the voice recognition may be performed by at least one second voice recognition model trained to recognize a normal person's voice and convert text.

As a result, when the pronunciation accuracy of one text is equal to or higher than the threshold value, the processor 120 determines that the text is communicable without converting the target voice, and outputs the user's voice for the corresponding text without conversion.

On the other hand, when the pronunciation accuracy of one text is less than the threshold value, the processor 120 converts the user's voice for the corresponding text into a target voice and outputs it.

To this end, the processor 120 may obtain a user's voice for each of a plurality of texts, perform voice recognition, and match and store pronunciation accuracy for each of a plurality of texts.

On the other hand, the processor 120, among the plurality of texts, if the pronunciation accuracy of the user's newly input voice, which utters the first text with previously matched pronunciation accuracy less than the threshold value, is greater than or equal to the threshold value, the user who utters the first text If the first test voice, which is a voice, is acquired a certain number of times, and the average pronunciation accuracy of the first test voice acquired a certain number of times is greater than or equal to a threshold value, the first test voice is determined based on the average pronunciation accuracy of the first test voice. Pronunciation accuracy matched to can be updated.

For example, it is assumed that the pronunciation accuracy of the user's first text is 40% and the threshold is 50%.

When a voice in which the user utters the first text is newly input to the processor 120, if the processor 120 calculates that the pronunciation accuracy of the first text of the corresponding voice is 60% higher than the threshold value, the processor ( 120) may perform a test to obtain a specific numerical value of pronunciation accuracy higher than a threshold in the voice of the user who utters the first text.

Specifically, the processor 120 acquires the first test voice, which is the voice of the user who utters the first text, a predetermined number of times (eg, 5 times), and each of the five first test voices has a corresponding value for the first text. Pronunciation accuracy can be calculated.

At this time, when the pronunciation accuracy of each of the five first test voices with respect to the first text is 48%, 55%, 52%, 63%, and 72%, the processor 120 determines that each of the five first test voices Pronunciation accuracy for the first text may be updated by calculating 58%, which is an average value of pronunciation accuracy for one text, and matching and storing the average value with the first text.

On the other hand, the processor 120, among the plurality of texts, when the pronunciation accuracy of the user's newly input voice, which utters the second text with previously matched pronunciation accuracy equal to or greater than the threshold value, is less than the threshold value, the processor 120 determines the accuracy of the user uttering the second text. If the second test voice, which is a voice, is acquired a certain number of times, and the average pronunciation accuracy of the test voice obtained a certain number of times is less than a threshold value, the second test voice is matched to the second text based on the average pronunciation accuracy of the second test voice. pronunciation accuracy can be updated.

For example, it is assumed that the pronunciation accuracy of the user's second text is 60% and the threshold is 50%.

When a voice in which the user utters the second text is newly input to the processor 120, if the processor 120 calculates that the pronunciation accuracy of the second text of the corresponding voice is 32% lower than the threshold value, the processor ( 120) may perform a test to obtain a specific numerical value of pronunciation accuracy that is lower than a threshold value in the voice of the user who utters the second text.

Specifically, the processor 120 acquires second test voices a predetermined number of times (eg, 5 times), which is the voice of a user who utters the second text, and each of the five second test voices has a corresponding value for the second text. Pronunciation accuracy can be calculated.

At this time, when the pronunciation accuracy of each of the five second test voices with respect to the second text is 44%, 55%, 48%, 33%, and 28%, the processor 120 determines that each of the five second test voices is the second text. Pronunciation accuracy for the second text may be updated by calculating an average pronunciation accuracy of 41.6% for the two texts, matching the average value with the second text, and storing the result.

In one embodiment, when a plurality of texts are acquired for the user's voice, the processor 120 identifies pronunciation accuracy for each text and determines whether to generate a target voice for each text based on the pronunciation accuracy. can

Specifically, the processor 120 acquires first subtext and second subtext constituting text that is a speech target of the second voice, and each of the first subtext and the second subtext among the audio data of the second voice Identifies the matched audio segment.

Then, the processor 120 identifies pronunciation accuracy of the first audio section for the first subtext and pronunciation accuracy of the second audio section for the second subtext.

At this time, if the pronunciation accuracy of the first subtext is less than the threshold and the pronunciation accuracy of the second subtext is greater than or equal to the threshold, a target voice for the first subtext is obtained based on the output of the artificial intelligence model 111; , based on the audio data including the target voice for the first subtext and the second audio section, a voice converted from the second voice may be output.

On the other hand, as shown in FIGS. 5, 7, and 9, the text may be a monosyllable word/language, a monopthong, and a word, and in various embodiments, a phoneme, a sentence (sentence), morpheme (morpheme), and the like.

Meanwhile, in learning the artificial intelligence model 111 for controlling the deep learning-based dysarthria voice improvement conversion system according to another aspect of the present invention, the server 300 utters one text from one user terminal 200. The artificial intelligence model obtains audio data of the first voice of the user who speaks text, obtains audio data of normal voice uttering text, and outputs a target voice based on the audio data of the first voice and the audio data of normal voice. (111) is trained.

On the other hand, when the user's second voice is received from the user terminal 200 (S210), the server 300 inputs the audio data of the received second voice to the artificial intelligence model 111, and the artificial intelligence model 111 A target voice obtained by converting the second voice may be acquired from (S220).

At this time, the server 300 outputs the target voice converted from the second voice through the user terminal 200 (S230).

In addition, in the server 300 performing step S220, the server 300 includes a plurality of first feature vectors extracted from the audio data of the first voice and a plurality of first feature vectors extracted from the audio data of the second voice. By comparing the two feature vectors to identify a text that is a speech target of the second voice, a target voice obtained by converting the second voice to be matched with the identified text may be obtained.

In this case, the plurality of feature vectors may include a plurality of acoustic feature vectors and a plurality of Mel Frequency Cepstral Coefficients (MFCC) feature vectors extracted from one audio data.

In addition, when the server 300 trains the artificial intelligence model 111 to output the target voice, the server 300 obtains the frequency band of the audio data for the first voice as user standard data, and the user standard data Based on the frequency band of , a target voice obtained by converting the user's second voice may be obtained.

As an embodiment, the processor identifies the type of dysarthria of the user, classifies difficult-to-pronounce text according to the type of dysarthria, obtains a first voice for the text, and based on the extracted audio data, the artificial intelligence model (111 ) can be learned.

To this end, the artificial intelligence model 111 according to one or more aspects of the present invention classifies the selected feature vectors according to at least one type of dysarthria, and performs clustering according to the type of dysarthria. A separate first model and a second model for classifying the type may be further included.

Specifically, the first model selects at least one feature vector from among a plurality of input feature vectors, and the processor 120 inputs the selected feature vector to the second model.

At this time, the first model is ica (: independent component analysis, independent component analysis), pca (: principal component analysis, principal component analysis), rp (: random projection, random projection), and dae (: deep autoencoder, deep auto- At least one feature vector may be selected by at least one of methods including an encoder).

The second model can determine the type of dysarthria by classifying the input feature vector, svm (: support vector machine, support vector machine), rf (: random (decision) forest, random forest), mlp (: multilayer perceptron , multi-layer perceptron) and cnn (: convolutional deep neural networks), the feature vector may be classified by at least one of classification methods.

The feature vector selected by the first model may be set according to a result of comparing audio data of a patient corresponding to at least one type of dysarthria with audio data of a normal person.

For example, when feature vectors of audio data of a normal person and feature vectors of audio data of a patient with dysarthria are compared, a feature vector having a difference greater than or equal to a threshold value may be a feature vector selected by the first model.

As a specific example, an average value of first feature vectors may be calculated for the audio data of a plurality of normal persons, and an average value of the first feature vectors may be calculated for the audio data of a plurality of patients. When is greater than a certain value, the first feature vector may be set as a feature vector selected by the first model.

Meanwhile, according to an embodiment, information on feature vectors of different groups may be stored according to the type of dysarthria.

For example, the feature vectors of the first group may be set to match for the first type of dysarthria, and the feature vectors of the second group may be set to match for the second type of dysarthria.

Specifically, the processor 120 may identify, for each type of dysarthria, a feature vector having a difference from a normal person greater than or equal to a threshold value, and store the corresponding feature vector as a group matched to the type of dysarthria.

Here, for a plurality of (feature vector) groups matched to each type of dysarthria, the order of being input to the second model may be set, and the order may be set according to the frequency of occurrence of each type of dysarthria.

For example, it is assumed that the frequency of occurrence is high in the order of the first dysarthria type, the second dysarthria type, and the third dysarthria type.

In this case, the processor 120 first controls the first model to extract a first feature vector group matching the first dysarthria type from the user's audio data, and inputs the extracted first feature vector group to the second model. can do. As a result, whether the user corresponds to the first type of dysarthria may be output.

Next, the processor 120 inputs the second feature vector group, which is the next order of the first feature vector group, as a second model, and as a result, whether the user corresponds to the second dysarthria type can be output, The processor 120 inputs the third feature vector group in the next order to the second model, and as a result, whether the user corresponds to the third dysarthria type is output.

As an additional embodiment, the processor 120 may flexibly change the order in which each feature vector group is input to the second model according to the association between speech disorder types.

Specifically, the processor 120 may identify the frequency/probability of simultaneous occurrence of two or more types of dysarthria based on the occurrence history of each type of dysarthria. In addition, the processor 120 may set whether to match between speech disorder types according to the identified frequency/probability.

For example, if the frequency of simultaneous occurrence of the first type of dysarthria and the third type of dysarthria is equal to or greater than a certain number, the processor 120 may set the first type of dysarthria and the third type of dysarthria to match each other.

In this regard, as an example, it is assumed that the first feature vector group, the second feature vector group, and the third feature vector group are input to the second model in order.

In this case, in general, the processor 120 inputs the first feature vector group as a second model, outputs whether the user corresponds to the first dysarthmic disorder type, and then, the next order of the first feature vector group The second feature vector group is input as the second model, and whether the user corresponds to the second dysarthria type is output.

However, if it is determined that the user has the first dysarthria type as a result of the input of the first feature vector group, the processor 120 replaces the second feature vector group, which is next to the first feature vector group, with the third feature vector group. can be input as the second model.

Specifically, when the user is determined to be the first dysarthria type, the processor 120 identifies a third dysarthria type that is a dysarthria type matched to the first dysarthria type, and matches the third dysarthria type to the third dysarthria type. The third feature vector group is input as the second model.

As a result, it is possible to output whether the user corresponds to the third dysarthria type.

Thereafter, if it is determined that the user is not the third dysarthria type, or if the user is determined to be the third dysarthria type, but the dysarthria type matched to the third dysarthria type does not exist, the processor 120 determines the third dysarthria type. By inputting the second feature vector group, which is the next sequential number of the first feature vector group, as the second model, it is possible to output whether the user corresponds to the second dysarthria type.

On the other hand, if the user is determined to be the third dysarthria type and the fourth dysarthria type is identified as the dysarthria type matched to the third dysarthria type, the processor 120 may perform the first dysarthria matched to the fourth dysarthria type. A group of 4 feature vectors may be first input into the second model.

Additionally, the first model may select at least one feature vector from among the plurality of acoustic feature vectors and the plurality of MFCC feature vectors extracted from the audio data, according to the text.

Specifically, the first model selects at least one feature vector according to the text, and the second model compares audio data of a voice of a normal person uttering the text with the audio data based on the selected feature vector.

To this end, audio data is matched with text and input to the processor 120, and the processor 120 pre-processes the audio data to input a plurality of feature vectors together with the text into the first model.

On the other hand, if the user's actual dysarthria type does not match the type of dysarthria output from the artificial intelligence model 111, the processor 120 updates the number of occurrences of errors matched to the text, and the number of occurrences of errors is a certain value. If , we can update the first model to select a different feature vector for the text.

At this time, if the speech impairment type obtained by selection through the updated first model and classification by the existing second model does not actually match the speech impairment type, the processor 120 classifies the text in a different method. The second model can be updated to

For example, as an existing dysarthria classification method, the first model according to the present invention applies ica to select 40 feature vectors (primary selection feature vectors) among 98 feature vectors, and the second model is mlp (primary selection feature vector). When classification is performed by applying the classification method), while the user's dysarthria type for the text "day bird" is Parkinson's, if it is output to the larynx by the second model, the processor 120 matches the text "day bird" The first model may be updated to select 40 new feature vectors (secondary selected feature vectors) from the remaining 58 feature vectors excluding the first selected feature vectors among the 98 feature vectors.

In this case, the artificial intelligence model 111, when a plurality of feature vectors matched to the audio data for the text “day bird” obtained after the update is input, automatically selects the pre-matched secondary selection feature vector to perform classification. By doing this, it is possible to increase the accuracy of speech disorder classification for the text “day bird”.

At this time, if the output from the second model is not Parkinson's despite the update of the first model for the text "day bird", the processor 120 may select a classification method of the second model as a method other than the primary classification method. And, by applying different methods to obtain a method (secondary classification method) classified as laryngeal type, and updating the classification method of the second model matched to the text "day bird" to the second classification method, the text "day bird" It can improve the accuracy of the classification of dysarthria.

Also, when the number of occurrences of errors in text exceeds a predetermined value, the processor 120 may update the first model to change the number of feature vectors selected for one text.

For example, even after each update of the first model and the second model, if the speech impairment classification result of the second model for the text "daytime" is a different type from the user's speech impairment type, a total of three errors have occurred, so the processor (120) may update the number of occurrences of errors matched with the text "daytime" to three times.

At this time, with respect to the number of errors exceeding 2 times, the processor 120 determines that a certain value (2 times) has been exceeded, and selects a feature vector of the first model for the audio data that utters the text “daytime”. You can update the number to fluctuate to 20.

Meanwhile, the present invention may include a computer program stored in a computer-readable recording medium to be combined with a computer, which is hardware, to perform the method according to another aspect of the present invention.

8 is a server configuration diagram according to an embodiment of the present invention.

As shown, the server 300 may include a memory 310 , a communication unit 320 and a processor 330 .

The memory 310 may store various programs and data necessary for the operation of the server 300 . The memory 310 may be implemented as a non-volatile memory 310, a volatile memory 310, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD).

The communication unit 320 may communicate with an external device. In particular, the communication unit 320 may include various communication chips such as a Wi-Fi chip, a Bluetooth chip, a wireless communication chip, an NFC chip, and a Bluetooth Low Energy (BLE) chip. At this time, the Wi-Fi chip, the Bluetooth chip, and the NFC chip perform communication in a LAN method, a WiFi method, a Bluetooth method, and an NFC method, respectively. In the case of using a Wi-Fi chip or a Bluetooth chip, various connection information such as an SSID and a session key is first transmitted and received, and various information can be transmitted and received after communication is connected using this. The wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, ZigBee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and 5th Generation (5G).

The processor 330 may control overall operations of the server 300 using various programs stored in the memory 310 . The processor 330 may include a RAM, a ROM, a graphic processing unit, a main CPU, first through n interfaces, and a bus. At this time, the RAM, ROM, graphic processing unit, main CPU, first to n interfaces, etc. may be connected to each other through a bus.

RAM stores O/S and application programs. Specifically, when the server 300 is booted, O/S is stored in RAM, and various application data selected by the user may be stored in RAM.

The ROM stores instruction sets for system booting. When a turn-on command is input and power is supplied, the main CPU copies the O/S stored in the memory 310 to the RAM according to the command stored in the ROM, and executes the O/S to boot the system. When booting is completed, the main CPU copies various application programs stored in the memory 310 to RAM, and executes the application programs copied to RAM to perform various operations.

The graphic processing unit uses a calculation unit (not shown) and a rendering unit (not shown) to create a screen including various objects such as items, images, and text. Here, the calculation unit may be configured to calculate attribute values such as coordinate values, shape, size, color, etc. of each object to be displayed according to the layout of the screen by using a control command received from the input unit. And, the rendering unit may be configured to generate screens of various layouts including objects based on the attribute values calculated by the calculation unit. The screen created by the rendering unit may be displayed within the display area of the display.

The main CPU accesses the memory 310 and performs booting using the OS stored in the memory 310 . And, the main CPU performs various operations using various programs, contents, data, etc. stored in the memory 310 .

The first through n interfaces are connected to the various components described above. One of the first through n interfaces may be a network interface connected to an external device through a network.

On the other hand, further, the processor 330 may control the artificial intelligence model. In this case, of course, the processor 330 may include a graphics-only processor (eg, GPU) for controlling the artificial intelligence model.

Meanwhile, the artificial intelligence model according to the present invention may be a model based on supervised learning or unsupervised learning. Furthermore, the artificial intelligence model according to the present invention may include a support vector machine (SVM), a decision tree, a neural network, and the like, and methodologies to which they are applied.

As an embodiment, the artificial intelligence model according to the present invention may be an artificial intelligence model based on convolutional deep neural networks (CNN) learned by inputting training data. However, it is not limited thereto, and it goes without saying that various artificial intelligence models can be applied to the present invention. For example, models such as a deep neural network (DNN), a recurrent neural network (RNN), and a bidirectional recurrent deep neural network (BRDNN) may be used as an artificial intelligence model, but are not limited thereto.

At this time, convolutional deep neural networks (CNNs) are a type of multilayer perceptrons designed to use a minimum of preprocessing. A convolutional neural network consists of one or several convolutional layers and general artificial neural network layers placed on top of them, and additionally utilizes weights and pooling layers. Thanks to this structure, convolutional neural networks can fully utilize input data with a two-dimensional structure. Also, convolutional neural networks can be trained via standard back-propagation. Convolutional neural networks are easier to train than other feedforward artificial neural network techniques and have the advantage of using fewer parameters.

In addition, deep neural networks (DNNs) are artificial neural networks (ANNs) consisting of a plurality of hidden layers between an input layer and an output layer.

At this time, the structure of the deep neural network may be composed of a perceptron. Perceptron consists of several inputs, one processor, and one output value. The processor multiplies each of a plurality of input values by a weight, and then sums all the input values multiplied by the weight. Then, the processor substitutes the summed value into the activation function and outputs one output value. If a specific value is desired as an output value of the activation function, a weight value multiplied with each input value may be modified, and an output value may be recalculated using the corrected weight value. At this time, each perceptron may use a different activation function. In addition, each perceptron accepts the outputs passed from the previous layer as inputs, and then obtains the outputs using the activation function. The obtained output is passed as input to the next layer. Through the process as described above, several output values can finally be obtained.

Recurrent Neural Network (RNN) refers to a neural network in which the connections between units constituting an artificial neural network constitute a directed cycle. Unlike feed-forward neural networks, recurrent neural networks can utilize the memory inside the neural network to process arbitrary inputs.

Deep Belief Networks (DBN) is a generative graphical model used in machine learning. In deep learning, it means a deep neural network consisting of multiple layers of latent variables. There are connections between layers, but there is no connection between units within a layer.

Due to the characteristics of a generative model, the deep trust neural network can be used for prior learning, and after learning initial weights through prior learning, the weights can be fine-tuned through backpropagation or other discrimination algorithms. This characteristic is very useful when there is little training data, because the smaller the training data, the stronger the influence of the initial weight values on the resulting model. The pre-learned initial value of the weight becomes closer to the optimal weight than the arbitrarily set initial value of the weight, which enables the performance and speed of the fine-tuning step to be improved.

The above-described artificial intelligence and its learning method are described for illustrative purposes, and the artificial intelligence and its learning method used in the above-described embodiments are not limited. For example, all kinds of artificial intelligence technologies and learning methods that can be applied by a person skilled in the art to solve the same problem can be used to implement the system according to the disclosed embodiment.

Meanwhile, the processor 330 may include one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals to and from other components. can

Processor 330 according to one embodiment performs the method described in connection with the present invention by executing one or more instructions stored in memory 310 .

For example, the processor 330 acquires new training data by executing one or more instructions stored in the memory 310, performs a test on the acquired new training data using a learned model, and performs the test. As a result, first training data for which the labeled information is obtained with an accuracy equal to or higher than a predetermined first reference value is extracted, the extracted first training data is deleted from the new training data, and the new training data from which the extracted training data is deleted The learned model may be retrained using the training data.

Meanwhile, the processor 330 includes RAM (Random Access Memory, not shown) and ROM (Read-Only Memory) temporarily and/or permanently storing signals (or data) processed in the processor 330. , not shown) may be further included. In addition, the processor 330 may be implemented in the form of a system on chip (SoC) including at least one of a graphics processing unit, RAM, and ROM.

The memory 310 may store programs (one or more instructions) for processing and control of the processor 330 . Programs stored in the memory 310 may be divided into a plurality of modules according to functions.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Components of the present invention may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. Components of the present invention may be implemented as software programming or software elements, and similarly, embodiments may include various algorithms implemented as data structures, processes, routines, or combinations of other programming constructs, such as C, C++ , Java (Java), can be implemented in a programming or scripting language such as assembler (assembler). Functional aspects may be implemented in an algorithm running on one or more processors.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: electronic device
110: memory
111: AI model
120: processor
130: voice input/output unit
130a: microphone
130b: speaker
140: display
150: Ministry of Communication
200: user terminal
300: server
310: memory
320: communication department
330: processor

Claims (12)

In electronic devices,
a memory in which an artificial intelligence model for generating a target voice is stored; and
Including; a processor connected to the memory;
the processor,
training the artificial intelligence model based on audio data of a user's first voice uttering a text and audio data of a normal voice uttering the text;
When the user's second voice is input, audio data of the input second voice is input to the artificial intelligence model, and a target voice obtained by converting the user's second voice is obtained based on the output of the artificial intelligence model. do,
Determining pronunciation accuracy of a second voice for a text that is a speech target of the second voice;
When the pronunciation accuracy of the second voice is equal to or greater than a threshold value, outputting the second voice as a target voice;
When the pronunciation accuracy of the second voice is less than a threshold value, obtaining a target voice converted from the second voice of the user based on an output of the artificial intelligence model;
obtaining a first subtext and a second subtext constituting text that is a speech target of the second voice;
Among the audio data of the second voice, an audio section in which the first subtext and the second subtext are matched is identified;
identify pronunciation accuracy of a first audio interval for the first subtext and pronunciation accuracy of a second audio interval for the second subtext;
If the pronunciation accuracy of the first subtext is less than the threshold and the pronunciation accuracy of the second subtext is greater than or equal to the threshold, the first subtext is based on the output of the artificial intelligence model to which the first subtext is input. Obtaining a target voice for
Based on audio data including a target voice for the first subtext and the second audio section, outputting a converted voice of the second voice;
Obtaining the user's voice for each of a plurality of texts, performing voice recognition to match and store pronunciation accuracy for each of the plurality of texts,
Among the plurality of texts, when the pronunciation accuracy of the user's newly input voice, which utters the first text with previously matched pronunciation accuracy less than the threshold value, is greater than or equal to the threshold value, the first text is the voice of the user who utters the first text. Acquire a certain number of test voices,
If the average of the pronunciation accuracies of the first test voice obtained the predetermined number of times is equal to or greater than a threshold, updating the pronunciation accuracy matched to the first text based on the average of the pronunciation accuracies of the first test voice;
Among the plurality of texts, when the pronunciation accuracy of the user's newly input voice, which utters the second text with previously matched pronunciation accuracy equal to or greater than the threshold value, is less than the threshold value, the second text is the voice of the user who utters the second text. Acquire a certain number of test voices,
If the average of the pronunciation accuracies of the test voices obtained the predetermined number of times is less than a threshold, updating the pronunciation accuracy matched to the second text based on the average of the pronunciation accuracies of the second test voice. According to claim 1,
The artificial intelligence model,
A plurality of first feature vectors extracted from the audio data of the first voice are compared with a plurality of second feature vectors extracted from the audio data of the second voice to identify text that is a speech target of the second voice; ,
Acquiring a target voice converted from the second voice to match the identified text. According to claim 1,
the processor,
Obtaining a frequency band of audio data for the first voice as user standard data;
Acquiring a target voice converted from the second voice of the user based on the frequency band of the user standard data. delete delete delete According to claim 1,
the processor,
noise filtering the audio data;
Sectioning the filtered audio data into a plurality of sections, overlapping adjacent sections by a predetermined time and framing;
The electronic device characterized in that a plurality of acoustic feature vectors and a plurality of Mel Frequency Cepstral Coefficients (MFCC) feature vectors are extracted from the framed audio data. According to claim 7,
the processor,
Obtaining at least one first MFCC feature vector by applying an MFCC technique to the framed audio data;
Obtaining at least one second MFCC feature vector by differentiating a value of the at least one first MFCC feature vector;
Obtaining at least one third MFCC feature vector by differentiating a value of the at least one second MFCC feature vector;
The plurality of MFCC feature vectors,
The electronic device characterized in that it comprises the at least one first MFCC feature vector, the at least one second MFCC feature vector, and the at least one third MFCC feature vector. delete delete delete delete