KR20230087872A - Deep learning-based sound impairment classification apparatus, system control method, and computer program - Google Patents

Abstract

The present invention provides a memory in which an artificial intelligence model for determining the type of dysarthria of a patient is stored, and audio data corresponding to a user's voice uttering a text is obtained by connecting to the memory, and the text and audio data are converted into an artificial intelligence model. Provided is an electronic device including a processor for inputting to and identifying a dysarthria type matched to audio data.

Description

DEEP LEARNING-BASED SOUND IMPAIRMENT CLASSIFICATION APPARATUS, SYSTEM CONTROL METHOD, AND COMPUTER PROGRAM}

The present invention relates to a deep learning-based speech impairment classification technology, and more particularly, to an apparatus and system for learning an artificial intelligence model through deep learning to determine the type of speech impairment of a user when a user's voice is input. It is a technology that provides control methods and computer programs.

The conventional dysarthria evaluation was conducted through online or offline language proficiency tests and filling out questionnaires, which were answered through self-assessment of the user and could not be objectified, resulting in inability to derive accurate types of dysarthria. there has been

In addition, as disclosed in Patent Document 1, there are time constraints in performing a language proficiency test and filling out a questionnaire according to a complicated procedure, and a problem has arisen in which a user is tired and gives up the speech impairment evaluation in the middle.

On the other hand, the matters described as the background art above are only for improving understanding of the background of the present invention, and should not be taken as an admission that they correspond to prior art already known to those skilled in the art. will be.

Registered Patent Publication No. 10-1921890, 2018.11.20.

The problem to be solved by the present invention is to provide a speech disorder classification technology based on deep learning that acquires audio data based on the input user's voice and determines the type of dysarthria matched to the audio data through an artificial intelligence model. .

In addition, the present invention is a technology researched from 2021.06.01 to 2021.12.31 with dysarthria voice data (assignment number 21002131305088212100101BS) of the Korea Institute for Intelligence and Information Society Promotion, AI learning data construction project.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

An electronic device according to an aspect of the present invention for solving the above problems includes a memory storing an artificial intelligence model for determining a patient's dysarthria type, and a processor connected to the memory, wherein the processor includes one text Acquisition of audio data corresponding to the voice of the user who utters the audio data, and inputting the audio data to an artificial intelligence model to identify a type of dysarthria matched to the audio data.

In addition, the processor performs noise filtering on the audio data, sections the filtered audio data into a plurality of sections, overlaps adjacent sections by a predetermined time, and frames the audio data from the framed audio data. It is characterized in that an acoustic feature vector and a plurality of MFCC (Mel Frequency Cepstral Coefficients) feature vectors are extracted.

Additionally, the processor obtains at least one first MFCC feature vector by applying the MFCC technique to the framed audio data, and differentiates values of the at least one first MFCC feature vector to obtain at least one second MFCC feature vector. Obtaining, and differentiating the value of the at least one second MFCC feature vector to obtain at least one third MFCC feature vector, wherein the plurality of MFCC feature vectors include at least one first MFCC feature vector and at least one second MFCC It is characterized by including a feature vector and at least one third MFCC feature vector.

At this time, the artificial intelligence model is a first model for selecting at least one feature vector from among a plurality of acoustic feature vectors and a plurality of MFCC feature vectors extracted from audio data, and determining the type of dysarthria based on the selected feature vector It may include a second model that does.

Additionally, the first model may select at least one feature vector according to the text, and the second model may compare audio data of a voice of a normal person uttering the text with the audio data based on the selected feature vector.

In addition, the processor updates the number of error occurrences matched to the text when the user's actual dysarthria type does not match the type of dysarthria output from the artificial intelligence model, and if the number of errors exceeds a certain value, the text The first model can be updated to select a different feature vector for .

In the method for controlling a deep learning-based dysarthria classification system according to another aspect of the present invention, the step of obtaining, by a server, audio data corresponding to a voice of a user uttering a text from a user terminal; The method includes identifying a dysarthria type matched to the user's voice through an artificial intelligence model for determining the type of dysarthria, and providing, by a server, the identified dysarthria type to a user terminal.

In addition, the step of identifying the type of dysarthria that matches the user's voice through the artificial intelligence model for determining the type of dysarthria of the patient is the step of noise filtering the audio data by the server, the filtering by the server Sectioning the audio data into a plurality of sections, but overlapping adjacent sections by a certain amount of time for framing; extracting, by the server, a plurality of acoustic feature vectors from the framed audio data; , obtaining at least one first MFCC feature vector by applying a Mel Frequency Cepstral Coefficients (MFCC) technique to the framed audio data, wherein the server differentiates the value of the at least one first MFCC feature vector to obtain at least one first MFCC feature vector It may include obtaining 2 MFCC feature vectors, and acquiring, by a server, at least one third MFCC feature vector by differentiating values of at least one second MFCC feature vector.

In addition, the step of identifying the dysarthria type matched to the user's voice through the artificial intelligence model for determining the type of dysarthria of the patient includes: a plurality of acoustic feature vectors and at least one first MFCC feature vector, at least Inputting one second MFCC feature vector and at least one third MFCC feature vector to a first model included in an artificial intelligence model, obtaining, by a server, at least one feature vector from the first model, server (a) inputting the obtained feature vector into a second model included in the artificial intelligence model; and acquiring, by a server, a dysarthria type matched to the user's voice from the second model.

At this time, the control method of the classification system according to this aspect of the present invention includes selecting at least one feature vector according to the text by the first model, and uttering the text based on the selected feature vector by the second model. Comparing the audio data of the normal person's voice with the audio data, and if the user's actual dysarthria type does not match the dysarthria type output from the artificial intelligence model, the server updating the number of occurrences of errors matched to the text. , and if the error occurrence count exceeds a predetermined value, the server may update the first model to select another feature vector for the text.

In addition, updating the first model to select another feature vector for the text may include, if the accuracy of the second model is less than a certain value, the server determines, from the first model, at least one value that causes the accuracy to be greater than or equal to the certain value. The method may include selecting a feature vector, and storing, by the server, at least one feature vector having an accuracy greater than or equal to a predetermined value for the text.

On the other hand, it is combined with a computer that is hardware and further includes a computer program stored in a computer-readable recording medium to perform the method of the classification system according to another aspect of the present invention.

Other specific details of the invention are included in the detailed description and drawings.

According to the deep learning-based dysarthria classification technology of the present invention, by determining and providing the type of dysarthria that the user has only with the user's voice input, increasing user satisfaction by providing a dysarthria evaluation service free from time and space constraints. effect occurs.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a configuration diagram of an electronic device according to an embodiment of the present invention.
2 is a flow chart of dysarthria classification according to an embodiment of the present invention.
3 and 4 are system configuration diagrams according to an embodiment of the present invention.
5 is an operational flow diagram of a system for learning and classification according to an embodiment of the present invention.
6 shows the result of an experiment for classifying dysarthria according to an embodiment of the present invention.
7 is a server configuration diagram according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

The term "unit" or "module" used in the specification means a hardware component such as software, FPGA or ASIC, and "unit" or "module" performs certain roles. However, "unit" or "module" is not meant to be limited to software or hardware. A “unit” or “module” may be configured to reside in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, a “unit” or “module” may refer to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and "units" or "modules" may be combined into smaller numbers of components and "units" or "modules" or may be combined into additional components and "units" or "modules". can be further separated.

The spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe a component's correlation with other components. Spatially relative terms should be understood as including different orientations of elements in use or operation in addition to the orientations shown in the drawings. For example, if you flip a component that is shown in a drawing, a component described as "below" or "beneath" another component will be placed "above" the other component. there is. Thus, the exemplary term “below” may include directions of both below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

In this specification, a computer means any kind of hardware device including at least one processor, and may be understood as encompassing a software configuration operating in a corresponding hardware device according to an embodiment. For example, a computer may be understood as including a smartphone, a tablet PC, a desktop computer, a laptop computer, and user clients and applications running on each device, but is not limited thereto.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of an electronic device 100 according to an embodiment of the present invention, and FIG. 2 is a flow chart for classifying dysarthria according to an embodiment of the present invention.

In one embodiment, the electronic device 100 includes a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, and a desktop PC. ), laptop PC, netbook computer, workstation, server, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, camera, or wearable device It may include at least one of (wearable device) and artificial intelligence speaker (AI speaker).

An electronic device 100 for classification of dysarthria based on deep learning according to an aspect of the present invention includes a memory 110 storing an artificial intelligence model 111 for determining the type of dysarthria of a patient, and a memory 110 connected to the memory 110. It includes a processor 120.

Speech disorder refers to a condition in which it is difficult to speak due to abnormalities in the vocal organs due to disorders and/or diseases caused by acquired or congenital factors. disability, speech impairment, and Parkinson's disease.

At this time, the processor 120 obtains audio data corresponding to the voice of the user uttering one text, inputs the text and audio data to the artificial intelligence model 111, and identifies the dysarthria type matched to the audio data. do.

Specifically, as shown in FIG. 2, in order to perform deep learning-based dysarthria classification, the processor 120 obtains audio data corresponding to the voice of a user uttering a text, and the artificial intelligence model 111 It is possible to identify a type of dysarthria that matches the audio data obtained through.

In this case, the audio data is a digitized sound source extracted from the user's voice, and the processor 120 may match and store the audio data with text.

The obtained audio data is subjected to noise filtering by the processor 120, and the processor 120 divides the filtered audio data into a plurality of sections, but overlaps adjacent sections by a certain amount of time to perform framing. (framing), and a plurality of acoustic feature vectors and a plurality of Mel Frequency Cepstral Coefficients (MFCC) feature vectors are extracted from the framed audio data.

At this time, the plurality of acoustic feature vectors are at least among meanF0Hz, stdevF0Hz, meanF1Hz, stdevF1Hz, meanF2Hz, stdevF2Hz, HNR, localjitter, localabsolutejitter, rapjitter, ppq5jitter, ddpjitter, localShimmer, localdbShimmer, apq3Shimmer, apq5Shimmer, apq11Shimmer, ddaShimmer, pitch_max, and pich_min. can be extracted by processor 120 from framed audio data, including one.

At this time, the plurality of MFCC feature vectors include at least one first MFCC feature vector, at least one second MFCC feature vector, and at least one third MFCC feature vector.

The first MFCC feature vector is extracted as a maximum of 13 vectors by applying the MFCC technique to the framed audio data by the processor 120.

The second MFCC feature vector is extracted by differentiating the value of at least one first MFCC feature vector by the processor 120, and the third MFCC feature vector is extracted by the processor 120 at least one second MFCC It can be extracted by differentiating the value of the feature vector.

Accordingly, the processor 120 has a maximum of 20 acoustic feature vectors and a maximum of 78 MFCC feature vectors (for each frame, the maximum number of first MFCC feature vectors is 13, the maximum number due to the first derivative is 13, After obtaining the maximum number of 　13, 　total 13*3=39 due to the 3rd derivative, and then calculating the average and standard deviation of each of them for all frames, use them as a feature vector. Up to 98 feature vectors including 3*2=78 MFCC feature vectors can be extracted from audio data.

3 and 4 are system configuration diagrams according to an embodiment of the present invention, and FIG. 5 is an operational flowchart of a system for learning and classification according to an embodiment of the present invention.

According to the present invention, the first model 111a selects at least one feature vector from among a plurality of input feature vectors, and the processor 120 inputs the selected feature vector to the second model 111b.

At this time, the first model 111a is ica (: independent component analysis, independent component analysis), pca (: principal component analysis, principal component analysis), rp (: random projection, random projection), and dae (: deep autoencoder, At least one feature vector may be selected using at least one of methods including a deep auto-encoder.

The second model 111b can determine the type of dysarthria by classifying the input feature vector, svm (: support vector machine, support vector machine), rf (: random (decision) forest, random forest), mlp ( : The feature vector may be classified by at least one of classification methods including multi-layer perceptron) and cnn (: convolutional deep neural networks).

At this time, the processor 120 classifies the collected data into learning data and prediction data at a certain ratio (ex. learning 8: prediction 2) to train the artificial intelligence model 111 through the learning data, and predictive data It is possible to determine the accuracy of the artificial intelligence model 111 based on the output classification result by inputting to the learned artificial intelligence model 111.

The accuracy of the artificial intelligence model 111 for determining the type of dysarthria may be targeted at 99% or more, and learning may be performed.

Additionally, the feature vector selected by the first model according to the present invention may be set according to a result of comparing audio data of a patient corresponding to at least one type of dysarthria with audio data of a normal person.

For example, when feature vectors of audio data of a normal person and feature vectors of audio data of a patient with dysarthria are compared, a feature vector having a difference greater than or equal to a threshold value may be a feature vector selected by the first model.

As a specific example, an average value of first feature vectors may be calculated for the audio data of a plurality of normal persons, and an average value of the first feature vectors may be calculated for the audio data of a plurality of patients. When is greater than a certain value, the first feature vector may be set as a feature vector selected by the first model.

Meanwhile, according to an embodiment, information on feature vectors of different groups may be stored according to the type of dysarthria.

For example, the feature vectors of the first group may be set to match for the first type of dysarthria, and the feature vectors of the second group may be set to match for the second type of dysarthria.

Specifically, the processor 120 may identify, for each type of dysarthria, a feature vector having a difference from a normal person greater than or equal to a threshold value, and store the corresponding feature vector as a group matched to the type of dysarthria.

Here, for a plurality of (feature vector) groups matched to each dysarthmic disorder type, the order input to the second model 111b may be set, and the order may be set according to the occurrence frequency of each dysarthmic disorder type. there is.

For example, it is assumed that the frequency of occurrence is high in the order of the first dysarthria type, the second dysarthria type, and the third dysarthria type.

In this case, the processor 120 first controls the first model 111a to extract a first feature vector group matching the first dysarthria type from the user's audio data, and converts the extracted first feature vector group into a second feature vector group. It can be input into the model 111b. As a result, whether the user corresponds to the first type of dysarthria may be output.

Next, the processor 120 inputs the second feature vector group, which is the next order of the first feature vector group, to the second model 111b, and as a result, whether the user corresponds to the second dysarthria type is output. And, by the processor 120, the third feature vector group in the next order is input to the second model 111b, and as a result, whether the user corresponds to the third dysarthria type is output.

As an additional embodiment, the processor 120 may flexibly change the order in which each feature vector group is input to the second model 111b according to the association between speech impairment types.

Specifically, the processor 120 may identify the frequency/probability of simultaneous occurrence of two or more types of dysarthria based on the occurrence history of each type of dysarthria. In addition, the processor 120 may set whether to match between speech disorder types according to the identified frequency/probability.

For example, if the frequency of simultaneous occurrence of the first type of dysarthria and the third type of dysarthria is equal to or greater than a certain number, the processor 120 may set the first type of dysarthria and the third type of dysarthria to match each other.

In this regard, as an example, it is assumed that the first feature vector group, the second feature vector group, and the third feature vector group are input to the second model 111b in order.

In this case, in general, the processor 120 inputs the first feature vector group to the second model 111b, outputs whether the user corresponds to the first dysarthmic disorder type, and then, the first feature vector group The second feature vector group, which is the next step, is input to the second model 111b, and whether the user corresponds to the second dysarthria type is output.

However, if it is determined that the user has the first dysarthria type as a result of the input of the first feature vector group, the processor 120 replaces the second feature vector group, which is next to the first feature vector group, with the third feature vector group. may be input as the second model 111b.

Specifically, when the user is determined to be the first dysarthria type, the processor 120 identifies a third dysarthria type that is a dysarthria type matched to the first dysarthria type, and matches the third dysarthria type to the third dysarthria type. The third feature vector group is input to the second model 111b.

As a result, it is possible to output whether the user corresponds to the third dysarthria type.

Thereafter, if it is determined that the user is not the third dysarthria type, or if the user is determined to be the third dysarthria type, but the dysarthria type matched to the third dysarthria type does not exist, the processor 120 determines the third dysarthria type. By inputting the second feature vector group, which is the next sequential number of the first feature vector group, to the second model 111b, it is possible to output whether the user corresponds to the second dysarthria type.

On the other hand, if the user is determined to be the third dysarthria type and the fourth dysarthria type is identified as the dysarthria type matched to the third dysarthria type, the processor 120 may perform the first dysarthria matched to the fourth dysarthria type. A group of 4 feature vectors may be first input to the second model 111b.

In addition, as shown in FIGS. 3 and 4, the artificial intelligence model 111 selects at least one feature vector from among a plurality of acoustic feature vectors and a plurality of MFCC feature vectors extracted from audio data. It may include a first model 111a and a second model 111b for determining the type of dysarthria based on the selected feature vector.

Specifically, the first model 111a selects at least one feature vector according to the text, and the second model 111b converts audio data of the voice of a normal person uttering the text into audio based on the selected feature vector. Compare with data.

To this end, as shown in FIG. 5, audio data (: wav in FIG. 5) is matched with text (: txt in FIG. 5) and input to the processor 120, and the processor 120 preprocesses the audio data to A plurality of feature vectors are input together with text into the first model 111a.

On the other hand, if the user's actual dysarthria type does not match the type of dysarthria output from the artificial intelligence model 111, the processor 120 updates the number of occurrences of errors matched to the text, and the number of occurrences of errors is a certain value. If it exceeds , the first model 111a can be updated to select another feature vector for the text.

At this time, when the dysarthria type obtained by selection through the updated first model 111a and classification by the existing second model 111b does not actually match the dysarthria type, the processor 120 determines the text. The second model 111b can be updated to perform the classification in a different way.

For example, as an existing dysarthria classification method, the first model 111a according to the present invention applies ica to select 40 feature vectors (primary selection feature vectors) among 98 feature vectors, and the second model 111b ) applies mlp (primary classification method) to perform classification, while the user's dysarthria type for the text "daytime" is Parkinson's, if it is output to the larynx by the second model 111b, the processor 120 ) updates the first model 111a to select new 40 feature vectors (secondary selected feature vectors) from the remaining 58 feature vectors excluding the first selected feature vectors among the 98 feature vectors matched with the text “day bird” can do.

In this case, the artificial intelligence model 111, when a plurality of feature vectors matched to the audio data for the text “day bird” obtained after the update is input, automatically selects the pre-matched secondary selection feature vector to perform classification. By doing this, it is possible to increase the accuracy of speech disorder classification for the text “day bird”.

At this time, if the output from the second model 111b is not Parkinson's despite the update of the first model 111a for the text "daytime", the processor 120 sets the classification method of the second model 111b to 1 It is possible to select a method other than the primary classification method, and a method (secondary classification method) classified as an occipital type is obtained by applying the other method, respectively, and the classification method of the second model 111b matched to the text "daytime" By updating with the secondary classification method, it is possible to increase the accuracy of the dysarthmic classification of the text “day bird”.

Also, if the number of occurrences of text errors exceeds a predetermined value, the processor 120 may update the first model 111a to change the number of feature vectors selected for one text.

For example, even after each update of the first model 111a and the second model 111b, if the speech impairment classification result of the second model 111b for the text “day bird” is a different type from the user's dysarthria type, Since a total of 3 errors have occurred, the processor 120 can update the number of occurrences of errors matched with the text “daytime” to 3 times.

At this time, the processor 120 determines that the number of occurrences of the error exceeds 2 times that a certain value (2 times) has been exceeded, and the first model 111a for the audio data uttering the text “daytime”. It can be updated to change the number of feature vector selections to 20.

In this case, as shown in FIG. 6 , when the dysarthria type is Parkinson's, it is allowed to reduce the number of feature vectors as the result for 20 feature vectors is more accurate than the result for 40 feature vectors.

Meanwhile, as shown in FIG. 3 , the electronic device 100 according to one aspect of the present invention may include the microphone 130 to obtain user audio data.

Alternatively, as shown in FIG. 4 , the electronic device 100 may communicate with the user terminal 200 through the communication unit 140 .

At this time, the user terminal 200 is a terminal of a user using the electronic device 100 of the present invention, and in an embodiment, the user terminal 200 includes a smartphone, a tablet personal computer (PC), Mobile phone, video phone, e-book reader, desktop PC, laptop PC, netbook computer, workstation, server, PDA It may include at least one of a personal digital assistant, a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, a wearable device, and an AI speaker.

Accordingly, the electronic device 100 may output the type of dysarthria output through the artificial intelligence model 111 through the user terminal 200 and provide it to the user, and through the microphone included in the user terminal 200. It is also possible to obtain the user's audio data.

6 shows the result of an experiment for classifying dysarthria according to an embodiment of the present invention.

On the other hand, looking at the experimental results according to the selection method and classification method for each type of dysarthria, as shown in FIG. 6, the feature vector selection method and number of selections of the first model 111a (data dimensionality reduction) for one text, and the second model The classification accuracy of Parkinson's patients (PD) and normal people (HC) according to the selected feature vector classification method of (111b) (classification) is quantified.

As shown, 20 feature vectors among 98 feature vectors are selected in the first model 111a by the ica method, and the result of classification by the svm method in the second model 111b shows the highest accuracy.

In addition, as a result of classification using the mlp method of the second model 111b for each of speech, hearing, and laryngeal dysarthria, the accuracy was calculated to be close to or exceed 99%.

Accordingly, it is proved that the deep learning-based dysarthria classification of the present invention is an effective invention.

On the other hand, in the control method of the deep learning-based speech impairment classification system according to the second aspect of the present invention, the speech impairment classification system is configured in the server 300 including the memory 310, the communication unit 320, and the processor 330. In addition, the server 300 may communicate with the user terminal 200 through the communication unit 320 .

Specifically, as shown in FIG. 2 , in order to classify dysarthria based on deep learning, the server 300 obtains audio data corresponding to the voice of a user uttering a text from the user terminal 200 (S210). ), and identifies the type of dysarthria that matches the user's voice through an artificial intelligence model that determines the type of dysarthria of the patient (S220), and provides the identified dysarthria type to the user terminal 200.

At this time, the server 300 may match and store audio data with text.

When the server 300 identifies the type of dysarthria that matches the user's voice through an artificial intelligence model that determines the type of dysarthria of the patient, the server 300 performs noise filtering on the audio data, The audio data is sectioned into a plurality of sections, and framed by overlapping adjacent sections by a predetermined time.

At this time, the server 300 outputs meanF0Hz, stdevF0Hz, meanF1Hz, stdevF1Hz, meanF2Hz, stdevF2Hz, HNR, localjitter, localabsolutejitter, rapjitter, ppq5jitter, ddpjitter, localShimmer, localdbShimmer, apq3Shimmer, apq5Shimmer, apq11Shimmer, ddaShimmer, pitch_max from the framed audio data. And including at least one of pich_min, a plurality of acoustic feature vectors may be extracted.

In addition, the server 300 may extract a plurality of MFCC feature vectors including a first MFCC feature vector, a second MFCC feature vector, and a third MFCC feature vector from the framed audio data.

Specifically, the server 300 may acquire up to 13 first MFCC feature vectors by applying a Mel Frequency Cepstral Coefficients (MFCC) technique to framed audio data.

At this time, the server 300 differentiates the value of the at least one first MFCC feature vector to obtain at least one second MFCC feature vector, and differentiates the value of the at least one second MFCC feature vector to obtain at least one third MFCC feature vectors can be obtained.

Accordingly, the server 300 has a maximum of 20 acoustic feature vectors and a maximum of 78 MFCC feature vectors (for each frame, the maximum number of 1st MFCC feature vectors is 13, the maximum number due to the first derivative is 13, 3 After obtaining the maximum number of 　13, 　total 13*3=39 due to the gray derivative, and then calculating the average and standard deviation of each of them for all frames, use them as a feature vector. *2 = Up to 98 feature vectors including 78 MFCC feature vectors can be extracted from audio data.

Meanwhile, as shown in FIGS. 3 and 4, the artificial intelligence model includes a first model for selecting at least one feature vector from among a plurality of acoustic feature vectors and a plurality of MFCC feature vectors extracted from audio data; and A second model for determining the type of dysarthria based on the selected feature vector may be included.

Accordingly, in the server 300 identifying the type of dysarthria that matches the user's voice through the artificial intelligence model for determining the type of dysarthria of the patient, the server 300 uses a plurality of acoustic feature vectors and a plurality of MFCC features Vectors (: at least one first MFCC feature vector, at least one second MFCC feature vector, and at least one third MFCC feature vector) are input to the first model.

The server 300 may obtain at least one feature vector from the first model, input the acquired feature vector to the second model, and obtain a dysarthria type matching the user's voice from the second model.

Specifically, the server 300 selects at least one feature vector according to the text through the first model, and based on the selected feature vector through the second model, audio data of the voice of a normal person uttering the text. Compare with audio data.

To this end, as shown in FIG. 5, audio data (: wav in FIG. 5) is matched with text (: txt in FIG. 5) and stored in the server 300, and the server 300 preprocesses the audio data to A plurality of feature vectors are input to the first model together with text.

The first model selects at least one feature vector from among a plurality of input feature vectors, and the processor inputs the selected feature vector to the second model.

At this time, the first model is ica (: independent component analysis, independent component analysis), pca (: principal component analysis, principal component analysis), rp (: random projection, random projection), and dae (: deep autoencoder, deep auto- At least one feature vector may be selected using at least one of methods including an encoder).

The second model can determine the type of dysarthria by classifying the input feature vector, svm (: support vector machine, support vector machine), rf (: random (decision) forest, random forest), mlp (: multilayer perceptron , multi-layer perceptron) and cnn (: convolutional deep neural networks), the feature vector may be classified by at least one of classification methods.

At this time, the server 300 classifies the collected data into learning data and prediction data at a certain ratio (ex. learning 8: prediction 2), trains the artificial intelligence model through the learning data, and learns the prediction data. The accuracy of the artificial intelligence model can be judged based on the classification result output by inputting it to the artificial intelligence model.

The accuracy of the artificial intelligence model aims at 99% or more, and the trained artificial intelligence model (disability classification model) is 99% It is possible to determine the type of dysarthria that is abnormally matched.

On the other hand, in the control method of the classification system according to this aspect of the present invention, when the first model selects at least one feature vector according to the text, the second model selects the normal person who utters the text based on the selected feature vector. The audio data of the voice is compared with the audio data, and if the user's actual dysarthria type does not match the dysarthria type output from the artificial intelligence model, the server 300 may update the number of occurrences of errors matched to the text. there is.

In this case, if the number of occurrences of errors exceeds a predetermined value, the server 300 may update the first model to select another feature vector for the text.

Specifically, when the server 300 updates the first model to select a different feature vector for text, when the accuracy of the second model is less than a certain value, the server 300 determines the accuracy from the first model to a certain value. At least one feature vector having an accuracy of at least a certain value may be selected, and at least one feature vector having an accuracy of at least a certain value may be matched with text and stored.

On the other hand, it is combined with a computer that is hardware and further includes a computer program stored in a computer-readable recording medium to perform the method of the classification system according to another aspect of the present invention.

7 is a server configuration diagram according to an embodiment of the present invention.

As shown, the server 300 may include a memory 310 , a communication unit 320 and a processor 330 .

The memory 310 may store various programs and data necessary for the operation of the server 300 . The memory 310 may be implemented as a non-volatile memory 310, a volatile memory 310, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD).

The communication unit 320 may communicate with an external device. In particular, the communication unit 320 may include various communication chips such as a Wi-Fi chip, a Bluetooth chip, a wireless communication chip, an NFC chip, and a Bluetooth Low Energy (BLE) chip. At this time, the Wi-Fi chip, the Bluetooth chip, and the NFC chip perform communication in a LAN method, a WiFi method, a Bluetooth method, and an NFC method, respectively. In the case of using a Wi-Fi chip or a Bluetooth chip, various connection information such as an SSID and a session key is first transmitted and received, and various information can be transmitted and received after communication is connected using this. The wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, ZigBee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE).

The processor 330 may control overall operations of the server 300 using various programs stored in the memory 310 . The processor 330 may include a RAM, a ROM, a graphic processing unit, a main CPU, first through n interfaces, and a bus. At this time, the RAM, ROM, graphic processing unit, main CPU, first to n interfaces, etc. may be connected to each other through a bus.

RAM stores O/S and application programs. Specifically, when the server 300 is booted, O/S is stored in RAM, and various application data selected by the user may be stored in RAM.

The ROM stores instruction sets for system booting. When a turn-on command is input and power is supplied, the main CPU copies the O/S stored in the memory 310 to the RAM according to the command stored in the ROM, and executes the O/S to boot the system. When booting is completed, the main CPU copies various application programs stored in the memory 310 to RAM, and executes the application programs copied to RAM to perform various operations.

The graphic processing unit uses a calculation unit (not shown) and a rendering unit (not shown) to create a screen including various objects such as items, images, and text. Here, the calculation unit may be configured to calculate attribute values such as coordinate values, shape, size, color, etc. of each object to be displayed according to the layout of the screen by using a control command received from the input unit. And, the rendering unit may be configured to generate screens of various layouts including objects based on the attribute values calculated by the calculation unit. The screen created by the rendering unit may be displayed within the display area of the display.

The main CPU accesses the memory 310 and performs booting using the OS stored in the memory 310 . And, the main CPU performs various operations using various programs, contents, data, etc. stored in the memory 310 .

The first through n interfaces are connected to the various components described above. One of the first through n interfaces may be a network interface connected to an external device through a network.

On the other hand, further, the processor 330 may control the artificial intelligence model. In this case, of course, the processor 330 may include a graphics-only processor (eg, GPU) for controlling the artificial intelligence model.

Meanwhile, the artificial intelligence model according to the present invention may be a model based on supervised learning or unsupervised learning. Furthermore, the artificial intelligence model according to the present invention may include a support vector machine (SVM), a decision tree, a neural network, and the like, and methodologies to which they are applied.

As an embodiment, the artificial intelligence model according to the present invention may be an artificial intelligence model based on convolutional deep neural networks (CNN) learned by inputting training data. However, it is not limited thereto, and it goes without saying that various artificial intelligence models can be applied to the present invention. For example, models such as a deep neural network (DNN), a recurrent neural network (RNN), and a bidirectional recurrent deep neural network (BRDNN) may be used as an artificial intelligence model, but are not limited thereto.

At this time, convolutional deep neural networks (CNNs) are a type of multilayer perceptrons designed to use a minimum of preprocessing. A convolutional neural network consists of one or several convolutional layers and general artificial neural network layers placed on top of them, and additionally utilizes weights and pooling layers. Thanks to this structure, convolutional neural networks can fully utilize input data with a two-dimensional structure. Also, convolutional neural networks can be trained via standard back-propagation. Convolutional neural networks are easier to train than other feedforward artificial neural network techniques and have the advantage of using fewer parameters.

In addition, deep neural networks (DNNs) are artificial neural networks (ANNs) consisting of a plurality of hidden layers between an input layer and an output layer.

At this time, the structure of the deep neural network may be composed of a perceptron. Perceptron consists of several inputs, one processor, and one output value. The processor multiplies each of a plurality of input values by a weight, and then sums all the input values multiplied by the weight. Then, the processor substitutes the summed value into the activation function and outputs one output value. If a specific value is desired as an output value of the activation function, a weight value multiplied with each input value may be modified, and an output value may be recalculated using the corrected weight value. At this time, each perceptron may use a different activation function. In addition, each perceptron accepts the outputs passed from the previous layer as inputs, and then obtains the outputs using the activation function. The obtained output is passed as input to the next layer. Through the process as described above, several output values can finally be obtained.

Recurrent Neural Network (RNN) refers to a neural network in which the connections between units constituting an artificial neural network constitute a directed cycle. Unlike feed-forward neural networks, recurrent neural networks can utilize the memory inside the neural network to process arbitrary inputs.

Deep Belief Networks (DBN) is a generative graphical model used in machine learning. In deep learning, it means a deep neural network consisting of multiple layers of latent variables. There are connections between layers, but there is no connection between units within a layer.

Due to the characteristics of a generative model, the deep trust neural network can be used for prior learning, and after learning initial weights through prior learning, the weights can be fine-tuned through backpropagation or other discrimination algorithms. This characteristic is very useful when there is little training data, because the smaller the training data, the stronger the influence of the initial weight values on the resulting model. The pre-learned initial value of the weight becomes closer to the optimal weight than the arbitrarily set initial value of the weight, which enables the performance and speed of the fine-tuning step to be improved.

The above-described artificial intelligence and its learning method are described for illustrative purposes, and the artificial intelligence and its learning method used in the above-described embodiments are not limited. For example, all kinds of artificial intelligence technologies and learning methods that can be applied by a person skilled in the art to solve the same problem can be used to implement the system according to the disclosed embodiment.

Meanwhile, the processor 330 may include one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals to and from other components. can

Processor 330 according to one embodiment performs the method described in connection with the present invention by executing one or more instructions stored in memory 310 .

For example, the processor 330 acquires new training data by executing one or more instructions stored in the memory 310, performs a test on the acquired new training data using a learned model, and performs the test. As a result, first training data for which the labeled information is obtained with an accuracy equal to or higher than a predetermined first reference value is extracted, the extracted first training data is deleted from the new training data, and the new training data from which the extracted training data is deleted The learned model may be retrained using the training data.

Meanwhile, the processor 330 includes RAM (Random Access Memory, not shown) and ROM (Read-Only Memory) temporarily and/or permanently storing signals (or data) processed in the processor 330. , not shown) may be further included. In addition, the processor 330 may be implemented in the form of a system on chip (SoC) including at least one of a graphics processing unit, RAM, and ROM.

The memory 310 may store programs (one or more instructions) for processing and control of the processor 330 . Programs stored in the memory 310 may be divided into a plurality of modules according to functions.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Components of the present invention may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. Components of the present invention may be implemented as software programming or software elements, and similarly, embodiments may include various algorithms implemented as data structures, processes, routines, or combinations of other programming constructs, such as C, C++ , Java (Java), can be implemented in a programming or scripting language such as assembler (assembler). Functional aspects may be implemented in an algorithm running on one or more processors.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: electronic device
110: memory
111: AI model
111a: first model
111b: second model
120: processor
130: microphone
140: Ministry of Communication
200: user terminal
300: server
310: memory
320: communication department
330: processor

Claims (13)

In electronic devices,
a memory in which an artificial intelligence model for determining the type of dysarthria of the patient is stored; and
Including; a processor connected to the memory;
the processor,
Obtaining audio data corresponding to a user's voice uttering one text;
The electronic device, characterized in that by inputting the audio data to the artificial intelligence model, to identify a dysarthria type matching the audio data. According to claim 1,
the processor,
Noise filtering the audio data;
Sectioning the filtered audio data into a plurality of sections, overlapping adjacent sections by a predetermined time and framing;
The electronic device characterized in that a plurality of acoustic feature vectors and a plurality of Mel Frequency Cepstral Coefficients (MFCC) feature vectors are extracted from the framed audio data. According to claim 2,
the processor,
Obtaining at least one first MFCC feature vector by applying an MFCC technique to the framed audio data;
Obtaining at least one second MFCC feature vector by differentiating a value of the at least one first MFCC feature vector;
Obtaining at least one third MFCC feature vector by differentiating a value of the at least one second MFCC feature vector;
The plurality of MFCC feature vectors,
The electronic device characterized in that it comprises the at least one first MFCC feature vector, the at least one second MFCC feature vector, and the at least one third MFCC feature vector. According to claim 2,
The artificial intelligence model,
a first model for selecting at least one feature vector from among a plurality of acoustic feature vectors and a plurality of MFCC feature vectors extracted from the audio data; and
An electronic device comprising: a second model for determining a type of dysarthria based on the selected feature vector. According to claim 4,
The first model,
select at least one feature vector according to the text;
The second model,
Based on the selected feature vector, audio data of a voice of a normal person uttering the text is compared with the audio data. According to claim 5,
the processor,
If the user's actual dysarthria type does not match the dysarthria type output from the artificial intelligence model, updating the number of occurrences of errors matched to the text;
and updating the first model to select another feature vector for the text when the number of occurrences of the error exceeds a predetermined value. In the control method of the deep learning-based dysarthria classification system,
obtaining, by a server, audio data corresponding to a voice of a user uttering a text from a user terminal;
identifying, by the server, a type of dysarthria matched to the voice of the user through an artificial intelligence model for determining the type of dysarthria of the patient; and
The control method of the classification system, comprising: providing, by the server, the identified dysarthria type to the user terminal. According to claim 7,
Identifying the type of dysarthria that matches the user's voice through an artificial intelligence model that determines the type of dysarthria of the patient,
performing, by the server, noise filtering on the audio data;
framing, by the server, dividing the filtered audio data into a plurality of sections, overlapping adjacent sections by a predetermined time; and
and extracting, by the server, a plurality of acoustic feature vectors from the framed audio data. According to claim 8,
Identifying the type of dysarthria that matches the user's voice through an artificial intelligence model that determines the type of dysarthria of the patient,
obtaining, by the server, at least one first MFCC feature vector by applying a Mel Frequency Cepstral Coefficients (MFCC) technique to the framed audio data;
obtaining, by the server, at least one second MFCC feature vector by differentiating a value of the at least one first MFCC feature vector; and
and obtaining, by the server, at least one third MFCC feature vector by differentiating a value of the at least one second MFCC feature vector. According to claim 9,
Identifying the type of dysarthria that matches the user's voice through an artificial intelligence model that determines the type of dysarthria of the patient,
The server may include the plurality of acoustic feature vectors, the at least one first MFCC feature vector, the at least one second MFCC feature vector, and the at least one third MFCC feature vector included in the artificial intelligence model. 1 input into the model;
obtaining, by the server, at least one feature vector from the first model;
inputting, by the server, the obtained feature vector to a second model included in the artificial intelligence model; and
Acquiring, by the server, a dysarthria type matching the voice of the user from the second model; including, a method for controlling a classification system. According to claim 10,
The control method of the classification system,
selecting, by the first model, at least one feature vector according to the text;
comparing, by the second model, audio data of a voice of a normal person uttering the text with the audio data based on the selected feature vector;
updating, by the server, the number of occurrences of errors matched to the text when the user's actual dysarthria type does not match the dysarthria type output from the artificial intelligence model; and
and updating, by the server, the first model to select another feature vector for the text when the number of occurrences of the error exceeds a predetermined value. According to claim 11,
Updating the first model to select a different feature vector for the text comprises:
when the accuracy of the second model is less than a predetermined value, causing the server to select at least one feature vector having the accuracy of the first model or higher; and
and storing, by the server, at least one feature vector for the text such that the accuracy is greater than or equal to a predetermined value. A computer program stored in a computer-readable recording medium to be combined with a computer, which is hardware, to perform the method of claim 8.

Images