TWI811097B - Method and apparatus for determining a degree of dementia of a user - Google Patents

Abstract

In order to determine the degree of dementia of the user, output the content through the user terminal, and continuously receive the user's response to the observed content, and generate biomarker information through the visualized response, and pass the convolutional neural network (CNN) and deep neural network based on the biomarker information (DNN) to determine the degree of dementia of the user.

Description

Method and apparatus for determining the degree of dementia in a user

The technical field relates to technologies for determining the degree of dementia of a user, and more particularly, to an apparatus and method for providing content to a user and determining the degree of dementia of the user based on the user's response to the provided content.

With the aging of society, dementia has become one of the most serious diseases of the elderly. It has shown a rapid upward trend in the past 10 years, and the social and economic costs are also rising. In addition, dementia brings great pain not only to the life of the patient himself but also to the family members who care about the patient, as the patient cannot live independently and can lead to disappearance or himself, etc. With early diagnosis and appropriate treatment, further cognitive decline can be prevented or delayed, but existing early diagnosis of this disease remains problematic. In the past, because it was necessary to see a professional medical institution such as a hospital, many patients who came to see a doctor who felt that their forgetfulness had worsened had developed mild cognitive impairment (MCI) or Alzheimer's disease (AD), Neurocognitive function tests for diagnosis (SNSB-II, CERAD-K, etc.) ), positron emission tomography (PET), and cerebrospinal fluid testing are very expensive and cause a lot of inconvenience to the patients receiving the diagnosis.

technical issues

An embodiment may provide an apparatus and method for determining a user's degree of dementia.

An embodiment may provide an apparatus and method for determining a user's dementia level based on a user's voice.

Technical method

According to an embodiment, the method for determining the degree of dementia of a user performed by an electronic device includes the following steps: outputting first content pre-produced for determining the degree of dementia of the user through a user terminal;

receiving the first voice of the user for the first content acquired through the microphone of the user terminal; outputting the pre-made second content through the user terminal; receiving the user's voice for the first content acquired through the microphone A second speech of second content; generating a first spectrogram image by visualizing at least one feature of the first speech; generating a second spectrogram image by visualizing at least one feature of the second speech ; By inputting the first spectrogram image into a pre-updated first convolutional neural network (CNN, convolutional neural network), generating a preset number of first features for the first speech; by adding the first The two spectrogram images are input to the pre-updated second convolutional neural network to generate a preset number of second features for the second speech; determine a preset number of features in the first feature and the second feature target features; and by inputting the target features into a pre-updated deep neural network (DNN, deep neural network), determining the dementia degree of the user, and outputting the determined dementia degree through the user terminal.

The first content may include an instruction (instruction) for receiving the first voice.

The first content may be content that makes the user read a sentence, guess the name of the output image, describe the output image, use language fluency, use number operations, and induce storytelling ( one of the content of story telling).

The step of generating a first spectrogram image by visualizing at least one feature of the first speech may include the step of: generating the first spectrogram image of the first speech by a librosa tool.

A size of the first spectrogram image and a size of the second spectrogram image may be the same as each other.

The first CNN may be pre-updated based on the VGG16 model.

The first CNN pass may include an input layer, 5 pre-convolutional layer blocks, a fully connected layer, and 2 post-convolutional layer blocks and does not include softmax to generate the The first feature of the first spectrogram image.

The method of determining the degree of dementia of a user may also include the step of updating said first CNN.

The step of updating the first CNN may include the following steps: receiving a first test speech of a test user for the first content; generating a first test spectrogram by visualizing at least one feature of the first test speech image, wherein the first test spectrogram image is marked as the GT (ground truth) dementia degree of the test user; by inputting the first test spectrogram image into the first complete CNN to determine the The first test dementia degree of the test user, wherein the first complete CNN includes an input layer, more than one pre-convolution layer block, a fully connected layer, more than one back-convolution layer block, and softmax; and based on the first testing the dementia degree and the GT dementia degree to update the complete first CNN, wherein the first CNN includes only the input layer, the one or more layers in the layers of the updated complete first CNN a pre-convolutional layer block, said fully-connected layer, and said one or more post-convolutional layer blocks.

The method for determining the degree of dementia of a user may further include the following step: updating the DNN after completing updating of a plurality of CNNs including the first CNN and the second CNN.

The step of updating the DNN may include the following steps: a preset number of first test features generated based on the first test spectrogram image and a preset number of second test features generated based on the second test spectrogram image , determine a preset number of test target features, wherein the test target feature is marked as the test user's GT dementia degree; by inputting the test target feature into the DNN to determine the test user's first two test dementia levels; and updating the DNN based on the second test dementia level and the GT dementia level.

The device for determining the degree of dementia of a user according to an embodiment includes: a memory recording a program for determining the degree of dementia of a user; and a processor for executing the program, wherein the program performs the following steps: The pre-made first content is used to determine the degree of dementia of the user; receiving the first voice of the user for the first content obtained through the microphone of the user terminal; outputting the pre-made second content through the user terminal content; receiving a second speech of the user for the second content captured through the microphone; generating a first spectrogram image by visualizing at least one characteristic of the first speech; visualizing the second speech at least one feature of the first speech to generate a second spectrogram image; by inputting the first spectrogram image into a pre-updated first convolutional neural network (CNN), a preset number of the first speech is generated for the first speech a feature;

By inputting the second spectrogram image into the pre-updated second convolutional neural network, generating a preset number of second features for the second speech; among the first features and the second features determining a preset number of target features; and determining the degree of dementia of the user by inputting the target features into a pre-updated deep neural network (DNN), and outputting the determined degree of dementia through the user terminal .

According to an embodiment, the method for updating the convolutional neural network used to determine the user's dementia degree by an electronic device includes the following steps: outputting the pre-made first content for determining the user's dementia degree through the user terminal; a first test utterance of a test user of the first content; generating a first test spectrogram image by visualizing at least one feature of the first test utterance, wherein the first test spectrogram image is labeled as The GT dementia degree of the test user; the test dementia degree of the test user is determined by inputting the first test spectrogram image into a complete CNN, wherein the complete CNN includes an input layer, more than one front a convolutional layer block, a fully connected layer, one or more post-convolutional layer blocks, and softmax; and updating the full CNN based on the test dementia level and the GT dementia level, and the CNN is The layers of the CNN may only include the input layer, the one or more pre-convolution layer blocks, the fully connected layer and the one or more post-convolution layer blocks.

An electronic device for updating a convolutional neural network for determining the degree of dementia of a user according to an embodiment includes: a memory recording a program for updating the CNN; and a processor for executing the program, wherein the processing The device executes the steps of: outputting the pre-made first content for determining the degree of dementia of the user through the user terminal; receiving the first test voice of the test user for the first content; by visualizing at least the first test voice A feature to generate a first test spectrogram image labeled as the test user's GT dementia level; by inputting the first test spectrogram image into a full CNN to determine the test dementia degree of the test user, wherein the complete CNN includes an input layer, more than one pre-convolution layer block, a fully connected layer, one or more post-convolution layer blocks, and softmax; and based on the test dementia degree and the GT dementia degree to update the complete CNN, and the CNN may only include the input layer, the one or more pre-convolution layer blocks, the The fully connected layer and the one or more post-convolutional layer blocks.

technical effect

An apparatus and method for determining the degree of dementia of a user may be provided.

An apparatus and method for determining a degree of dementia of a user based on a user's voice may be provided.

Hereinafter, embodiments will be described in detail with reference to the drawings. However, the scope of rights of the present invention is not limited or defined by the examples. Like drawing numbers in each drawing represent like elements.

Various modifications can be made to the following embodiments. It should be understood that the embodiments described below are not intended to limit these embodiments, and include all modifications, equivalents, and substitutions thereto.

The terms used in the embodiments are for describing specific embodiments only, and are not intended to limit the embodiments. Unless otherwise specified in the content, a singular expression includes a plural meaning. In this specification, terms such as "comprising" or "having" are used to express the presence of features, numbers, steps, operations, constituent elements, accessories or combinations thereof described in the specification, and do not exclude the presence of one or more other features , numbers, steps, operations, constituent elements, accessories or combinations thereof, or additional functions.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the usual meanings understood by those of ordinary skill in the art. Commonly used terms that are the same as those defined in dictionaries should be understood as meanings that are consistent with the usual content of related technologies, and should not be overly idealized or interpreted as formal meanings unless explicitly mentioned in this application.

In addition, in the process of describing with reference to the drawings, the same constituent elements are assigned the same reference symbols regardless of the reference symbols, and overlapping description thereof will be omitted. In explaining the embodiments, when it is judged that the specific descriptions of related well-known technologies will unnecessarily obscure the embodiments, the detailed descriptions thereof are omitted.

FIG. 1 is a block diagram illustrating a system for determining a user's degree of dementia, according to an example.

According to an aspect, the system for determining a user's dementia degree may include an electronic device 110 for determining a user's dementia degree, a user terminal 120 for outputting content, and a monitoring terminal 130 for a medical institution. For example, the electronic device 110 may be a server.

The electronic device 110 may provide pre-made content to the user terminal 120 to determine the degree of dementia of the user. For example, the content may be content for acquiring voice from the user. The content for acquiring the user's voice will be described in detail below with reference to FIG. 5 .

The user terminal 120 may be connected to the electronic device 110 offline or online to communicate with each other. The electronic device 110 provides content to the user terminal 120, and the user terminal 120 outputs the content to the user through a display. For example, the user terminal 120 may acquire the user's voice through the microphone as a response to the content, and send the acquired voice to the electronic device 110 .

The electronic device 110 may determine the user's dementia degree based on the user's voice, and transmit the determined dementia degree to the user terminal 120 .

The user terminal 120 may be a mobile terminal such as a tablet computer or a smart phone. When the user terminal 120 is a mobile terminal, the user is not limited by time and place, and can measure the degree of dementia at low cost.

Hereinafter, a method for determining the degree of dementia of a user will be described in detail with reference to FIGS. 2 to 17 .

FIG. 2 illustrates an image output to a user terminal to determine a user's dementia degree according to an example.

The images below ( 210 to 240 ) may be images of an application used to determine the degree of dementia. For example, a user of the electronic device 110 can create and distribute an application program, and the user can execute the application program through the user terminal 120 .

The first image 210 is the start screen of the application.

The second image 220 indicates the functions supported by the application.

The third image 230 is an example of content provided to the user. A plurality of contents may be provided to the user.

The fourth image 240 represents the determined degree of dementia of the user. For example, normal, mild cognitive impairment (MCI), or Alzheimer's disease (AD) determined to be the user's degree of dementia may be output. In addition to the degree of attention to individual diseases, comprehensive judgments can also be output together.

FIG. 3 is a block diagram illustrating an electronic device for determining a degree of dementia of a user according to an embodiment.

The electronic device 300 includes a communication unit 310 , a processor 320 and a memory 330 . For example, the electronic device 300 may be the above-mentioned electronic device 110 described with reference to FIG. 1 .

The communication section 310 is connected to the processor 320 and the memory 330 to transmit and receive data. The communication section 310 may be connected to another device outside to transmit/receive data. Hereinafter, the expression "sending and receiving A" may mean sending and receiving "information or data representing A".

The communication unit 310 may be realized as a circuit in the electronic device 300 . For example, the communication unit 310 may include an internal bus (internal bus) and an external bus (external bus). As another example, the communication part 310 may be an element connecting the electronic device 300 and an external device. The communication unit 310 may be an interface. The communication part 310 may receive data from an external device and transmit it to the processor 320 and the memory 330 .

The processor 320 processes data received by the communication unit 310 and data stored in the memory 330 . A "processor" may be a hardware-implemented data processing device having circuits of a physical structure for performing desired operations. For example, the desired operations may include codes or instructions included in the program. For example, a data processing device implemented as hardware may include a microprocessor (microprocessor), a central processing unit (central processing unit), a processor core (processor core), a multi-core processor (multi-core processor), a multiprocessor (multiprocessor) ), Application-Specific Integrated Circuit (ASIC) and Field Programmable Gate Array (FPGA, Field Programmable Gate Array).

Processor 320 executes computer readable code (eg, software) stored in memory (eg, memory 330 ) and instructions issued by processor 320 .

The memory 330 stores data received by the communication unit 310 and data processed by the processor 320 . For example, the memory 330 may store programs (or application programs, software). The stored program may be a set of syntax (syntax) coded to determine the level of dementia of the user and executable by the processor 320 .

According to an aspect, memory 330 may include one or more of volatile memory, non-volatile memory and random access memory (RAM), flash memory, hard disk drive, and optical disk drive.

The memory 330 stores an instruction set (eg, software) for operating the electronic device 300 . A set of instructions for operating the electronic device 300 is executed by the processor 320 .

The communication part 310, the processor 320, and the memory 330 will be described in detail below with reference to FIGS. 4 to 17 .

FIG. 4 is a flow chart illustrating a method for determining the degree of dementia of a user according to an embodiment.

The following steps 410 to 440 are performed by the above-mentioned electronic device 300 described with reference to FIG. 3 .

In step 410 , the electronic device 300 outputs pre-generated content to determine the user's dementia level through the display of the user terminal (eg, the user terminal 120 ). The content is output to the user terminal, and the user reacts to the content.

The user terminal may receive the user's voice as the above reaction by using the camera. The user terminal can generate a voice in response by using a microphone. The generated speech may be in the form of a data file.

A plurality of contents may be provided to the user, and a user's voice for each of the plurality of contents may be generated.

According to an embodiment, the following [Table 1] is used to describe the content for generating the user's voice.

[Table 1] voice task instructions Step 1. Read the sentence Now please listen carefully to my sentences and read them. After each sentence, please hear the beep to start. The yard is full of roses. Step 2. Read the sentence Again, please listen carefully to my sentences and follow along. After each sentence, please hear the beep to start. Yesterday, it rained, and I stayed at home. Step 3. Read the sentence Again, please listen carefully to my sentences and follow along. After each sentence, please hear the beep to start. The walls have seams, and the walls have ears. Step 4. Say the name Next, name the animals you see. After the beep, say the names of the animals you see in turn. Step 5. Describe the picture Please look at the picture next, please describe the picture in as much detail as possible within 1 minute. Please describe in as much detail as possible where it is, what is there, what the animal or person is doing, etc. Please hear the beep to start. Step 6. Linguistic fluency (phonemic form) Next, say the word that begins with the letter that appears. For example, if you see the letter "a," say as many words that begin with "a" as you can. Can say words like apple, ant, astronaut, etc. Are there any other words that start with "a"? Now say the words that start with another letter, ie "b". You have a minute, say as many words starting with "b" as you can, are you ready? Please hear the beep to start. Step 7. Linguistic fluency (semantic form) If we tell you about a genre, please tell us the name that belongs to that genre as soon as possible. For example, if we say "type of animal", you can name dogs, cats, lions, etc. Is there anything else of the animal type? Now name all the names that belong to another type, that is, to fruits. You have a minute, please name the fruit you think of in one minute, are you ready? Please hear the beep to start. Step 8. Subtraction Now we have a simple calculation problem. What is 100 minus 3? Subtract 3 from 100 and the answer is 97. Well, subtract 3 from there. Subtract 3 from 97, so the answer is 94. Please continue to subtract 3. Start at 100 and keep subtracting 3, are you ready? Please hear the beep to start. Step 9. Tell a story (front) What is the happiest thing you have experienced so far? Please tell us in as much detail as you can about the happiest thing you have ever experienced in one minute. Please hear the beep to start. Step 10. Tell the Story (Negative) What is the saddest thing that has happened to you so far? Please tell us in as much detail as you can about the saddest thing that has happened to you in one minute. Please hear the beep to start. Step 11. Tell a story (interlude) What happened yesterday? Please tell us what happened yesterday in as much detail as you can within one minute. Please hear the beep to start.

In step 420, the electronic device 300 continuously receives the user's response to watching the content from the user terminal. For example, the electronic device 300 may receive a user voice of content acquired through a microphone of the user terminal.

When producing multiple contents, steps 410 and 420 may be performed repeatedly. Steps 410 and 420 are repeatedly performed to receive user voices of multiple contents. For example, the plurality of contents may include first to eleventh contents for receiving a user's voice. First to eleventh voices of the first to eleventh contents may be received.

In step 430, the electronic device 300 generates a biomarker message based on the received response.

According to an embodiment, the electronic device 300 generates a spectrogram image of the speech as the biomarker information by visualizing at least one feature of the received speech. For example, the electronic device 300 can generate a speech spectrogram image through the librosa tool. The spectrogram image may be a mel spectrogram image.

For example, a spectrogram image of each of the first to eleventh voices may be generated. The spectrogram image is described in detail below with reference to FIG. 6 .

In step 440, the electronic device 300 determines the dementia level of the user based on the biomarker information.

According to an embodiment, the electronic device 300 can determine the dementia level of the user by inputting the spectrogram image as biomarker information into a preset dementia level classification model. For example, a dementia classification model can be pre-trained on a neural network. A method of determining a user's dementia degree based on a dementia degree classification model will be described in detail below with reference to FIGS. 7 to 12 .

After step 440 is executed, the electronic device 300 may output the determined dementia degree through the user terminal.

FIG. 5 illustrates content pre-produced to receive user voice according to an example.

For example, the content 500 provided to the user may be content to guess the names of the output images 520 , 530 , 540 . The content 500 may include an instruction 510 for a user voice of the content 500 in addition to the images 520 , 520 , 540 . The instruction 510 can be displayed in text form, or can be output by voice. The user can generate voice by speaking the name of the image 520 , 530 , 540 .

Although an example of content for receiving a user's voice has been described with reference to FIG. 5 , the content may be produced in various ways according to the user's voice to be measured. For example, the content may be content that makes the user say a value obtained by subtracting 3 from 100, that makes the user say the output voice again, or makes the user say as much as possible within a given time. The content of the word beginning with ".

According to an embodiment, a preset number (for example, 11) of contents may be sequentially provided to the user through the user terminal. The user terminal can generate a plurality of voice data files by recording the user voice of each content, and send the generated voice data files to the electronic device 300 .

Fig. 6 shows a raw spectrogram image generated for speech according to an example.

According to one aspect, the electronic device 300 can generate the original spectrogram image 600 of speech through the Librosa tool. The horizontal axis of the original spectrogram image 600 may be a time axis, and the vertical axis may be a frequency axis. The original spectrogram image 600 represents a difference in amplitude according to changes in the time axis and the frequency axis as a difference in printing density/display color. The display color of the corresponding position may be determined based on the magnitude of the changed amplitude difference. For example, a legend 610 showing the magnitude of the amplitude difference may be output together with the original spectrogram image 600 . In order to display the determined color, the values of the R, G, and B channels of the pixel corresponding to the coordinates may be determined.

According to an embodiment, the plurality of spectrogram images to be input to the model may be generated based on the plurality of raw spectrogram images for each of the plurality of utterances. For example, a first spectrogram image may be generated for a first speech, and a second spectrogram image may be generated for a second speech. The scale of the time axis and the frequency axis of the original spectrogram image can be different according to the total time of each speech, but the size of the final generated spectrogram image can be the same.

According to an embodiment, the electronic device 300 may convert the first original spectrogram image generated for the first content into a first adjusted spectrogram image with a preset first time range for the first content. For example, the first time range of the first content may be preset based on the average response time of multiple users to the first content. For example, the first time range may be the sum of the average response time and the median of the response times (or the standard deviation of the response times).

According to an embodiment, statistical data on user's response time to the first to fourth contents is shown in [Table 2] below.

[Table 2] first content Second content third content fourth content average response time 6.423967571 6.517637474 7.738273502 10.29516905 Standard deviation value 9.641737921 9.008077433 9.55999683 10.85853336 Median 4.120746667 4.47616 5.258986667 6.893226667 maximum value 60.78869333 60.74616889 60.76743111 59.97546667 minimum value 1.3056 1.314133333 1.258666667 1.32096

For example, according to [Table 2] above, when the first time range of the first content is determined as the sum of the average response time and the median value, its sum is 10.544714238 (seconds), and 9 seconds with no significant difference from 10.5447 14238 can be Determined as the first time frame.

For example, when the length of the user's first original spectrogram image for the first content is 10 seconds, the first adjusted spectrogram image may be generated by cutting a part of 9 seconds or longer. As another example, when the length of the user's first original spectrogram image for the first content is 8 seconds, the first adjusted spectrogram image may be generated by adding a silence time from 8 seconds to 9 seconds.

According to an embodiment, first to eleventh adjusted spectrogram images for the first to eleventh contents may be generated. For example, each of the first to eleventh adjusted spectrogram images may represent a different time interval. Certain parts of the upper part of the image where the difference between the first adjusted spectrogram image and the eleventh adjusted spectrogram image are not large may be removed. Based on the first to eleventh adjusted spectrogram images with certain parts removed, the first to eleventh adjusted spectrogram images may be processed Image processing such that the images have the same size. The image-processed adjusted spectrogram image may be referred to as a spectrogram image. For example, the size of the first spectrogram image and the size of the second spectrogram image may be 300x300, which are the same as each other. For example, the size of a spectrogram image can be in pixels. For example, the value of a pixel can be represented by 16 bits.

Although an example of content for receiving a user's voice has been described with reference to FIGS. 5 and 6 , the content may be produced in various ways according to the user's voice to be measured. For example, the content may include content depicting a photo and content for reading an output sentence.

FIG. 7 is a flowchart illustrating a method of determining a degree of dementia of a user using a CNN and a DNN according to an example.

According to one aspect, the above step 440 described with reference to FIG. 4 may include the following steps ( 710 to 730 ).

In step 710 , the electronic device 300 generates a preset number of features of the content by inputting the spectrogram image into a pre-updated convolutional neural network (CNN) corresponding to the spectrogram image. The CNN used to generate features may vary depending on the content. For example, when there are 11 contents, there is a CNN corresponding to each of the 11 contents, and these 11 CNNs may be called a CNN set. Hereinafter, the word "update" may include the meaning of the word "training" and may be used interchangeably.

According to one aspect, the CNN can be pre-updated based on the VGG16 model. A CNN can be part of a complete CNN, which includes an input layer, one or more pre-convolutional layer blocks, a fully connected layer, one or more post-convolutional layers, and softmax. For example, a CNN can include an input layer, a pre-convolution block, a fully-connected layer, and a post-convolution block, but not softmax. Since the CNN does not include softmax, it is possible to output a preset number of features that are used to calculate the degree of dementia instead of calculating the degree of dementia as a result of the input spectrogram image. The full CNN and the partial CNN will be described in detail with reference to FIG. 8 .

For example, the electronic device 300 generates a preset number of first features for the first content by inputting the first spectrogram image into the pre-updated first CNN, and inputs the second spectrogram image into the pre-updated second CNN. CNN to generate a preset number of second features for the second content. As a specific example, when 11 spectrogram images are received and 256 features are generated for one spectrogram, a total of 2816 features can be generated.

In step 720, the electronic device 300 determines a target feature among the features of the plurality of contents (or the plurality of corresponding spectrogram images). The determined characteristic of interest may be a marker for determining the degree of dementia. Features determined to be target features may be predetermined as markers. The markers may be predetermined through the steps of updating the CNN and updating the deep neural network (DNN), which will be described later with reference to FIGS. 12 to 16 .

Fig. 8 shows a full CNN and a partial CNN capable of determining the degree of dementia of a user according to an example.

According to one aspect, a complete CNN 800 includes an input layer 810, a first convolutional layer block 820, a second convolutional layer block 830, a third convolutional layer block 840, a fourth convolutional layer block 840, a fifth convolutional layer block 850, a fully connected Layer 870, sixth convolutional layer block 880, seventh convolutional layer block 890, and softmax 895. For ease of identification, the first convolutional layer block 820, the second convolutional layer block 830, the third convolutional layer block 840, the fourth convolutional layer block 850, and the fifth convolutional layer block 860 located in front of the fully connected layer 879 can be referred to as pre- Convolutional layer blocks (pre-convolutional layer blocks), and the sixth convolutional layer block 880 and the seventh convolutional layer block 890 located behind the fully connected layer 879 are called post-convolutional layer blocks (post-convolutional layer blocks).

According to an embodiment, a convolutional layer block may include more than one convolutional layer and pooling layer. In addition, each of the sixth convolutional layer block 880 and the seventh convolutional layer block 890 may further include a drop-out layer block.

The full CNN 800 may be a full CNN updated by a full CNN update method (described later with reference to FIG. 12 ). A complete CNN that is different for each content can be updated in advance.

The partial CNN 805 may only include an input layer 810, a first convolutional layer block 820, a second convolutional layer block 830, a third convolutional layer block 840, a fourth convolutional layer block 850, a fifth convolutional layer block 860, a fully connected layer 870, Sixth convolutional layer block 880 and seventh convolutional layer block 890 , excluding softmax 895 . That is, the partial CNN 805 may be a CNN with the softmax 895 removed from the full CNN 800 after the update of the full CNN 800 is completed. For example, the CNN used in the above step 710 described with reference to FIG. 7 may be the partial CNN 805 .

Since part of the CNN 805 does not include softmax895, it can output various features of the spectrogram image.

FIG. 9 illustrates generated features for each of multiple sets of user images and target features determined based thereon, according to an example.

According to an aspect, a predetermined number of features of the target speech are generated through a target CNN corresponding to the target content. For example, the preset number of features may be 256. When the number of multiple spectrogram images according to multiple contents is n, the number of all features 900 generated may be 256×n.

A preset number of target features 910 among all features 900 are determined. The determined target characteristics 910 may be preset markers for determining the degree of dementia. The method of predetermining the target feature 910 as a marker will be described in detail below with reference to step 1310 of FIG. 13 .

FIG. 10 illustrates a DNN for determining a user's degree of dementia, according to an example.

According to an aspect, a DNN for determining the degree of dementia of a user may include an input layer 1010 , one or more hidden layers 1020 , 1030 , 1040 and an output layer 1050 . For example, the DNN may be a DNN updated by a method of updating a DNN (described later with reference to FIG. 13 ).

The DNN may output the user's degree of dementia as an output of the input target feature 910 . The DNN can input any of a number of preset dementia levels. For example, the preset multiple degrees of dementia may include confirmed normal, mild cognitive impairment (MCI), and Alzheimer's disease (AD).

Figure 11 shows a classification of two steps performed to improve the accuracy of determining the degree of dementia, according to an example.

Unlike a method of determining any one of a plurality of dementia levels through a single model, a method of stepwise determining a dementia level through a plurality of models can improve the determination accuracy of the dementia level.

For example, rather than a single model to identify normal, mild cognitive impairment (MCI) and Alzheimer's disease (AD), normal or abnormal (mild cognitive impairment (MCI) ) and Alzheimer's disease (AD)), and identify mild cognitive impairment (MIC) or Alzheimer's disease (AD) in the second stage of the classification.

In order to use the above method, the first set of CNNs and the first DNN used in the first stage of classification and the second set of CNNs and the second DNN used in the second stage of classification are prepared in advance, respectively.

For example, when the above steps ( 410 to 440 ) referring to FIG. 4 are performed for the first classification stage, and the user's dementia degree is determined to be abnormal through the first classification stage, step 440 for the second classification stage may be executed. When the dementia level of the user is determined to be normal through the first classification stage, the second classification stage may not be executed. The first set of CNNs and the first DNN used for the first classification stage and the second set of CNNs and the second DNN used for the second classification stage are respectively different from each other.

FIG. 12 illustrates a two-step operation performed to improve the accuracy of determining the degree of dementia according to another example.

Unlike the method of determining any one of a plurality of dementia degrees by a single model, the method of determining the degree of dementia by a plurality of models can improve the accuracy of determining the degree of dementia.

According to an embodiment, instead of using one model to determine any of normal, mild cognitive impairment (MCI), and Alzheimer's disease (AD), it is calculated separately using multiple models trained for different classification purposes The probability of mild cognitive impairment (MCI) and Alzheimer's disease (AD), and a method for determining the degree of dementia based on the calculated probabilities.

In step 1210, a plurality of models may be used to calculate partial probabilities for each of the user's normal, mild cognitive impairment (MCI), and Alzheimer's disease (AD). For example, when generating 256 features for 11 spectrogram images, a total of 2816 features can be generated and 2816 features can be input to each of the multiple models. For example, multiple models may include: a first model for classifying normal and mild cognitive impairment (MCI) and Alzheimer's disease (AD), a second model for classifying normal and Alzheimer's disease (AD) model, a third model for classifying normal and mild cognitive impairment (MCI), and a fourth model for classifying mild cognitive impairment (MCI) and Alzheimer's disease (AD).

The first normal probability P _SCI1 and the first mild cognitive impairment (MCI) probability P _MCI1 can be calculated as partial probabilities through the first model. The first Alzheimer's disease (AD) probability P _AD1 may be the same as the first mild cognitive impairment (MCI) probability P _MCI1 . The second normal probability P _SCI2 and the second Alzheimer's disease (AD) probability P _AD2 can be calculated as partial probabilities through the second model. The third normal probability P _SCI3 and the second mild cognitive impairment (MCI) probability P _MCI2 can be calculated as partial probabilities through the third model. The third mild cognitive impairment (MCI) probability P _MCI3 and the third Alzheimer's disease (AD) probability P _AD3 can be calculated as partial probabilities through the fourth model.

To use the above method, prepare in advance the first CNN set and the first DNN used in the first model, the second CNN set and the second DNN used in the second model, and the third CNN set used in the third model and a third DNN, a fourth set of CNNs used in a fourth model, and a fourth DNN.

In step 1220 , a first probability of normal, a second probability of mild cognitive impairment (MCI), and a third probability of Alzheimer's disease (AD) may be determined based on the partial probabilities calculated through the plurality of models.

For example, the sum of the first probability of normality _PSCI1 , the second probability of normality _PSCI2 and the third probability of normality _PSCI3 may be calculated as the first probability of normality. The sum of the first mild cognitive impairment (MCI) probability P _MCI1 , the second mild cognitive impairment probability P _MCI2 , and the third mild cognitive impairment probability P _MCI3 may be determined as the second probability of mild cognitive impairment (MCI). The sum of the first Alzheimer's disease (AD) probability P _AD1 , the second Alzheimer's disease (AD) probability P _AD2 and the third Alzheimer's disease (AD) probability P _AD3 can be determined as Alzheimer The third probability of silent disease (AD).

In step 1230, the classification corresponding to the maximum value among the first probability, the second probability and the third probability may be determined as the dementia degree of the user. For example, when the second probability is the largest among the first probability, the second probability and the third probability, the degree of dementia of the user may be determined as mild cognitive impairment (MCI).

According to an embodiment, in the description with reference to FIG. 12 , it has been described that four models are used to determine the degree of dementia of the user, but the number of models used to determine the degree of dementia is not limited to the disclosed embodiment. For example, to determine the degree of dementia, more than two models may be used.

FIG. 13 is a flowchart illustrating a method for updating a complete CNN according to an example.

According to one aspect, before performing the above step 410 described with reference to FIG. 4 , the following step 1300 is performed in advance. Step 1300 relates to a method for updating a full CNN, which may include the following steps (1310 to 1350).

In step 1310, the electronic device 300 outputs pre-made content to the test user to determine the degree of dementia of the user. For example, the electronic device 300 may output content through a user terminal of a test user.

A test user may be a person whose degree of dementia has been determined by a professional diagnosis by a doctor. For example, the test user may be normal, or suffer from mild cognitive impairment (MCI) or Alzheimer's disease (AD).

In step 1320, the electronic device 300 receives the test voice of the content test user through the microphone of the user terminal. When multiple contents are provided, multiple test voices may be received.

In step 1330, the electronic device 300 generates a test spectrogram image of the test voice by visualizing at least one feature of the received test voice. The test spectrogram image may be labeled with the test user's GT dementia level.

In step 1340, the electronic device 300 determines the test dementia level of the test user by inputting the test spectrogram image into the complete CNN. Since the full CNN includes softmax, the full CNN can determine the degree of test dementia. For example, determined test dementia levels may include normal, with mild cognitive impairment (MCI), and Alzheimer's disease (AD).

According to an embodiment, the first complete CNN corresponding to the first content determines the test dementia degree of the test user based only on the first test spectrogram image, and the nth complete CNN corresponding to the nth content is based only on the nth Test the spectrogram image to determine the degree of dementia of the test user.

In step 1350, the electronic device 300 updates the full CNN based on the test dementia degree and the GT dementia degree. For example, if there is a difference between the test dementia level and the GT dementia level, you can use that difference as an error value to perform backpropagation to update the full CNN. The way to update the full CNN can be supervised learning.

In an embodiment of FIG. 8, when the complete CNN 800 includes an input layer 810, a first convolutional layer block 820, a second convolutional layer block 830, a third convolutional layer block 840, a fourth convolutional layer block 850, a fifth convolutional layer block When stacking block 860, fully connected layer 870, sixth convolutional layer block 880, seventh convolutional layer block 890, and softmax 895, only the fifth convolutional layer block 860 is updated, and other layers may not be updated.

According to an embodiment, the complete CNN can be updated repeatedly through a large number of test users, and when the output accuracy of the updated complete CNN becomes greater than a preset threshold, the update of the complete CNN can be terminated.

According to an aspect, when the degree of dementia is gradually determined through multiple models as shown in the method described with reference to FIGS. each classification step. For example, the first completed CNN is updated to determine normal or abnormal (mild cognitive impairment (MCI) and Alzheimer's disease (AD)), the second completed CNN is updated to determine mild cognitive impairment (MCI) or Alzheimer's disease (AD) Alzheimer's disease (AD).

The CNN used in step 710 may be a neural network that removes the softmax (eg, softmax895) from the full CNN after updating the full CNN. That is, the CNN used in step 510 can be used as a feature extractor for the corresponding spectrogram image.

FIG. 14 is a flowchart illustrating a method of updating a DNN according to an example.

According to an embodiment, the following step 1400 relates to a method for updating a DNN, which may be performed after performing the above step 1300 described with reference to FIG. 13 and before performing the above step 410 described with reference to FIG. 4 . For example, step 1400 may be performed after the update of the full CNN (or CNN) is completed.

Step 1400 may include the following steps (1410 to 1440).

In step 1410, the electronic device 300 determines a preset number of features generated by the second CNN based on the preset number of first test features and the second spectrogram image generated by the first CNN according to the first test spectrogram image. A preset number of test target features in the second test feature. Although only the first test feature and the second test feature are described, for example, when generating n test spectrogram images for n contents, the test target can be determined from the first test feature to the nth test feature feature. The test target characteristic may be a marker for determining the degree of dementia. Hereinafter, a method of determining a characteristic of a test target will be described in detail with reference to FIGS. 15 and 16 .

The test target feature can be marked with the test user's GT dementia degree.

In step 1420, the electronic device 300 may verify the determined test target characteristics. For example, test target features can be validated by k-fold cross validation. The method of verifying the characteristics of the test object will be described in detail below with reference to FIGS. 17 and 18 . According to an embodiment, when the determined test target feature does not need to be verified, step 1420 may not be performed.

Step 1430 may be performed when the test target feature has been verified (or does not need to be verified). According to an example, it is necessary to verify the test target feature, but if it is not verified, it is considered that the CNN needs to be updated again, and step 1300 can be performed again.

In step 1430, the electronic device 300 determines the test dementia degree of the test user by inputting the test target features into the DNN. To distinguish it from the test degree of dementia determined in step 1340, the test degree of dementia of step 1340 is referred to as the first test degree of dementia, and the test degree of dementia of step 1430 is referred to as the second test degree of dementia. When step 1430 is performed for the first time, the DNN used may be an initial DNN or a basic DNN.

In step 1440, the electronic device 300 updates the DNN based on the second test dementia degree and the GT dementia degree. For example, when there is a difference between the second test dementia level and the GT dementia level, backpropagation may be performed using the difference as an error value to update the DNN. The method of updating DNN can be supervised learning.

According to an embodiment, the DNN can be updated repeatedly through a large number of test users, and when the output accuracy of the updated DNN becomes greater than or equal to a preset threshold, the update of the DNN can be terminated.

According to an embodiment, when the degree of dementia is determined step by step through multiple models as shown in the method described with reference to FIGS. classification steps. For example, the first DNN is updated to determine normal or abnormal (mild cognitive impairment (MCI) and Alzheimer's disease (AD)), the second DNN is updated to determine mild cognitive impairment (MCI) or Alzheimer's disease (AD) disease (AD).

FIG. 15 is a flowchart illustrating a method for determining test target characteristics according to an example.

According to one aspect, the above step 1410 described with reference to FIG. 14 may include the following steps ( 1510 to 1550 ).

In step 1510, the overall test signature including the first test signature and the second test signature may be divided into a plurality of sub-feature sets. For example, when the number of overall test features is 2816, each of the sub-feature sets may be generated to include 200 test features, and the fifteenth sub-feature set may include 16 test features. Each of the overall test features may have an index number, and the first set of sub-features includes test feature numbers 1 to 200.

In step 1520, a portion of the plurality of sub-feature sets (15) is selected. For example, five sets from the first sub-feature set to the fifteenth sub-feature set may be selected. The selected 5 sub-feature sets include a total of 1000 test features. The method of selecting a part of the subfeature set will be described in detail below with reference to FIG. 16 .

In step 1530, the selected sub-features (eg, 1000) are divided into multiple sub-feature sets. For example, if the selected features are 1000, each of the sub-feature sets (50) may be generated to include 20 test features.

In step 1540, a portion of the plurality of sub-feature sets (50) is selected. For example, ten sets from the first sub-feature set to the fiftieth sub-feature set may be selected. The selected 10 sub-feature sets include a total of 200 test features. The detailed description of step 1540 can be similarly applied to the description of FIG. 15 for step 1520 below.

In step 1550, the test features included in the selected sub-feature set are determined as test target features. An index may be identified for each identified test target characteristic.

The determined test target characteristics can be used as markers to determine the degree of dementia of the user. For example, when the 4th feature, the 46th feature and the 89th feature in the first feature and the 78th feature and the 157th feature in the second feature are determined as the test target feature, it will be described with reference to FIG. The target features determined in the above step 720 also include the 4th feature, the 46th feature, and the 89th feature in the first feature, and the 78th feature and the 157th feature in the second feature.

The specific numbers shown in the embodiment described with reference to FIG. 15 refer to examples, and the specific numbers may vary according to actual implementation.

FIG. 16 is a flowchart illustrating a method of selecting sub-features according to an example.

According to one aspect, the above step 1520 described with reference to FIG. 15 may include the following steps ( 1610 to 1640 ).

Data from a large number of users is required to identify test target characteristics. In the following, the process of determining the test target characteristics will be described using the data of 1000 users as an example. Set the correct value together with the data of 1000 users.

For example, 1000 users may be classified into 600 training data users, 200 certification data users, and 200 test data users. For each of 600, 2816 features may be generated for the first to eleventh spectrogram images, and 600 first sub-feature sets with specific indices (eg, 1 to 200) may be generated. For example, 600 first to fifteenth sub-feature sets are generated for training data. Similarly, 200 first to fifteenth sub-feature sets are generated for the authentication data, and 200 first to fifteenth sub-feature sets are generated for the test data.

As another example, 1000 users may be classified into 800 training data users and 200 test data users if no verification of the test target features is required. For the first to eleventh spectrogram images of each person in 800, 2816 features may be generated, and 800 first sub-feature sets with specific indices (eg, 1 to 200) may be generated. For example, 800 first to fifteenth sub-feature sets for training data are generated. Similarly, 200 first to fifteenth sub-feature sets are generated for the test data.

In step 1610, based on the 600 first sub-feature sets of the training data (first training data) and the 200 first sub-feature sets of the authentication data (first authentication data), one training cycle of the initial DNN ( epoch). If there is no need to verify the test target features, one training cycle of the initial DNN can be performed based on the 800 first sub-feature sets of the training data. Weights of edges or parameters of nodes in the DNN are adjusted based on the 600 (or 800) first sub-feature sets. The result of the input first authentication data is outputted through the weighted DNN. The number of output results can be 200. The administrator can adjust the number of preset cycles executed for learning by referring to the 200 output results.

In step 1620, a preset number of training cycles are performed on the DNN. For example, a training cycle of 30 may be performed. A learning (or training) can be considered complete when a preset number of epochs are performed.

In step 1630, a first learning accuracy may be calculated based on the 200 first sub-feature sets of the test data (first test data). For example, the first test data can be input to the learned DNN, and the accuracy of 200 results can be calculated as the first learned accuracy.

Additional learned accuracy can be calculated by repeating steps (1610 to 1630) a preset number of times. Since the initial DNN provided in step 1610 is different, the result of DNN learning may also be different, therefore, the learning accuracy of multiple learning will also be different. When the steps (1610 to 1630) are repeated 10 times, the first to tenth learning accuracies may be calculated.

In step 1640, a first average learning accuracy of the first training data is calculated. For example, an average value of the first to tenth learning precisions may be calculated as the first average learning precision.

For example, if the steps (1610 to 1640) are performed on a first sub-feature set including features with indexes 1 to 200, a first average learning accuracy of the first sub-feature set may be calculated.

As another example, when the steps ( 1610 to 1640 ) are performed on a second sub-feature set including features with indexes 201 to 400 , a second average learning accuracy of the second sub-feature set may be calculated.

For example, the first to fifteenth average learned accuracies for each of the 15 sub-feature sets may be calculated. Among the 15 average learned accuracies, the top 5 sub-feature sets can be selected.

As another example, the 15 sub-feature sets may be classified into a preset number of groups, and the group average learning accuracy of the corresponding groups may be calculated. A sub-feature set within a selected group may be selected by selecting some groups from among multiple groups based on the group average learned accuracy.

If 5 sub-feature sets are selected, 1000 indices are selected. Due to the selection of each sub-feature set, geographical features among the features generated by the CNN based on the spectrogram image can be automatically considered.

The description of steps ( 1610 to 1640 ) can be similarly applied to the detailed description of step 1540 .

FIG. 17 is a flowchart illustrating a method of verifying test object characteristics according to an example.

According to one aspect, the above step 1420 described with reference to FIG. 14 may include the following steps ( 1710 to 1730 ).

In step 1710, the electronic device 300 divides the test target feature set into K groups. Define the test target characteristics determined for each test user as a set. For example, if there are 1000 test users, there are 1000 test target feature sets, and the 1000 sets can be divided into K groups. K is a natural number of 2 or more. When K is 5, 5 groups can be generated, and each group includes 200 sets.

In step 1720, the electronic device 300 generates K test DNNs by respectively updating the initial DNNs based on the K groups. When generating the first group to the fifth group, the first test DNN can be updated using the second group to the fifth group; the second test DNN can be updated using the first group, the third group to the fifth group; , the second group, the fourth group and the fifth group to update the third test DNN; use the first group to the third group and the fifth group to update the fourth test DNN; and use the first group to the fourth group to update the first 5. Test DNN.

In step 1730, the electronic device 300 verifies the test target features based on the accuracies of the K test DNNs. In the above embodiments, by inputting the first group into the first test DNN, the results of the first group can be output, and the first accuracy of the output results can be calculated. Similarly, second to fourth accuracies may be calculated for each of the second to fourth test DNNs.

When the calculated average of the first to fifth accuracies is equal to or greater than a preset threshold, it may be determined that the test target feature has been verified. When the calculated average of the first to fifth accuracies is less than the preset threshold, it may be determined that the test target feature has not been verified. If the test target features are not verified, the CNN that extracts the test features can be re-updated.

FIG. 18 illustrates a K-fold cross-validation method for validating target features according to an example.

According to an example, the test target feature set 1810 may be divided into a first group 1801 , a second group 1802 , a third group 1803 , a fourth group 1804 and a fifth group 1805 . When the test target feature set 1810 includes 1000 sets, each of the groups ( 1801 to 1805 ) includes 200 sets. Each set includes test target characteristics for a particular test user.

The first test DNN 1820 may be updated with the second to fifth groups (1802 to 1805). For example, the first test DNN 1820 may be updated 800 times based on 800 sets.

An updated first test DNN 1820 may receive as input the first group 1801 to determine the degree of dementia of the first group 1802 of test users. For example, the first test DNN 1820 may determine 200 second test dementia levels of 200 sets.

The accuracy of the first test DNN 1820 can be calculated based on the GT dementia levels for each of the first set 1801 of 200 sets and the 200 second test dementia levels. Similarly, the accuracies of the second to fourth test DNNs can be calculated. Finally, the test target features can be verified based on the average accuracy of the first to fifth test DNNs.

The embodiments described above can be realized by hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments can utilize, for example, a processor, a controller, an arithmetic logic unit (arithmetic logic unit, ALU), a digital signal processor (digital signal processor), a microcomputer, a field programmable array ( field programmable array (FPA), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, capable of being embodied by more than one general purpose or special purpose computer . The processing device is capable of executing an operating system (OS) and one or more application software executing within the operating system. And, the processing means is responsive to the execution of software to access, store, manipulate, process and generate data. For ease of understanding, only one processing device is described, but those skilled in the art should understand that a processing device can include multiple processing elements (processing elements) and/or multiple types of processing elements. For example, a processing device can include multiple processors or a processor and a controller. Also, other processing configurations like parallel processors can also be included.

Software can include a computer program, code, instruction, or a combination of one or more of these, capable of causing a processing device to operate in a desired manner, or, individually or collectively, commanding a processing device . Software and/or data can be permanently or temporarily embodied (embodied) in any type of equipment, component, physical device, virtual device, A computer storage medium or device, or a transmitted signal wave. Software is distributed on computer systems connected via a network, enabling distributed storage or execution. Software and data can be stored in more than one computer read-write storage medium.

The methods according to the embodiments are embodied in the form of program commands that can be executed by various computer means, and are recorded in computer read-write media. The computer read-write medium can include program commands, data files, data structures, etc. in a single or combined form. The program instructions recorded in the medium can be instructions specially designed and constructed to implement the embodiments, or instructions that can be used by those skilled in the field of computer software based on known knowledge. Computer read-write recording media can include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media similar to CD-ROMs and DVDs; magneto-optical media similar to floptical disks ( magneto-optical media), and hardware devices specially configured to store and execute program commands, such as read-only memory (ROM), random-access memory (RAM), flash memory, etc. Examples of program instructions include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer by using an interpreter or the like. In order to implement the operations of the embodiments, the hardware device can be configured in such a way that more than one software module can implement the operation, and vice versa.

To sum up, the embodiments have been described through limited embodiments and drawings, and those skilled in the art can make various modifications and variations to the above descriptions. For example, the described techniques are performed in a different order than the described method, and/or the described components are combined or combined in a different form than the described method, or are replaced by other components or equivalents or Replacement can also achieve the same effect.

Therefore, other embodiments, other embodiments, and equivalents of the scope of the patent application all belong to the scope of the patent claims.

110: Electronic device

120: user terminal

130: Monitoring terminal

210: first image

220: Second image

230: Third image

240: The fourth image

300: electronic device

310: Department of Communications

320: Processor

330: memory

410~440,710~730,800,805,810~890,895,1300,1310~1350,1400,1410~1440,1450,1510~1540,1610~1640,1710~1730: steps

500: content

510: instruction

520, 530, 540: image

600: Original spectrogram image

610: Legend

900: all features

910: target features

1010: input layer

1020, 1030, 1040: hidden layer

1050: output layer

1210~1230: steps

1801: The first group

1802: The second group

1803: The third group

1804: The fourth group

1805: Fifth group

1810: Test target feature set

1820: First test of DNN

FIG. 1 is a block diagram illustrating a system for determining a user's degree of dementia, according to an example. FIG. 2 illustrates an image output to a user terminal to determine a user's dementia degree according to an example. FIG. 3 is a block diagram illustrating an electronic device for determining a degree of dementia of a user according to an embodiment. FIG. 4 is a flow chart illustrating a method for determining the degree of dementia of a user according to an embodiment. FIG. 5 illustrates content pre-produced to receive user voice according to an example. Fig. 6 shows a raw spectrogram image generated for speech according to an example. FIG. 7 is a flowchart illustrating a method of determining a degree of dementia of a user using a CNN and a DNN according to an example. Fig. 8 shows a full CNN and a partial CNN capable of determining the degree of dementia of a user according to an example. FIG. 9 illustrates generated features for each of multiple sets of user images and target features determined based thereon, according to an example. FIG. 10 illustrates a DNN for determining a user's degree of dementia, according to an example. Figure 11 shows a classification of two steps performed to improve the accuracy of determining the degree of dementia, according to an example. FIG. 12 illustrates a two-step operation performed to improve the accuracy of determining the degree of dementia according to another example. FIG. 13 is a flowchart illustrating a method for updating a complete CNN according to an example. FIG. 14 is a flowchart illustrating a method of updating a DNN according to an example. FIG. 15 is a flowchart illustrating a method for determining test target characteristics according to an example. FIG. 16 is a flowchart illustrating a method of selecting sub-features according to an example. FIG. 17 is a flowchart illustrating a method of verifying test object characteristics according to an example. FIG. 18 illustrates a K-fold cross-validation method for validating target features according to an example.

410~440: Steps

Claims (15)

A method for determining the degree of dementia of a user performed by an electronic device, including the following steps: outputting, through a user terminal, first content pre-produced for determining the degree of dementia of the user; the first voice of the first content acquired by the microphone; output the pre-made second content through the user terminal; receive the second voice of the user for the second content acquired through the microphone; electronically The device generates a first spectrogram image by visualizing at least one feature of the first speech; the electronic device generates a second spectrogram image by visualizing at least one feature of the second speech; The device generates a preset number of first features for the first speech by inputting the first spectrogram image into a pre-updated first convolutional neural network (CNN); The second spectrogram image is input to the pre-updated second convolutional neural network to generate a preset number of second features for the second speech; use the electronic device to compare the first features and the second features Determine a preset number of target features in ; and By inputting the target features into a pre-updated deep neural network (DNN), the degree of dementia of the user is determined, wherein the determined degree of dementia is output through the user terminal. The method for determining the degree of dementia of a user according to claim 1, wherein the first content includes an instruction for receiving the first voice. The method for determining the degree of dementia of the user as claimed in claim 2, wherein the first content includes the content of making the user read the sentence, the content of guessing the name of the output image, the content of describing the output image, and the content used for language fluency Sexual content, content for number crunching, and content that induces storytelling. The method for determining the degree of dementia of a user according to claim 1, wherein the step of generating a first spectrogram image by visualizing at least one feature of the first speech includes the following steps: generating the first spectrogram image through a librosa tool The first spectrogram image of speech. The method for determining the degree of dementia of a user according to claim 1, wherein the size of the first spectrogram image and the size of the second spectrogram image are the same as each other. The method for determining the degree of dementia of a user according to claim 1, wherein the first convolutional neural network is pre-updated based on the VGG16 model. The method for determining the degree of dementia of a user as claimed in claim 1, wherein the first convolutional neural network includes an input layer, 5 pre-convolutional layer blocks (pre-convolutional layer blocks), a fully connected layer and 2 post- Convolutional layer blocks (post- convolutional layer blocks) and do not include softmax to generate the first feature of the first spectrogram image. The method for determining the degree of dementia of the user according to claim 1, further comprising a step of updating the first convolutional neural network with the electronic device. The method for determining the degree of dementia of a user according to claim 8, wherein the step of updating the first convolutional neural network with the electronic device includes the following steps: receiving the first content of the test user for the first content Test speech; generate a first test spectrogram image by visualizing at least one feature of the first test speech, wherein the first test spectrogram image is marked as the test user's GT (ground truth) dementia degree ; determining a first test dementia level of the test user by inputting the first test spectrogram image into a first full convolutional neural network, wherein the first full convolutional neural network includes an input layer , one or more pre-convolutional layer blocks, a fully connected layer, one or more post-convolutional layer blocks, and softmax; and updating the complete first convolutional neural network based on the first test dementia degree and the GT dementia degree , wherein the first convolutional neural network includes only the input layer, the one or more pre-convolutional layer blocks, the fully connected layer, and the layers of the updated complete first convolutional neural network. Describe one or more post-convolutional layer blocks. The method for determining the degree of dementia of a user according to claim 9, wherein, It also includes performing the following step with the electronic device: updating the deep neural network after updating a plurality of convolutional neural networks including the first convolutional neural network and the second convolutional neural network. The method for determining the degree of dementia of a user according to claim 10, wherein the step of updating the deep neural network with the electronic device includes the following steps: in the first preset number of generated based on the first test spectrogram image Determining a preset number of test target features among a test feature and a preset number of second test features generated based on the second test spectrogram image, wherein the test target feature is marked as GT dementia of the test user degree; determining a second test dementia degree of the test user by inputting the test target feature into the deep neural network; and updating the depth based on the second test dementia degree and the GT dementia degree Neural Networks. A computer-readable recording medium storing a program for executing the method of any one of claims 1 to 11. A device for determining the degree of dementia of a user, including: a memory recording a program for determining the degree of dementia of the user; and a processor for executing the program, wherein the program performs the following steps: pre-produced first content for determining the degree of dementia of the user; receiving the first voice of the user for the first content acquired through the microphone of the user terminal; outputting the pre-made second content through the user terminal; receiving the user's voice for the first content acquired through the microphone A second speech of the second content; generating a first spectrogram image by visualizing at least one characteristic of the first speech; generating a second spectrogram image by visualizing at least one characteristic of the second speech; generating a second spectrogram image by visualizing at least one characteristic of the second speech; The first spectrogram image is input to a pre-updated first convolutional neural network (CNN), generating a preset number of first features for the first speech; by inputting the second spectrogram image into The pre-updated second convolutional neural network generates a preset number of second features for the second speech; determines a preset number of target features among the first features and the second features; The target features are input to a pre-updated deep neural network (DNN) to determine the dementia degree of the user, wherein the determined dementia degree is output through the user terminal. A method performed by an electronic device for updating a convolutional neural network for determining a degree of dementia in a user, comprising the steps of: Outputting pre-made first content for determining the degree of dementia of the user through the user terminal; receiving a first test voice of a test user for the first content; generating a second test voice by visualizing at least one feature of the first test voice A test spectrogram image, wherein the first test spectrogram image is marked as the GT dementia level of the test user; by inputting the first test spectrogram image into a complete convolutional neural network determining a test dementia level for the test user, wherein the complete convolutional neural network includes an input layer, one or more pre-convolutional blocks, a fully connected layer, one or more post-convolutional blocks, and softmax; and based on the test The degree of dementia and the degree of GT dementia are used to update the complete convolutional neural network, wherein the convolutional neural network only includes the input layer, the one of the layers of the updated complete convolutional neural network The above pre-convolutional layer blocks, the fully-connected layer, and the one or more post-convolutional layer blocks. An electronic device for updating a convolutional neural network for determining a degree of dementia in a user, comprising: a memory recording a program for updating the convolutional neural network; and a processor for executing the program, wherein the The processor performs the following steps: Outputting pre-made first content for determining the degree of dementia of the user through the user terminal; receiving a first test voice of a test user for the first content; generating a second test voice by visualizing at least one feature of the first test voice A test spectrogram image, wherein the first test spectrogram image is marked as the GT dementia level of the test user; by inputting the first test spectrogram image into a complete convolutional neural network determining a test dementia level for the test user, wherein the complete convolutional neural network includes an input layer, one or more pre-convolutional blocks, a fully connected layer, one or more post-convolutional blocks, and softmax; and based on the test The degree of dementia and the degree of GT dementia are used to update the complete convolutional neural network, wherein the convolutional neural network only includes the input layer, the one of the layers of the updated complete convolutional neural network The above pre-convolutional layer blocks, the fully-connected layer, and the one or more post-convolutional layer blocks.

Images