KR102475750B1 - Device and method for trauma screening in voice based on deep-learning - Google Patents

Abstract

The present invention relates to an apparatus and method for voice-based trauma screening using deep learning. According to an embodiment of the present invention, a voice-based screening apparatus and method using deep learning can screen trauma through a voice that can be obtained in a non-contact manner without restrictions of space or situation.

Description

Voice-based trauma screening apparatus and method using deep learning {DEVICE AND METHOD FOR TRAUMA SCREENING IN VOICE BASED ON DEEP-LEARNING}

The present invention relates to an apparatus and method for voice-based trauma screening using deep learning, and more particularly, to an apparatus and method for voice-based trauma screening using deep learning that recognizes emotions and infers the possibility of trauma by deep learning voice data. It is about.

Today, modern people are exposed to various stresses ranging from everyday events such as study and employment to serious events such as traffic accidents and crimes. In medical terms, trauma is called post-traumatic stress disorder (PTSD), and it refers to psychological injuries caused by external traumatic events. People who experience trauma have difficulty regulating and stabilizing their emotions, and spontaneous recovery rates are as high as 60% or more within one year of trauma, but drop rapidly thereafter. Therefore, early treatment within 1 year after experiencing trauma is very important for recovery from traumatic sequelae. For initial treatment, it is essential to visit a hospital and consult to diagnose trauma, but diagnosis and treatment often fail because social prejudice against mental illness delays treatment or fails to recognize trauma.

In recent years, deep learning is being used to combine engineering techniques with the medical field to help doctors make early diagnoses. In particular, voice is widely used because it contains emotions and intentions that are effective in identifying the patient's emotions and can be obtained in a non-contact manner in a natural environment without the patient feeling repulsive. In addition, many studies using voice, such as age classification and emotion recognition, are being conducted, but trauma screening studies using voice analysis have not been conducted.

The background art of the present invention is published in Korean Patent Registration No. 10-189765.

The present invention provides a voice-based trauma screening apparatus and method using deep learning that screens trauma using voice, which can be acquired relatively without objection compared to images in a non-contact manner.

The present invention provides a voice-based trauma screening apparatus and method using deep learning for trauma screening by converting voice data into image data.

The present invention provides a voice-based trauma screening apparatus and method using deep learning in which the emotion of voice is recognized through deep learning and then the accuracy of recognition is increased through post-processing.

The present invention provides an apparatus and method for voice-based trauma screening using deep learning, which is helpful in trauma diagnosis, in which emotions can be conveniently recognized with only voice, even if it is not in a specific situation or space.

According to one aspect of the present invention, a voice-based trauma screening device using deep learning is provided.

An audio-based trauma screening apparatus using deep learning according to an embodiment of the present invention includes an input unit for acquiring audio data, a pre-processing unit for editing audio data, a conversion unit for converting edited audio data into image data, and image data. It may include a deep learning unit that recognizes emotion and a determination unit that post-processes the result value of the deep learning unit.

According to another aspect of the present invention, a voice-based trauma screening method using deep learning and a computer-readable recording medium on which a computer program executing the method is recorded are provided.

According to an embodiment of the present invention, a voice-based trauma screening method using deep learning and a recording medium storing a computer program executing the same include steps of acquiring voice data, preprocessing the voice data, and converting the preprocessed voice data into image data. It may include converting to , performing deep learning on image data, and post-processing the result of deep learning.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to an embodiment of the present invention, a voice-based screening apparatus and method using deep learning can screen trauma through a voice that can be obtained in a non-contact manner without restrictions of space or situation.

According to an embodiment of the present invention, emotion can be recognized by deep learning by converting voice data into image data, and post-processing can increase the accuracy of trauma screening.

1 is a block diagram of an apparatus for voice-based trauma screening using deep learning according to an embodiment.
2 is a diagram for explaining a method of converting voice data into image data according to an exemplary embodiment.
3 is a diagram illustrating an example of image data according to an exemplary embodiment.
4 is a diagram illustrating an example of a deep learning model according to an embodiment.
5 is a diagram for explaining a result value of a deep learning model according to a window size according to an embodiment.
6 is a diagram for explaining a voice-based trauma screening method using deep learning according to an embodiment.
7 is a flowchart of a voice-based trauma screening method using deep learning according to an embodiment.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail through detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. Also, as used in this specification and claims, the terms "a" and "an" are generally to be construed to mean "one or more" unless stated otherwise.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. do it with

1 to 5 are diagrams for explaining an apparatus for voice-based trauma screening using deep learning according to an embodiment of the present invention.

1 is a block diagram of an apparatus for voice-based trauma screening using deep learning according to an embodiment.

Referring to FIG. 1 , the speech-based trauma screening apparatus 10 using deep learning includes an input unit 100, a pre-processing unit 200, a conversion unit 300, a deep learning unit 400, and a determination unit 400. include

The voice-based trauma screening device 10 using deep learning screens trauma with voice data. The voice-based trauma screening device 10 using deep learning screens for trauma rather than determining the presence or absence of trauma, and can acquire voice without feeling repulsive in a natural environment in a non-contact manner.

The voice-based trauma screening device 10 using deep learning recognizes four emotions of happiness, neutrality, sadness, and fear from voice data and screens trauma. The reason why the voice-based trauma screening device using deep learning (10) uses 4 emotions is that many people feel a lot of fear when trauma occurs, and the intensity of sadness increases over time after trauma and often develops into depression. Because. At the beginning of the trauma, feelings of fear, sadness, surprise, and anger are prominent, and over time, the anger weakens and the fear and sadness intensify. The voice-based trauma screening apparatus 10 using deep learning assumes that the trauma probability is high when fear and sad emotions are recognized in voice data, and the trauma probability is low when neutral and happy emotions appear, and screens for trauma.

The input unit 100 receives voice and generates voice data or receives and obtains voice data.

2 is a diagram for explaining a method of converting voice data into image data according to an exemplary embodiment.

Referring to FIG. 2 , the preprocessor 200 eliminates a length difference between data and increases the number of audio data in order to convert acquired audio data into image data. For example, if the pre-processing unit 200 shifts by 0.1 second units, it edits by 2 second units.

The conversion unit 300 converts audio data into image data. In detail, the transform unit 300 converts one-dimensional audio data into two-dimensional spectrogram image data using a short-time Fourier transform (STFT) spectrogram. For example, the transform unit 300 designates the number of samples per second as 1024 for the preprocessed voice data, performs Fast Fourier Transform (FFT, Fast Fourier Transform), overlaps by 512 samples, and moves ( shift).

The conversion unit 300 scales all image data values to be between 0 and 1 using a min-max scaler. The min-max scaler standardizes data to a range between '0 and 1' using the minimum value (Min) and maximum value (Max).

3 is a diagram illustrating an example of image data according to an exemplary embodiment.

Referring to FIG. 3 , a result of the transformation unit 300 converting audio data into spectrogram image data can be confirmed.

The deep learning unit 400 uses the spectrogram image data as an input value of the deep learning model and learns to recognize emotions.

The deep learning unit 400 learns using a Visual Geometry Group-13 (VGG-13) model among convolutional neural network (CNN) models. The deep learning unit 400 uses a Korean voice data set created by extracting voices containing six basic emotions (happiness, sadness, disgust, anger, fear, and surprise) from domestic broadcasting and movies. In the Korean voice data set, each voice data has a length of 2 to 11 seconds, and there are a total of about 600 voice data for each emotion. The deep learning unit 400 learns using only voice data of four emotions of fear, sadness, neutral, and happiness among Korean voice data sets.

4 is a diagram illustrating an example of a deep learning model according to an embodiment.

Referring to FIG. 4, the deep learning unit 400 includes 10 convolutional layers (Conv-layer, convolutional layer) of a 3 * 3 kernel, 5 max pooling layers, and 3 fully connected layers (Fully connected layer). For example, the deep learning unit 400 uses spectrogram image data having a size of 288*432*3 as an input value. The deep learning unit 400 may perform a maximum pooling layer of a 2*2 kernel after two 3*3 convolutional layers (Conv-layer, convolutional layer). Then, the deep learning unit 400 outputs the trauma screening value as a binary classification value of 0 or 1 through a fully connected layer. That is, the deep learning unit 400 outputs a case in which there is a high probability of trauma or a case in which there is no trauma by binary classification.

The determination unit 500 increases accuracy by processing the result of the deep learning unit 400 later. When the result of the deep learning unit 400 is maintained constant for a certain period of time, the determination unit 500 screens for trauma to improve the reliability of the final result. For example, the determination unit 500 may preset a window size of 2-10. When the result of the deep learning unit 400 is maintained at 0 or 1 as much as the set window size, the determination unit 500 finally screens for trauma.

5 is a diagram for explaining a result value of a deep learning model according to a window size according to an embodiment.

Referring to FIG. 5 , the determination unit 500 sets the window size to 4 to 8 and extracts only the resultant value when the value is maintained as much as the set window size. In this way, it can be confirmed that the accuracy of the emotion recognized by the determination unit 500 in the voice data is 100%. A window size of 1 is a case where the determination unit 500 does not perform post-processing, and the accuracy is 99.03. However, when the window size is set to 4 to 8 and the determination unit 500 performs post-processing, it can be confirmed that the accuracy is 100%. have. In detail, the voice-based trauma screening device 10 using deep learning uses voice for 400 to 800 ms instead of 100 ms voice to increase accuracy in recognizing accurate emotions from voice data. The voice-based trauma screening device 10 using deep learning may determine the corresponding emotion when the voice data maintains the same emotion for 400 to 800 ms. Therefore, the voice-based trauma screening apparatus 10 using deep learning uses only the resultant value when the window size is 4 to 8.

6 is a diagram illustrating a voice-based trauma screening method using deep learning according to an embodiment.

Each process described below is a process performed by each functional unit constituting a voice-based trauma screening device using deep learning, but for a concise and clear description of the present invention, the subject of each step is voice-based trauma screening using deep learning Let's call it a device.

In step S610, the voice-based trauma screening apparatus 10 using deep learning acquires voice data required for emotion recognition. The voice-based trauma screening apparatus 10 using deep learning receives voice directly and generates voice data or receives voice data and acquires voice data.

In step S620, the voice-based trauma screening apparatus 10 using deep learning pre-processes the input voice data to be suitable for emotion recognition. The voice-based trauma screening apparatus 10 using deep learning edits the length of voice data to be the same and increases the number of voice data (augmentation). For example, the voice-based trauma screening apparatus 10 using deep learning cuts voice data in 2s units while shifting in units of 0.1s.

In step S630, the voice-based trauma screening apparatus 10 using deep learning converts the pre-processed voice data into image data. In detail, the voice-based trauma screening device 10 using deep learning converts one-dimensional voice data edited in 2s units to two-dimensional spectrogram image data using Short Time Fourier Transform (STFT). convert to For example, the voice-based trauma screening device 10 using deep learning specifies the number of samples per second as 1024 for preprocessed voice data, performs Fast Fourier Transform (FFT), and overlaps by 512 samples. ) and shift. Then, scaling is performed through a min-max scaler so that all data values are between 0 and 1.

In step S640, the voice-based trauma screening apparatus 10 using deep learning deep-learns the scaled spectrogram image data. The speech-based trauma screening apparatus 10 using deep learning performs emotion recognition using a convolutional neural network (CNN) model, a Visual Geometry Group-13 (VGG-13) model. The deep learning model is described in FIG. 4 .

In step S650, the speech-based trauma screening apparatus 10 using deep learning increases the accuracy of speech-based emotion recognition by post-processing the result value obtained using the deep learning model. In the voice-based trauma screening device 10 using deep learning, when the window size is 4 to 8, the accuracy of voice-based emotion recognition is 100%, so the deep learning result value corresponds to the window size 4 to 8 In this case, the result is used as the result of trauma screening.

7 is a flowchart of a voice-based trauma screening method using deep learning according to an embodiment.

Since the method shown in FIG. 7 is related to the embodiments described in the drawings described above, even if the content is omitted below, the content described in the drawings may also be applied to the method of FIG. 10 .

Referring to FIG. 7 , in step 710, the processor may acquire voice data.

In the voice data, not only content information but also emotions of the speaker may be reflected.

In step 720, the processor may pre-process the voice data.

In one embodiment, the voice data may be pre-processed by shifting the voice data by a predetermined time unit so that the voice data becomes data having a predetermined length. The processor may eliminate a length difference between voice data and increase the number of voice data through a preprocessing process.

In step 730, the processor may convert the preprocessed voice data into image data.

The processor may perform short-time Fourier transform on the preprocessed voice data to generate 2D data and utilize the generated 2D data as image data.

In step 740, the processor may input the image data to the deep learning model and obtain a trauma result value as an output value of the deep learning model.

If the trauma result value is 1, the probability that the speaker of the corresponding voice data has trauma is high, and if the trauma result value is 0, the probability that the speaker of the corresponding voice data has trauma may be low.

In one embodiment, the processor may input image data to a deep learning model and obtain an emotion result value as an output value of the deep learning model. Emotion result values may be classified into a first emotion classification or a second emotion classification. For example, the first emotion classification may include neutral and happy, which have a low relationship with trauma, and the second emotion classification may include fear and sadness, which have a high relationship with trauma. , anger, and surprise, but the first emotion classification and the second emotion classification are not limited to the above example.

In one embodiment, the processor may determine the stage of trauma based on the emotional result value obtained as the output value of the deep learning model. Emotion result values may be classified into a first emotion classification, a second emotion classification, and a third emotion classification.

For example, the first emotion classification may include neutral and happy, which are not related to trauma, and the second emotion classification includes fear, surprise ( surprise), and the third emotion classification may include sadness and anger that become prominent after a predetermined period of time has elapsed after the occurrence of trauma.

For example, when the emotion result obtained as the output value of the deep learning model is the second emotion classification, the processor may determine that the trauma stage of the speaker of the voice data is an initial stage.

8 is a block diagram of a device according to an embodiment.

Referring to FIG. 8 , the device 1100 may include a communication unit 1110, a processor 1120, and a DB 1130. In the device 1100 of FIG. 8 , only components related to the embodiment are shown. Accordingly, those skilled in the art can understand that other general-purpose components may be further included in addition to the components shown in FIG. 8 .

The communication unit 810 may include one or more components that enable wired/wireless communication with an external server or external device. For example, the communication unit 810 may include at least one of a short-range communication unit (not shown), a mobile communication unit (not shown), and a broadcast reception unit (not shown).

The DB 830 is hardware for storing various data processed in the device 800, and may store programs for processing and controlling the processor 820.

The DB 830 includes random access memory (RAM) such as dynamic random access memory (DRAM) and static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and CD-ROM. ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

The processor 820 controls the overall operation of the device 800. For example, the processor 820 may generally control an input unit (not shown), a display (not shown), a communication unit 810, and the DB 830 by executing programs stored in the DB 830. The processor 820 may control the operation of the lane determining device 800 by executing programs stored in the DB 830 .

The processor 820 may control at least some of the operations of the lane determining device described above with reference to FIGS. 1 to 7 .

The processor 820 may include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, and microcontrollers. It may be implemented using at least one of micro-controllers, microprocessors, and electrical units for performing other functions.

Meanwhile, the processor 820 may include a data learning unit and a data recognizing unit that perform learning and inference of a deep learning model.

The data learning unit may learn criteria for determining a situation. The data learning unit may learn criteria for what kind of data to use to determine a predetermined situation and how to determine a situation using the data. The data learning unit may acquire data to be used for learning and learn criteria for determining a situation by applying the obtained data to a data recognition model to be described later.

The data recognizing unit may determine a situation based on the data. The data recognizer may recognize a situation from predetermined data using the learned data recognition model. The data recognizing unit may acquire predetermined data according to a predetermined criterion by learning, and determine a predetermined situation based on the predetermined data by using a data recognition model using the acquired data as an input value. In addition, result values output by the data recognition model using the acquired data as input values may be used to update the data recognition model.

At least one of the data learning unit and the data recognizing unit may be manufactured in the form of at least one hardware chip and mounted in an electronic device. For example, at least one of the data learning unit and the data recognizing unit may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or a conventional general-purpose processor (eg, CPU or application processor) or graphics It may be manufactured as a part of a dedicated processor (eg GPU) and mounted in various electronic devices described above.

In this case, the data learning unit and the data recognizing unit may be mounted on one electronic device (eg, lane determining device) or may be mounted on separate electronic devices. For example, one of the data learning unit and the data recognizing unit may be included in the lane determining device, and the other may be included in the server. In addition, the data learning unit and the data recognizing unit may provide model information built by the data learning unit to the data recognizing unit through wired or wireless communication, and data input to the data recognizing unit may be provided to the data learning unit as additional learning data.

Embodiments according to the present invention may be implemented in the form of a computer program that can be executed on a computer through various components, and such a computer program may be recorded on a computer-readable medium. At this time, the medium is a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a ROM hardware devices specially configured to store and execute program instructions, such as RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. An example of a computer program may include not only machine language code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like.

According to one embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store™) or between two user devices. It can be distributed (e.g., downloaded or uploaded) directly or online. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

The steps constituting the method according to the present invention may be performed in any suitable order unless an order is explicitly stated or stated to the contrary. The present invention is not necessarily limited according to the order of description of the steps. The use of all examples or exemplary terms (eg, etc.) in the present invention is simply to explain the present invention in detail, and the scope of the present invention is limited due to the examples or exemplary terms unless limited by the claims. it is not going to be In addition, those skilled in the art can appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments and should not be determined, and all scopes equivalent to or equivalently changed from the claims as well as the claims described below are within the scope of the spirit of the present invention. will be said to belong to

10: Voice-based trauma screening device using deep learning
100: input unit
200: pre-processing unit
300: conversion unit
400: deep learning unit
500: judgment unit

Claims (5)

In the voice-based trauma screening device using deep learning,
a memory in which at least one program is stored; and
a processor that performs calculations by executing the at least one program;
the processor,
Acquiring voice data;
pre-processing the voice data;
converting the preprocessed audio data into image data;
Inputting the image data to a deep learning model, obtaining a trauma result value as an output value of the deep learning model,
the processor,
As the output value of the deep learning model, an emotion result value including a first emotion classification, a second emotion classification, and a third emotion classification is obtained, and the emotion result value is obtained only when a predetermined emotion classification is maintained over a predetermined window size. to obtain,
When the emotion result value indicates at least one of a second emotion classification and a third emotion classification, it is determined that trauma exists;
and determining that the stage of trauma is an initial stage when the emotional result value is a second emotion classification, and that the stage of trauma is an initial stage or later when the emotional result value is a third emotion classification. According to claim 1,
the processor,
and pre-processing the voice data by shifting the voice data by a predetermined time unit so that the voice data becomes data having a predetermined length. According to claim 1,
the processor,
generating two-dimensional data by short-time Fourier transforming the preprocessed voice data;
Utilizing the two-dimensional data as image data and inputting the deep learning model to the apparatus. In the voice-based trauma screening method using deep learning,
acquiring voice data;
pre-processing the voice data;
converting the preprocessed audio data into image data; and
inputting the image data to a deep learning model, and obtaining a trauma result value as an output value of the deep learning model;
Including,
The obtaining step is
As the output value of the deep learning model, an emotion result value including a first emotion classification, a second emotion classification, and a third emotion classification is obtained, and the emotion result value is obtained only when a predetermined emotion classification is maintained over a predetermined window size. obtaining;
determining that trauma exists when the emotion result value indicates at least one of a second emotion classification and a third emotion classification; and
determining that the trauma stage is initial when the emotion result value is a second emotion classification, and that the trauma stage is an initial or later trauma stage when the emotion result value is a third emotion classification;
Including, method. A computer-readable recording medium recording a program for executing the method of claim 4 on a computer.

Images