KR20220106029A - Method and apparatus for psychological counselingusing artificial intelligence - Google Patents

Abstract

The present invention relates to a psychological counseling method using artificial intelligence and a device thereof which receive image and audio data of a user, measure biorhythms of the user in the image data received through a first deep learning model, measure and classify an emotion in the image and audio data through a second deep learning model, perform analysis based on an emotion measurement index and text through text data extracted from the audio data and survey response result text data inputted from the user, and derive a comprehensive psychological state result based on the measured biorhythms, a classified emotion and a text analysis result.

Description

Psychological counseling method and apparatus using artificial intelligence {Method and apparatus for psychological counselingusing artificial intelligence}

Embodiments of the present invention relate to a psychological counseling method and apparatus using artificial intelligence for identifying and giving advice to a speaker's actual emotional state by using artificial intelligence technology.

In general, in the conventional psychological counseling, the user discusses the current daily life and concerns with a psychological counselor through one-on-one face-to-face counseling with a psychological counselor or a psychotherapist, while the counselor analyzes the client's feelings and psychological state. After identifying them, they provide advice and jointly resolve their concerns and carry out treatment. However, such face-to-face counseling takes a lot of time, there is a burden of having to meet in person, and there is a considerable cost burden.

Accordingly, various studies are being conducted to identify a person's emotional state in real time through a smart phone or a computing device and provide a customized service accordingly. As one example, as a method using a human face image, a facial image including various emotions is trained using artificial intelligence to derive a probability of a specific emotion for the human face image.

However, emotion analysis using such a person's face image derives a result value using only one extracted image. However, studies that try to identify facial expressions by images have a problem in that it is difficult to grasp the appearance and expression at the moment of speaking. There was also the problem of being difficult to predict.

In addition, even if voice data is added, facial expressions and tones of voice are artificially adjusted, so the data such as emotional restraint or poker face that shows different facial expressions and actual emotional states are not well predicted. Since emotion analysis is stopped at the level of performing classification for a specific emotion, it is impossible to grasp the cause of the emotional state and the degree of ups and downs of the emotion as the details of the actual speaker's emotion.

The above-mentioned background art is technical information possessed by the inventor for the derivation of the present invention or acquired in the process of derivation of the present invention, and cannot necessarily be said to be a known technique disclosed to the general public prior to the filing of the present invention.

Republic of Korea Patent Publication No. 2020-0085696

An embodiment of the present invention for solving the problems of the prior art provides a psychological counseling method and apparatus using artificial intelligence for identifying and giving advice to a speaker's actual emotional state using artificial intelligence technology.

Another embodiment of the present invention utilizes deep learning technology, which is a field of artificial intelligence, to derive emotion, biorhythm, and text analysis result values using the user's image and voice data, and through questionnaire data It provides a psychological counseling method and apparatus using artificial intelligence to analyze the emotional state of the user along with the result value obtained as a psychological value to understand the actual emotional state of the speaker and give advice.

One aspect of the present invention, the step of receiving a user's video and audio data; Measuring the user's biorhythm from the image data input through the first deep learning model; measuring and classifying emotions in the video and audio data through a second deep learning model; performing an emotion measurement index and text-based analysis based on the text data extracted from the voice data and the text data of the questionnaire response input from the user; And based on the measured biorhythm, classification emotion, and text analysis results, a comprehensive psychological state result is derived.

In addition, the measuring of the biorhythm may include predicting one of six emotions from the image data; and calculating a numerical value for the predicted emotion and calculating a heart rate and blood flow corresponding to the calculated numerical value.

In addition, the step of measuring and classifying the emotion is characterized in that it is calculated by predicting the emotional state through the facial expression and tone of voice when speaking from the video and audio data.

In addition, the step of performing the analysis based on the emotion measurement index and the text may include: calculating a corresponding emotion measurement index according to a survey response result; calculating a correlation value between texts included in a sentence based on the moving image data extraction text data and the questionnaire response result text data; and outputting a psychological state analysis result based on the emotion measurement index and the correlation value.

In addition, the text data as a result of the questionnaire response may include: transmitting a psychological counseling questionnaire from a computing device to a user terminal; receiving text data as a response to the questionnaire from the user terminal; and calculating a text correlation index for the text data and the text data extracted from the video.

Another aspect of the present invention includes an input unit for receiving the user's image and audio data, a biorhythm prediction result calculation unit for measuring the user's biorhythm from the image data input through the first deep learning model, and a second deep learning model An emotional state prediction result calculation unit for measuring and classifying emotions from the video and audio data through a model, an emotion measurement index and text through the text data extracted from the voice data and the questionnaire response result text data input from the user It includes a psychological state analysis result calculation unit for performing a basic analysis and a psychological state result derivation unit for deriving a comprehensive psychological state result based on the measured biorhythm, classification emotion, and text analysis results.

In addition, the biorhythm prediction result calculator predicts one of six emotions from the image data, calculates a numerical value for the predicted emotion, and calculates a heart rate and blood flow corresponding to the calculated numerical value.

In addition, the emotional state prediction result calculation unit predicts and calculates the emotional state based on the facial expression and voice tone when speaking from the video and audio data.

In addition, the psychological state analysis result calculation unit calculates a corresponding emotion measurement index according to a questionnaire response result, and calculates a correlation value between texts included in a sentence based on the moving image data extraction text data and the questionnaire response result text data and output a psychological state analysis result based on the emotion measurement index and correlation value.

In addition, the text data as a result of the questionnaire response is transmitted from a computing device to a user terminal for psychological counseling, receives text data in response to the questionnaire from the user terminal, and text data extracted from the text data and the video. Calculate the text correlation index for .

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

A psychological counseling method and apparatus using artificial intelligence according to an embodiment of the present invention accurately predicts the actual emotional state and psychological state based on the user's biological rhythm, measured emotions, and survey results, and based on the predicted values, the user psychological advice may be available.

In addition, users can not only know their emotional state and biorhythm, but also receive psychological advice and receive psychological relief services. Through this, people who are subjected to emotional labor or psychological stress can receive self-diagnosis in real time rather than in a hospital or face-to-face with people.

1 is a diagram illustrating a psychological counseling system using artificial intelligence according to an embodiment of the present invention.
2 is a diagram specifically illustrating a computing device for psychological counseling according to an embodiment of the present invention.
3 is a flowchart illustrating a psychological counseling method using artificial intelligence according to an embodiment of the present invention.
4 is a diagram illustrating data input to a deep learning model and output results according to an embodiment of the present invention.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. Effects and features of the present invention, and a method of achieving them, will become apparent with reference to the embodiments described below in detail in conjunction with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when described with reference to the drawings, the same or corresponding components are given the same reference numerals, and the overlapping description thereof will be omitted. .

In the following embodiments, terms such as first, second, etc. are used for the purpose of distinguishing one component from another, not in a limiting sense.

In the following examples, the singular expression includes the plural expression unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have means that the features or components described in the specification are present, and the possibility that one or more other features or components may be added is not excluded in advance.

In the following embodiments, when it is said that a part such as a film, region, or component is on or on another part, not only when it is directly on the other part, but also another film, region, component, etc. is interposed therebetween. Including cases where there is

In the drawings, the size of the components may be exaggerated or reduced for convenience of description. For example, since the size and thickness of each component shown in the drawings are arbitrarily indicated for convenience of description, the present invention is not necessarily limited to the illustrated bar.

An object of the present invention is to provide a method and apparatus for identifying and giving advice to a speaker's actual emotional state using artificial intelligence technology.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a psychological counseling system using artificial intelligence according to an embodiment of the present invention.

Referring to FIG. 1 , the psychological counseling system 100 is for performing psychological advice based on data received from a user, and may include a user terminal 200 and a computing device 300 . The user terminal 200 may include an input unit 202 , a control unit 204 , a data transmission/reception unit 206 , a display unit 208 , a storage unit 210 , and the like. The user terminal 200 may be connected to the computing device 300 through a wired/wireless network to transmit/receive data through the data transmission/reception unit 206 . In addition, the user terminal 200 may receive video data including video and audio data for the user through the input unit 202 . The moving image data may be photos and images taken for a user's daily time zone and psychological counseling, and the input data is stored in the storage unit 210 and a computing device through the data transmission/reception unit 206 . may be transmitted to 300 .

The control unit 204 controls the components of the user terminal 200, can perform video or voice counseling with a psychological counseling expert connected through the computing device 300, and can transmit and receive a document or image questionnaire. have. The display unit 208 may reproduce the input video or output the contents of consultation with a psychological counseling expert.

The computing device 300 may include a data input unit 302 , a preprocessor 304 , a control unit 306 , an output unit 308 , and the like. The computing device 300 may be connected to at least one user terminal 200 , and receives data from the connected user terminal 200 in the data input unit 302 , and receives data from the user terminal 200 in the preprocessor 304 . In order to transmit one data to the controller 306 , image data, audio data, and text data may be extracted.

The control unit 306 controls each component of the computing device 300 , analyzes the biological rhythm, emotional state, and psychological state based on the user's image, voice, and text data, and based on the performed result, the user's By deriving the final mental state result, the derived result may be provided through the output unit 308 , or feedback on advice matching the derived result may be performed.

On the other hand, although the psychological counseling system 100 in FIG. 1 is composed of at least one user terminal 200 and a computing device 300 , the user directly transmits data to the computing device 300 rather than the user terminal 200 . You can also enter That is, the computing device 300 may be a personal computer, a notebook computer, a tablet computer, or the like, and may include a plurality of interlocked computers, servers, and clouds.

2 is a diagram specifically illustrating a computing device for psychological counseling according to an embodiment of the present invention.

Referring to FIG. 2 , the data input unit 302 of the computing device 300 receives data from the connected user terminal 200 , and the preprocessor 304 receives the data received from the user terminal 200 from the control unit 306 . Image data, audio data, and text data can be extracted for delivery to the . Specifically, the pre-processing unit 304 extracts image data including the user's face from the moving picture data, extracts face, gesture, and hand motion images and images for each time period, resizes if necessary, and outputs the image to the control unit 306 , It is possible to extract and output voice data for each time period from the data. Then, text data corresponding to voice in the video data is output for each time period and output to the controller 306 .

The control unit 306 controls each component of the computing device 300, analyzes the biological rhythm, emotional state, and psychological state based on the user's image, voice, and text data, and based on the performed result, the user's By deriving the final psychological state result value, the derived result value may be transmitted to the output unit 308 and output, or feedback on advice matching the derived result may be performed to the user.

Specifically, the controller 306 may receive the image data 3000 , the audio data 3010 , and the text data 3020 output from the preprocessor 304 . Then, the double image data 3000 is passed through the first deep learning model 3030 . The first deep learning model 3030 is a method using a convolutional neural network (CNN), which uses a convolution layer to find feature values of the image while pixels of the image pass through the convolution layer. have characteristics.

Accordingly, in emotion classification, an image is extracted from an image, and the extracted image passes through a convolutional network, for example, as six specific emotion labels, joy, sadness, anger, happiness, calmness, dullness, etc. It is possible to predict the emotion label represented in the image by finding the eigenvector of the time, and output the calculated value through the biorhythm prediction result calculation unit 3060 .

In addition, the biorhythm prediction result calculator 3060 may measure the emotional biorhythm based on the label corresponding to the emotion. That is, the predicted value for emotion may be calculated as a label or a numerical value, and a heart rate, blood flow, etc. corresponding to the calculated numerical value may be predicted. Such emotional biorhythms may include biorhythms for emotions for each day, month, and season cycle.

4 is a diagram illustrating data input to a deep learning model and output results according to an embodiment of the present invention.

Referring to FIG. 4 , for example, when image data such as (a) is passed through the deep learning model, results such as the table of (b) and (c) may be output. Video_label in (b) means the emotion value represented by the data during training, and predict_label is the emotion value predicted by the deep learning model.

And Video_label in (c) means the emotion value represented by the data during training, and predict_label means the emotion value predicted by the deep learning model. HR_label means the heart rate that the data actually represents during training, and HR_predict means the heart rate predicted by the deep learning model. HR_MAE/BPM means the error between the actual correct answer data and the value predicted by the deep learning model. BR_label means the blood flow actually represented by the data during training, and BR_predict means the blood flow predicted by the actual model. BR_MAE/BPM means the error between the actual correct answer data and the value predicted by the deep learning model.

In addition, the control unit 306 passes the image data 3000 and the audio data 3010 through the second deep learning model 3040, derives the prediction result for the emotional state, and through the emotional state prediction result calculation unit 3070 The emotional state can be calculated and output. Speech information can be analyzed with only a small number of sentences, and is sometimes more accurate than reading facial expressions, and especially high-pitched voices indicate active emotions such as anger and fear, while low-pitched voices indicate sadness and indifference. indicates. Accordingly, the second deep learning model 3040 is to perform emotion analysis by additionally using voice data such as the user's intonation and the pitch of the voice included in the video data. CNN and Recurrent Neural Network (RNN) can be used to measure emotional labels from image data, and convert the voice data into graphs to classify the speaker's emotional state using the pitch and intensity of the speaker's voice.

Accordingly, when the image data 3000 and the audio data 3010 are passed through the second deep learning model 3040, the facial expression and mouth shape, the appearance and expression at the moment of speaking, the facial expression and the tone of the voice are measured. You can predict your emotional state.

Meanwhile, as a deep learning neural network for image data, a deep learning model using RNN and Long Shot Term Memory (LSTM) can be applied. As a model for sequence data, RNN/LSTM is different from existing neural networks because it can memorize input values in units of time, so it can be applied to data whose contents change over time, such as video or audio. .

Therefore, when RNN/LSTM is applied to the first and second deep learning models, the image is finally processed through the RNN/LSTM neural network for each frame. It may be possible to measure the cause of the emotional state and the degree of ups and downs of the emotion.

In addition, the control unit 306 may receive the text data 3020 output from the preprocessor 304 and transmit it to the text correlation index calculation unit 3050 . That is, the user can obtain correlation values between texts included in sentences as data converted from voice to text when shooting video data, and evaluate the psychological state through sentence analysis.

Accordingly, in the control unit 306, when the prediction results calculated by the biorhythm prediction result calculation unit 3060, the emotional state prediction result calculation unit 3070, and the psychological state analysis result calculation unit 3080 are in the form of a data frame, it is converted into a CSV file. to the psychological state result derivation unit 3090 , and the psychological state result derivation unit 3090 may derive a comprehensive analysis result based on the biorhythm prediction result, the emotional state prediction result, and the psychological state prediction result. In addition, advice phrases corresponding to the results of the comprehensive analysis, advice texts, and customized services for psychological relief may be fed back to the user terminal 200 through the output unit 308 and provided.

Meanwhile, the data input unit 302 may receive text data for a questionnaire response result in addition to video data. In addition, the input text data may be input to the text correlation index calculator 3050 .

The user may request a psychological counseling expert before and after inputting the video data to fill out a questionnaire for psychological counseling, and transmit the questionnaire response to the user terminal 200 to be input to the computing device 300 . After accessing the user terminal 200 using the computing device 300 or another user terminal interlocked with the computing device 300, the psychological counseling expert may conduct psychological counseling through video or voice, and at this time, The psychological state evaluation may be performed through the data, and the user's voice may be extracted and input to the text correlation index calculator 3050 as text data.

Alternatively, the user may check a questionnaire for psychological counseling through on/offline, extract the response as text data, and input it into the text correlation index calculation unit 3050 . The psychological counseling expert may input a value derived for a corresponding psychological state based on the user's response result through the psychological counseling questionnaire into the text correlation indicator calculating unit 3050 . In this case, when the psychological counseling questionnaire is objectively received and a response to the questionnaire is received, the psychological state may be derived through the corresponding psychological state evaluation table based on the response corresponding to the multiple choice in the text correlation index calculation unit 3050 .

The questionnaire can be conducted in a way that receives multiple-choice or subjective responses in a way that examines whether there are emotional or psychological concerns or concerns such as family, study, friends, living/residence, health care/exercise, work life, etc. This can be done in order to receive specific comments about the key emotional or psychological states identified in the data.

Accordingly, the control unit 306 obtains correlation values between texts included in a sentence based on the emotion measurement index according to the questionnaire response and the text, and evaluates the psychological state through sentence analysis and based on the content of advice according to the emotional state. The user analyzes the texts extracted from the simple questionnaire and the user's image.

As described above, in the present invention, emotion prediction using image data, video and audio data is performed using a deep learning model, respectively, and the emotion prediction result using text data is integrated and used. In addition, by additionally utilizing the data extracted from the user's image data as well as the actual psychological state of the user through the survey feedback with a psychological counselor, it is possible to make more accurate psychological predictions for the user. have.

Through this, the user inputs an image of the current state of the user using a user terminal or computing device without direct face-to-face with a psychological counseling expert, video call with a psychological counseling expert before and after inputting image data, and outputs from the user terminal or computing device Psychological counseling can be performed by responding to the questionnaire using at least one of the digital questionnaire and the offline questionnaire.

The psychological counseling system extracts video data, audio data and text data including face images from the input video data, and passes the video data, video and audio data through each deep learning model to predict biorhythms and emotions. can be derived Then, based on the text data extracted from the image data, the emotional state analysis is performed through the text correlation index to derive the emotion prediction result, and then the emotion prediction result passed through the deep learning model and the emotion prediction result using the text data are integrated. Comprehensive psychological state results can be derived for

In this case, the text correlation index may be analyzed based on the text data of the questionnaire received from the user in addition to the text data extracted from the image data. That is, it is possible to derive a more accurate emotion prediction result in real time by performing emotion prediction in a state in which the psychological state of the user is more understood through a psychological counseling expert.

3 is a flowchart illustrating a psychological counseling method using artificial intelligence according to an embodiment of the present invention.

Referring to FIG. 3 , in step S300 , the user terminal 200 or the computing device 300 receives the user's emotion-containing image data and questionnaire data input from the user. Image, image, voice, and text data are extracted from the input image data, and in step S310 , the control unit 306 of the computing device 300 uses the first deep learning model 3030 to obtain the user's biorhythm from the image data of the image data. will measure In step S320 , the control unit 306 of the computing device 300 measures the emotional state of the user from the image and audio data through the second deep learning model 3040 .

In step S330, the control unit 306 of the computing device 300 analyzes the text correlation index from the input text data and the questionnaire text to measure the psychological state, and in step S340, the control unit 306 of the computing device 300 measures Based on the biorhythm, emotional state and psychological state, a comprehensive psychological state result is derived and output, and customized advice is fed back to the user.

In this way, psychological relief can be made possible by conducting psychological analysis based on the data of the video, face image, voice, and text extracted from the user's video, as well as a questionnaire built by a psychological expert for more detailed psychological measurement. Users can not only know their emotional state and biorhythm, but also receive psychological advice and receive psychological relief services.

The psychological counseling method using artificial intelligence according to an embodiment of the present invention utilizes deep learning technology, which is a field of artificial intelligence, based on the user's image and voice data, and analyzes the user's emotions, biorhythms, and text analysis result values. There is an advantage in that the user's emotional state can be analyzed together with the psychological numerical result value through the questionnaire data to understand the user's actual emotional state and give advice.

Meanwhile, the present invention can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device.

In addition, the computer-readable recording medium may be distributed in a network-connected computer system, and the computer-readable code may be stored and executed in a distributed manner. And functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention pertains.

The steps constituting the method according to the present invention may be performed in an appropriate order, unless the order is explicitly stated or there is no description to the contrary. The present invention is not necessarily limited to the order in which the steps are described.

The use of all examples or exemplary terminology (eg, etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is not limited by the examples or exemplary terms unless limited by the appended claims. It is not limited. In addition, those skilled in the art can appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and not only the claims described below, but also all ranges equivalent to or changed from these claims are of the spirit of the present invention. would be said to belong to the category.

As such, the present invention has been described with reference to one embodiment shown in the drawings, but this is merely exemplary, and those skilled in the art will understand that various modifications and variations of the embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: psychological counseling system
200: user terminal
300: computing device

Claims (10)

receiving user's video and audio data;
measuring the user's biorhythm from the image data input through the first deep learning model;
measuring and classifying emotions in the video and audio data through a second deep learning model;
performing an emotion measurement index and text-based analysis based on the text data extracted from the voice data and the text data of the questionnaire response input from the user; and
A psychological counseling method using artificial intelligence that derives comprehensive psychological state results based on measured biorhythms, classification emotions, and text analysis results.
According to claim 1,
Measuring the biorhythm comprises:
predicting one of six emotions from the image data; and
calculating a numerical value for the predicted emotion and calculating a heart rate and blood flow corresponding to the calculated numerical value;
A psychological counseling method using artificial intelligence, including According to claim 1,
The step of measuring and classifying the emotion is
A psychological counseling method using artificial intelligence, characterized in that predicting and calculating an emotional state through facial expressions and voice tones when speaking from the video and audio data. The method of claim 1,
The step of performing the analysis based on the emotion measurement index and the text,
calculating a corresponding emotion measurement index according to the survey response result;
calculating a correlation value between texts included in a sentence based on the moving image data extraction text data and the questionnaire response result text data; and
outputting a psychological state analysis result based on the emotion measurement index and the correlation value;
A psychological counseling method using artificial intelligence, including The method of claim 1,
The text data of the survey response result is,
transmitting a psychological counseling questionnaire from the computing device to the user terminal;
receiving text data as a response to the questionnaire from the user terminal; and
calculating a text correlation index for the text data and the text data extracted from the video;
A psychological counseling method using artificial intelligence, including an input unit for receiving user's video and audio data;
a biorhythm prediction result calculator for measuring the user's biorhythm from the image data input through the first deep learning model;
an emotional state prediction result calculation unit for measuring and classifying emotions in the video and audio data through a second deep learning model;
a psychological state analysis result calculation unit for performing an emotion measurement index and text-based analysis through the text data extracted from the voice data and the questionnaire response result text data input from the user; and
A psychological state result derivation unit that derives a comprehensive psychological state result based on the measured biorhythm, classification emotion, and text analysis result
A psychological counseling device using artificial intelligence, including a. 7. The method of claim 6,
The biorhythm prediction result calculation unit,
Predicting one of six emotions from the image data,
A psychological counseling apparatus using artificial intelligence, characterized in that it calculates a numerical value for a predicted emotion and calculates a heart rate and blood flow corresponding to the calculated numerical value. 7. The method of claim 6,
The emotional state prediction result calculation unit,
Psychological counseling apparatus using artificial intelligence, characterized in that predicting and calculating the emotional state through the facial expression and tone of voice when speaking from the video and audio data. 7. The method of claim 6,
The psychological state analysis result calculation unit,
Calculate the corresponding emotion measurement index according to the survey response result,
Based on the video data extraction text data and the questionnaire response result text data, a correlation value between texts included in a sentence is calculated,
Psychological counseling apparatus using artificial intelligence, characterized in that outputting a psychological state analysis result based on the emotion measurement index and the correlation value. 7. The method of claim 6,
The text data of the survey response result is,
Transmitting the psychological counseling questionnaire from the computing device to the user terminal,
receiving text data as a response to the questionnaire from the user terminal;
Psychological counseling apparatus using artificial intelligence, characterized in that for calculating a text correlation index for the text data and the text data extracted from the video.

Images