WO2021139310A1

WO2021139310A1 - Emotion recognition method, apparatus, computer device, and computer-readable storage medium

Info

Publication number: WO2021139310A1
Application number: PCT/CN2020/122418
Authority: WO
Inventors: 黄晓君; 庄伯金
Original assignee: 平安科技（深圳）有限公司
Priority date: 2020-07-30
Filing date: 2020-10-21
Publication date: 2021-07-15
Also published as: CN111797817B; CN111797817A

Abstract

An emotion recognition method, an apparatus, a computer device and a computer-readable storage medium, belonging to the technical field of artificial intelligence. The method comprises: acquiring a plurality of fingertip images corresponding to finger tips (S101): converting the format corresponding to the fingertip images to obtain a PPG signal corresponding to said fingertip images (S102): by means of a preset neural network model, extracting a heart rate feature and a heart rate variation feature from the PPG signal (S103); acquiring, on the basis of said heart rate feature and the heart rate variation feature, an emotional state index corresponding to the PPG signal (S104); on the basis of the emotional state index, and on a preset matching relationship between an emotional state index and emotional states, obtain the emotional state corresponding to the emotional state index (S105).

Description

Emotion recognition method, device, computer equipment and computer readable storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 202010752926.1, and the application name is "emotion recognition method, device, computer equipment and computer readable storage medium" on July 30, 2020, and its entire content Incorporated in this application by reference.

Technical field

This application relates to the field of artificial intelligence technology, and in particular to an emotion recognition method, device, computer equipment, and computer-readable storage medium based on a neural network model.

Background technique

People’s negative emotions can lead to abnormal states such as insomnia and even depression. If this emotional state cannot be relieved and released in time for a long time, it will cause a series of discomforts such as headache, insomnia, and anxiety, and will increase cardiovascular and cerebrovascular diseases. A series of chronic disease risks such as diabetes, diabetes and cancer. Emotion is the primary line of defense for human immunity, and good emotional management can effectively help users build a good physique.

Emotion management is generally divided into emotional state assessment and emotional regulation. The traditional method of emotional state evaluation is subjective evaluation method, according to the description of the subject’s self-perception, and the score is based on a professionally designed psychological scale. The inventor realizes that although this method is simple and common in operation, it has poor objectivity and is more suitable for large sample statistical analysis, and it is difficult for individuals to perform long-term and multiple evaluations. Therefore, in traditional technology, the efficiency of recognizing or evaluating emotions is low.

Summary of the invention

This application provides an emotion recognition method, device, computer equipment, and computer readable storage medium based on a neural network model, which can solve the problem of low emotion recognition efficiency in traditional technologies.

In the first aspect, the present application provides a method for emotion recognition based on a neural network model, the method includes: acquiring a plurality of fingertip images corresponding to the fingertips; converting the format corresponding to the fingertip images, To obtain the PPG signal corresponding to the fingertip image; extract the heart rate feature and the heart rate variability feature from the PPG signal through a preset neural network model; obtain the PPG signal according to the heart rate feature and the heart rate variability feature The corresponding emotional state index; according to the emotional state index and the preset matching relationship between the emotional state index and the emotional state, the emotional state corresponding to the emotional state index is obtained.

In a second aspect, the present application also provides an emotion recognition device based on a neural network model, including: a first acquisition unit for acquiring a plurality of fingertip images corresponding to the fingertips of the fingers; a conversion unit for converting the The format corresponding to the fingertip image is converted to obtain the PPG signal corresponding to the fingertip image; the extraction unit is configured to extract the heart rate feature and the heart rate variability feature from the PPG signal through a preset neural network model; second The obtaining unit is used for obtaining the emotional state index corresponding to the PPG signal according to the heart rate characteristic and the heart rate variability characteristic; the third obtaining unit is used for obtaining the emotional state index and the emotional state index and the emotional state according to the emotional state index And obtain the emotional state corresponding to the emotional state index.

In a third aspect, an embodiment of the present application also provides a computer device, which includes a memory and a processor, and a computer program is stored on the memory. When the processor executes the computer program, the following steps are performed: Obtain a number of fingers The fingertip image corresponding to the fingertip; the format corresponding to the fingertip image is converted to obtain the PPG signal corresponding to the fingertip image; the heart rate feature is extracted from the PPG signal through a preset neural network model And heart rate variability characteristics; according to the heart rate characteristics and the heart rate variability characteristics, obtain the emotional state index corresponding to the PPG signal; according to the emotional state index, and the preset matching relationship between the emotional state index and the emotional state To obtain the emotional state corresponding to the emotional state index.

In a fourth aspect, the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the following steps: Obtain several The fingertip image corresponding to the fingertip of the finger; the format corresponding to the fingertip image is converted to obtain the PPG signal corresponding to the fingertip image; the heart rate is extracted from the PPG signal through a preset neural network model Characteristics and heart rate variability characteristics; according to the heart rate characteristics and the heart rate variability characteristics, obtain the emotional state index corresponding to the PPG signal; according to the emotional state index, and a preset match between the emotional state index and the emotional state Relationship to obtain the emotional state corresponding to the emotional state index.

This application provides a method, device, computer equipment, and computer-readable storage medium for emotion recognition based on a neural network model. This application acquires several fingertip images corresponding to the fingertips, transforms the format corresponding to the fingertip images, to obtain the PPG signal corresponding to the fingertip images, and obtains the PPG signal corresponding to the fingertip image through a preset neural network model. The heart rate feature and heart rate variability feature are extracted from the PPG signal, the emotional state index corresponding to the PPG signal is obtained according to the heart rate feature and the heart rate variability feature, and the emotional state index and the emotional state index are obtained according to the emotional state index, and the emotional state index and emotional state To obtain the emotional state corresponding to the emotional state index, the embodiment of the application obtains the emotional state indicator according to the fingertip image to obtain the emotional state, which improves the objectivity of emotion recognition, and Can improve the efficiency of emotion recognition.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following will briefly introduce the drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present application. Ordinary technicians can obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic flowchart of a method for emotion recognition based on a neural network model provided by an embodiment of the application;

FIG. 2 is a schematic diagram of an emotional state calculation method of an emotion recognition method based on a neural network model provided by an embodiment of the application;

3 is a schematic diagram of a sub-process in the method for emotion recognition based on a neural network model provided by an embodiment of the application;

FIG. 4 is a schematic block diagram of an emotion recognition device based on a neural network model provided by an embodiment of the application; and

Fig. 5 is a schematic block diagram of a computer device provided by an embodiment of the application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of a method for emotion recognition based on a neural network model provided by an embodiment of the application. As shown in Figure 1, the method includes the following steps S101-S105:

S101. Acquire several fingertip images corresponding to the fingertips of the fingers.

Specifically, to acquire the fingertip image, the fingertip image can be captured by the image acquisition device, for example, the fingertip image can be captured by the camera of the mobile terminal device itself, or multiple images within a preset time can be continuously captured by continuous shooting. A good description of the continuous fingertip conditions within the preset time to more accurately reflect the emotional state. In addition, you can also use the camera of the mobile terminal to record a fingertip video, and then extract each frame of the video to obtain several fingertip images. Moreover, due to the continuity of the video itself, each video extracted from the video A frame of fingertip images has continuity and can accurately reflect the fingertip conditions within a preset time. When emotion recognition is performed through fingertip images, the accuracy of emotional state recognition can also be improved.

S102: Convert the format corresponding to the fingertip image to obtain a PPG signal corresponding to the fingertip image.

Among them, PPG signal, English for Photoplethysmography, abbreviated as PPG, is a photoplethysmography method, also known as photoplethysmography pulse wave, is a detection method to detect changes in blood volume in living tissues by photoelectric means.

Specifically, when a light beam of a certain wavelength irradiates the skin surface of the fingertip, the contraction and expansion of blood vessels will affect the transmission of light during each heartbeat. For example, in the transmission PPG, the light passing through the fingertip, when the light passes through the skin tissue and then When it is reflected to the photosensitive sensor, the light will be attenuated to a certain extent. The absorption of light by muscles, bones, veins and other connected tissues is basically unchanged (provided that there is no significant movement of the measurement site), but the arteries will be different. Because of the pulsation of blood in the arteries, the absorption of light will naturally change, resulting in changes in the fingertip image, which is reflected in the changes in the pixels of the fingertip image. Therefore, by changing the pixels contained in the fingertip image The analysis can reflect the characteristics of blood flow, and then reflect the characteristics of heart rate through blood flow.

Therefore, after the fingertip image is obtained, the format corresponding to the fingertip image is converted, and the fingertip image can be described by converting the fingertip image into RGB three-channel mode, and then by comparing the RGB three-channel Perform analysis, such as analyzing the green light channel or the red light channel of the RGB three channels to obtain the PPG signal corresponding to the fingertip image. Among them, the RGB three-channel is because RGB images are also called full-color images, which have three channels: R (red), G (green), and B (blue). For example, use the Halcon program and Halcon's own images to understand RGB images and grayscale values. For example, the color image can also be separated by three channels through Opencv and Matlab. When Opencv and Matlab process color images, the storage order of the channels is different. The order of Matlab is R, G, B, while in Opencv , The arrangement order is B, G, R.

S103: Extract the heart rate feature and the heart rate variability feature from the PPG signal through a preset neural network model.

Among them, heart rate variability (ie, HRV) refers to the small difference between heartbeat intervals.

Specifically, in the embodiment of the present application, pre-training is performed on a preset neural network model in advance, for example, for a recurrent neural network (Recurrent Neural Network in English, or RNN for short), since the neural network model has the ability to automatically learn, After the training samples including the sample PPG signal and the heart rate characteristics and heart rate variability characteristics corresponding to the sample PPG signal are input to the preset neural network, the preset neural network model will be based on the sample PPG signal and the sample PPG signal The corresponding heart rate feature and heart rate variability feature are automatically learned, so that after the PPG signal is subsequently obtained, the heart rate feature and heart rate variability feature can be extracted from the PPG signal. After the preset neural network model is trained, the training effect of the preset neural network model can be tested by the verification sample. If the preset neural network model passes the verification, the preset neural network model is applied to the production environment.

Because the shape of PPG signal is similar to the waveform of arterial blood pressure (ABP), and the heart rate is closely related to blood pressure, the emotional state of a person directly affects the heart rate and blood pressure, and the blood pressure and heart rate also reflect the emotional state of the person. Therefore, this makes the PPG signal non- An invasive heart rate monitoring tool can measure a person's heart rate and blood pressure by analyzing the PPG signal, and identify the person's emotional state. Since the periodicity of the PPG signal corresponds to the heart rhythm, the heart rate can be estimated based on the PPG signal.

The characteristics of heart rate variability can be obtained through frequency domain analysis and time domain analysis of the PPG signal.

Among them, time domain analysis, English is Timedomainanalysis, HRV time domain analysis is to quantitatively describe the change characteristics of the cardiac cycle by various statistical methods, for example, by measuring and calculating the average RR interval and the longest RR interval within a certain period of time The difference or ratio with the shortest RR interval, as well as the standard deviation of all RR intervals, so as to achieve filtering processing of the original PPG signal to calculate the number of peaks of the PPG signal within a certain period of time, and then the heart rate value can be calculated, For example, if the sampling time is 5 seconds, the number of peaks in 5 seconds is N, then the heart rate is N*12.

Frequency domain analysis, also known as frequency domain analysis in English, uses a special calculation method to decompose the heart rate fluctuation curve that changes with time into the sum of sinusoidal curves of different frequencies and different amplitudes to obtain the HRV spectrum. The advantage is that the periodicity of heart activity can be quantified. The human HRV power spectrum is often divided into 4 regions: high frequency band, low frequency band, very low frequency band and ultra low frequency band. For example, the frequency can be obtained by performing FFT on the PPG signal. The characteristics of the domain.

After the pre-trained preset neural network model is put into the production environment, after receiving the PPG signal, the PPG signal can be analyzed in the time domain and the frequency domain to extract the heart rate characteristics and the PPG signal. The characteristics of heart rate variability, the embodiment of the application uses AI healthy fingertip detection technology to well solve the subjective evaluation method and improve the objectivity and accuracy of emotional evaluation.

S104. Obtain an emotional state index corresponding to the PPG signal according to the heart rate characteristic and the heart rate variability characteristic.

Specifically, after the heart rate feature and the heart rate variability feature are extracted, the emotional state index corresponding to the PPG signal is obtained according to the heart rate feature and the heart rate variability feature. Please refer to FIG. 2, which is a schematic diagram of an emotional state calculation method of a neural network model-based emotion recognition method provided by an embodiment of the application. As shown in FIG. 2, in this embodiment, according to the heart rate characteristics and the heart rate variability characteristics, the emotional state indexes such as the vitality index, the stress index, and the fatigue index corresponding to the PPG signal are obtained, so as to be used for quantification. Assess the activity and balance of the autonomic nervous system, as well as the fatigue and stress status of the human body.

For example, based on big data analysis of existing data, with the help of training samples used in pre-training, the statistical relationship between heart rate characteristics and heart rate variability characteristics and emotional state index can be obtained. For example, heart rate characteristics and heart rate can be obtained through training samples The statistical relationship between the fatigue index, the stress index and the vitality index corresponding to the variability feature, and then after the heart rate feature and the heart rate variability feature corresponding to the PPG signal are obtained, the fatigue corresponding to the PPG signal is obtained according to the statistical relationship Index, stress index and vitality index, that is, the emotional state index corresponding to the PPG signal is obtained. For example, in an example, the statistical relationship between the heart rate characteristics and the heart rate variability characteristics and the emotional state index is shown in the following Tables 1 to 3, where Table 1 is the fatigue index level and evaluation table, and Table 2 is the stress index level and Evaluation table, Table 3 is the vitality index grade and evaluation table, Table 1 to Table 3 are as follows:

Table 1

Form 2

Form 3

S105. Obtain the emotional state corresponding to the emotional state index according to the emotional state index and the preset matching relationship between the emotional state index and the emotional state.

Specifically, based on prior knowledge and accumulated data, the emotional state is classified to form a preset matching relationship between the emotional state index and the emotional state, after obtaining the emotional state index, according to the emotional state index, and The matching relationship is preset, and the emotional state corresponding to the emotional state index is obtained. For example, please continue to refer to Table 1 to Table 3 to obtain the emotional state corresponding to the emotional index to realize the recognition of the emotional state. When realizing the quantification of emotional health, the embodiments of this application use smart mobile devices to perform emotional quantification, use smart mobile devices to perform fingertip detection, collect electrocardiogram pulse waves (PPG) combined with neural network RNN algorithms, and extract heart rate and heart rate variability (HRV). ) Features to detect the user’s vitality, stress, fatigue state, objectively quantify the current and long-term emotional state, and achieve a low threshold for emotion recognition based on the neural network model, without the need for additional external equipment to obtain the user’s physical and mental health data index.

Please refer to FIG. 3, which is a schematic diagram of a sub-process in the method for emotion recognition based on a neural network model provided by an embodiment of the application. As shown in FIG. 3, in this embodiment, the step of acquiring a plurality of fingertip images corresponding to the fingertips includes: S301, in response to the instruction to collect the fingertip video, turn on the flash of the camera of the mobile terminal; S302 , Prompt the user to block the camera through the fingertip of the finger; S303, determine whether the camera is blocked by the fingertip of the finger; S304, if the camera is blocked by the fingertip of the finger, point the finger to the finger through the camera Recording with the tip of the finger to obtain a fingertip video, if the camera is not blocked by the fingertip of the finger, return to the step of prompting the user to block the camera with the fingertip of the finger; S305, image the fingertip video Extract to get a fingertip image.

Specifically, when emotion recognition and emotional management are realized through a smart mobile terminal, the camera of the smart mobile terminal can be used to perform fingertip detection, and then the emotional state can be evaluated. When it is necessary to collect the ECG pulse wave (ie PPG) signal through the camera of the smart mobile device, for example, when the user opens the emotion management application for emotion recognition, the mobile terminal responds to the user's operation instruction on the preset function, that is, the response Command to collect fingertip video, and then turn on the flash of the camera of the mobile terminal to match the flash of the mobile device, and prompt the user to block the camera with the fingertip of the finger, so that the user can use the finger to block the camera, and judge the camera through image recognition Whether it is blocked by the fingertip of the finger, such as judging whether the image feature corresponding to the extracted occluder matches the preset fingertip image feature, if the image feature corresponding to the extracted occluder matches the preset fingertip image feature , It is determined that the camera is blocked by the fingertip of the finger, and then video recording is performed on the fingertip of the finger, if the camera is not blocked by the fingertip of the finger, return to the execution of the prompting the user to block the camera with the fingertip of the finger A step of. Because the fingertip blood flow will fluctuate periodically with the change of blood vessel volume, the recorded video pixel value will also periodically change, so that the fingertip video image contained in the recorded fingertip video can reflect the user's heart rate , And then reflect the user's emotional state through the heart rate.

In one embodiment, the step of transforming the format corresponding to the fingertip image to obtain the PPG signal corresponding to the fingertip image includes: performing, according to a preset separation method, each frame of the fingertip image Perform RGB classification to obtain the RGB three channels corresponding to each frame of the fingertip image; separate the red channel from the RGB three channels; calculate the red channel to extract the PPG signal corresponding to the red channel ; Combine all the PPG signals corresponding to the red channel to obtain the PPG signal corresponding to the fingertip image.

Specifically, RGB classification is performed on each frame of the fingertip image according to a preset separation method. The preset separation method includes using the Halcon program, Halcon's own image to understand the RGB image and gray value, or through Opencv and Matlab. The image is separated into three channels to obtain the RGB three channels corresponding to each frame of the fingertip image, and then the heart rate-related pixel data such as the red channel or the green channel are separated from the RGB three channels to obtain the PPG signal , To extract the PPG signal corresponding to the red channel, and combine all the PPG signals corresponding to the red channel to obtain the PPG signal corresponding to the fingertip image so as to convert the format of each frame of video to The PPG signal is obtained, and then combined with a preset neural network model, such as a neural network RNN algorithm, to extract features of heart rate and heart rate variability (HRV).

In one embodiment, after the step of obtaining the emotional state index corresponding to the PPG signal, the method further includes: obtaining all the emotional state indexes within a preset period of time; and calculating the emotional state indexes corresponding to all the emotional state indexes. Average value of emotional state index;

The step of obtaining the emotional state corresponding to the emotional state index according to the emotional state index and the preset matching relationship between the emotional state index and the emotional state includes: according to the emotional state index average value and the emotional state The preset matching relationship between the state index and the emotional state is used to obtain the emotional state corresponding to the average value of the emotional state index.

Specifically, record the emotional state results of each fingertip detection, such as vitality index, stress index, and fatigue index. According to the emotional state index, analyze people's health trends, record daily detection values as a curve, and monitor the trend of emotional changes . And can obtain all the emotional state indexes within a preset time period, calculate the average value of the emotional state index corresponding to all the emotional state indexes, and calculate the average value of the emotional state index according to the emotional state index and the relationship between the emotional state index and the emotional state. To obtain the emotional state corresponding to the average value of the emotional state index, for example, calculate the average value of the emotional state results for 7 days as the Baseline (benchmark index reference), and intuitively compare and understand the current emotional state. For users who insist on using emotion detection for 7 days, calculate the 7-day average and provide health evaluation and suggestions. The scoring system of vitality, stress, and fatigue index obtained according to the algorithm is shown in Table 1 to Table 3 above. By measuring the emotional state within a preset period of time by the average value of the emotional state index, the emotional state within a period of time can be obtained more accurately, thereby measuring the stability of the user's emotional state. Through emotional monitoring of users’ emotions within a preset time period, users who persist in using them for a long time can analyze health trends and digitally display changes in emotional states, thereby identifying mental health risks and quantitatively assessing users’ mental health. Identify the risk of mental illness in time.

In one embodiment, after the step of obtaining the emotional state corresponding to the average value of the emotional state index, the method further includes: drawing all the emotional state indexes into emotional trend diagrams; and comparing the emotional trend diagrams with the emotional state indexes. The emotional state corresponding to the average value of the emotional state index is output to display the emotional state.

Specifically, all the emotional state indexes within a preset time period are drawn into emotional trend diagrams, and the emotional state corresponding to the emotional trend diagram and the average value of the emotional state indexes are output for emotional state display, which can be intuitive Let users understand their own emotional state, thereby improving the efficiency of emotional management.

Specifically, after the step of obtaining the emotional state corresponding to the emotional state index, the method further includes: judging whether the emotional state index is less than or equal to a preset emotional state index threshold; if the emotional state index is less than or equal to all The preset emotional state index threshold is output, and the preset emotional guidance prompt is output; it is judged whether the instruction corresponding to the user agreeing to perform the emotional guidance operation is received; if the instruction corresponding to the user agrees to the emotional guidance operation is received, follow the pre-defined Set up an emotional guidance method to guide users to adjust their emotions.

Specifically, if the emotional state of the user is not in a good state, the emotional state of the user can be guided through a preset emotional guidance method to improve the emotional state of the user, for example, to determine whether the emotional state index is less than or equal to the preset emotional state index Threshold, if the emotional state index is less than or equal to the preset emotional state index threshold, determine that the user's emotional state is in a bad state, and output a preset emotional guidance prompt to remind the user that the emotional state needs to be improved, otherwise it will bring physical and mental health Can't affect. After the user is prompted with emotional guidance, it is determined whether the instruction corresponding to the user's consent to the emotional guidance operation is received, and if the instruction corresponding to the user's consent to the emotional guidance operation is received, the user is guided according to the preset emotional guidance method Perform emotional regulation. Through the embodiments of this application, it is possible to implement quantified emotion and emotion monitoring of the user’s emotions by providing it on a smart mobile device to identify mental health risks, and use short-term emotion regulation services when the user’s emotions are in a bad state, Reduce the user's risk of suffering from mental illness and play a role in assisting the adjustment of mental health.

Further, the step of guiding the user to adjust emotions according to a preset emotion guidance manner includes: obtaining the emotional state index; obtaining a preset type of healing music corresponding to the emotional state index according to the emotional state index; The preset type of healing music is combined with preset mindfulness training to guide the user to adjust emotions.

Specifically, you can create meditation music through AI, learn specific styles of healing music through AI machine learning, use multi-layer sequence models and high-latitude music feature extraction methods to create the main theme of healing music, and use balanced breathing methods, The breathing pattern of professional breathing prevention such as abdominal breathing method adjusts the rhythm and tone of the music, and adds white noises such as wind, rain, and water droplets according to different theme scenes, and finally forms a preset type of healing music, and the preset type The healing music and the emotional state index form a corresponding matching relationship, that is, different emotional state indexes correspond to different types of healing music, and different emotional states can be targeted to differently assist in regulating emotions and improve the effect of regulating the emotional state.

Then, the preset healing lingo corresponding to the meditation music created by AI is combined with the preset mindfulness training method, and the users whose health evaluation is "attention needed" provide effective emotional adjustment services on the mobile device, thereby achieving music therapy Combined with mindfulness training and considering the device characteristics of mobile devices to be used anytime and anywhere, it provides a short-term emotional regulation and enhances the effect of emotional regulation. Since traditional mindfulness training methods generally require users to last for more than 30 minutes, the short-term emotional adjustment provided by the embodiments of this application can provide flexible mindfulness voice guidance on mobile devices, for example, provide 8 kinds of 5-10 minutes mindfulness voice guidance , According to the user’s different life scenarios, we recommend services for emotional regulation on mobile devices. The recommended content includes emotional first aid, mindful breathing, body scanning, preparation before going to bed, restful sleep, alleviating anxiety, accepting anxiety, enhancing concentration, etc. . Users can choose the guidance of mindfulness speech they need according to their emotional state, and cooperate with soothing meditation music and breathing rhythm adjustment, can effectively relax, slowly "thinking" from mode to "feeling" mode, and thinking And attention returns to the body itself, which can effectively calm emotions, and provide an effective and convenient method of emotional adjustment for people who have been in an anxious and stressed life environment for a long time, so as to combine mindfulness training methods with AI-created meditation music, and move intelligently. The device provides a set of short-term emotional adjustment services, allowing users to use fragmented time to adjust emotions, effectively solving users' anxiety, tension, and stress, and improving the efficiency of emotional adjustment.

In the embodiments of this application, the emotion management based on smart mobile devices can be realized through the above-mentioned methods. By combining AI health fingertip detection technology, AI creative meditation music technology and mindfulness training methods, a kind of "emotion detection-analysis-quantification" is constructed. The closed-loop system of "assessment-emotion regulation" provides effective emotional management for people. Compared with traditional technology, through the traditional telephone hotline or psychological counseling room, the psychological intervention method guidance provided by professional psychologists or the psychological treatment plan are provided. Such a traditional method of emotional regulation has a high threshold, which is not conducive to people’s daily emotional regulation, and it is easy to cause a long-term backlog of emotional problems or even turn into a mental illness before emotional regulation is carried out. The embodiments of this application improve the efficiency and effectiveness of emotional regulation. Convenience.

It should be noted that the neural network model-based emotion recognition method described in the above embodiments can recombine the technical features contained in the different embodiments as needed to obtain a combined implementation plan, but they are all in this application. Within the scope of protection required.

Please refer to FIG. 4, which is a schematic block diagram of an emotion recognition apparatus based on a neural network model provided by an embodiment of the application. Corresponding to the foregoing neural network model-based emotion recognition method, an embodiment of the present application also provides a neural network model-based emotion recognition device. As shown in FIG. 4, the emotion recognition device based on the neural network model includes a unit for executing the above-mentioned emotion recognition method based on the neural network model. The emotion recognition device based on the neural network model can be configured in a computer such as a mobile terminal. In the device. Specifically, referring to FIG. 4, the emotion recognition apparatus 400 based on a neural network model includes a first acquisition unit 401, a conversion unit 402, an extraction unit 403, a second acquisition unit 404 and a third acquisition unit 405. Wherein, the first acquiring unit 401 is configured to acquire several fingertip images corresponding to the fingertips of the fingers; the conversion unit 402 is configured to convert the format corresponding to the fingertip images to obtain the fingertip images. Corresponding PPG signal; extraction unit 403, used to extract the heart rate feature and heart rate variability feature from the PPG signal through a preset neural network model; the second acquisition unit 404, used to extract the heart rate feature and the heart rate variability feature according to the heart rate feature and the heart rate variability feature , Obtain the emotional state index corresponding to the PPG signal; the third obtaining unit 405 is configured to obtain the emotional state index according to the emotional state index and the preset matching relationship between the emotional state index and the emotional state Corresponding emotional state.

In one embodiment, the first acquiring unit 401 includes: a response subunit, which is used to turn on the flashlight of the camera of the mobile terminal in response to an instruction to collect a fingertip video; and a prompt subunit, which is used to prompt the user to use the fingertip Blocking the camera; a judging subunit for judging whether the camera is blocked by the fingertip of the finger; a recording subunit for detecting the finger through the camera if the camera is blocked by the fingertip of the finger The fingertips are recorded to obtain a fingertip video; and the extraction subunit is used to perform image extraction on the fingertip video to obtain a fingertip image.

In one embodiment, the device 400 for emotion recognition based on the neural network model further includes: a fourth obtaining unit, configured to obtain all the emotional state indexes within a preset time period; and a calculating unit, configured to calculate all the emotional state indexes. The average value of the emotional state index corresponding to the emotional state index; the third obtaining unit 405 is configured to obtain the emotional state index according to the average value of the emotional state index and the preset matching relationship between the emotional state index and the emotional state. The emotional state corresponding to the average state index.

In one embodiment, the device 400 for emotion recognition based on the neural network model further includes: a drawing unit for drawing all the emotional state indexes into emotional trend diagrams; a display unit for displaying the emotional trend diagrams and The emotional state corresponding to the average value of the emotional state index is output to display the emotional state.

In one embodiment, the device 400 for emotion recognition based on the neural network model further includes: a first judging unit, configured to judge whether the emotional state index is less than or equal to a preset emotional state index threshold; and a prompt unit, configured to: The emotional state index is less than or equal to the preset emotional state index threshold, and a preset emotional guidance prompt is output; a second judgment unit is used to judge whether an instruction corresponding to the user agrees to perform an emotional guidance operation is received; the guidance unit, It is used for guiding the user to adjust emotions according to the preset emotion guidance mode if the instruction corresponding to the operation that the user agrees to perform the emotion guidance is received.

In one embodiment, the guiding unit includes: a first obtaining subunit for obtaining the emotional state index; a second obtaining subunit for obtaining the emotional state index corresponding to the emotional state index The preset type of healing music; the guiding subunit is used to combine the preset type of healing music with preset mindfulness training to guide the user to adjust emotions.

It should be noted that those skilled in the art can clearly understand that the above-mentioned neural network model-based emotion recognition device and the specific implementation process of each unit can refer to the corresponding description in the foregoing method embodiment, for the convenience and conciseness of the description. , I won’t repeat it here.

At the same time, the division and connection of the various units in the emotion recognition device based on the neural network model are only used for illustration. In other embodiments, the emotion recognition device based on the neural network model can be divided into different units as needed. The units in the emotion recognition device based on the neural network model can be connected in different order and manners to complete all or part of the functions of the emotion recognition device based on the neural network model.

The above-mentioned emotion recognition apparatus based on the neural network model can be implemented in the form of a computer program, and the computer program can be run on the computer device shown in FIG. 5.

Please refer to FIG. 5, which is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or component in other devices.

Referring to FIG. 5, the computer device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501. The memory may include a non-volatile storage medium 503 and an internal memory 504, and the memory may also include volatile storage. medium.

The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the aforementioned emotion recognition method based on the neural network model.

The processor 502 is used to provide calculation and control capabilities to support the operation of the entire computer device 500.

The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can make the processor 502 execute the aforementioned neural network model-based emotion recognition method. .

The network interface 505 is used for network communication with other devices. Those skilled in the art can understand that the structure shown in FIG. 5 is only a block diagram of part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structures and functions of the memory and the processor are the same as those of the embodiment shown in FIG. 5, which will not be repeated here.

Wherein, the processor 502 is configured to run a computer program 5032 stored in a memory to implement the neural network model-based emotion recognition method described in the embodiment of the present application.

It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (Central Processing Unit, CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSPs), and special purpose processors. Integrated circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor.

A person of ordinary skill in the art can understand that all or part of the processes in the methods of the foregoing embodiments can be implemented by a computer program, and the computer program can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the foregoing method embodiment.

Therefore, this application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program is executed by the processor When the processor executes the steps of the neural network model-based emotion recognition method described in the above embodiments.

The storage medium is a physical, non-transitory storage medium, such as a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a magnetic disk, or an optical disk, etc., which can store computer programs. medium.

A person of ordinary skill in the art may be aware that the units and algorithm steps of the examples described in the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both, in order to clearly illustrate the hardware and software Interchangeability, in the above description, the composition and steps of each example have been generally described in accordance with the function. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

The above are only specific implementations of this application, but the scope of protection stated in this application is not limited to this. Any person skilled in the art can easily think of various equivalents within the technical scope disclosed in this application. Modifications or replacements, these modifications or replacements shall be covered within the scope of protection of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

A method of emotion recognition based on neural network model, including:

Acquire several fingertip images corresponding to the fingertips of the fingers;

Converting the format corresponding to the fingertip image to obtain the PPG signal corresponding to the fingertip image;

Extracting heart rate features and heart rate variability features from the PPG signal through a preset neural network model;

Obtaining the emotional state index corresponding to the PPG signal according to the heart rate feature and the heart rate variability feature;

According to the emotional state index and the preset matching relationship between the emotional state index and the emotional state, the emotional state corresponding to the emotional state index is obtained.
The method for emotion recognition based on a neural network model according to claim 1, wherein the step of obtaining fingertip images corresponding to a plurality of fingertips comprises:

In response to the instruction to collect the fingertip video, turn on the flash of the camera of the mobile terminal;

Prompt the user to block the camera with a fingertip;

Judging whether the camera is blocked by the fingertip of the finger;

If the camera is blocked by the fingertip of the finger, recording the fingertip of the finger through the camera to obtain a fingertip video;

Image extraction is performed on the fingertip video to obtain a fingertip image.
The method for emotion recognition based on a neural network model according to claim 1, wherein after the step of obtaining the emotional state index corresponding to the PPG signal, the method further comprises:

Acquiring all the emotional state indexes in a preset time period;

Calculating the average value of the emotional state index corresponding to all the emotional state indexes;

The step of obtaining the emotional state corresponding to the emotional state index according to the emotional state index and the preset matching relationship between the emotional state index and the emotional state includes:

According to the average value of the emotional state index and the preset matching relationship between the emotional state index and the emotional state, the emotional state corresponding to the average value of the emotional state index is obtained.
The method for emotion recognition based on a neural network model according to claim 3, wherein after the step of obtaining the emotional state corresponding to the average value of the emotional state index, the method further comprises:

Drawing all the emotional state indexes into emotional trend diagrams;

The emotional state corresponding to the emotional trend diagram and the emotional state index average value are output for emotional state display.
The method for emotion recognition based on a neural network model according to claim 1, wherein after the step of obtaining the emotional state corresponding to the emotional state index, the method further comprises:

Judging whether the emotional state index is less than or equal to a preset emotional state index threshold;

If the emotional state index is less than or equal to the preset emotional state index threshold, output a preset emotional guidance prompt;

Determine whether an instruction corresponding to the user's consent to perform an emotionally guided operation is received;

If the instruction corresponding to the operation that the user agrees to perform emotional guidance is received, the user is guided to perform emotional adjustment according to a preset emotional guidance manner.
The method for emotion recognition based on a neural network model according to claim 5, wherein the step of guiding the user to adjust emotion according to a preset emotion guidance method comprises:

Obtaining the emotional state index;

Obtaining a preset type of healing music corresponding to the emotional state index according to the emotional state index;

The preset type of healing music is combined with preset mindfulness training to guide the user to adjust emotions.
The method for emotion recognition based on a neural network model according to claim 1, wherein the step of converting the format corresponding to the fingertip image to obtain the PPG signal corresponding to the fingertip image comprises:

Performing RGB classification on each frame of the fingertip image according to a preset separation method to obtain the RGB three channels corresponding to each frame of the fingertip image;

Separating the red channel from the three RGB channels;

Calculating the red channel to extract the PPG signal corresponding to the red channel;

Combine all the PPG signals corresponding to the red channel to obtain the PPG signal corresponding to the fingertip image.
An emotion recognition device based on a neural network model, including:

The first acquiring unit is configured to acquire a plurality of fingertip images corresponding to the fingertips of the fingers;

A conversion unit for converting the format corresponding to the fingertip image to obtain the PPG signal corresponding to the fingertip image;

An extraction unit, configured to extract heart rate characteristics and heart rate variability characteristics from the PPG signal through a preset neural network model;

The second acquiring unit is configured to acquire the emotional state index corresponding to the PPG signal according to the heart rate characteristic and the heart rate variability characteristic;

The third acquiring unit is configured to acquire the emotional state corresponding to the emotional state index according to the emotional state index and the preset matching relationship between the emotional state index and the emotional state.
A computer device includes a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program to perform the following steps:

Acquire several fingertip images corresponding to the fingertips of the fingers;

Converting the format corresponding to the fingertip image to obtain the PPG signal corresponding to the fingertip image;

Extracting heart rate features and heart rate variability features from the PPG signal through a preset neural network model;

Obtaining the emotional state index corresponding to the PPG signal according to the heart rate feature and the heart rate variability feature;

According to the emotional state index and the preset matching relationship between the emotional state index and the emotional state, the emotional state corresponding to the emotional state index is obtained.
9. The computer device according to claim 9, wherein the step of acquiring a plurality of fingertip images corresponding to the fingertips comprises:

In response to the instruction to collect the fingertip video, turn on the flash of the camera of the mobile terminal;

Prompt the user to block the camera with a fingertip;

Judging whether the camera is blocked by the fingertip of the finger;

If the camera is blocked by the fingertip of the finger, recording the fingertip of the finger through the camera to obtain a fingertip video;

Image extraction is performed on the fingertip video to obtain a fingertip image.
The computer device according to claim 9, wherein after the step of obtaining the emotional state index corresponding to the PPG signal, the method further comprises:

Acquiring all the emotional state indexes in a preset time period;

Calculating the average value of the emotional state index corresponding to all the emotional state indexes;

The step of obtaining the emotional state corresponding to the emotional state index according to the emotional state index and the preset matching relationship between the emotional state index and the emotional state includes:

According to the average value of the emotional state index and the preset matching relationship between the emotional state index and the emotional state, the emotional state corresponding to the average value of the emotional state index is obtained.
11. The computer device according to claim 11, wherein after the step of obtaining the emotional state corresponding to the average value of the emotional state index, the method further comprises:

Drawing all the emotional state indexes into emotional trend diagrams;

The emotional state corresponding to the emotional trend diagram and the emotional state index average value are output for emotional state display.
The computer device according to claim 9, wherein after the step of obtaining the emotional state corresponding to the emotional state index, the method further comprises:

Judging whether the emotional state index is less than or equal to a preset emotional state index threshold;

If the emotional state index is less than or equal to the preset emotional state index threshold, output a preset emotional guidance prompt;

Determine whether an instruction corresponding to the user's consent to perform an emotionally guided operation is received;

If the instruction corresponding to the operation that the user agrees to perform emotional guidance is received, the user is guided to perform emotional adjustment according to a preset emotional guidance manner.
The computer device according to claim 13, wherein the step of guiding the user to adjust emotions according to a preset emotion guidance method comprises:

Obtaining the emotional state index;

Obtaining a preset type of healing music corresponding to the emotional state index according to the emotional state index;

The preset type of healing music is combined with preset mindfulness training to guide the user to adjust emotions.
9. The computer device according to claim 9, wherein the step of converting the format corresponding to the fingertip image to obtain the PPG signal corresponding to the fingertip image comprises:

Performing RGB classification on each frame of the fingertip image according to a preset separation method to obtain the RGB three channels corresponding to each frame of the fingertip image;

Separating the red channel from the three RGB channels;

Calculating the red channel to extract the PPG signal corresponding to the red channel;

Combine all the PPG signals corresponding to the red channel to obtain the PPG signal corresponding to the fingertip image.
A computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, the following steps can be implemented:

Acquire several fingertip images corresponding to the fingertips of the fingers;

Converting the format corresponding to the fingertip image to obtain the PPG signal corresponding to the fingertip image;

Extracting heart rate features and heart rate variability features from the PPG signal through a preset neural network model;

Obtaining the emotional state index corresponding to the PPG signal according to the heart rate feature and the heart rate variability feature;

According to the emotional state index and the preset matching relationship between the emotional state index and the emotional state, the emotional state corresponding to the emotional state index is obtained.
15. The computer-readable storage medium according to claim 16, wherein the step of acquiring a plurality of fingertip images corresponding to the fingertips comprises:

In response to the instruction to collect the fingertip video, turn on the flash of the camera of the mobile terminal;

Prompt the user to block the camera with a fingertip;

Judging whether the camera is blocked by the fingertip of the finger;

If the camera is blocked by the fingertip of the finger, recording the fingertip of the finger through the camera to obtain a fingertip video;

Image extraction is performed on the fingertip video to obtain a fingertip image.
15. The computer-readable storage medium according to claim 16, wherein after the step of obtaining the emotional state index corresponding to the PPG signal, the method further comprises:

Acquiring all the emotional state indexes in a preset time period;

Calculating the average value of the emotional state index corresponding to all the emotional state indexes;

The step of obtaining the emotional state corresponding to the emotional state index according to the emotional state index and the preset matching relationship between the emotional state index and the emotional state includes:

According to the average value of the emotional state index and the preset matching relationship between the emotional state index and the emotional state, the emotional state corresponding to the average value of the emotional state index is obtained.
18. The computer-readable storage medium according to claim 18, wherein after the step of obtaining the emotional state corresponding to the average value of the emotional state index, the method further comprises:

Drawing all the emotional state indexes into emotional trend diagrams;

The emotional state corresponding to the emotional trend diagram and the emotional state index average value are output for emotional state display.
15. The computer-readable storage medium according to claim 16, wherein after the step of obtaining the emotional state corresponding to the emotional state index, the method further comprises:

Judging whether the emotional state index is less than or equal to a preset emotional state index threshold;

If the emotional state index is less than or equal to the preset emotional state index threshold, output a preset emotional guidance prompt;

Determine whether an instruction corresponding to the user's consent to perform an emotionally guided operation is received;

If the instruction corresponding to the operation that the user agrees to perform emotional guidance is received, the user is guided to perform emotional adjustment according to a preset emotional guidance manner.