KR20230081012A - Method, device, and program for education comprising deep-learning module enabling generating indicators of emotion - Google Patents

Abstract

The present invention relates to a video educational method based on changes in emotion and concentration states performed by a computing device. A video educational method includes the steps of: receiving image data from an external electronic device; converting the received image data into Mat data; inputting the Mat data into a deep learning module learned based on Korean input data to convert the Mat data into emotion and concentration state values; and inserting the emotional and concentration state values into a control logic.

Description

Educational device, method, and program for calculating emotional quotient based on emotion index and concentration index

The present invention relates to an educational apparatus, method, and program for calculating an emotional index based on an emotional index and a concentration index.

In the past, education to enhance intellectual ability was valued, but in the 21st century, the importance of emotional ability is increasing. In particular, many research results have been announced that emotional education in early childhood and elementary school plays an important role in ego-emotional development, empathy formation, sociality, and creativity promotion. Although this emotional care is becoming a key factor in increasing the learning effect, education sites are showing a passive attitude to utilize the emotional part of students, so the reality is that the improvement of learning efficiency is not being achieved. As the form of learning shifts from offline to online after the COVID-19 pandemic, the importance of emotional care is on the rise due to the lack of emotional management in face-to-face situations.

Therefore, it is required to develop a technology that recognizes students' emotional expressions and physical reactions in real time in a non-face-to-face learning situation and measures various emotional indicators such as the student's current feeling and concentration state. In particular, it is required to develop a technology that can sympathize with the emotions students are feeling and lead them in a positive direction through various devices. In order to effectively derive these students' emotions, it is necessary to develop related technologies because it is necessary to derive qualitative indicators as well as quantitative indicators.

An object of the present invention is to provide an educational device, method, and program for calculating an emotional index based on an emotional index and a concentration index in order to solve the above problems.

An image education method based on changes in emotion and concentration state performed by a computing device according to an embodiment of the present invention includes receiving image data from an external electronic device; detecting face data from the received image data; obtaining an emotion index by inputting the facial data to an emotion recognition model based on facial expression recognition technology; and acquiring a concentration index by inputting the face data into an rPPG model based on a change in optical blood flow.

The emotion recognition model may include a first emotion recognition model that determines any one of positive, negative, and neutral emotion types and calculates each probability value.

The emotion recognition model may include a second emotion recognition model that determines any one emotion type among joy, surprise, sadness, anger, fear, displeasure, and calmness and calculates a probability value for each emotion.

The rPPG model derives heart rate and heart rate variability after detecting a face from an input image, determines a concentration state in normal, concentration, and immersion stages, and calculates respective probability values.

The emotion recognition model,

It may be characterized in that it is learned based on input data of Koreans and elementary school students.

The emotion recognition model,

Generate network data through convolutional networks and max pooling,

Deep learning may be performed using a Rectified Linear Unit (ReLu) obtained by rectifying the generated network data as an activation function.

A learning index may be calculated based on the concentration index and the emotion index.

According to another embodiment of the present invention, an electronic device includes a camera, a recording device, an output device, a memory, a transceiver, and at least one processor, wherein the at least one processor performs the education method according to the above description. is configured to

Describes a computer program recorded in a computer readable storage medium configured to perform the education method according to the above-described content through an electronic device according to another embodiment of the present invention.

Using this image recognition technology, it is possible to measure three detailed types of emotion indicators as follows. The first emotional state indicator and the second emotional state indicator use facial expression recognition technology based on AI deep learning technology, and the concentration state indicator uses rPPG technology that analyzes the amount of light blood flow change in a facial image.

A learning index can be calculated based on the measured emotion and concentration indicators.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 illustrates a communication system according to various embodiments of the present invention.
2 is a block diagram of a configuration of a video electronic device based on changes in emotion and concentration state according to various embodiments of the present disclosure.
Figure 3 shows a block diagram of the configuration of a server according to various embodiments of the present invention.
4 illustrates an education method performed by an electronic device to enable conversion of emotions and concentration states according to various embodiments of the present disclosure.
5 illustrates the structure of a machine learning model according to various embodiments of the present invention.
6 illustrates an example in which the artificial intelligence logic according to the present invention derives result values for emotion, posture, and concentration.
7 is an exemplary diagram for explaining the architecture and learning of a deep learning module according to the present invention.
8 illustrates an example of a process of detailed models constituting an emotion recognition model based on transfer learning in detail.
9 is a diagram exemplarily illustrating the learning effect of the emotion index derived in the present invention.
10 illustrates an education method capable of deriving an emotion index according to various embodiments of the present disclosure.
11 shows an example of an additional index required for calculating the emotional index in addition to the deep learning module.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. This invention may be embodied in many different forms and is not limited to the embodiments set forth herein.

An image education method based on changes in emotion and concentration performed by a computing device includes receiving image data from an external electronic device; converting the received image data into Mat data; converting the Mat data into emotional and concentration values by inputting the Mat data into an emotion recognition SDK module; inserting the emotional state and concentration state values into control logic; and calling a chatbot message UI linked to the control logic;

1 illustrates a communication system according to various embodiments of the present invention.

Referring to FIG. 1 , a communication system according to various embodiments of the present disclosure includes an electronic device 110, a wired/wireless communication network 120, and a server 130. The server 130 obtains image data from the user's electronic device 110 through the wired/wireless communication network 120, derives an emotional state and a concentration state, and then displays a chatbot message UI corresponding to the corresponding state. It is transmitted back to the electronic device 110 of the user through the wireless communication network 120 .

The electronic device 110 captures and transmits image data including face and posture information for the learning state of the user according to a request of the server 130 through the wired/wireless communication network 120 . The electronic device 110 includes a memory that can store information, a transceiver that can transmit and receive information, and at least one processor that can perform information calculation, such as a personal computer, a cellular phone, a smart phone, and a tablet computer. It may be an electronic device including. The type of electronic device 110 is not limited.

The wired/wireless communication network 120 provides a communication path through which the electronic device 110 and the server 130 can transmit and receive signals and data to each other. The wired/wireless communication network 120 is not limited to a communication method according to a specific communication protocol, and an appropriate communication method may be used according to an implementation example. For example, when configured as an Internet Protocol (IP) based system, the wired/wireless communication network 120 may be implemented as a wired/wireless Internet network, and the electronic device 110 and the server 130 are implemented as mobile communication terminals. If possible, the wired/wireless communication network 120 may be implemented as a wireless network such as a cellular network or a wireless local area network (WLAN) network.

The server 130 receives image data including face and posture information for the learning state of the user from the electronic device 110 through the wired/wireless communication network 120 . The server 130 may be an electronic device including a memory capable of storing information, a transmitting/receiving unit capable of transmitting and receiving information, and at least one processor capable of performing information calculation.

2 illustrates a block diagram of a configuration of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 2 , an electronic device 110 according to various embodiments of the present disclosure includes a memory 111, a transceiver 112, and a processor 113.

The memory 111 may include volatile memory, non-volatile memory, or a combination of volatile and non-volatile memories. Also, the memory 111 may provide stored data according to a request of the processor 113 .

The transceiver 112 is connected to the processor 113 and transmits and/or receives signals. All or part of the transceiver 113 may be referred to as a transmitter, a receiver, or a transceiver. The transceiver 112 is a wired access system and a wireless access system, such as an institute of electrical and electronics engineers (IEEE) 802.xx system, an IEEE Wi-Fi system, a 3rd generation partnership project (3GPP) system, and a 3GPP long term evolution (LTE) system. , 3GPP 5G new radio (NR) system, 3GPP2 system, at least one of various wireless communication standards such as Bluetooth may be supported.

The processor 113 may be configured to implement the procedures and/or methods proposed in the present invention. The processor 113 controls overall operations of the electronic device 110 to provide content based on machine learning analysis of biometric information. For example, the processor 116 transmits or receives information or the like through the transceiver 115 . Processor 116 also writes data to and reads data from memory 112 . Processor 116 may include at least one processor.

Figure 3 shows a block diagram of the configuration of the server 130 according to various embodiments of the present invention.

Referring to FIG. 3 , a server 130 according to various embodiments of the present disclosure includes a memory 131 , a transceiver 132 and a processor 133 . The server 130 may be a type of electronic device.

The server 130 receives image data including face and posture information about the learning state of the user from the electronic device 110 through the wired/wireless communication network 120 . The server 130 converts the received image data into Mat data, converts the Mat data into emotional and concentration values by inputting the Mat data into the emotion recognition SDK module, and converts the emotional state and concentration state values into control logic. and call the chatbot message UI linked to the control logic.

The memory 131 is connected to the transceiver 132 and may store information received through communication. In addition, the memory 131 is connected to the processor 133 and may store data such as a basic program for operation of the processor 133, an application program, setting information, and information generated by operation of the processor 133. . The memory 131 may include volatile memory, non-volatile memory, or a combination of volatile and non-volatile memories. Also, the memory 131 may provide stored data according to a request of the processor 133 .

The transceiver 132 is connected to the processor 133 and transmits and/or receives signals. All or part of the transceiver 132 may be referred to as a transmitter, a receiver, or a transceiver. The transceiver 132 is a wired access system and a wireless access system, such as an institute of electrical and electronics engineers (IEEE) 802.xx system, an IEEE Wi-Fi system, a 3rd generation partnership project (3GPP) system, and a 3GPP long term evolution (LTE) system. , 3GPP 5G new radio (NR) system, 3GPP2 system, at least one of various wireless communication standards such as Bluetooth may be supported.

The processor 133 may be configured to implement the procedures and/or methods proposed in the present invention. The processor 133 converts image data into emotional and concentration values by inputting them to the emotion recognition SDK module, and controls overall operations of the server 130, such as control logic using the emotional state and concentration values as input values. For example, the processor 133 transmits or receives information or the like through the transceiver 132 . Also, the processor 133 writes data to and reads data from the memory 131 . The processor 135 may include at least one processor.

4 illustrates an education method performed by an electronic device to enable conversion of emotions and concentration states according to various embodiments of the present disclosure.

Referring to FIG. 4, receiving image data from an external electronic device (S100), converting the received image data into Mat data (S200), inputting the Mat data to the emotion recognition SDK module to enter a state of emotion and concentration Converting into values (S300), inserting the emotional state and concentration state values into control logic, and calling a chatbot message UI linked to the control logic (S400).

Step S100 is a step of receiving image data including a face captured or transmitted in real time from the user's electronic device 110 . The training method according to the present invention derives facial expression-based emotion and concentration state based on image data including the user's facial expression, and provides a reactive chatbot UI to the user through logic control according to quantitative numerical values of the corresponding emotion and concentration state. By providing the electronic device 110 of, it is to perform the role of an assistant teacher capable of responding to the user's reaction in a non-face-to-face situation. Accordingly, it may be required that the image data be data having a resolution and image quality by which facial expressions of at least a plurality of points in time can be confirmed. The aforementioned server 130 may receive data from the electronic device 100 through the wired/wireless communication network 120 .

Step S200 may process image data received through an OpenCV module or the like. For example, in this step, image data can be converted into a Mat (matrix) format handled by the OpenCV image processing library. OpenCV stands for Open Source Computer Vision C Library, which represents a language used for real-time image processing.

Mat data represents the data format of m by n channel structure, and can include not only variables such as basic data (char, int), but also various types of data defined by OpenCV, such data formats include complex numbers, mono.color , solids, vectors, clusters, tensors, and histograms.

Step S300 is a step of converting the Mat data into emotion and concentration values by inputting the Mat data to the emotion recognition SDK module. In the present invention, the types of emotions that the user wants to show during learning and recognize in the module may include seven. In the present invention, the concentration state indicated by the user during learning may include three states.

The emotional state may include a plurality of emotional state values with a sum of 1.0, and the concentration state may include a plurality of concentration state values with a sum of 1.0. Specifically, the emotional state value and the concentration state value represent probability values, and the sum of the corresponding states must always satisfy 1.0. The 7 emotions and 3 concentration states are calculated as probability values, and the sum of the 7 emotions always has a probability value of 1.0, and the sum of the 3 concentration states also always has a probability value of 1.0. That is, it is possible to form a closed variable space without additional variables.

The emotion recognition SDK module can discriminate emotions based on facial expression by inputting the user's facial image. In addition, the module may determine the concentration state by calculating cardiac response information.

The emotion recognition SDK module returns the concentration/immersion value as a probability value in the form of a decimal point, and based on the value, emotion recognition/discrimination may be possible.

Referring to FIG. 7 , specifically, the seven emotions may include joy, surprise, fear, anger, displeasure, sadness, and calmness.

Specifically, the three concentration states may represent normal, concentration, and immersion.

Step S400 may be a step of inserting emotional state and concentration state values into control logic. The control logic may control the chatbot to output an appropriate response to enhance the user's learning efficiency according to changes in emotional state and concentration state values or corresponding values, and changes per unit time.

The control logic calls a corresponding chatbot message UI when the emotional state value or the concentration state value reaches a preset value. More specifically, the control logic calls a corresponding chatbot message UI when the emotional state value or the concentration state value reaches a preset value when converted into an average value for the preset reference time. can Details will be described below with reference to FIG. 6 .

Specifically, look at the control logic according to FIG. 6 . After recognizing the learner's face from the image data provided by the user's electronic device, the first depth can be divided into emotion (emotion) determination, face detection, and concentration state determination. The learner's emotional state value and concentration state value derived from the above-described emotion recognition SDK module are allocated to the emotion determination unit and the concentration state determination unit, respectively, and basic face detection data may correspond to the face detection unit.

In the second depth corresponding to the emotion determination unit, one of seven types of emotional states of the learner may be derived based on the result input to the emotion determination unit. This emotion determination unit can derive a corresponding result through machine learning (machine learning) modules such as deep learning and artificial intelligence, and is not limited to a specific technology.

In the third depth corresponding to the emotion determination unit, an appropriate control logic may be calculated based on emotional state values corresponding to the seven emotions determined in the second depth. For example, when a positive emotional state value corresponds to level 0 (less than 0.5) for a long period of time, control logic may be activated to drive a care chatbot UI for user management. For example, if the positive emotional state value is 1st level (0.5 to 0.75) and 2nd level (0.75 to 1.0) for a long period of time, the control logic is such that the chatbot UI responds to the question in a text interface with short-answer responses and response screens, end of activity and It can be controlled to implement guides in various types such as screen switching, activity end, and activity recommendation screens.

In the second depth corresponding to the face detection unit, based on the extracted face detection data, a result of the appropriateness of the posture and a determination of whether to leave the seat may be derived through the posture determination unit and the seat departure determination unit.

In the third depth corresponding to the face detection unit, when the posture judgment value derived from the posture determination unit exceeds the appropriate area, the output signal is sent to the user's electronic device to implement a guide to the dialog and face position reset screen by controlling the chatbot UI can be sent

In the third depth corresponding to the seat departure determination unit, when the derived seat departure judgment value exceeds the appropriate area, the chatbot UI is controlled, and the user's electronic interface is implemented to implement a guide through a voice interface during learning It can transmit an output signal to the device.

In the second depth corresponding to the concentration state determiner, it can be divided into a normal state, a concentration state, and an immersion state corresponding to three categories.

In the third depth corresponding to the concentration state determination unit, if the normal state is extended for a longer period of time than a preset time, the chatbot UI is controlled, and if the normal state continues for a long time, it is determined that care is required, and guidance through a voice interface during learning An output signal can be sent to the user's electronic device to implement.

If the concentration and immersion state decreases per unit time than the preset value, the control logic is such that the chatbot UI responds to questions in various types such as short-answer responses and response screens in the text interface, activity end and screen transition, activity end and activity recommendation screen, etc. Guides can be controlled to implement.

As described above, in the 3-step depth structure of the control logic, when the control logic determines that the learning efficiency is reduced based on the user's emotion/concentration value, the seat is empty, or a change is required, the control logic According to the control, a chatbot message UI may be called and a signal may be transmitted to be controlled in the user's electronic device.

In short, the present method (S400) calls the care chatbot message UI when the positive emotional state value in the emotion derivation result is less than the reference value for a preset time or when the concentration state value is less than the reference value for a preset time in the concentration derivation result can do.

In short, in the present method (S400), when the positive emotional state value in the emotion derivation result has a large negative change amount during the unit time, or the concentration state value in the concentration deduction result has a large negative change amount during the unit time, the guide chatbot message UI is displayed. can be called

The method includes a step of calculating a user state change before and after the UI appearance time after calling the chatbot message UI (S500).

Step S500 may be a step of calculating a change in the user's emotion and/or concentration state after calling the chatbot message UI and determining the effectiveness of the chatbot message UI as feedback. Specifically,

According to a change in at least one of a change in emotional state value, a change in concentration state value, a change in emotional state category, or a change in concentration state category

The method may include feeding back a chatbot message UI corresponding to a corresponding control logic when the emotional state changes from the first state to the second state.

When the first state is negative emotion and the second state is positive emotion, performing reinforcement feedback on the chatbot message UI corresponding to the control logic;

When the first state is positive emotion and the second state is negative emotion, performing negative reinforcement on the chatbot message UI corresponding to the control logic.

In addition, the present invention, the chatbot message UI may further include a step (not shown) of transmitting voice data to an external electronic device by communicating with the STT/TTS API.

5 illustrates the structure of a machine learning model according to various embodiments of the present invention.

The structure of a multi-layer perceptron (MLP) for providing content based on machine learning analysis of biometric information according to various embodiments of the present invention is shown.

Deep learning, as one of the emerging technologies in the field of machine learning, is a neural network composed of a plurality of hidden layers and a plurality of hidden units included in them. When low-level features are input to a deep learning model, these basic features are transformed into high-level features that can better explain the problem to be predicted while passing through a plurality of hidden layers. In this process, since prior knowledge or intuition of an expert is not required, subjective factors in feature extraction can be removed, and a model with higher generalization ability can be developed. Furthermore, in the case of deep learning, since feature extraction and model construction are composed of one set, there is an advantage in that the final model can be formed through a simpler process compared to existing machine learning theories.

A multi-layer perceptron (MLP) is a type of artificial neural network (ANN) with multiple nodes based on deep learning. Each node uses a non-linear activation function with neurons similar to animal connection patterns. This nonlinear property makes it possible to linearly distinguish inseparable data.

Referring to FIG. 5 , the artificial neural network 500 of the MLP model according to various embodiments of the present invention includes one or more input layers 510, a plurality of hidden layers 530, and one or more outputs. It consists of an output layer (550).

Input data such as an RGB value of each pixel in at least one ultrasound image per unit time is input to a node of the input layer 510 . Here, the user's biometric information, eg, electrocardiogram information, concentration level, happiness emotion intensity information, and adjusted content information, eg, content genre, content topic, and content channel information 511 are deep It corresponds to the basic characteristic (low level feature) of the learning model.

A node of the hidden layer 530 performs calculations based on input factors. The hidden layer 530 is a layer in which units defined by a plurality of nodes formed by integrating the user's biometric information and the information 511 of the adjusted content are stored. As shown in FIG. 5 , the hidden layer 530 may include a plurality of hidden layers.

For example, when the hidden layer 530 is composed of a first hidden layer 531 and a second hidden layer 533, the first hidden layer 531 includes user biometric information and adjusted content information 511 As a layer in which first units 532 defined by a plurality of nodes formed by consolidating are stored, the first unit 532 corresponds to a higher characteristic of the user's biometric information and the information 511 of the adjusted content. The second hidden layer 533 is a layer in which second units 534, defined as a plurality of nodes formed by consolidating the first units of the first hidden layer 531, are stored. Corresponds to the upper characteristics of 1 unit 532.

Nodes of the output layer 550 represent calculated prediction results. The output layer 550 may include a plurality of prediction result units 551 . Specifically, the plurality of prediction result units 551 may include two units of a true unit and a false unit. Specifically, the true unit is a prediction result unit meaning that the emotional state value and concentration state value of the user's face data are highly likely to be higher than the threshold value after adjusting the content to the adjusted content, and the false unit is the content adjustment to the adjusted content. It is a prediction result unit that means that the possibility that the emotion state value and the concentration state value of the user's face data are higher than the threshold value is low.

Weights are assigned to connections between the prediction result units 551 and the second units 534 included in the second hidden layer 533, which is the last layer among the hidden layers 530. Based on these weights, it is predicted whether the emotional state value and the concentration state value of the user's face data are greater than or equal to a threshold value after adjusting the content to the adjusted content.

The artificial neural network 500 of the MLP model learns by adjusting learning parameters. According to one embodiment, the learning parameters include at least one of a weight and a variance. The learning parameters are iteratively adjusted through an optimization algorithm called gradient descent. Each time a prediction result is computed from a given data sample (forward propagation), the performance of the network is evaluated through a loss function that measures the prediction error. Each learning parameter of the artificial neural network 500 is adjusted by gradually increasing in the direction of minimizing the value of the loss function, and this process is called back-propagation.

Referring to FIG. 7 , the facial expression recognition deep learning module based on ai deep learning technology according to the present invention is a model learned using an image dataset of Korean elementary school students (5 to 13 years old) as input data. 7 is an exemplary diagram for explaining the architecture and learning of a deep learning module according to the present invention.

Referring to FIG. 7 , pre-processing may be performed on an image data set of 5,000 to 10,000 or Korean elementary school students at a predetermined resolution and size.

Thereafter, the preprocessed image dataset is inserted into a first convolutional network, and at this time, the number of channels of the first convolutional network may be determined as n by n, where n may be 24 by way of example.

Thereafter, max-pooling is performed on the first convolution to prevent overfitting, and the size of the network can be reduced by extracting the maximum value from the data set. For example, the network after max pooling may be determined as n/2 by n/2, and may be illustratively a 12 by 12 network, but is not limited thereto.

Thereafter, the max-pooled data may be inserted into a second convolutional network. In this case, the number of channels of the second convolutional network may be determined by m by m, for example, m may be 8 by way of example, but is not limited thereto.

The second convolution can be max-pooled again.

Afterwards, the Rectified Linear Unit (ReLu), which rectifies the final network, can be inserted into the front of the neural network for deep learning learning as an active function. Through this learning, deep learning can be learned to derive whether an input image corresponds to any one of a plurality of emotional states or a plurality of concentration states.

The present invention can derive quantitative indicators for the emotional state, learning state, and concentration state of Korean students by deep learning through the transfer learning architecture of these images of Korean elementary school students.

Referring to FIG. 8 , the emotion recognition derivation model according to the present invention may be composed of the following three detailed models.

The first emotion recognition model may perform facial expression analysis by applying the ai deep learning module after detecting a face from an input image, and calculate probability values for each of three emotion types (positive, negative, and neutral). A quantitative emotional index may be derived based on the probability value.

The second emotion recognition model performs facial expression analysis by applying the above-described ai deep learning module after detecting a face from an input image, and probability values for each of the seven emotion types (joy, surprise, sadness, anger, fear, displeasure, calmness) can be calculated. A quantitative emotional index may be derived based on the probability value. The first emotion recognition model and the second emotion recognition model may be applied alternatively or complementary to each other, so that a more precise emotion index may be calculated.

The concentration recognition model may derive heart rate and heart rate variability after face detection from an input image. By applying rPPG (remote photoplethysmography) technology to the derived heart rate variability data, heart rate and heart rate variability can be measured and analyzed. Based on the heart rate variability analysis result, the concentration recognition model calculates a probability value for each of the three stages of concentration (normal → concentration → immersion), and derives the concentration state index. A concentration index can be derived based on the concentration state indicator. The concentration state index can be used as a basic value for deriving a learning index.

Finally, the emotion recognition derivation model may derive a learning index based on quantitative index values derived from the first emotion recognition model, the second emotion recognition model, and the concentration recognition model. The learning index is an indicator for quantitatively grasping the user's learning status, and more precise calculation is possible by additionally utilizing student data and data generated in the learning situation, such as learning time, number of questions, and absentee status. It could be possible.

Specifically, referring to FIG. 10 , the education method according to the present invention includes receiving image data from an external electronic device (S110), detecting face data from the received image data (S210), and Obtaining an emotion index by inputting it into an emotion recognition model based on facial expression recognition technology (S310), and acquiring a concentration index by inputting the face data into an rPPG model based on light blood flow change (S410), and obtaining a concentration index and the emotion index A step of calculating a learning index based on the index (S510) may be included.

Step S110 is a step of receiving image data including a face captured or transmitted in real time from the user's electronic device 110 .

Step S210 is a step of extracting a user's facial expression from the received image data and extracting a data set for deriving emotion and concentration based on the facial expression.

In step S310, based on the first emotion recognition module and the second emotion recognition module, probability values for each of three emotion types (positive, negative, neutral) are calculated or seven emotion types (joy, surprise, sadness, anger, Fear, displeasure, calmness) may be a step of calculating each probability value.

In step S410, heart rate and heart rate variability may be derived after face detection from the input image. This step measures and analyzes heart rate and heart rate variability by applying rPPG (remote photoplethysmography) technology to the derived heart rate variability data.

Step S510 is a step of calculating a learning index based on the concentration index and the emotion index.

Emotional index and learning index may be built into a database. The database may store the emotional index and learning index in the form of a time series to enable comprehensive analysis.

Referring to FIG. 11 , an example of an additional index necessary for calculating the emotional index in addition to the deep learning module will be described below.

In the case of implementing the embodiment of the present invention using hardware, ASICs (application specific integrated circuits) or DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices) configured to perform the present invention , FPGAs (field programmable gate arrays), etc. may be provided in the processor of the present invention.

Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable medium. In addition, the structure of data used in the above-described method may be recorded on a computer-readable storage medium through various means. Program storage devices, which may be used to describe a storage device containing executable computer code for performing various methods of the present invention, should not be construed as including transitory objects such as carrier waves or signals. do. The computer-readable storage media includes storage media such as magnetic storage media (eg, ROM, floppy disk, hard disk, etc.) and optical reading media (eg, CD-ROM, DVD, etc.).

The embodiments described above are those in which elements and features of the present invention are combined in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form not combined with other components or features. In addition, it is also possible to configure an embodiment of the present invention by combining some elements and/or features. The order of operations described in the embodiments of the invention may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment. It is obvious that claims that do not have an explicit citation relationship in the claims can be combined to form an embodiment or can be included as new claims by amendment after filing.

It will be clear to those skilled in the art that the present invention can be embodied in other forms without departing from the technical spirit and essential characteristics of the present invention. Accordingly, the above embodiments should be considered in all respects as illustrative rather than restrictive. The scope of the present invention should be determined by reasonable interpretation of the appended claims and all possible changes within the equivalent scope of the present invention.

100: user 110: electronic device
111: memory 112: transceiver
113: processor 114: camera
115: recording device 116: output device
120: wired/wireless communication network 130: server
131: memory 132: transceiver
133: processor 500: artificial neural network
510: input layer 511: input information
530: hidden layer 531: first hidden layer
532: first unit 533: second hidden layer
534: second unit 550: output layer
551: prediction result unit

Claims (9)

In the image education method based on the change of emotion and concentration state performed by a computing device,
Receiving image data from an external electronic device;
detecting face data from the received image data;
obtaining an emotion index by inputting the facial data to an emotion recognition model based on facial expression recognition technology; and
and acquiring a concentration index by inputting the face data into an rPPG model based on the amount of change in optical blood flow. According to claim 1,
The emotion recognition model includes a first emotion recognition model that determines any one of positive, negative, and neutral emotion types and calculates each probability value. According to claim 2,
The emotion recognition model includes a second emotion recognition model that determines any one emotion type among joy, surprise, sadness, anger, fear, displeasure, and calmness and calculates each probability value. According to claim 1,
The rPPG model derives heart rate and heart rate variability after face detection from the input image, determines the concentration state in normal, concentration, and immersion stages, and calculates each probability value. According to claim 1,
The emotion recognition model,
An educational method characterized in that it is learned based on input data of Korean students and elementary school students. According to claim 1,
The emotion recognition model,
Generate network data through convolutional networks and max pooling,
A training method characterized in that deep learning is performed by using a Rectified Linear Unit (ReLu) obtained by rectifying the generated network data as an activation function. According to claim 1,
Calculating a learning index based on the concentration index and the emotion index; education method characterized in that it further comprises. In electronic devices,
a memory, a transceiver and at least one processor;
The at least one processor is configured to perform the education method according to any one of claims 1 to 7,
educational device. A computer program recorded on a computer readable storage medium configured to perform the education method according to any one of claims 1 to 7 through an electronic device.

Images