KR20220029850A - Method and apparatus for supporting user's learning concentration - Google Patents

Abstract

The present invention relates to a method and apparatus for supporting the learning concentration of a user. Specifically, the present invention relates to a method and apparatus for obtaining and analyzing biometric information while a user is watching learning content, outputting a ventilation sequence when the concentration and immersion of the user are low, and adaptively adjusting the intensity of the ventilation sequence and an output element type according to a change in the concentration of the user.

Description

METHOD AND APPARATUS FOR SUPPORTING USER'S LEARNING CONCENTRATION

The present invention relates to a method and apparatus for supporting a user's learning concentration. Specifically, the present invention obtains and analyzes biometric information while the user is watching learning content, outputs a ventilation sequence when the user's concentration and immersion are low, and according to the change in the concentration of the user, the intensity and output elements of the ventilation sequence It relates to a method and apparatus for adaptively adjusting a type.

Recently, due to the outbreak of the Corona 19 virus, group meeting learning has decreased, and each user is studying alone while leaving a study video in a time lapse format or sharing it via social network service (SNS). In this way of studying alone, it is difficult for the user to check whether he or she is concentrating and immersed by himself because he does not study with others.

If biometric information including concentration and immersion during user learning is acquired non-face-to-face using an electronic device and a special effect sequence can be output to a user interface/user experience (UI/UX) when a decrease in concentration is detected, the user It can improve the overall concentration of learning by reinvigorating students' motivation to learn.

Accordingly, there is a need for a method and apparatus for outputting a ventilation sequence when the user's concentration level is low and adaptively adjusting the intensity of the sequence and the type of output element according to the change in the user's concentration and immersion level.

Republic of Korea Patent No. 10-1656072 (2020.03.30.) (Multisensory service system and multisensory service method using the same for improving concentration in the childcare environment)

The present invention aims to solve the following problems in order to solve the above problems.

The present invention recognizes biometric information according to non-verbal factors such as electrocardiogram information, concentration level, and immersion level, so that accurate biometric information is identified even if the user does not directly express it, and based on this, to provide a ventilation sequence to support learning concentration do.

An object of the present invention is to provide a method and apparatus for outputting a ventilation sequence when the user's concentration level is low, and adaptively adjusting the intensity of the sequence and the type of output element according to the change of the user's concentration and immersion level.

The problems to be solved of the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

According to various embodiments of the present disclosure, a method of operating an electronic device for supporting a user's learning concentration is provided. The electronic device includes a camera, an output device, a memory, a transceiver, and at least one processor, and the output device includes at least one of a display and a speaker. The operation method includes a process of outputting content, a process of acquiring a video image of an upper body including a user's face, and a living body including one or more cardiac response effective variables from micro-movements of the user's face in the video image a process of acquiring information; a process of acquiring the concentration value and the concentration value of the user from the one or more cardiac response effective variables; whether the average value per minute of the concentration value of the user falls below a threshold concentration value; and The process of determining whether to output a ventilation sequence based on whether the average value per minute of the engagement value falls below a threshold engagement value, wherein the ventilation sequence is composed of a plurality of elements, and the output of the ventilation sequence is When it is determined, the process of outputting the ventilation sequence as an initially set intensity and an output element type, the output element type is at least one output element among the plurality of elements, and based on a change in the concentration value, The process of adjusting the intensity and the type of the output element, and when the average value per minute of the concentration value and the average value per minute of the engagement value are restored to more than the threshold concentration value and the threshold engagement value, the output of the ventilation sequence including the process of stopping

According to various embodiments of the present disclosure, an electronic device for supporting a user's learning concentration is provided. The electronic device includes a camera, an output device, a memory, a transceiver, and at least one processor, and the processor performs an operating method of the electronic device for supporting a user's learning concentration according to various embodiments of the present disclosure. is composed

According to various embodiments of the present disclosure, there is provided a computer program configured to perform an operating method of an electronic device for supporting a user's learning concentration and recorded in a computer-readable storage medium.

The present invention recognizes biometric information according to non-verbal factors such as electrocardiogram information, concentration, and immersion, so that the user does not directly express biometric information, and based on this, it is possible to provide a ventilation sequence to support learning concentration.

The present invention may provide a method and apparatus for outputting a ventilation sequence when the user's concentration level is low, and adaptively adjusting the intensity of the sequence and the type of output element according to the change in the user's concentration and immersion level.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

1 illustrates a communication system according to various embodiments of the present invention.
2 is a block diagram illustrating a configuration of an electronic device according to various embodiments of the present disclosure.
3 is a block diagram illustrating a configuration of a server according to various embodiments of the present disclosure.
4 illustrates a method of operating an electronic device according to various embodiments of the present disclosure.
5 illustrates an example of a biometric information acquisition process according to various embodiments of the present disclosure.
6 illustrates an example of a process of detecting face points from a user's video image according to various embodiments of the present disclosure.
7 illustrates an example of a process of recognizing heart response information and user expression information based on a user's image according to various embodiments of the present disclosure.
8 illustrates a process of collecting user's biometric information and providing a service solution, according to various embodiments of the present disclosure.
9 illustrates an example of biometric information according to various embodiments of the present disclosure.
10 illustrates a relationship between focus and engagement according to various embodiments of the present disclosure.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art can easily implement them. The present invention may be implemented in several different forms and is not limited to the embodiments described herein.

1 illustrates a communication system according to various embodiments of the present invention.

Referring to FIG. 1 , a communication system according to various embodiments of the present disclosure includes an electronic device 110 , a wired/wireless communication network 120 , and a server 130 . The electronic device 110 obtains biometric information of the user 100 and outputs audiovisual content to the user 100 .

The electronic device 110 receives, from the server 130 through the wired/wireless communication network 120 , the time series mapping information for the user's concentration level, the intensity of the ventilation sequence, and the type of output element in the ventilation sequence, and the server 130 ) is a terminal device capable of transmitting information on the newly acquired concentration and immersion level of the user. The electronic device 110 includes a memory capable of storing information, a transceiver capable of transmitting and receiving information, and at least one processor capable of performing an operation of information, such as a personal computer, a cellular phone, a smart phone, and a tablet computer. It may be an electronic device including

The wired/wireless communication network 120 provides a communication path through which the electronic device 110 and the server 130 can transmit and receive signals and data to each other. The wired/wireless communication network 120 is not limited to a communication method according to a specific communication protocol, and an appropriate communication method may be used according to an implementation example. For example, when configured as an Internet Protocol (IP)-based system, the wired/wireless communication network 120 may be implemented as a wired/wireless Internet network, and the electronic device 110 and the server 130 are implemented as mobile communication terminals. In this case, the wired/wireless communication network 120 may be implemented as a wireless network such as a cellular network or a wireless local area network (WLAN) network.

The server 130 transmits, to the electronic device 110 through the wired/wireless communication network 120 , time series mapping information for the user's concentration level, the intensity of the ventilation sequence, and the type of output element in the ventilation sequence, and the electronic device ( 110) is a terminal device capable of receiving information on the newly acquired concentration and immersion level of the user. The server 130 may be an electronic device including a memory capable of storing information, a transceiver capable of transmitting and receiving information, and at least one processor capable of performing an operation of information.

2 is a block diagram illustrating a configuration of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 2 , an electronic device 110 according to various embodiments of the present disclosure includes a memory 111 , a transceiver 112 , a processor 113 , a camera 114 , and an output device 115 .

The memory 111 is connected to the transceiver 112 and received from the server 130 through communication, the intensity of the ventilation sequence, and time series mapping information for the type of output element in the ventilation sequence, ventilation sequence data, etc. can be saved. In addition, the memory 111 is connected to the processor 113 and may store data such as a basic program for the operation of the processor 113 , an application program, setting information, and information generated by the operation of the processor 113 . . The memory 111 may be configured as a volatile memory, a non-volatile memory, or a combination of a volatile memory and a non-volatile memory. In addition, the memory 111 may provide stored data according to the request of the processor 113 .

The transceiver 112 is connected to the processor 113 and transmits and/or receives a signal. All or part of the transceiver 113 may be referred to as a transmitter, a receiver, or a transceiver. The transceiver 112 is a wired access system and wireless access systems, such as an Institute of Electrical and Electronics Engineers (IEEE) 802.xx system, an IEEE Wi-Fi system, a 3rd generation partnership project (3GPP) system, and a 3GPP long term evolution (LTE) system. , 3GPP 5G NR (new radio) system, 3GPP2 system, may support at least one of various wireless communication standards such as Bluetooth (bluetooth).

The processor 113 may be configured to implement the procedures and/or methods proposed in the present invention. The processor 113 controls overall operations of the electronic device 110 to support the user's learning concentration. For example, the processor 113 transmits or receives information through the transceiver 112 . In addition, the processor 113 writes data to and reads data from the memory 111 . The processor 113 may include at least one processor.

The camera 114 may be configured to acquire a video image of the upper body including the user's face. The image image acquired by the camera 114 may be stored in the memory 111 and analyzed by the processor 113 .

The output device 115 includes one or more of a display and a speaker. The output device 115 may be configured to output content or output a ventilation sequence under the control of the processor 113 .

3 is a block diagram illustrating a configuration of a server according to various embodiments of the present disclosure.

Referring to FIG. 3 , the server 130 according to various embodiments of the present disclosure includes a memory 131 , a transceiver 132 , and a processor 133 .

The server 130 refers to a server capable of storing accumulated data for mapping of content information and changes in biometric information according to a user's content viewing, and transmitting content data according to a request from the electronic device 110 .

The memory 131 is connected to the transceiver 132 and may store information received through communication and the like. Also, the memory 131 may be connected to the processor 133 and store data such as a basic program for the operation of the processor 133 , an application program, setting information, and information generated by the operation of the processor 133 . . The memory 131 may be configured as a volatile memory, a non-volatile memory, or a combination of a volatile memory and a non-volatile memory. In addition, the memory 131 may provide stored data according to the request of the processor 133 .

The transceiver 132 is connected to the processor 133 and transmits and/or receives a signal. All or part of the transceiver 132 may be referred to as a transmitter, a receiver, or a transceiver. The transceiver 132 is a wired access system and wireless access systems, which are an Institute of Electrical and Electronics Engineers (IEEE) 802.xx system, an IEEE Wi-Fi system, a 3rd generation partnership project (3GPP) system, and a 3GPP long term evolution (LTE) system. , 3GPP 5G NR (new radio) system, 3GPP2 system, may support at least one of various wireless communication standards such as Bluetooth (bluetooth).

The processor 133 may be configured to implement the procedures and/or methods proposed in the present invention. The processor 133 controls overall operations of the server 130 in order to support the user's concentration on learning. For example, the processor 133 transmits or receives information and the like through the transceiver 132 . In addition, the processor 133 writes data to and reads data from the memory 131 . The processor 135 may include at least one processor.

4 illustrates a method of operating an electronic device according to various embodiments of the present disclosure. 4 , the electronic device includes a camera, an output device, a memory, a transceiver, and at least one processor, and the output device includes one or more of a display and a speaker.

Referring to FIG. 4 , in step S401 , the electronic device outputs content through an output device.

In step S402 , the electronic device acquires an image of the upper body including the user's face through the camera.

In step S403 , the electronic device acquires biometric information including one or more cardiac response effective variables from the micro-movement of the user's face in the image image through the processor.

According to various embodiments of the present disclosure, in step S403, the process of detecting the facial feature region of the user from the image image, extracting unconscious facial microtremor data from the facial feature region, and removing noise due to conscious movement It may include the process of performing the process, and the process of acquiring the heart rate and amplitude information of the user from the unconscious facial micro-tremor data.

In step S404 , the electronic device obtains the user's concentration level and immersion level from one or more cardiac response effective variables through the processor.

According to various embodiments of the present disclosure, in step S404, the process of obtaining the concentration value of the user from the one or more cardiac response effective variables including the heart rate and amplitude information of the user, and maintaining the concentration value above a threshold value It may include a process of acquiring the level of immersion according to the degree.

In step S405, the electronic device, through the processor, based on whether the average value per minute of the concentration value of the user falls below the threshold concentration value and whether the average value per minute of the engagement value falls below the threshold concentration value, based on whether Determines whether the ventilation sequence is output or not. The ventilation sequence consists of a plurality of elements.

According to various embodiments of the present disclosure, the plurality of elements constituting the ventilation sequence output a recommendation message of a learning break, an output of a stretching video, a counting display output of a break time, a background color change output of the screen outputting the learning content , an output of a beep sound, an output of white noise, wherein the intensity of the ventilation sequence may be related to at least one of an output frequency per unit time of the ventilation sequence and an audiovisual output intensity of the ventilation sequence there is.

In step S406, when the output of the ventilation sequence is determined, the electronic device outputs the ventilation sequence with an initially set intensity and an output element type through the output device. The output element type is at least one output element among the plurality of elements.

In step S407, the electronic device adjusts the intensity of the ventilation sequence and the type of the output element based on the change in the concentration value through the processor.

According to various embodiments of the present disclosure, in step S407, when the concentration level falls from the threshold concentration level, adjusting the output intensity of the beep sound, and when the concentration level rises below the threshold concentration level , adjusting the output intensity of the white noise.

According to various embodiments of the present disclosure, step S407 may include adjusting the number of types of the output element among a plurality of elements according to a section and a downward slope of the concentration value.

According to various embodiments of the present disclosure, in the embodiment of FIG. 4 , a process of storing time series mapping information for a change in concentration level, the intensity of the ventilation sequence, and the type of the output element, and transmitting the time series mapping information to a server and receiving accumulated time series mapping information from the server, and step S407 may be performed based on the accumulated mapping information.

According to various embodiments of the present disclosure, in step S407, based on the accumulated mapping information, the intensity of the ventilation sequence before and after adjustment of the intensity of the ventilation sequence and the type of the output element is among the intensity and the output element type in which the concentration value is converted from falling to rising This can be done by applying in order starting with the lowest intensity and the type of the output element. Through such an operation, even if concentration and immersion are lowered, rather than outputting more than necessary intensity or many types of output elements that are more than necessary, the minimum intensity or minimum intensity that can achieve the purpose of increasing concentration and immersion It can output any kind of output element.

In step S408 , the electronic device determines, through the processor, whether the average value per minute of the concentration level and the average value per minute of the level of immersion are restored to greater than or equal to the threshold level of concentration and the threshold level of engagement.

In step S409, the electronic device stops the output of the ventilation sequence through the output device when the average value per minute of the concentration value and the average value per minute of the immersion value are restored to more than the threshold concentration value and the threshold immersion value do.

5 illustrates an example of a biometric information acquisition process according to various embodiments of the present disclosure.

Referring to FIG. 5 , an example of a process of obtaining biometric information of a user from an image image of an upper body including a user's face is shown.

Specifically, the electronic device may acquire information on the user's emotion type and emotion ratio by analyzing a pattern of specific emotion points in the face and a pattern of specific movement points in the upper body of the image image. More specifically, a standard for obtaining a facial image of a basic facial expression (non-expression) from a user and an emotional expression according to a plurality of emotional states compared thereto, and defining a reference vertex for each facial region in the facial image of the basic facial expression or emotional expression After mapping a number of vertices for each facial region on the user's facial image by masking (matching) the model, the coordinate change value between vertices in the same facial region is extracted from the facial image of the basic expression and emotion expression, and the vertices are applied to the human face It is grouped by expression unit defined in information can be obtained.

6 illustrates an example of a process of detecting face points from a user's video image according to various embodiments of the present disclosure.

Specifically, FIG. 6 illustrates a process of extracting face image data from an image image of the upper body including the user's face.

First, the electronic device performs Haar feature extraction. Specifically, the electronic device calculates the probability of the presence of a face from the input image image by comparing each part of the image in units of pixels using kernels corresponding to facial features such as eyes and nose. The electronic device subtracts brightness values corresponding to the black region and the white region from the input image to find one that is greater than or equal to the threshold value.

Next, the electronic device performs cascade classification. Specifically, the electronic device detects a face by using a cascade classifier capable of recognizing whether a face is a face by the Haar feature. In the embodiment of FIG. 6 , a pre-trained Frontalface_alt2 model is applied. The electronic device reduces candidates from certain Haar features such as eyes, and then detects the final face with Haar features of detailed face elements.

Next, the electronic device detects the landmark. In the example of FIG. 6 , the DLIB-based Regression Trees ensemble model trained on the iBUG 300-W dataset is applied to the electronic device. For the front face detected from the Haar model, 68 face points for eyes, nose, mouth, eyebrows, and chin are detected. Each face point has an x, y coordinate value in 2D space.

Next, the electronic device performs coordinate system normalization. Specifically, the detected face point is located in a coordinate system having the upper left corner of the screen as the origin. Although it is the same expression, different values may be displayed depending on the position of the face, which acts as noise. Therefore, this problem can be solved by normalizing it to a relative coordinate system with the forehead as the origin.

7 illustrates an example of a process of recognizing heart response information and user expression information based on a user's image according to various embodiments of the present disclosure.

Specifically, FIG. 7 illustrates a non-contact image-based sensing technology according to various embodiments of the present disclosure.

According to various embodiments of the present disclosure, the electronic device may sense the user's inner reaction in real time through image-based non-contact, and analyze the user's heart reaction and emotional reaction based on the micro-tremor of the face detected in the image.

Referring to FIG. 7 , in step 1, the electronic device may detect and track an image-based non-contact real-time face image. The electronic device may detect the image-based non-contact facial feature region. The electronic device may track and analyze a real-time signal for each facial feature area of the user.

In a second step, the electronic device may perform facial micro-tremor extraction and conscious motion noise removal. Specifically, the electronic device may analyze a frequency spectrum of the signal analyzed in step 1, noise-filter the motion, and extract the user's unconscious fine motion.

In step 3, the electronic device may analyze the user's heart response based on the user's subconscious minute tremors. Specifically, by extracting an electrocardiogram (ECG) and RRI (RR interval) based on the user's unconscious fine movement data, analyzing the time domain for ECG and RRI, and analyzing the frequency domain after converting the frequency power spectrum to ECG and RRI, Effective parameters of cardiac response can be extracted. The electronic device may analyze the user's cardiac response based on effective parameters of the cardiac response.

In step 4, the electronic device may analyze the user's facial expression based on the user's subconscious minute tremors. Specifically, the electronic device maps the AU (action units)-feature points in the face image, derives a frame average for all AUs, selects a specific AU from among all AU candidates based on a threshold value, and uses an expression from the selected AU variables can be extracted. The electronic device may analyze the user's emotion based on the effective expression variables.

The electronic device according to various embodiments of the present disclosure may non-contact analyze the user's inner reaction based on the camera in consideration of the organic relationship of various variables such as the user's heart reaction, facial expression, gaze, and voice.

The electronic device may obtain electrocardiogram information including heart rate information and amplitude information by using the facial microtremor information in the image image. In this case, microscopic motion information may be extracted from the image using a non-contact type camera, and electrocardiogram information may be obtained therefrom. Specifically, the electrocardiogram information can be obtained in the following way.

1) (Image Acquisition) Continuous image data is generated from the upper body image including the user's face.

2) (Face Tracking) To extract minute movements of the human body from the head, image information is separated through face recognition using OpenCV. A facial recognition system based on OpenCV is a computer-aided application that automatically identifies each person through a digital image, by comparing selected facial features appearing in the video image with a facial database. A rectangle appearing in the original image of FIG. 6 indicates the tracking area of the face. In response to the user's movement, tracking of the facial part is performed.

3) (Spatial Decomposition) A well-known image spatial separation technique that separates the space for each frequency through Gaussian blur and down sampling.

4) (Extraction of fine motion information) Select a frequency band with correlation between biosignals and acquire fine motion information in the selected band.

5) (Temporal Processing) Using temporal processing of the extracted fine motion information, data of a frequency band related to the fine motion information according to the passage of time is extracted and the extracted data is amplified.

6) (Extraction of electrocardiogram information) The difference between the average of the average of one frame of the motion data of the image measured every predetermined time for the data value of the frequency band related to the micro-motion information is calculated, and the average difference between the previous and current motions By calculating the value, the amount of micro-movement can be extracted and converted into ECG information.

Furthermore, the electronic device may acquire information on the concentration value of the user based on heart rate and amplitude information, which are electrocardiogram information.

For example, the user is stimulated through interesting content such as a game. In this case, when a quest is given, sympathetic activation is indicated, and a specific relationship between heart rate and amplitude can be derived according to the concentration level.

That is, in general, in a neutral state with less stimulation to the user, the heart rate is slow in the bradycardia (slow pulse) range, and the stroke volume should increase accordingly, whereas in High Engagement and Low Engagement it is neutral. It shows a pattern in which the heart rate is relatively high and the amplitude is small compared to the state.

Specifically, it is possible to quantitatively interpret concentration by analyzing the rules of the interaction between heart rate and amplitude, detect the user's ECG signal concentrating on a specific stimulus, and extract the heart rate (BPM) and actual amplitude (actual) from the ECG signal. amplitude) data is detected. By analyzing the detected data, it is possible to derive the concentration through the correlation between the heart rate and the actual amplitude.

According to various embodiments of the present disclosure, the electronic device may additionally acquire user voice information while acquiring a video image. The electronic device may acquire information on the user's emotion type and emotion ratio by analyzing a prosody characteristic and a voice spectrum characteristic in the voice information. That is, the electronic device can obtain information on the user's emotion type and emotion ratio based on the prosody characteristic and the voice spectrum characteristic in the voice information along with the pattern of specific emotional points in the face and the pattern of specific movement points in the upper body in the image image. there is.

Meanwhile, in another embodiment of the present invention, the electronic device may acquire the user's biometric information according to the electrocardiogram information, the image information, and the voice information received from the wearable device. That is, the user wears a wearable device such as a smart watch, and the wearable device may measure the electrocardiogram information while the wearable device comes into contact with the user's body. In this case, the electronic device may receive measured ECG information from the wearable device through wireless communication such as Bluetooth. The electronic device may calculate biometric information such as the user's heart rate and concentration based on the received electrocardiogram information. In addition, the electronic device may calculate the user's biometric information, such as a heart rate and concentration, by considering both the above-described image image and the electrocardiogram information received from the wearable device.

8 illustrates a process of collecting user's biometric information and providing a service solution, according to various embodiments of the present disclosure.

According to various embodiments of the present disclosure, an electronic device analyzes a learner's non-verbal expression and interaction with a learning medium that occurs during non-face-to-face learning through driving a program and a processor stored in a memory using artificial intelligence technology to analyze user emotions and It is possible to implement an emotion recognition engine that quantitatively measures the cognitive state. For example, the electronic device i) collects and accumulates camera and microphone-based user's video and/or audio data, ii) facial response elements (expression, gaze processing), biometric response elements (immersion, tension, concentration) ), extract and analyze voice response elements (rhythmic characteristics, spectral characteristics, sound/sound quality characteristics, stress, pronunciation, intonation), and iii) process and process data for each response element (integration of measurement values for each response element, analysis of effective variables) , emotion recognition discrimination by effective variable, filtering and integrated analysis, multi-modal data purification and processing), and iv) analysis and interlocking of emotional data (creating reports based on emotion analysis results, designing and recommending customized activity professors) .

9 illustrates an example of biometric information according to various embodiments of the present disclosure.

The graphs of FIG. 9 show information on heart rate (HR), focus, and engagement among the user's biometric information obtained according to various embodiments of the present disclosure.

The graph on the left of FIG. 9 shows relative changes over time compared to average values of a user's heart rate (HR), focus, and engagement over time. Specifically, the y-axis shows relative values for each time when the average value of heart rate and concentration is 50, and the x-axis shows time. According to the graph on the left of FIG. 9 , it can be seen that an engagement state appears when the focused state continues for a predetermined time. In other words, the level of concentration increases after a certain period of time when the level of concentration is increased and maintained.

The graph on the right of FIG. 9 shows the user's concentration by time. Specifically, the y-axis represents the numerical value of concentration, and the x-axis represents time. On the y-axis, 2 means the highest concentration, and 0 means the lowest concentration. With respect to the y-axis value, 0 may indicate normal, 1 may indicate focus, and 2 may indicate engagement. For example, when the concentration increases from the normal state of 0, the user enters the concentration state, and when the concentration of the user becomes higher from the concentration state and reaches 2, the user enters the immersion state.

Various embodiments of the present invention aim to increase the concentration of the extracted user as shown in the graphs of FIG. 9 .

10 illustrates a relationship between focus and engagement according to various embodiments of the present disclosure.

According to various embodiments of the present disclosure, the electronic device may perform emotion recognition based on a heart response. A cardiac response mechanism according to various embodiments of the present invention is as follows. Heart rate (HR) is determined by the autonomic nervous system's influence on the spontaneity of pacemaker cells in the sinus node. The sinus node determines the heart rate by the balance of the sympathetic and parasympathetic nervous systems of the autonomic nervous system. The sympathetic nervous system directs the body to an aroused state, while the parasympathetic nervous system tends to direct the body to a calm and relaxed state. The analysis of heart reactions and emotions according to various embodiments of the present invention is as follows. When heart variability occurs in response to respiration, blood pressure, hormones, emotions, etc. and the heart rate rises accordingly, it is determined as focused, which is an awakening state through a sympathetic nervous system response. When this concentration state continues for a certain period of time, it is determined as an engagement state. Accordingly, the electronic device may perform emotion recognition through a heart response through i) input of an upper body image including a user's face using a camera, ii) extraction of a user's heart rate through an image, and iii) heart rate-based emotion recognition.

Engagement is defined by the Hungarian psychologist Csikszentmihalyi Mihaly, and it means to cut off all distractions and distractions around you and to focus all your mind on one desired place. Immersion has to do with the pleasure created by the activity, and it is this emotion that motivates you to do the activity again. Therefore, the state of immersion is closely related to enjoyment and focused attention.

In the state of psychological immersion, the body reaction is as follows. Human emotions induce physical and unconscious physiological responses, which can be observed and measured. According to the Inverted-U theory, it is possible to increase work performance (performance) through arousal state. Therefore, immersion can be defined as a state in which concentration is made more than usual due to awakening. According to the graph on the right of FIG. 10 , it can be seen that an engagement state appears when the focused state continues for a predetermined time.

According to various embodiments of the present disclosure, the electronic device collects emotional data and concentration data according to non-verbal factors such as heart rate, concentration, and immersion. The electronic device may analyze and match a ventilation sequence in which a user's responsiveness is high by accumulating bio-information data such as concentration and immersion obtained through bio-information recognition such as heart rate.

According to various embodiments of the present disclosure, the electronic device collects biologs such as heart rate during a user's activity as data, analyzes a point in time when a user's concentration and immersion is high and a point in time when the user's concentration is low, It is possible to provide an appropriate ventilation sequence, such as when

According to various embodiments of the present disclosure, the electronic device outputs a message indicating that the high concentration and immersion states are continuing when the user's concentration and immersion levels continue to exceed the critical concentration and immersion levels for a predetermined period of time or longer. , may motivate the user to continue to maintain the state of concentration and immersion. Such a message, like a kind of cheering message, can further keep the user focused and immersed in learning. According to an embodiment, a message indicating how much the concentration and immersion state is maintained may be output after a predetermined time elapses or at every predetermined period while the user's concentration and immersion levels are greater than or equal to the threshold concentration and immersion. there is. In addition, when the user enters the concentration and immersion state, by changing the background image or color of UI/UX other than the learning content part of the output device, the user himself enters the concentration and immersion state while viewing the learning content can make you aware of In addition, when the user enters the concentration state and immersion state, by changing the background image or color of UI/UX other than the learning content part of the output device, the user's own concentration and immersion while viewing the learning content on their own are the critical concentration and critical immersion It can be made to recognize the fact that it has fallen below the degree.

According to various embodiments of the present disclosure, the electronic device analyzes biometric information on the user's emotional and cognitive state in real time, and accumulates and stores data on which activity or content the user is enjoying when the concentration is high in a certain section. can

When the electronic device according to various embodiments of the present disclosure is applied to the field of a child learning service application, the electronic device randomly provides a previously prepared ventilation sequence when the user's average concentration during learning falls below a certain value and increases the level of interest. can check The electronic device may improve the average concentration of the user by exposing a ventilation sequence suitable for a category that has achieved a high degree of interest based on the accumulated data.

According to various embodiments of the present disclosure, when data for each user's content is accumulated over a long period of time, the electronic device analyzes big data through machine learning to derive the user's biometric information response expected after content adjustment, and based on the user's current ecological information, It is possible to provide a ventilation sequence in which positive changes are expected with high probability.

When implementing the embodiment of the present invention using hardware, ASICs (application specific integrated circuits) or DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices) configured to perform the present invention , FPGAs (field programmable gate arrays), etc. may be provided in the processor of the present invention.

Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable medium. In addition, the structure of data used in the above-described method may be recorded in a computer-readable storage medium through various means. Program storage devices that may be used to describe a storage device comprising executable computer code for performing the various methods of the present invention should not be construed as including transitory objects such as carrier waves or signals. do. The computer-readable storage medium includes a storage medium such as a magnetic storage medium (eg, a ROM, a floppy disk, a hard disk, etc.) and an optically readable medium (eg, a CD-ROM, a DVD, etc.).

The embodiments described above are those in which elements and features of the present invention are combined in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form that is not combined with other components or features. In addition, it is also possible to configure an embodiment of the present invention by combining some elements and/or features. The order of operations described in embodiments of the present invention may be changed. Some configurations or features of one embodiment may be included in other embodiments, or may be replaced with corresponding configurations or features of other embodiments. It is obvious that claims that are not explicitly cited in the claims can be combined to form an embodiment or included as a new claim by amendment after filing.

It will be apparent to those skilled in the art that the present invention can be embodied in other forms without departing from the spirit and essential characteristics of the present invention. Accordingly, the above embodiments should be considered in all respects as illustrative and not restrictive. The scope of the present invention should be determined by a reasonable interpretation of the appended claims and all possible changes within the equivalent scope of the present invention.

100: user 110: electronic device
111: memory 112: transceiver unit
113: processor 114: camera
115: output device 120: wired/wireless network
130: server 131: memory
132: transceiver 133: processor

Claims (10)

A method of operating an electronic device for supporting a user's learning concentration, wherein the electronic device includes a camera, an output device, a memory, a transceiver, and at least one processor, the output device includes at least one of a display and a speaker, ,
the process of outputting the content; and
A process of obtaining an image image of the upper body including the user's face;
obtaining biometric information including at least one cardiac response effective variable from the micro-movement of the user's face in the image image;
obtaining the user's concentration level and immersion level from the one or more cardiac response effective variables;
The process of determining whether to output a ventilation sequence based on whether the average value per minute of the concentration value of the user falls below a threshold concentration value and whether the average value per minute of the immersion value falls below a threshold concentration value And, the ventilation sequence consists of a plurality of elements,
When the output of the ventilation sequence is determined, the process of outputting the ventilation sequence with an initially set intensity and an output element type, wherein the output element type is at least one output element among the plurality of elements,
adjusting the intensity of the ventilation sequence and the type of the output element based on the change in the concentration value;
When the average value per minute of the concentration value and the average value per minute of the immersion value are restored to more than the threshold concentration value and the threshold immersion value, stopping the output of the ventilation sequence comprising the step of,
how it works.
The method according to claim 1,
The process of obtaining the biometric information including the one or more cardiac response effective variables from the micro-movement of the user's face in the image image,
detecting the facial feature region of the user from the image image;
The process of extracting unconscious facial micro-tremor data from the facial feature area and removing noise due to conscious movement;
Including the process of obtaining the heart rate and amplitude information of the user from the unconscious facial microtremor data,
how it works.
The method according to claim 1,
The process of obtaining the user's concentration level and immersion level from the one or more cardiac response effective variables includes:
obtaining a concentration value of the user from the one or more cardiac response effective variables including heart rate and amplitude information of the user;
Comprising the process of obtaining the level of immersion according to the degree of maintaining more than the threshold value of the level of concentration,
method.
The method according to claim 1,
The plurality of elements constituting the ventilation sequence are output of a learning break recommendation message, output of a stretching video, a counting display output of a break time, an output of a background color change of a screen outputting the learning content, and a beep sound. output, including the output of white noise;
The intensity of the ventilation sequence is related to at least one of an output frequency per unit time of the ventilation sequence and an audiovisual output intensity of the ventilation sequence,
how it works.
5. The method according to claim 4,
The process of adjusting the intensity of the ventilation sequence based on the change in the concentration value,
adjusting the output intensity of the beep sound when the concentration level falls from the threshold concentration level;
and adjusting the output intensity of the white noise when the concentration level rises below the threshold concentration level.
how it works.
The method according to claim 1,
The process of adjusting the type of the output element of the ventilation sequence based on the change in the concentration value comprises:
Comprising the process of adjusting the number of the types of the output element among the plurality of elements according to the section and the falling slope of the concentration value,
how it works.
The method according to claim 1,
The process of storing time series mapping information for the change of the concentration value, the intensity of the ventilation sequence, and the type of the output element;
The process of transmitting the time series mapping information to a server;
Further comprising the step of receiving the accumulated time series mapping information from the server,
The process of adjusting the intensity of the ventilation sequence and the type of output element based on the change in the concentration value is performed based on the accumulated mapping information,
how it works.
8. The method of claim 7,
The process of adjusting the intensity of the ventilation sequence and the type of the output element comprises:
Based on the accumulated mapping information, before and after the adjustment of the intensity of the ventilation sequence and the type of output element, by applying the lowest intensity and the output element type of the intensity and output element type in which the concentration value is converted from falling to rising in order performed,
how it works.
An electronic device for supporting a user's learning concentration, comprising:
a camera, an output device, a memory, a transceiver, and at least one processor;
The processor is configured to perform the method of operation according to any one of claims 1 to 8,
electronic device.
A computer program configured to perform the method of operating an electronic device according to any one of claims 1 to 8, and recorded in a computer-readable storage medium.

Images