KR20190140767A - Method and Apparatus for Measuring Degree of Immersion - Google Patents

Abstract

Disclosed are an immersion level measurement device and a method thereof. According to an embodiment of the present invention, the immersion level measurement device inputs image information collected from an image photographing device to an expression recognition model learned by a user to calculate an interest level score, and aggregates the number of blink and a ratio of face front stare of the user to calculate a concentration level score, thereby consequently determining an immersion level for an interest part of the user.

Description

Immersion degree measuring apparatus and method {Method and Apparatus for Measuring Degree of Immersion}

This embodiment relates to an immersion measurement apparatus and method. More specifically, the present invention relates to an apparatus and method for immersion measurement using face analysis, for use in a computer-based multi-intelligence evaluation system.

The contents described in this section merely provide background information on the present embodiment and do not constitute a prior art.

The IQ test is a common way to measure human intelligence. Human intelligence should be measured on the basis of various intellectual characteristics, but IQ test has a disadvantage of focusing on language ability and math ability. Professor Howard Gardner presented a multi-intelligence theory that separates human intelligence into eight domains and tests each domain. Based on this, a multi-intelligent evaluation system that evaluates the user's intelligence in various aspects Is being developed.

 As a method of multi-intelligence evaluation, it is common to check the degree to which a certain problem is solved within a time limit. However, many studies have shown that measuring multi-intelligence in a uniform way cannot measure accurate multi-intelligence, and many opinions have to be separated and tested in various areas of human beings. Therefore, in order to perform a separate test for each area, a method of analyzing a questionnaire or a questionnaire by an inspector conducting a questionnaire or consultation and analyzing an examinee's answer is used.

However, such a survey method may involve subjective factors such as the examiner's experience in the multi-intelligence evaluation process, and requires a large number of experts, time, and cost for the multi-intelligence evaluation. In addition, there is a problem in that the reliability of the test results is low because the answer about the questionnaire of the questionnaire is not constant for infants with less developed language ability.

In an embodiment of the present invention, by inputting the image information collected from the image capturing apparatus to the facial expression recognition model learned by the user to calculate the interest score, the concentration score is calculated by combining the number of eye blinks and the front face gaze ratio of the user The main object of the present invention is to provide an immersion measuring apparatus and method for calculating and consequently determining the degree of immersion of a user's interest.

According to the present embodiment, an image processing unit is configured to receive image information of a user photographed by an image capturing apparatus, specify a face region from the image information, and detect motion of a facial expression within the face region to generate motion data; An interest determination unit configured to calculate and output an interest score of the user based on a previously learned facial expression recognition model and the motion data; A concentration determination unit for extracting face data based on the face area to calculate and output a concentration score of the user; And an immersion determination unit for calculating and outputting an immersion score of the user based on at least one of the interest score and the concentration score.

According to another aspect of the present embodiment, the image capturing apparatus receives image information of a user, specifies a face region from the image information, and detects facial expression within the face region to generate motion data. process; Calculating and outputting an interest score of the user based on a previously learned facial expression model and the motion data; Extracting face data based on the face area to calculate and output a concentration score of the user; And calculating and outputting an immersion score of the user based on at least one of the interest score and the concentration score.

As described above, according to the present embodiment, the immersion degree for the user's interest field may be measured and used for the multi-intelligence evaluation based on the user's interest and concentration. By measuring the degree of immersion using the user's image information, it can be used even for infants who lack the development of language ability when performing multi-intelligence evaluation.

The immersion measuring apparatus according to the present embodiment can be used in a computer-based multi-intelligence evaluation system, and it is possible to secure objectivity and reliability because subjective factors such as the ability or experience of a counselor are not involved.

1 is a block diagram schematically showing an immersion measurement apparatus according to the present embodiment.
FIG. 2 is a diagram exemplarily illustrating an interest level determining unit according to an exemplary embodiment to calculate an interest level score corresponding to a facial expression of a user.
3 is a diagram illustrating an example of an eye blink calculation unit for converting a number of eye blinks into a score according to an exemplary embodiment.
4 is a flowchart for calculating an interest score according to the present embodiment.
5 is a flowchart for calculating a concentration score according to this embodiment.
6 is an exemplary view for explaining a immersion measurement method according to the present embodiment.
7 (a) and 7 (b) are exemplary views illustrating a immersion measurement result screen according to the present embodiment.

Hereinafter, the present embodiments will be described in detail with reference to exemplary drawings. In describing the present embodiments, when it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easily understanding the embodiments disclosed herein, the technical spirit disclosed in the specification by the accompanying drawings are not limited, and all changes included in the spirit and scope of the present invention. It should be understood to include equivalents and substitutes.

In the present embodiment, the term 'comprising' means that there is a component, a feature, a step, or a combination thereof described in the specification, and excludes in advance the possibility of the existence of one or a plurality of components or other features, steps, or a combination thereof. It should be understood that it does not.

In the present embodiment, the term 'face data' refers to a concept including feature points as face information for grasping a user's expression, eye blink, and face gaze in the image information.

In the present embodiment, the term 'immersion' is a combination of interests and concentrations of the user, and is referred to as 'immersion' for a person skilled in the art to easily understand the present embodiment.

With reference to the accompanying drawings will be described in detail this embodiment.

1 is a block diagram schematically showing an immersion measurement apparatus according to the present embodiment.

The immersion measuring apparatus 100 according to the present exemplary embodiment includes an image processor 110, an interest determining unit 120, a concentration determining unit 130, and an immersion determining unit 140.

The image processor 110 includes a photographing apparatus 112, a face specifying unit 114, and a motion data generator 116. The image processor 110 acquires and processes the user's image information. More specifically, the image processor 110 acquires the user's image information from the photographing apparatus 112, and based on the image information, the face region 114 of the user is determined. The motion data generation unit 116 generates a specific motion data.

Motion data is information for detecting a facial expression of a user on a captured image and is based on dense motion flow. Dense motion flow refers to a modification that is easy to detect facial movement based on an optical flow. The photographing apparatus 112 refers to any device capable of photographing a user's face, including a camera. The face specifying unit 114 specifically extracts the face region of the user to easily detect the change in the expression of the user from the image information acquired by the photographing apparatus 112.

The interest determining unit 120 includes a facial expression database 122, an expression recognition model learning unit 124, and an interest score output unit 126. The interest determination unit 120 inputs motion data generated by the image processor 110 to the facial expression recognition model to calculate an interest score. More specifically, the facial recognition model is trained using a database storing various facial expressions, and the generated motion data is input to the learned facial expression model to calculate an interest score corresponding to the facial expression of the user.

The facial expression database 122 stores various facial expressions, and the expression recognition model learning unit 124 learns an expression recognition model based on the stored facial expressions. More specifically, the facial expressions are learned to classify the facial expressions into categories of negative facial expressions, neutral facial expressions, or positive facial expressions, and each category is divided into at least one or more stages. For example, since negative expressions include various expressions such as anger, fear, and sadness, negative expressions are classified according to motion data intensity of facial expressions as anger, fear, and sadness. The facial expression recognition model is a regression model, which is trained by a convolutional neural network (CNN) using the facial expression database 122, but is not limited thereto. On the other hand, the interest score output unit 126 calculates the score of the analyzed facial expression by inputting motion data into the facial expression recognition model based on the score stored for each step of the learned facial expression.

The concentration determining unit 130 includes a blinking unit 132, a front ratio calculating unit 134, and a concentration score output unit 136. The concentration determining unit 130 calculates a concentration score by combining the eye blink of the user and the front gaze ratio of the face. More specifically, in determining the concentration of the user, two factors, eye blinking and face-to-eye ratio, are considered, and it is determined that the less eye blink for a certain time and the higher the face-to-face ratio, the higher the concentration. .

The blinking unit 132 extracts a feature point in a specified face region by using a conditional neural model, estimates the position of the user's eyes and nose from the feature point, and averages the shape to produce a rotational effect and a scaling effect ( remove the scaling effect. A histogram is used to detect motion units in a standardized face and to estimate the blinking of the user based on the motion unit. Herein, the operation unit refers to one unit of operation in which motion data is moved. On the other hand, the minimum, maximum and average number of blinks of the user during the specific time is estimated and divided into a plurality of sections, the smaller the blinks are calculated to the higher score. The score calculation will be described later with reference to FIGS. 3 and 5.

The front ratio calculator 134 defines a range of pitch, roll, and yaw in order to determine whether the face is stared at the face. On the other hand, it should be understood that the present invention is not limited to the pitch, roll and yaw method, but also includes other methods for determining the face gaze. The front ratio calculator 134 determines whether the face gazes at each face in each frame, calculates a ratio of the front gaze frame of the user's face among all frames of the image information for a specific time, and converts it to a score. The score calculation will be described later with reference to FIG. 5.

The concentration score output unit 136 calculates the concentration score by weighting the front ratio score when the score calculated by the blinking unit 132 and the score calculated by the front ratio calculator 134 are combined. This takes into account the fact that the concentration can be judged more accurately when using the face gaze than when using eye blink. In other words, eye blinking is a factor that helps in assessing concentration, but it takes more personal and environmental factors such as eye dryness than frontal gaze ratio.

The immersion degree determination unit 140 calculates an immersion score based on at least one of the calculated interest score and concentration score. More specifically, the interest-concentration synthesis unit 142 synthesizes the interest score and the concentration score, and the immersion output unit 144 is the interest-concentration score synthesized based on the immersion level preset by the user. Determine the immersion level corresponding to. The immersion level is an index for determining the immersion degree and may be defined by the user.

FIG. 2 is a diagram exemplarily illustrating an interest level determining unit according to an exemplary embodiment to calculate an interest score corresponding to a facial expression of a user.

Referring to FIG. 2, the expression recognition model learning unit 124 is trained based on the facial expression database 122, and classifies the learned facial expressions into three facial expression categories. The three facial expression categories consist of negative facial expressions, neutral facial expressions, or positive facial expressions, and each category is divided into detailed facial expression categories. For example, the detailed facial expression types may include anger, fear, sadness, surprise, expressionlessness, happiness, and the like. The facial expression recognition model learning unit 124 stores a learning expression corresponding to the type of facial expression as a result of learning and receives motion data.

The interest score output unit 126 stores the interest score for each type of facial expression, compares the received motion data with the learning expression of the facial expression recognition model learning unit 124, and outputs the corresponding interest score. The interest score for the facial expression type is set based on the intensity of facial data. More specifically, as illustrated in FIG. 2, a score is set by differentially positive and negative facial expressions according to the strength of facial expressions with a neutral facial expression as a reference point. In the image information taken for a specific time, the lowest value of negative expression is -10 points for each frame, 0 points for neutral expressions and 10 points for positive expressions, and the average score is extracted. As a result, the interest score is calculated. In order to calculate the immersion score at the same rate as the concentration score, the average score is calculated so that there is a difference of 10 points from the minimum -5 to the maximum 5 points. Meanwhile, categories, facial expression types, and interest scores may be differently defined by a user.

3 is a diagram illustrating an example of an eye blink calculation unit for converting a number of eye blinks into a score according to an exemplary embodiment.

Referring to FIG. 3, the blinking unit 132 measures the number of blinks and converts the score into a corresponding score. More specifically, the minimum, maximum, and average value of eye blinks are counted by counting the number of blinks of the user for a specific time period, and then divided into at least one blink interval. When dividing into a plurality of sections, in addition to the minimum, maximum, and average values, a middle value between the minimum and average values may be set as a reference point. For example, the minimum value is 0, the maximum value is 63.43, and the average value is 6.43, and the median value of 3.23, 6.47, and 63.43, which is the median of 0 and 6.47, is 34.95, and the values 0, 3.23, 6.47, 34.95, and 63.43 are the reference. . The plurality of section settings may be defined differently according to the user. The blinking unit 132 sets the number of blinking points according to the number of blinking intervals and converts the score to the blinking points. The blink score can be given from a minimum of zero to a maximum of three. The combination of the blinking score and the face front gaze score will be described later with reference to FIG. 5.

4 is a flowchart for calculating an interest score according to the present embodiment.

As illustrated in FIG. 4, the immersion measurement apparatus 100 allows the user to photograph the image using the photographing apparatus 112 and specifies the face region of the user from the photographed image information (S402). Motion data is generated by detecting a user's face movement in a specified face region based on a motion flow (S404). Here, the motion flow is a modification of the optical flow to detect the movement of the object, and the present embodiment is based on a dense motion flow for detecting the movement of the facial expression of the user. The immersion measurement apparatus 100 inputs the generated motion data to the facial expression recognition model trained using the facial expression database 122 (S406). The facial expression database 122 stores various expressions of the photographic subject, and the facial expression recognition model is trained based on the CNN. In operation S408, the immersion measurement apparatus 100 specifies a facial expression type corresponding to the input motion data and outputs an interest score that matches the facial expression type. The interest score is preset by the user and used as ground truth data of the facial expression recognition model. The interest score is set so that the difference between the highest point and the lowest point is 10 points in order to be reflected at an equal ratio with the concentration point.

In FIG. 4, steps S402 to S408 are described in order, but this is merely illustrative of the technical spirit of the exemplary embodiment of the present invention. Various modifications and variations can be applied, such as executing one or more of steps S402 to S408 in parallel, without departing from the essential characteristics of one embodiment of the present invention.

5 is a flowchart for calculating a concentration score according to this embodiment.

As illustrated in FIG. 5, the immersion measuring apparatus 100 captures a user using the photographing apparatus 112, specifies a face region of the user from the captured image information (S502), and determines a specified face region. The feature point is extracted (S504). If the feature point is not extracted, step S502 is repeated. Feature points are extracted using a conditional local neural field model. The feature points are shown in Figs. 7A and 7B.

In operation S506, the feature point is used to determine whether the face of the user or the photographer is staring at the front. Based on the extracted feature points, the number of blinks and the front face ratio of the user are converted into scores. In order to measure the number of blinks, the estimation of eyes, nose and mouth of the user's face must be performed first. In operation S508, the immersion measurement apparatus 100 estimates the position of the user's eyes, nose, and mouth using the feature points, and normalizes the shape to remove the rotation effect and the scaling effect. Based on the histogram extracted from the normalized face shape, it is determined whether the user blinks in motion units for a specific time and counts the number of blinks (S512). Herein, the operation unit refers to one unit of operation in which motion data is moved.

Based on the extracted feature points, it is determined whether the user or the photographer is staring at the front (S510). If the user or the photographer does not stare at the front face, the process returns to step S502. If the user or the photographer stares at the front face, the frame on which the face of the user or the photographer is looking at the front face is counted based on the extracted feature point (S514). In determining whether the user's face is facing front, a range of rolling, pitching, and yawing is defined, and whether or not the face is stared at each frame is determined. When a specific time elapses in S516, the blinking count and the face front gaze frame count test are finished (S516).

The counted number of blinks and the face front gaze frame are respectively converted into scores (S518). The counted number of blinks is converted into a score as described above with reference to FIG. 3 (S520). The immersion measuring apparatus 100 calculates a ratio of the front gaze frame of the face among the entire frames of the image information for a specific time, and converts it into a score based on this (S522). In this case, the frame refers to each of the still image data constituting the image information. For example, if a total of 100 frames and 20 frontal gaze frames during a specific time period, the ratio of 1/5 to a score.

In S524, the immersion measuring apparatus 100 calculates the concentration score by summing the eye blink score and the face front gaze score. More specifically, the concentration score is calculated by weighting the face front gaze score, not the simple sum of the two scores. This is to enhance the reliability of the concentration score, which is considered to have a higher reliability of face front gaze than the blinking element when determining concentration. In other words, the blinking of the eye is weighted to the face frontal gaze, since personal or external environmental factors such as dry eye or fine dust are easy to act. Referring to FIG. 3 again, in order to weight the face front gaze score, the highest point of the blinking score of the total 10 points of concentration scores is 3 points, and the highest point of the face front gaze score is 7 points. It is to be understood that this method of weighting is merely a mere example, and includes both methods of giving a difference between the blink score and the face-to-face score. The concentration score is set to 10 points as well as the interest score in order to be reflected in the ratio equal to the interest score in the determination of immersion.

6 is an exemplary view for explaining a immersion measurement method according to the present embodiment.

Referring to FIG. 6, the user is positioned in front of the display 600, and the photographing apparatus 112 photographs the user. The display 600 outputs an image or item for measuring the user's immersion degree. Since each image or item has a different output time, the interest score and the concentration score are calculated based on the same specific time in order to determine the immersion level for each image or item based on the same criteria. For example, if one item blinks twice in 30 seconds for one question and four times blinks per minute for the other item, the scores for the two blinks are the same.

7 (a) and 7 (b) are exemplary views illustrating a immersion measurement result screen according to the present embodiment.

As shown in FIG. 7A, a feature point is displayed on a face of a user on a result screen 710 of photographing a user, and a portion for outputting a status display 720 is displayed on a lower right side of the result screen 710. exist. The feature points are used to determine eye blinking, face gaze, and the like. On the other hand, the status display window 720 for outputting the status display includes information on whether the face is staring at the front and facial expression.

Referring to FIG. 7B, the immersion measurement apparatus 100 determines that the front face is not stared when the pitch, roll, or yaw is out of a range preset by the user using the feature points of the face. Since the remaining frames, except for the frame that does not stare at the front side, as shown in the drawing, are frames that stare at the front side of the entire frame of the image for a specific time, the score is calculated by calculating the front gaze ratio.

The above description is merely illustrative of the technical idea of the present embodiment, and those skilled in the art will appreciate that various modifications and changes can be made without departing from the essential characteristics of the present embodiment. Therefore, the exemplary embodiments are not intended to limit the technical spirit of the present exemplary embodiment but to describe the exemplary embodiments, and the scope of the technical spirit of the present exemplary embodiment is not limited by the exemplary embodiments. The scope of protection of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present embodiment.

100: immersion measurement device 110: image processing unit
112: photographing apparatus 114: face special government
116: motion data generation unit 120: interest degree determination unit
122: facial expression database 124: facial expression recognition model learning unit
126: interest score output unit 130: concentration determination unit
132: eye blink calculation unit 134: front ratio calculation unit
136: concentration score output unit 140: immersion determination unit
142: interest-concentration synthesis unit 144: immersion output unit
600: display 710: results screen
720: status display window

Claims (17)

An image processor for receiving a user's image information captured by the image capturing apparatus, specifying a face region from the image information, and detecting motion of a face expression within the face region to generate motion data;
An interest determination unit configured to calculate and output an interest score of the user based on a previously learned facial expression recognition model and the motion data;
A concentration determination unit for extracting face data based on the face area to calculate and output a concentration score of the user; And
An immersion degree determination unit for calculating and outputting an immersion score of the user based on at least one of the interest score and the concentration score.
Immersion degree measuring device comprising a. The method of claim 1,
The interest degree determination unit,
An expression recognition model learning unit which stores the output value learned by learning the expression recognition model based on a database of facial expressions; And
The interest score output unit for inputting the motion data to the facial expression recognition model to determine the facial expression and output the score corresponding to the facial expression.
Immersion degree measuring device comprising a. The method of claim 2,
The expression recognition model learning unit,
The immersion measurement apparatus, characterized in that for learning the facial expression recognition model based on the database divided the learned output value into a plurality of categories, the category is divided into at least two steps. The method of claim 3, wherein
The category,
Immersion degree measuring device, characterized in that any one of a negative expression, a neutral expression and a positive expression. The method of claim 2,
The interest score output unit,
And setting scores corresponding to the steps classified by the facial expression recognition model and inputting the motion data into the facial expression recognition model to calculate the scores of the corresponding steps. The method of claim 1,
The concentration determination unit,
An eye blink calculation unit for counting and scoring the number of eye blinks of the user to calculate an eye blink score;
A front ratio calculation unit configured to calculate and score a ratio of the user looking at the front to calculate a front ratio score; And
A concentration score output unit for adding up the scores calculated by the blinking unit and the front ratio calculator.
Immersion degree measuring device comprising a. The method of claim 6,
The concentration score output unit,
And when calculating the concentration score by adding the eye blink score and the front ratio score, the concentration score is calculated by applying a specific weight to the front ratio score. The method of claim 6,
The eye blink calculation unit,
The user extracts at least one feature point from the face area to analyze the shape of the user's face, determines whether the user blinks by using an operation unit, counts the number of eye blinks of the user, and converts the score into a score. Immersion measuring device, characterized in that the movement unit of the feature point. The method of claim 8,
The eye blink calculation unit,
And dividing the light into a plurality of stages based on the minimum, maximum, and average values of the number of blinks, and setting a higher score to obtain a higher score as the number of blinks decreases. The method of claim 8,
The front ratio calculation unit,
The feature point is analyzed to determine whether the user's face is in front, and the ratio of the front position of the entire frames of the photographed image is calculated and converted into a score, wherein each frame constitutes image information. Immersion degree measuring device, characterized in that the data. The method of claim 10,
The front ratio calculation unit,
An immersion measuring apparatus, characterized in that the frame of the user's face is counted and converted into an average score based on the entire frame of the image information. The method of claim 1,
The image processor,
A photographing apparatus for obtaining the image information;
Face specificity specifying the face region based on the image information; And
Motion data generation unit for generating the motion data by detecting the motion from the face specific portion
Immersion degree measuring device comprising a. The method of claim 12,
The face special government,
And dividing the image information into a pixel unit and extracting a face region corresponding to a face part of the user from the pixel unit. The method of claim 12,
The motion data generator,
The immersion measurement apparatus, characterized in that for generating the motion data based on dense motion flow in the face area. The method of claim 1,
The immersion degree determination unit,
And an immersion score is calculated based on at least one of the interest score and the concentration score. Receiving image information of a user photographed by the image capturing apparatus, specifying a face region from the image information, and detecting motion of a face expression within the face region to generate motion data;
Calculating and outputting an interest score of the user based on a previously learned facial expression model and the motion data;
Extracting face data based on the face area to calculate and output a concentration score of the user; And
Calculating and outputting an immersion score of the user based on at least one of the interest score and the concentration score
Immersion degree measurement method comprising a. The method of claim 16,
The process of calculating and outputting the interest score,
Learning the facial expression recognition model based on a database of facial expressions, storing the learned output value, inputting the motion data to the facial expression recognition model, determining the facial expression, and outputting the score corresponding to the facial expression; Immersion degree measurement method.

Images