JP6991401B2 - Information processing equipment, programs and information processing methods - Google Patents

Description

The present invention relates to an information processing apparatus, a program and an information processing method.

Technological development for estimating the drowsiness of a person is progressing in a car or in a factory to prevent accidents. For example, there is a technique for estimating a drowsiness state by extracting a feature amount that changes according to a drowsiness state from a heartbeat, an electroencephalogram, a blink, or the like, and comparing the feature amount with a threshold value.

Biological signals such as heartbeats or brain waves can directly measure a person's condition and are a possible technique, but they require a sensor to be attached and may be annoying. Further, if the sensor is forgotten to be attached, the function itself is impaired, and there is a problem that the cost is high.

On the other hand, the method of estimating the drowsiness state of a person from a moving image captured by a camera has the advantages of low cost, long sensor life, non-contact with the user, and simple system.

As a method for estimating a person's drowsiness state from a moving image, for example, there is a technique for estimating a drowsiness state by setting threshold values for each of blink frequency, blink speed, and blink duration. However, when wearing sunglasses, there is a problem that the drowsiness state cannot be estimated accurately because the blink information cannot be acquired accurately.

As a method for estimating a person's drowsiness state from a moving image without using blink information, for example, a technique for estimating a drowsiness state using the frequency of yawning is disclosed in Patent Document 1. The technique described in Patent Document 1 determines yawning by pattern matching of changes in openings (paragraph 882, etc.).

Japanese Unexamined Patent Publication No. 2005-199078

However, the technique described in the above-mentioned Patent Document 1 has a problem that the erroneous determination rate becomes high depending on the method of setting the pattern because the yawning is determined by the pattern matching of the change of the opening.

Therefore, one or more aspects of the present invention are intended to enable accurate determination of the yawning state.

The information processing apparatus according to one aspect of the present invention has a face area extraction unit that extracts a face area, which is a face area of a person, from each of a plurality of frames included in a moving image, and a face area extraction unit that extracts the face area from the face area in advance. From the face feature point extraction unit that extracts a plurality of defined feature points, the face feature amount calculation unit that calculates the face feature amount indicating the facial feature from the plurality of feature points, and the face feature amount. The opening continuation, which is the degree to which the mouth is continuously opened, by the opening degree specifying portion that specifies the opening degree, which is the degree to which the mouth is open on the face, and the plurality of the opening degrees specified from the plurality of frames. The opening continuity calculation unit that calculates the degree determines whether or not the person is opening due to a predetermined factor other than the missing stretch, and it is determined that the person is opening due to the predetermined factor. A value lower than a predetermined first threshold is associated with the frame as the extension opening continuity, and the opening continuity is defined as the extension opening in the frame determined that the opening is not performed due to the predetermined factor. The number of frames to which the stretch opening continuity, which is equal to or higher than the first threshold value, is associated with the predetermined number of consecutive frames including the latest frame, which is associated with the continuity, is predetermined. It is characterized by comprising a determination processing unit for determining that the person has a defect when the value is equal to or higher than the second threshold value.

The program according to one aspect of the present invention uses a computer in advance from a face area extraction unit that extracts a face area, which is a face area of a person, from each of a plurality of frames included in a moving image, and from the face area. A face feature point extraction unit that extracts a plurality of defined feature points, a face feature amount calculation unit that calculates a face feature amount indicating the face feature from the plurality of feature points, and a face feature amount that calculates the face feature amount from the face feature amount. The opening continuity, which is the degree to which the mouth is continuously open, is calculated from the opening degree specifying portion that specifies the opening degree, which is the degree to which the mouth is open, and the plurality of the opening degrees specified from the plurality of frames. The opening continuity calculation unit and the frame determined to determine whether or not the person has opened due to a predetermined factor other than yawning, and the frame determined to have opened due to the predetermined factor. , A value lower than the predetermined first threshold value is associated as the yawning opening continuity, and the opening continuity is set as the yawning opening continuity to the frame determined that the opening is not performed due to the predetermined factor. Of the predetermined number of consecutive frames including the latest frame, the number of frames to which the yawning opening continuity, which is equal to or higher than the first threshold value, is associated is predetermined. It is characterized in that it functions as a determination processing unit for determining that the person has yawned when the value is equal to or higher than the second threshold value.

In the information processing method according to one aspect of the present invention, a face region, which is a region of a person's face, is extracted from each of a plurality of frames included in a moving image, and a plurality of predetermined faces are extracted from the face region. The feature points are extracted, the facial feature amount indicating the facial feature is calculated from the plurality of feature points, and the degree of opening, which is the degree to which the mouth is open in the face, is specified from the facial feature amount. Whether the person opens by a predetermined factor other than yawning by calculating the opening continuity, which is the degree to which the mouth is continuously opened, from the plurality of openings specified from the plurality of frames. It is determined whether or not the frame is opened due to the predetermined factor, and a value lower than the predetermined first threshold value is associated with the frame as the yawning opening continuity, which is predetermined. The opening continuity is associated with the frame determined not to be opened due to the above factor as the yawning opening continuity, and the first threshold value among the predetermined number of consecutive frames including the latest frame. When the number of frames associated with the above-mentioned yawning opening continuity is equal to or greater than a predetermined second threshold value, it is determined that the person has yawned.

According to one or more aspects of the present invention, the yawning state can be accurately determined.

It is a block diagram which shows schematic structure of the yawning determination apparatus which concerns on Embodiment 1. FIG. It is a schematic diagram which shows an example of the opening distinction determination condition in Embodiment 1. FIG. It is a schematic diagram which shows an example of a yawning determination condition. It is a block diagram which shows schematic structure of the processing part in Embodiment 1. FIG. It is a schematic diagram which shows an example of the face feature point information. It is a schematic diagram for demonstrating the amount of facial features. It is a schematic diagram which shows the facial feature amount information. (A) and (B) are graphs for explaining the opening speed. It is a block diagram which shows the hardware composition of the yawning determination device schematicly. It is a flowchart which shows the process in the yawning determination part in Embodiment 1. It is a block diagram which shows schematic structure of the yawning determination apparatus which concerns on Embodiments 2 and 3. It is a schematic diagram which shows an example of the opening distinction determination condition in Embodiment 2. It is a schematic diagram which shows an example of the facial expression characteristic model in Embodiment 2. It is a block diagram which shows schematic structure of the processing part in Embodiment 2. FIG. It is a flowchart which shows the process in the yawning determination part in Embodiment 2. It is a schematic diagram which shows an example of the opening distinction determination condition in Embodiment 3. FIG. It is a block diagram which shows schematic structure of the processing part in Embodiment 3. FIG. It is a flowchart which shows the process in the yawning determination part in Embodiment 3. FIG.

Embodiment 1.
FIG. 1 is a block diagram schematically showing a configuration of a yawning determination device 100 as an information processing device according to the first embodiment.
The yawn determination device 100 includes an image pickup unit 110, a database unit 120, a processing unit 130, and a display unit 150.

The image pickup unit 110 acquires a moving image and gives the acquired moving image to the processing unit 130.
The database unit 120 is a storage unit that stores the yawning determination condition model 121.
The yawning determination condition model 121 includes an opening distinction determination condition and a yawning determination condition.

The opening distinction determination condition is a determination condition used to distinguish between an opening due to a predetermined factor and an opening due to a factor other than the predetermined factor.
FIG. 2 is a schematic view showing an example of an opening distinction determination condition.
In the first embodiment, when the opening speed evaluation value described later is equal to or higher than a predetermined threshold value (for example, 20), the opening distinction determination condition is determined to be an opening due to a factor other than the predetermined factor. When the opening speed evaluation value is less than a predetermined threshold value, it is a condition for determining the opening due to a predetermined factor. The details of the opening speed evaluation value will be described later. If the opening is performed by a factor other than the predetermined factor, it means that the opening is not performed by the predetermined factor.

The yawning determination condition is a condition for determining yawning.
FIG. 3 is a schematic view showing an example of yawning determination conditions.
In the first embodiment, the yawning determination condition is a predetermined value (for example, 40) in which the yawning opening continuity is the first threshold value among the predetermined number of frames (for example, 100 frames) in the past. When the number of frames as described above is equal to or greater than a predetermined number (for example, 20) as the second threshold value, it is a condition for determining yawning. The details of the yawning opening continuity will be described later. Here, the predetermined number of frames in the past is a predetermined number of consecutive frames including the latest frame.

Returning to FIG. 1, the processing unit 130 executes processing by the yawning determination device 100.
FIG. 4 is a block diagram schematically showing the configuration of the processing unit 130 according to the first embodiment.
The processing unit 130 includes an input unit 131, a face area extraction unit 132, a face feature point extraction unit 133, a face feature point storage unit 134, a face feature amount calculation unit 135, a face feature amount storage unit 136, and an opening. It includes a degree specifying unit 137, an opening degree storage unit 138, an opening continuity calculation unit 139, an opening continuity storage unit 140, a determination processing unit 141, a stretch determination result storage unit 146, and an output unit 147.

The determination processing unit 141 includes an opening speed evaluation value calculation unit 142, an opening speed evaluation value storage unit 143, a yawning determination unit 144, and a temporary storage unit 145.

The input unit 131 receives the input of the moving image acquired by the image pickup unit 110. The input moving image is given to the face area extraction unit 132.

The face region extraction unit 132 extracts a face region, which is a region of a person's face, from each of the plurality of frames included in the moving image.
For example, the face area extraction unit 132 extracts a person's face area from the input moving image by using a discriminator using the Haar-like feature by AdaBoost learning.
This is described, for example, in the following literature.
Paul Viola, Michael J. Jones, "Robust Real-Time Face Detection", International Journal of Computer Vision. Vol. 57, No. 2, pp137-154, 2004

The facial feature point extraction unit 133 extracts a plurality of facial feature points, which are a plurality of predetermined feature points such as contours, eyebrows, eyes, nose, and mouth, from the extracted face area.
As an extraction method for extracting facial feature points such as contours, eyebrows, eyes, nose or mouth in the extracted facial region image, for example, a known method described in the following documents may be used.
Wiskott L. , Fellous j. -M. , Kruger N. et al. , Von der Mlsburg C.I. , "Face Recognition by Elastic Bunch Graph Matching", IEEE Transitions on Pattern Analysis and Machine Integrity, Vol. 19, Issue 7, pp. 775-779, 1997

In addition, the face feature point extraction unit 133 also calculates the orientation of the face from the arrangement of the extracted plurality of face feature points. The orientation of the face is specified, for example, by the angles of Yaw, Pitch and Roll.

The face feature point storage unit 134 stores a plurality of face feature points extracted by the face feature point extraction unit 133 and face feature point information indicating the calculated face orientation.
FIG. 5 is a schematic diagram showing an example of face feature point information stored in the face feature point storage unit 134.
As shown in FIG. 5, the facial feature point information includes the coordinates of the facial feature points and the angle of the face orientation.

Returning to FIG. 4, the facial feature amount calculation unit 135 calculates the facial feature amount indicating the facial feature from the extracted plurality of facial feature points.
For example, the facial feature amount calculation unit 135 calculates a facial feature amount representing a facial feature based on a distance between two facial feature points, an angle consisting of three facial feature points, and the like.
Specifically, as shown in FIG. 6, the facial feature amount calculation unit 135 has a distance V1 between two points between the inner and outer corners of the left eye and the inner and outer corners of the right eye when viewed from the front of the person's face. The distance V2 between the two points, the distance V3 between the upper and lower lips, the angle between the straight lines L1 and L2 connecting the reference point (for example, the tip of the nose) P and the inner ends of both eyebrows (for example). V4), the angle V5 between the straight lines L3 and L4 connecting the reference point P and both ends under the nose, and the angle V6 between the straight lines L5 and L6 connecting the reference point P and the left and right mouth angles are facial features. Calculated as. These facial features are features that are said to appear in facial expression changes such as rising corners of the mouth, opening of the nose, and frowning of the eyebrows.

The facial feature amount calculated by the facial feature amount calculation unit 135 is not limited to the above examples. For example, the facial feature amount may be an image feature amount in the glabellar region, mouth region, or cheek region of the face.
At this time, for the image feature amount, for example, the HOG (Histograms of Oriented Gradients) feature amount is used.
This is described, for example, in the following literature.
N. Dalal, B. Triggs, "Histograms of Oriented Gradients for human Detection", Proc. IEEE Computer Vision and Pattern Recognition (CVPR), pp. 886-893, 2005

In addition to the HOG feature amount, SIFT (Scaled Invariant Feature Features) feature amount, SURF (Speed-up Robot Features), or Haar-like feature amount may be used as the image feature amount.

Returning to FIG. 4, the face feature amount storage unit 136 stores the face feature amount information indicating the face feature amount calculated by the face feature amount calculation unit 135.
FIG. 7 is a schematic diagram showing facial feature amount information stored in the face feature amount storage unit 136.
As shown in FIG. 7, the face feature amount information stores a value calculated as a face feature amount.

Returning to FIG. 4, the opening degree specifying unit 137 specifies the opening degree, which is the degree to which the mouth is open in the face of a person, from the facial feature amount stored in the facial feature amount storage unit 136.
For example, the opening degree specifying portion 137 specifies the distance V3 between the two points of the upper lip and the lower lip as the opening degree.

Further, the opening degree specifying unit 137 may be used by normalizing the distance V3 and converting it into a percentage value from 0 to 100.
Specifically, in a two-dimensional image, the distance V3 at 100% opening when the person is facing the front is different from the distance V3 at 100% opening when the person is looking up or down. .. Therefore, the opening degree specifying portion 137 is set so that the opening degree is 100% at a distance V3 = 5 cm when the person is facing the front, and the opening degree is opened at a distance V3 = 3 cm when the person is facing upward. A normalization process may be performed in which the value of the distance V3 at which the opening degree is 100% is changed according to the direction of the face stored in the face feature point storage unit 134 so that the degree becomes 100%.

The opening degree storage unit 138 stores the opening degree specified by the opening degree specifying unit 137.

The opening continuity calculation unit 139 calculates the opening continuity degree, which is the degree to which the mouth is continuously opened, from the plurality of opening degrees stored in the opening degree storage unit 138. Multiple apertures are specified from multiple frames.
For example, the aperture continuity calculation unit 139 weights and adds the aperture degree of the t-frame, which is the t-th frame (t is an integer of 1 or more) in the moving image, and the aperture degree of the past frame. Calculate the opening continuity. The formula for calculating the opening continuity x _{t + 1} in the t + 1 frame, which is the t + 1st frame, is the following formula (1).

Here, x _t is the opening continuity in the t frame, y _{t + 1} is the opening degree in the t + 1 frame, α is the adjustment value of the parameter, and 0 <α <1 is satisfied.
This opening continuity is a time-filtered value in which the numerical value becomes higher when the opening is continued.

The opening continuity storage unit 140 stores the opening continuity calculated by the opening continuity calculation unit 139.

The determination processing unit 141 determines whether or not the person in the frame has opened due to a predetermined factor other than yawning. The determination processing unit 141 associates a value lower than the predetermined first threshold value with the frame determined to be opened by a predetermined factor as the yawning opening continuity, and determines the predetermined factor. The frame determined not to be opened by the above is associated with the opening continuity calculated by the opening continuity calculation unit 139 as the yawning opening continuity.
Then, in the determination processing unit 141, among a predetermined number of consecutive frames including the latest frame, the number of frames associated with the yawning opening continuity that is equal to or higher than the first threshold value is determined in advance. If it is equal to or higher than the specified second threshold value, it is determined that the person has yawned.

In the first embodiment, the determination processing unit 141 determines that when a person in the frame is having a conversation, an opening is performed due to a predetermined factor.
Specifically, the determination processing unit 141 determines that the person is having a conversation when the opening speed, which is the speed at which the person's mouth opens, is less than a predetermined third threshold value.

The processing in the determination processing unit 141 is realized by the opening speed evaluation value calculation unit 142, the opening speed evaluation value storage unit 143, the yawning determination unit 144, and the temporary storage unit 145. This will be described below.

ここで、Ｖ３_ｔは、ｔフレームにおける距離Ｖ３、Ｖ３_ｔ－１は、ｔ－１フレームにおける距離Ｖ３である。The opening speed evaluation value calculation unit 142 calculates an opening speed evaluation value, which is a value that increases as the opening speed increases, from the opening continuity stored in the opening continuity storage unit 140.
The opening speed zt in the _t -frame can be calculated by the following equation (2).

Here, V3 _t is the distance V3 in the t frame, and V3 _t-1 is the distance V3 in the t-1 frame.

The opening speed evaluation value calculation unit 142 may calculate the opening speed _zt using the formula (2), but in the first embodiment, it is shown by the following formulas (3) to (5). Any one of the existing opening speed evaluation values z # 1 _t to z # 3 _t is calculated.

The opening speed evaluation values z # 1 _t to z # 3 _t are all values that increase as the opening speed z _t increases. Therefore, whether or not the opening speed is less than a predetermined third threshold value by determining whether or not the opening speed evaluation values z # 1 _t to z # 3 _t are less than the fourth threshold value. Can be determined.
In the first embodiment, the opening speed evaluation value calculation unit 142 calculates the opening speed evaluation value z # 3 _t . Here, in equation (5), attention is paid to the gradient of x _t , and when the value of α in equation (1) above is set small, the difference between x _t and y _t increases as the opening speed increases. Since it becomes large, the opening speed evaluation value z # 3 _t can be used as an evaluation value of the opening speed.

Here, the opening speed becomes higher when the inclination of the opening continuity is large. Therefore, as shown in FIGS. 8A and 8B, the opening speed is high during yawning and slow during conversation.

Returning to FIG. 4, the opening speed evaluation value storage unit 143 stores the opening speed evaluation value calculated by the opening speed evaluation value calculation unit 142.

The yawning determination unit 144 uses a predetermined factor (here) using the opening continuity stored in the opening continuity storage unit 140 and the opening speed evaluation value stored in the opening speed evaluation value storage unit 143. Then, the opening due to the conversation) and the opening due to a factor other than the predetermined factor are discriminated, and the yawning is determined.

Specifically, in the opening distinction determination condition in which the opening speed evaluation value stored in the opening speed evaluation value storage unit 143 is included in the yawning determination condition model 121, the opening speed evaluation value is a predetermined threshold value (for example). , 20), it is determined that the opening is due to a predetermined factor. The threshold value here is also referred to as a fourth threshold value.

Next, when the yawning determination unit 144 determines that the opening is due to a predetermined factor, the invalid value is set as the continuation value of the yawning opening, and when it is determined that the opening is due to a factor other than the predetermined factor, the invalid value is used. The degree of opening continuity of the frame is set as the degree of continuation of yawning, and is stored in the temporary storage unit 145 in association with the frame identification number which is the identification information for identifying the frame. The invalid value may be any value as long as it is smaller than the first threshold value.

Then, the yawning determination unit 144 determines that yawning is achieved when the yawning determination condition included in the yawning determination condition model 121 is satisfied.
Here, the yawning determination unit 144 is the number of frames in which the degree of continuation of the yawning opening is a predetermined value (for example, 40) or more among the predetermined number of frames in the past (for example, 100 frames). When is a predetermined number (for example, 20) or more, it is determined to be yawning.

The yawning determination result storage unit 146 stores the yawning determination result in the yawning determination unit 144. The yawning determination result here is, for example, "yawning" or "not yawning".

The output unit 147 sends the yawning determination result stored in the yawning determination result storage unit 146 to the display unit 150 shown in FIG. 1, and the display unit 150 receives the yawning determination result or the information corresponding to the yawning determination result. Is displayed.

FIG. 9 is a block diagram schematically showing a hardware configuration of the yawning determination device 100 according to the first embodiment.
The yawning determination device 100 according to the first embodiment can be configured by a computer 160 including a camera 161, an auxiliary storage device 162, a processor 163, a memory 164, and a display device 165.

Specifically, the image pickup unit 110 can be realized by a camera 161 as an image pickup device.
The database unit 120 can be realized by an auxiliary storage device 162 such as an HDD (Hard Disk Drive).

The processing unit 130 can be realized by the processor 163 and the memory 164.
The processor 163 can be realized by a CPU (Central Processing Unit) or the like, and the memory 164 can be realized by a non-volatile memory. The processor 163 reads the program stored in the auxiliary storage device 162 into the memory 164 and executes the program to execute various processes. Such a program may be provided through a network, or may be recorded and provided on a recording medium. That is, such a program may be provided, for example, as a program product.
The display unit 150 can be realized by the display device 165.

FIG. 10 is a flowchart showing the processing in the yawning determination unit 144 in the first embodiment.
First, the stretchout determination unit 144 acquires the opening continuity stored in the opening continuity storage unit 140 and the opening speed evaluation value stored in the opening speed evaluation value storage unit 143 (S10).

Next, the yawning determination unit 144 determines whether or not the acquired opening speed evaluation value is equal to or greater than the threshold value a in the opening discrimination determination condition (S11). When the opening speed evaluation value is less than the threshold value a (No in S11), the process proceeds to step S12, and when the opening speed evaluation value is equal to or more than the threshold value a (Yes in S11), the process proceeds to step S13. move on.

In step S12, the opening speed evaluation value is less than the threshold value a, and it is considered that the opening of the person shown in the moving image is a conversation other than yawning. It is an invalid value (for example, "-1") which is a value. Then, the process proceeds to step S13.

In step S13, the yawning determination unit 144 stores the opening continuity of the current frame as the yawning opening continuity in the temporary storage unit 145 together with the frame number. Here, the yawning opening continuity is the opening continuity calculated by the opening continuity calculation unit 139 corresponding to the current frame when the opening speed evaluation value is equal to or higher than the threshold value a, and the opening speed evaluation value is If it is less than the threshold value a, it becomes an invalid value set in step S12.

Next, the yawning determination unit 144 determines whether or not the yawning determination condition condition included in the yawning determination condition model 121 is satisfied (S14). Here, the yawning determination unit 144 is the number of frames in which the degree of continuation of the yawning opening is a predetermined value (for example, 40) or more among the predetermined number of frames in the past (for example, 100 frames). When is a predetermined number (for example, 20) or more, it is determined to be yawning. If the yawning determination condition is satisfied (Yes in S14), the process proceeds to step S15, and if the yawning determination condition is not satisfied (No in S14), the process proceeds to step S16.

In step S15, the yawning determination unit 144 determines that the yawning is due, and stores the determination result in the yawning determination result storage unit 146. The yawning determination unit 144 assumes that yawning does not occur continuously, and once yawning is determined, a certain frame (for example, 100 frames) is not determined to be yawning.
On the other hand, in step S16, the yawning determination unit 144 determines that the yawning is not, and stores the determination result in the yawning determination result storage unit 146.

As described above, according to the first embodiment, when the person shown in the moving image is talking, it is not determined to be yawning, so that the yawning is detected accurately from the state where the person's mouth is open. be able to.

Embodiment 2.
FIG. 11 is a block diagram schematically showing the configuration of the yawning determination device 200 as the information processing device according to the second embodiment.
The yawn determination device 200 includes an image pickup unit 110, a database unit 220, a processing unit 230, and a display unit 150.
The image pickup unit 110 and the display unit 150 of the yawn determination device 200 according to the second embodiment are the same as the image pickup unit 110 and the display unit 150 of the yawn determination device 100 according to the first embodiment.

The database unit 220 is a storage unit that stores the yawning determination condition model 221 and the facial expression feature model 222.
The yawn determination condition model 221 includes an opening distinction determination condition and a yawn determination condition.
The yawning determination condition of the yawning determination condition model 221 in the second embodiment is the same as the yawning determination condition of the yawning determination condition model 121 in the first embodiment.

The opening distinction determination condition is a determination condition used to distinguish between an opening due to a predetermined factor and an opening due to a factor other than the predetermined factor.
FIG. 12 is a schematic view showing an example of the opening distinction determination condition.
In the second embodiment, the opening distinction determination condition is a condition for determining whether the opening is due to a predetermined factor or a factor other than the predetermined factor according to the facial expression feature comparison result described later. There is. In the second embodiment, the predetermined factor is that the facial expression of the person is a smiling face or an angry face.

The facial expression feature model 222 includes a smile determination condition and a surprise determination condition.
The smile determination condition is a condition for determining that the facial expression of the person included in the moving image is a smile based on the facial feature amount.
The surprise determination condition is a condition for determining that the facial expression of the person included in the moving image is a surprised face based on the facial feature amount.

FIG. 13 is a schematic view showing an example of the facial expression feature model 222.
When the facial feature amount satisfies any of the conditions shown in FIG. 13, it is determined that the facial expression of the person is a smiling face or a surprised face.

Returning to FIG. 11, the processing unit 230 executes the processing by the yawning determination device 200.
FIG. 14 is a block diagram schematically showing the configuration of the processing unit 230 according to the second embodiment.
The processing unit 230 includes an input unit 131, a face area extraction unit 132, a face feature point extraction unit 133, a face feature point storage unit 134, a face feature amount calculation unit 135, a face feature amount storage unit 136, and an opening. It includes a degree specifying unit 137, an opening degree storage unit 138, an opening continuity calculation unit 139, an opening continuity storage unit 140, a determination processing unit 241 and a stretch determination result storage unit 146, and an output unit 147.

The determination processing unit 241 includes a yawn determination unit 244, a temporary storage unit 145, a facial expression feature comparison unit 248, and a facial expression feature comparison result storage unit 249.

Input unit 131, face area extraction unit 132, face feature point extraction unit 133, face feature point storage unit 134, face feature amount calculation unit 135, face feature amount storage unit 136, opening degree specification of the processing unit 230 in the second embodiment. Unit 137, opening degree storage unit 138, opening continuity calculation unit 139, opening continuity storage unit 140, temporary storage unit 145, stretch determination result storage unit 146, and output unit 147 are inputs of the processing unit 130 in the first embodiment. Unit 131, face area extraction unit 132, face feature point extraction unit 133, face feature point storage unit 134, face feature amount calculation unit 135, face feature amount storage unit 136, opening degree identification unit 137, opening degree storage unit 138, opening. This is the same as the continuity calculation unit 139, the opening continuity storage unit 140, the temporary storage unit 145, the stretchout determination result storage unit 146, and the output unit 147.

Also in the second embodiment, the determination processing unit 241 determines whether or not the person in the frame makes an opening due to a predetermined factor other than yawning.
However, in the second embodiment, the determination processing unit 241 determines that the opening is performed by a predetermined factor when the facial expression of the person is a predetermined facial expression different from the yawning.
The processing in the determination processing unit 241 in the second embodiment is realized by the yawn determination unit 244, the temporary storage unit 145, the facial expression feature comparison unit 248, and the facial expression feature comparison result storage unit 249. This will be described below.

The facial expression feature comparison unit 248 is a facial expression determination unit that determines whether or not a person in the frame has a predetermined facial expression based on the facial feature amount. Here, the facial expression feature comparison unit 248 determines the facial expression of a person by comparing the current facial feature with the facial expression feature (for example, the facial feature that appears at the time of a smile or surprise).

For example, the facial expression feature comparison unit 248 uses the facial feature amount stored in the facial feature amount storage unit 136 to determine whether facial features that do not occur during yawning can be seen. In general, when a person's face is smiling or surprised, facial features that do not occur during yawning appear. Therefore, when the facial expression feature comparison unit 248 is a facial feature that indicates a facial feature that does not occur during yawning, the facial expression feature comparison unit 248 determines that the opening is not due to yawning.

Specifically, in the facial expression feature comparison unit 248, any of the facial expressions stored in the facial feature amount storage unit 136 satisfies the determination condition shown in the facial expression feature model 222 of FIG. In some cases, it is determined that the opening is due to a smile or an opening due to surprise.

The facial feature comparison unit 248 uses the facial feature amount stored in the facial feature amount storage unit 136 to perform machine learning such as Random Forest, SVM (Support Vector Machine), AdaBoost, and CNN (Convolutional Neural Network). Techniques may be used to determine if the opening is due to a predetermined factor.

The facial expression feature comparison result storage unit 249 stores the comparison result (also referred to as a determination result) in the facial expression feature comparison unit 248.
The comparison result in the facial expression feature comparison unit 248 is, for example, "aperture due to a smile", "opening due to surprise", or "opening". The "opening" is a comparison result when it is not determined to be "opening due to a smile" or "opening due to surprise". Here, "aperture by smile" indicates that the facial expression of the person in the frame is a smile, and "aperture by surprise" indicates that the facial expression is a surprised face.

The yawning determination unit 244 uses an opening continuity stored in the opening continuity storage unit 140 and a comparison result stored in the facial expression feature comparison result storage unit 249 to determine an opening due to a predetermined factor. It is determined whether or not it is present, and yawning is determined. For example, the yawning determination unit 244 determines that when the determination result of the facial expression feature comparison unit 248 is a predetermined facial expression, the opening is performed by a predetermined factor. When the determination result is that the facial expression of the person is not a predetermined facial expression, it is determined that the opening is due to a factor other than the predetermined factor.

Specifically, the yawning determination unit 244 is based on a predetermined factor when the comparison result stored in the facial expression feature comparison result storage unit 249 is "opening due to smile" or "opening due to surprise". If it is determined to be an opening and the comparison result is "opening", it is determined to be an opening due to a factor other than a predetermined factor.

Next, when the yawning determination unit 244 determines that the opening is due to a predetermined factor, the invalid value is set as the continuation value of the yawning opening, and when it is determined that the opening is due to a factor other than the predetermined factor, the invalid value is used. The degree of opening continuity of the frame is set as the degree of continuation of yawning, and is stored in the temporary storage unit 145 in association with the frame identification number which is the identification information for identifying the frame.
Then, the yawning determination unit 244 determines that yawning is achieved when the yawning determination condition included in the yawning determination condition model 221 is satisfied.

FIG. 15 is a flowchart showing the processing in the yawning determination unit 244 in the second embodiment.
First, the yawning determination unit 244 acquires the opening continuity stored in the opening continuity storage unit 140 and the comparison result stored in the facial expression feature comparison result storage unit 249 (S20).

Next, the yawning determination unit 244 determines whether or not the acquired comparison result is an opening due to a predetermined factor (S21). For example, when the comparison result is "opening due to smile" or "opening due to surprise", the yawning determination unit 244 determines that the opening is due to a predetermined factor, and when the comparison result is "opening". Is determined to be an opening due to a factor other than a predetermined factor. If the opening is due to a predetermined factor (Yes in S21), the process proceeds to step S22, and if the opening is due to a factor other than the predetermined factor (No in S21), the process proceeds to step S23. Proceed to.

In step S22, the yawning determination unit 244 sets the opening continuity to an invalid value (for example, “-1”) which is a predetermined value. Then, the process proceeds to step S23.

In step S23, the yawning determination unit 244 stores the opening continuity of the current frame as the yawning opening continuity in the temporary storage unit 145 together with the frame number. Here, the yawning opening continuity is the opening continuity calculated by the opening continuity calculation unit 139 corresponding to the current frame when the opening is due to a factor other than a predetermined factor, and is predetermined. If the opening is due to a factor, the invalid value set in step S22 is used.

Next, the yawning determination unit 244 determines whether or not the yawning determination condition condition included in the yawning determination condition model 221 is satisfied (S24). Here, the yawning determination unit 244 is the number of frames in which the degree of continuation of the yawning opening is a predetermined value (for example, 40) or more among the predetermined number of frames in the past (for example, 100 frames). When is a predetermined number (for example, 20) or more, it is determined to be yawning. If the yawning determination condition is satisfied (Yes in S24), the process proceeds to step S25, and if the yawning determination condition is not satisfied (No in S24), the process proceeds to step S26.

In step S25, the yawning determination unit 244 determines that the yawning occurs, and stores the determination result in the yawning determination result storage unit 146. The yawning determination unit 244 assumes that yawning does not occur continuously, and once yawning is determined, a certain frame (for example, 100 frames) is not determined to be yawning.
On the other hand, in step S26, the yawning determination unit 144 determines that the yawning is not, and stores the determination result in the yawning determination result storage unit 146.

As described above, according to the second embodiment, when the facial expression of the person in the moving image is a smile or a surprised face, it is not determined to be yawning, so that the person's mouth is open. , Yawning can be detected accurately.

Embodiment 3.
As shown in FIG. 11, the yawning determination device 300 as the information processing device according to the third embodiment includes an imaging unit 110, a database unit 320, a processing unit 330, and a display unit 150.
The image pickup unit 110 and the display unit 150 of the yawn determination device 300 according to the third embodiment are the same as the image pickup unit 110 and the display unit 150 of the yawn determination device 100 according to the first embodiment.

The database unit 320 is a storage unit that stores the yawning determination condition model 321 and the facial expression feature model 222.
The facial expression feature model 222 in the third embodiment is the same as the facial expression feature model 222 in the second embodiment.

The yawn determination condition model 321 includes an opening distinction determination condition and a yawn determination condition.
The yawning determination condition of the yawning determination condition model 321 in the third embodiment is the same as the yawning determination condition of the yawning determination condition model 121 in the first embodiment.

The opening distinction determination condition is a determination condition used to distinguish between an opening due to a predetermined factor and an opening due to a factor other than the predetermined factor.
FIG. 16 is a schematic view showing an example of the opening distinction determination condition.
In the third embodiment, when the opening speed evaluation value is less than a predetermined threshold value (for example, 20), the opening discrimination determination condition determines that the opening is due to a predetermined factor, and the opening speed evaluation value is determined. If is equal to or greater than a predetermined threshold value, the opening is due to a factor other than the predetermined factor, and depending on the facial expression feature comparison result, the opening is due to the predetermined factor. It is a condition to judge whether the opening is due to a factor other than the specified factor. Also in the third embodiment, the predetermined factor is that the facial expression of the person is a smiling face or an angry face.

Returning to FIG. 11, the processing unit 330 executes processing by the yawning determination device 300.
FIG. 17 is a block diagram schematically showing the configuration of the processing unit 330 in the third embodiment.
The processing unit 330 includes an input unit 131, a face area extraction unit 132, a face feature point extraction unit 133, a face feature point storage unit 134, a face feature amount calculation unit 135, a face feature amount storage unit 136, and an opening. It includes a degree specifying unit 137, an opening degree storage unit 138, an opening continuity degree calculation unit 139, an opening continuity degree storage unit 140, a determination processing unit 341, a missing extension determination result storage unit 146, and an output unit 147.

The determination processing unit 341 includes an opening speed evaluation value calculation unit 142, an opening speed evaluation value storage unit 143, a stretch determination unit 344, a temporary storage unit 145, a facial expression feature comparison unit 248, and a facial expression feature comparison result storage. A unit 249 is provided.

Input unit 131, face area extraction unit 132, face feature point extraction unit 133, face feature point storage unit 134, face feature amount calculation unit 135, face feature amount storage unit 136, opening degree specification of the processing unit 330 in the third embodiment. Unit 137, opening degree storage unit 138, opening continuity calculation unit 139, opening continuity storage unit 140, temporary storage unit 145, stretch determination result storage unit 146, and output unit 147 are inputs of the processing unit 130 in the first embodiment. Unit 131, face area extraction unit 132, face feature point extraction unit 133, face feature point storage unit 134, face feature amount calculation unit 135, face feature amount storage unit 136, opening degree identification unit 137, opening degree storage unit 138, opening. This is the same as the continuity calculation unit 139, the opening continuity storage unit 140, the temporary storage unit 145, the stretchout determination result storage unit 146, and the output unit 147.

Further, the facial expression feature comparison unit 248 and the facial expression feature comparison result storage unit 249 of the processing unit 330 in the third embodiment store the facial expression feature comparison unit 248 and the facial expression feature comparison result storage of the processing unit 230 in the second embodiment. It is the same as the part 249.

Also in the third embodiment, the determination processing unit 341 determines whether or not the person in the frame makes an opening due to a predetermined factor.
However, in the third embodiment, the determination processing unit 341 determines a predetermined factor when the person is having a conversation or when the facial expression of the person is a predetermined facial expression different from the yawning. It is determined that the opening is performed by.
The processing in the determination processing unit 341 in the third embodiment includes the opening speed evaluation value calculation unit 142, the opening speed evaluation value storage unit 143, the stretch determination unit 344, the temporary storage unit 145, the facial expression feature comparison unit 248, and the facial expression feature. It is realized by the comparison result storage unit 249. This will be described below.

The missing stretch determination unit 344 is stored in the opening continuity storage unit 140, the opening speed stored in the opening speed evaluation value storage unit 143, and the facial expression feature comparison result storage unit 249. Using the comparison results, the opening due to a predetermined factor and the opening due to a factor other than the predetermined factor are distinguished, and the loss is determined.

Specifically, the yawning determination unit 344 sets the opening speed evaluation value in advance under the opening distinction determination condition in which the opening speed evaluation value stored in the opening speed evaluation value storage unit 143 is included in the yawning determination condition model 121. If it is less than a predetermined threshold (for example, 20), it is determined that the opening is due to a predetermined factor.

Further, the yawning determination unit 344 is an opening due to a predetermined factor when the comparison result stored in the facial expression feature comparison result storage unit 249 is "opening due to smile" or "opening due to surprise". If the comparison result is "opening", it is determined that the opening is due to a factor other than a predetermined factor.

Next, when the yawning determination unit 344 determines that the opening is due to a predetermined factor, the invalid value is set as the continuation value of the yawning opening, and when it is determined that the opening is due to a factor other than the predetermined factor, the invalid value is used. The degree of opening continuity of the frame is set as the degree of continuation of yawning, and is stored in the temporary storage unit 145 in association with the frame identification number which is the identification information for identifying the frame.
Then, the yawning determination unit 344 determines that yawning is achieved when the yawning determination condition included in the yawning determination condition model 321 is satisfied.
Here, the yawning determination unit 344 is the number of frames in which the degree of continuation of the yawning opening is a predetermined value (for example, 40) or more among the predetermined number of frames in the past (for example, 100 frames). When is a predetermined number (for example, 20) or more, it is determined to be yawning.

FIG. 18 is a flowchart showing the process in the yawning determination unit 344 in the third embodiment.
First, the stretchlessness determination unit 344 has an opening continuity stored in the opening continuity storage unit 140, an opening speed evaluation value stored in the opening speed evaluation value storage unit 143, and a facial expression feature comparison result storage unit 249. The comparison result stored in is acquired (S30).

Next, the yawning determination unit 344 determines whether or not the acquired opening speed evaluation value is equal to or greater than the threshold value a in the opening discrimination determination condition (S31). When the opening speed evaluation value is less than the threshold value a (No in S31), the process proceeds to step S32, and when the opening speed evaluation value is equal to or more than the threshold value a (Yes in S31), the process proceeds to step S33. move on.

In step S32, the opening speed evaluation value is less than the threshold value a, and it is considered that the opening of the person shown in the moving image is a conversation other than yawning. Therefore, the yawning determination unit 344 determines the opening continuity degree in advance. It is an invalid value (for example, "-1") which is a value. Then, the process proceeds to step S33.

In step S33, the yawning determination unit 344 determines whether or not the acquired comparison result is an opening due to a predetermined factor. For example, when the comparison result is "opening due to smile" or "opening due to surprise", the yawning determination unit 344 determines that the opening is due to a predetermined factor, and when the comparison result is "opening". Is determined to be an opening due to a factor other than a predetermined factor. If the opening is due to a predetermined factor (Yes in S33), the process proceeds to step S34, and if the opening is due to a factor other than the predetermined factor (No in S34), the process proceeds to step S35. Proceed to.

In step S34, the yawning determination unit 344 sets the opening continuity to an invalid value (for example, “-1”) which is a predetermined value. Then, the process proceeds to step S35.

In step S35, the yawning determination unit 344 stores the opening continuity of the current frame as the yawning opening continuity in the temporary storage unit 145 together with the frame number.

Next, the yawning determination unit 344 determines whether or not the yawning determination condition condition included in the yawning determination condition model 321 is satisfied (S36). Here, the yawning determination unit 344 is the number of frames in which the degree of continuation of the yawning opening is a predetermined value (for example, 40) or more among the predetermined number of frames in the past (for example, 100 frames). When is a predetermined number (for example, 20) or more, it is determined to be yawning. If the yawning determination condition is satisfied (Yes in S36), the process proceeds to step S37, and if the yawning determination condition is not satisfied (No in S36), the process proceeds to step S38.

In step S37, the yawning determination unit 344 determines that the yawning occurs, and stores the determination result in the yawning determination result storage unit 146. The yawning determination unit 344 assumes that yawning does not occur continuously, and once yawning is determined, a certain frame (for example, 100 frames) is not determined to be yawning.
On the other hand, in step S36, the yawning determination unit 344 determines that the yawning is not, and stores the determination result in the yawning determination result storage unit 146.

As described above, according to the third embodiment, when the person shown in the moving image is talking, smiling, or surprised, it is not determined to be yawning, so that the person's mouth is open. Yawning can be detected accurately from the open state.

In the above-described embodiments 1 to 3, the yawning determination devices 100, 200, and 300 include all of the imaging unit 110, the database unit 120, the processing units 130, 230, 330, and the display unit 150. Embodiments 1 to 3 are not limited to such examples. For example, at least one of the image pickup unit 110, the database unit 120, and the display unit 150 may be another device connected to the yawn determination devices 100, 200, and 300 via a network.

100,200,300 deficiency determination device, 110 imaging unit, 120, 220, 320 database unit, 121,221,321 deficiency determination condition model, 222 facial expression feature model, 130, 230, 330 processing unit, 131 input unit, 132 Face area extraction unit, 133 Face feature point extraction unit, 134 Face feature point storage unit, 135 Face feature amount calculation unit, 136 Face feature amount storage unit, 137 Openness specification unit, 138 Openness storage unit, 139 Opening continuity calculation unit Unit, 140 Opening continuity storage unit, 141,241,341 Judgment processing unit, 142 Opening speed evaluation value calculation unit, 143 Opening speed evaluation value storage unit, 144,244,344 Missing extension judgment unit, 145 Temporary storage unit, 146 Missing extension Judgment result storage unit, 147 output unit, 248 facial expression feature comparison unit, 249 facial expression feature comparison result storage unit, 150 display unit.

Claims (14)

A face area extraction unit that extracts a face area, which is a face area of a person, from each of a plurality of frames included in a moving image, and a face area extraction unit.
A face feature point extraction unit that extracts a plurality of predetermined feature points from the face area, and a face feature point extraction unit.
A face feature amount calculation unit that calculates a face feature amount indicating the face feature from the plurality of feature points, and a face feature amount calculation unit.
An opening degree specifying portion that specifies the opening degree, which is the degree to which the mouth is open in the face, from the facial feature amount.
An opening continuity calculation unit that calculates an opening continuity, which is the degree to which the mouth is continuously opened, by a plurality of opening degrees specified from the plurality of frames.
It is determined whether or not the person is opening due to a predetermined factor other than yawning, and a predetermined first threshold value is set in the frame determined to be opening due to the predetermined factor. A value lower than is associated with the yawning opening continuity, and the frame determined that the opening is not performed due to the predetermined factor is associated with the opening continuation as the yawning opening continuity, and the latest frame is included. When the number of frames associated with the yawning opening continuity that is equal to or greater than the first threshold value among the predetermined number of consecutive frames is equal to or greater than the predetermined second threshold value. An information processing apparatus including a determination processing unit for determining that the person has yawned. The information processing apparatus according to claim 1, wherein the determination processing unit determines that the person is making an opening due to a predetermined factor when the person is having a conversation. The second aspect of the present invention is characterized in that the determination processing unit determines that the person is having a conversation when the opening speed, which is the opening speed of the mouth, is less than a predetermined third threshold value. Information processing equipment. The determination processing unit
An opening speed evaluation value calculation unit that calculates an opening speed evaluation value, which is a value that increases as the opening speed increases from the opening continuity.
3. The third aspect of the present invention is provided with a yawning determination unit for determining that the opening speed is less than the third threshold value when the opening speed evaluation value is less than a predetermined fourth threshold value. The information processing device described in. The first aspect of claim 1, wherein the determination processing unit determines that the opening is performed by the predetermined factor when the facial expression of the person is a predetermined facial expression different from the yawning. Information processing equipment. The determination processing unit
A facial expression determination unit that determines whether or not the facial expression of the person is a predetermined facial expression from the facial feature amount.
When the determination result in the facial expression determination unit is that the facial expression of the person is the predetermined facial expression, it is determined that the opening is performed by the predetermined factor, and the facial expression determination unit determines that the opening is performed. 5. The fifth aspect of the present invention is characterized in that the determination result includes a stretch determination unit for determining that the opening due to the predetermined factor is not performed when the facial expression of the person is not the predetermined facial expression. The information processing device described in. The information processing apparatus according to claim 5 or 6, wherein the predetermined facial expression is a smiling face or a surprised face. When the person is having a conversation or the facial expression of the person is a predetermined facial expression different from the yawning, the determination processing unit is said to perform the opening due to the predetermined factor. The information processing apparatus according to claim 1, wherein the determination is made. The eighth aspect of the present invention is characterized in that the determination processing unit determines that the person is having a conversation when the opening speed, which is the opening speed of the mouth, is less than a predetermined third threshold value. Information processing equipment. The determination processing unit
An opening speed evaluation value calculation unit that calculates an opening speed evaluation value, which is a value that increases as the opening speed increases from the opening continuity.
9. The aspect 9 is provided with a yawning determination unit for determining that the opening speed is less than the third threshold value when the opening speed evaluation value is less than a predetermined fourth threshold value. The information processing device described in. The determination processing unit
A facial expression determination unit that determines whether or not the facial expression of the person is a predetermined facial expression from the facial feature amount.
When the determination result in the facial expression determination unit is that the facial expression of the person is the predetermined facial expression, it is determined that the opening is performed by the predetermined factor, and the determination in the facial expression determination unit. The eighth aspect of the present invention is characterized by comprising a stretchout determination unit for determining that the opening is not performed due to the predetermined factor when the facial expression of the person is not the predetermined facial expression. The information processing device described. The information processing apparatus according to claim 11, wherein the predetermined facial expression is a smiling face or a surprised face. Computer,
A face area extraction unit that extracts a face area, which is a face area of a person, from each of a plurality of frames included in a moving image.
A face feature point extraction unit that extracts a plurality of predetermined feature points from the face area,
A facial feature amount calculation unit that calculates a facial feature amount indicating the facial feature from the plurality of feature points.
An opening degree specifying portion that specifies the opening degree, which is the degree to which the mouth is open in the face, from the facial feature amount.
An opening continuity calculation unit that calculates an opening continuity, which is the degree to which the mouth is continuously open, based on the plurality of openings specified from the plurality of frames, and an opening continuity calculation unit.
It is determined whether or not the person is opening due to a predetermined factor other than yawning, and a predetermined first threshold value is set in the frame determined to be opening due to the predetermined factor. A value lower than is associated with the yawning opening continuity, and the frame determined that the opening is not performed due to the predetermined factor is associated with the opening continuation as the yawning opening continuity, and the latest frame is included. When the number of frames associated with the yawning opening continuity that is equal to or greater than the first threshold value among the predetermined number of consecutive frames is equal to or greater than the predetermined second threshold value. A program characterized by functioning as a determination processing unit for determining that the person has yawned. The face area, which is the area of the human face, is extracted from each of the multiple frames included in the moving image.
A plurality of predetermined feature points are extracted from the face area, and a plurality of predetermined feature points are extracted.
From the plurality of feature points, a facial feature amount indicating the facial feature is calculated.
From the facial features, the degree of opening, which is the degree to which the mouth is open in the face, is specified.
The opening continuity, which is the degree to which the mouth is continuously opened, is calculated from the plurality of openings specified from the plurality of frames.
It is determined whether or not the person is opening due to a predetermined factor other than yawning, and a predetermined first threshold value is set in the frame determined to be opening due to the predetermined factor. Corresponding to a value lower than the yawning opening continuity,
The opening continuity is associated with the frame determined not to be opened due to the predetermined factor as the yawning opening continuity.
Of the predetermined number of consecutive frames including the latest frame, the number of frames associated with the yawning opening continuity that is equal to or higher than the first threshold value is the predetermined second threshold value. An information processing method characterized in that it is determined that the person has yawned in the above cases.

Images