JP2020149361A - Expression estimating apparatus, feeling determining apparatus, expression estimating method, and program - Google Patents

Abstract

To provide an expression estimating apparatus, a feeling determining apparatus, an expression estimating method, and a program therefor which can estimate expression of a person with higher accuracy.SOLUTION: An expression estimating apparatus 1 has a captured data acquiring unit 2 for acquiring an image data of a face of a target person captured by an imaging device, a face image data acquiring unit 3 for acquiring a normal time face image data of the target person which is stored in advance in a storage device, an angle estimating unit 4 for estimating an angle of the face of the target person from the image data acquired by the captured data acquiring unit 2, a correction unit 4 for correcting the face image data acquired by the face image data acquiring unit 3 based on the estimation result of the angle estimating unit 4, and an expression estimating unit 6 for estimating an expression of the target person by comparing the image data acquired by the captured data acquiring unit 2 and the face image data corrected by the correction unit 5.SELECTED DRAWING: Figure 1

Description

The present invention relates to a facial expression estimation device and a facial expression estimation method for estimating a facial expression of a person from image data obtained by capturing a person's face, and an emotion determination device provided with the facial expression estimation device, and further, to realize these. About the program.

In recent years, a technique for reading a person's facial expression from image data obtained by capturing a person's face has been proposed. Patent Document 1 discloses a psychological state estimation method that analyzes not only the facial expression of a person but also the internal state of the person. The psychological state estimation method described in Patent Document 1 calculates the difference (age difference) between the age estimated from the image data of a person who has returned home and the age estimated from the standard facial image data of a normal facial expression. are doing. Taking advantage of the fact that when the age difference is large, the person looks older than usual, it is estimated whether the person who returned home has positive emotions or negative emotions. As a result, Patent Document 1 makes it possible to objectively determine the psychological state of the target person.

Japanese Unexamined Patent Publication No. 2016-093313

However, in Patent Document 1, in order to estimate the psychological state, it is necessary to use the image data captured with the face facing the front of the imaging device. Therefore, if the face of a person cannot be imaged from the front, the psychological state cannot be estimated, or the estimation accuracy may decrease.

An example of an object of the present invention is to provide a facial expression estimation device, an emotion determination device, a facial expression estimation method, and a program capable of increasing the estimation accuracy of a person's facial expression.

In order to achieve the above object, the facial expression estimation device in one aspect of the present invention is
An imaging data acquisition unit that acquires image data of the face of the target person captured by the imaging device, and
A face image data acquisition unit that acquires face image data of the target person in normal times, which is stored in advance in a storage device,
An angle estimation unit that estimates the face angle of the target person from the image data acquired by the imaging data acquisition unit, and an angle estimation unit.
A correction unit that corrects the face image data acquired by the face image data acquisition unit based on the estimation result of the angle estimation unit, and a correction unit.
A facial expression estimation unit that estimates the facial expression of the target person by comparing the image data acquired by the imaging data acquisition unit with the facial image data corrected by the correction unit.
It is characterized by having.

Further, in order to achieve the above object, the emotion determination device in one aspect of the present invention is used.
An imaging data acquisition unit that acquires image data of the face of the target person captured by the imaging device, and
A face image data acquisition unit that acquires face image data of the target person in normal times, which is stored in advance in a storage device,
An angle estimation unit that estimates the face angle of the target person from the image data acquired by the imaging data acquisition unit, and an angle estimation unit.
A correction unit that corrects the face image data acquired by the face image data acquisition unit based on the estimation result of the angle estimation unit, and a correction unit.
A facial expression estimation unit that estimates the facial expression of the target person by comparing the image data acquired by the imaging data acquisition unit with the facial image data corrected by the correction unit.
An emotion determination unit that determines the emotion of the target person based on the estimation result by the facial expression estimation unit, and
It is characterized by having.

Further, in order to achieve the above object, the facial expression estimation method in one aspect of the present invention is:
(A) A step of acquiring image data of the face of the target person captured by the imaging device, and
(B) A step of acquiring the normal face image data of the target person stored in the storage device in advance, and
(C) A step of estimating the face angle of the target person from the image data acquired in the step (a), and
(D) A step of correcting the face image data acquired in the step (b) based on the estimation result in the step (c), and
(E) A step of estimating the facial expression of the target person by comparing the image data acquired in the step (a) with the corrected face image data in the step (d).
It is characterized by having.

In addition, in order to achieve the above object, the program in one aspect of the present invention
To a computer that acquires image data of the face of the target person captured by the image pickup device
(A) A step of acquiring the normal face image data of the target person stored in advance in the storage device, and
(B) A step of estimating the face angle of the target person from the image data, and
(C) A step of correcting the face image data acquired in the step (a) based on the estimation result in the step (b), and a step of correcting the face image data.
(D) A step of estimating the facial expression of the target person by comparing the image data with the face image data corrected in the step (c).
Is characterized by executing.

According to the present invention, since the facial expression is estimated by correcting the reference facial image data in which the lack of facial information of the target person is small and comparing it with the image data obtained by imaging, the facial expression can be accurately obtained. Can be estimated.

FIG. 1 is a block diagram showing a configuration of a facial expression estimation device according to the present embodiment. FIG. 2 is a block diagram showing a specific configuration of the emotion determination device. FIG. 3 is a diagram showing feature points detected by the face feature point detection unit. FIG. 4 is a diagram showing the definition of the position of the facial organ and the measurement items in each organ. FIG. 5 is a diagram showing face image data before and after correction by the correction unit. FIG. 6 is a diagram showing a point addition / deduction table used when calculating the score for each facial expression. FIG. 7 is a diagram showing a point addition / deduction table used when calculating the score for each facial expression. FIG. 8 is a diagram showing a table used when estimating facial expressions from the results of addition and subtraction of facial expression scores. FIG. 9 is a diagram showing a table used when quantifying the emotional determination result. FIG. 10 is a diagram showing a table used when quantifying the emotional determination result. FIG. 11 is a diagram showing a determination result by the emotion determination device. FIG. 12 is a flow chart showing the operation of the facial expression estimation device and the emotion determination device in the embodiment. FIG. 13 is a block diagram showing an example of a computer that realizes the facial expression estimation device and the emotion determination device according to the present embodiment.

Hereinafter, the facial expression estimation device, the emotion determination device, and the facial expression estimation method according to the embodiment of the present invention will be described with reference to the drawings.

[Device configuration]
FIG. 1 is a block diagram showing a configuration of a facial expression estimation device 1 according to the present embodiment.

The facial expression estimation device 1 includes an imaging data acquisition unit 2, a face image data acquisition unit 3, an angle estimation unit 4, a correction unit 5, and a facial expression estimation unit 6.

The imaging data acquisition unit 2 acquires image data of the face of the target person captured by an imaging device (not shown).

The face image data acquisition unit 3 acquires the face image data of the target person in normal times, which is stored in advance in a storage device (not shown).

The angle estimation unit 4 estimates the angle of the face of the target person from the image data acquired by the imaging data acquisition unit 2.

The correction unit 5 corrects the face image data acquired by the face image data acquisition unit 3 based on the estimation result of the angle estimation unit 4.

The facial expression estimation unit 6 estimates the facial expression of the target person by comparing the image data acquired by the imaging data acquisition unit 2 with the face image data corrected by the correction unit 5.

This facial expression estimation device 1 corrects the reference facial image data in which there is little lack of facial information of the target person, and estimates the facial expression by comparing it with the image data obtained by imaging, so that the facial expression is accurately expressed. Can be estimated.

Subsequently, the configuration of the facial expression estimation device 1 of the present embodiment will be specifically described with reference to FIGS. 2 to 10. The facial expression estimation device 1 is used as a partial configuration of the emotion determination device. Hereinafter, the facial expression estimation device 1 will be described by specifically explaining the emotion determination device.

FIG. 2 is a block diagram showing a specific configuration of the emotion determination device 10. The emotion determination device 10 is mounted on, for example, an automobile. The automobile is equipped with an image pickup device 70. The image pickup device 70 is installed so as to image at least the driver's face. The emotion determination device 10 is connected to the image pickup device 70. The emotion determination device 10 and the image pickup device 70 may be integrated or may be independent devices. Further, the imaging device 70 does not have to be a device provided for the emotion determination device 10.

The facial expression estimation device 1 estimates the facial expression of the driver from the image data 70A captured by the image pickup device 70. Then, the emotion determination device 10 determines the driver's emotion from the driver's facial expression estimated by the facial expression estimation device 1. The place where the emotion determination device 10 is installed is not limited to the automobile, and may be installed in the building. In the following description, the driver is referred to as the target person.

The emotion determination device 10 includes an emotion determination unit 11 in addition to the facial expression estimation device 1 described above. Further, the facial expression estimation device 1 includes a face detection unit 7, a face feature point detection unit 8 and a face in addition to the imaging data acquisition unit 2, the face image data acquisition unit 3, the angle estimation unit 4, the correction unit 5, and the facial expression estimation unit 6. The organ measurement unit 9 is provided.

The image pickup data acquisition unit 2 acquires the image data 70A captured by the image pickup device 70.

The face detection unit 7 detects the face of the target person from the image data 70A. The face detection unit 7 extracts a face candidate region in the image data 70A by, for example, edge detection or shape pattern detection. The face detection unit 7 detects a face by dividing the extracted candidate area into small areas and collating the feature points of each small area with the face authentication pattern 71 acquired from the outside. The face recognition pattern 71 may be stored in a storage device installed in the emotion determination device 10, or may be stored in an external storage device connected to the emotion determination device 10 via a network. Further, the face recognition pattern 71 is updated every time the face detection process is performed by machine learning. The face detection method by the face detection unit 7 is not limited to the above, and a well-known technique can be appropriately used.

The face feature point detection unit 8 detects feature points of organs (for example, eyes, nose, mouth, etc.) constituting the face in the image data 70A detected by the face detection unit 7. FIG. 3 is a diagram showing feature points detected by the face feature point detection unit 8. The face feature point detection unit 8 has a total of 68, such as left eyebrows 17 to 21, right eyebrows 22 to 26, left eyes 36 to 41, and right eyes 42 to 47, from the face detected by the face detection unit 7 in the image data 70A. Detect individual feature points.

The facial organ measuring unit 9 classifies facial organs from the feature points detected by the facial feature point detecting unit 8. In addition, the facial organ measuring unit 9 measures a predetermined item for each of the classified facial organs. FIG. 4 is a diagram showing the definition of the position of the facial organ and the measurement items in each organ. The numbers shown in the item of "position" in FIG. 4 indicate the positions of the feature points in FIG. For example, the facial organ measuring unit 9 classifies the region surrounded by the feature points 5 to 11 (the shaded region in FIG. 3) into the “chin”. Then, the facial organ measuring unit 9 measures the position (coordinates) of the center of gravity of the region. The definition of the position of the facial organ shown in FIG. 4 and the measurement items in each organ are examples and can be changed as appropriate.

The angle estimation unit 4 estimates the angle of the face in the image data 70A. The angle of the face is based on the state in which the target person faces the front of the imaging device. As a method of estimating the angle, a conventional method can be used. For example, the angle estimation unit 4 detects the coordinates of a plurality of facial feature points in the image data 70A, for example, the tip of the nose, the chin, the left corner of the left eye, the right corner of the right eye, the left corner of the mouth, and the right corner of the mouth. To do. Then, the angle estimation unit 4 detects a plurality of feature points such as the tip of the nose described above by using a preset three-dimensional model. The angle estimation unit 4 performs calculations such as matrix conversion from the feature points detected from the image data 70A and the feature points detected from the three-dimensional model to obtain the angle.

The face image data acquisition unit 3 acquires the face image data 72A stored in the storage device 72. The storage device 72 stores in advance the face image data 72A of the target person, which is captured by the target person facing the surface of the image pickup device in normal times. Normal time is when the target person does not express emotions. That is, the facial expression in the face image data 72A is expressionless. The face image data acquisition unit 3 acquires the same target person's face image data 72A as the image data 70A acquired by the imaging data acquisition unit 2 from the storage device 72. The storage device 72 may be included in the facial expression estimation device 1 or the emotion determination device 10, or may be connected to the facial expression estimation device 1 or the emotion determination device 10 via a network.

The correction unit 5 corrects the face image data 72A acquired by the face image data acquisition unit 3 according to the estimation result of the angle estimation unit 4. Then, the correction unit 5 makes the angle of the face in the face image data 72A match the angle of the face in the image data 70A. FIG. 5 is a diagram showing face image data before and after correction by the correction unit 5. FIG. 5A is the face image data 72A acquired by the face image data acquisition unit 3, and FIG. 5B is the face image data 72B corrected by the correction unit 5.

As shown in FIG. 5A, the correction unit 5 is based on the face image data 72A of the facial feature points, for example, the tip of the nose, the chin, the left corner of the left eye, the right corner of the right eye, the left corner of the mouth, and the mouth. Detect the coordinates of the right corner. Further, the correction unit 5 maps the detected feature points to the three-dimensional model prepared in advance and facing the front. Then, the correction unit 5 moves the detected feature points as shown in FIG. 5 (B) by rotating the three-dimensional model according to the angle estimated by the angle estimation unit 4. As a result, the angle of the face in the face image data 72B coincides with the angle of the face in the image data 70A.

The facial expression estimation unit 6 compares the image data 70A with the face image data 72B (FIG. 5B) to estimate the facial expression of the target person. For example, the facial expression estimation unit 6 classifies the facial expressions of the image data 70A into six types of facial expressions: “happiness”, “surprise”, “anger”, “fear”, “sadness”, and “disgust”.

First, the facial expression estimation unit 6 detects facial feature points on the facial image data 72B, as described in FIGS. 3 and 4, to classify the facial organs and to classify each facial organ. Make a measurement. The facial expression estimation unit 6 compares the measurement result for the face image data 72B with the measurement result for the image data 70A, and calculates a score for each facial expression with reference to the tables shown in FIGS. 6 and 7.

6 and 7 are diagrams showing a point addition / deduction table used when calculating the score for each facial expression. The facial expression estimation unit 6 calculates a score for each facial expression based on how the measurement result for the image data 70A has changed based on the measurement result for the face image data 72B. For example, in the case of the "chin" organ, when the position of the center of gravity of the "chin" is lowered, the facial expression estimation unit 6 adds "+1" to the score of the "surprise" facial expression. Further, in the case of the "upper eyelid" organ, when the positions of both the left and right "upper eyelids" are raised, the facial expression estimation unit 6 adds "+1" to the points of "surprise" and "fear". The facial expression estimation unit 6 performs these additions and subtractions for each organ.

As a result of the addition and subtraction, the facial expression estimation unit 6 refers to the table of FIG. 8 and estimates the facial expression in the image data 70A according to the number of times the facial expression score is added or subtracted and the final value of the score. FIG. 8 is a diagram showing a table used when estimating facial expressions from the results of addition and subtraction of facial expression scores. The facial expression estimation unit 6 estimates the facial expression of the image data 70A from the number of additions and subtractions as a result of adding and subtracting the points of each facial expression and the points of the result of adding and subtracting using FIGS. 6 and 7. For example, the facial expression estimation unit 6 adds or subtracts the score of "happiness" twice or more, and when the final score of "happiness" is within the range of "-6 to +7", the face of the image data 70A. It is presumed that "happiness" appears in the expression of. The facial expression estimation unit 6 may estimate that a plurality of facial expressions appear on the face in one image data 70A depending on the facial expression score.

The definitions of addition and subtraction and each numerical value shown in FIGS. 6 to 8 are examples, and the present invention is not limited thereto.

The emotion determination unit 11 determines the emotion of the target person from the facial expression estimation result by the facial expression estimation unit 6. The emotions of the subject include positive emotions, which are "pleasant" emotions, negative emotions, which are "unpleasant" emotions, and disagreement emotions. The emotion determination unit 11 numerically represents the emotion determination result.

9 and 10 are diagrams showing a table used when quantifying the emotional determination result. When the facial expression estimation unit 6 determines that the facial expression of the image data 70A is "happiness, surprise, anger, fear, sadness", the emotion determination unit 11 uses FIG. 9 to make the target person feel comfortable or uncomfortable. Quantify the emotions of. Further, when the facial expression estimation unit 6 determines that the facial expression of the image data 70A is "disgust", the emotion determination unit 11 quantifies the disagreement emotion of the target person using FIG.

For example, when the facial expression estimation unit 6 estimates that the facial expression in the image data 70A is "happiness", the emotion determination unit 11 refers to FIG. 9, and the emotion of the target person is "pleasant". It is determined that there is, and the result is represented by a numerical value of "0.99081181". Further, when the facial expression estimation unit 6 estimates that the facial expressions in the image data 70A are "surprise" and "anger", the emotion determination unit 11 determines that the emotion of the target person is "pleasant". Determined to be both "unpleasant" feelings. Then, the emotion determination unit 11 expresses the feeling of "pleasant" by the numerical value of "0.3455514304" and the emotion of "discomfort" by the numerical value of "0.163369188".

Further, when the facial expression estimation unit 6 estimates that the facial expression of the image data 70A is "disgust", the emotion determination unit 11 determines that the emotion of the target person is a disagreement, and the determination result. Is expressed numerically. As a result of quantification, the closer it is to "1.0", the stronger the disagreement feeling, and the closer it is to "0", the stronger the non-disagreement feeling (agreement feeling).

The numerical values shown in FIGS. 9 and 10 are examples, and are not limited thereto.

FIG. 11 is a diagram showing a determination result by the emotion determination device 10. The judgment result includes the presence or absence (0 or 1) of six kinds of facial expressions estimated for one image data 70A, and the quantified feelings of "pleasant", "unpleasant", and "disagreement". Including. This determination result is transmitted from, for example, the emotion determination device 10 to the cloud via the network. And it is used for various services on the cloud side.

[Operation description]
Next, the operations of the facial expression estimation device 1 and the emotion determination device 10 in the present embodiment will be described with reference to FIG. FIG. 12 is a flow chart showing the operations of the facial expression estimation device 1 and the emotion determination device 10 in the present embodiment. In the following description, FIGS. 1 to 10 will be referred to as appropriate. Further, in the present embodiment, the facial expression estimation method is implemented by operating the facial expression estimation device 1. Therefore, the description of the facial expression estimation method in the present embodiment is replaced with the following description of the operation of the facial expression estimation device 1.

First, as a premise, the face of the target person is imaged by the imaging device 70. Under this premise, the image pickup data acquisition unit 2 acquires the image data 70A captured by the image pickup apparatus 70 (S1). Next, the face detection unit 7 detects the face in the image data 70A (S2). The face feature point detection unit 8 detects feature points of organs (for example, eyes, nose, mouth, etc.) that constitute the face of the target person detected by the face detection unit 7 in the image data 70A (S3). Then, the facial organ measuring unit 9 classifies the facial organs from the feature points detected by the facial feature point detecting unit 8 and measures each of the classified facial organs (S4).

The face image data acquisition unit 3 acquires the face image data 72A stored in the storage device 72 (S5), and the angle estimation unit 4 estimates the angle of the face of the target person in the image data 70A (S6). The correction unit 5 corrects the face image data 72A acquired by the face image data acquisition unit 3 according to the estimation result of the angle estimation unit 4, and obtains the face image data 72B that matches the face angle of the image data 70A. Generate (S7).

The facial expression estimation unit 6 compares the image data 70A acquired in S1 with the facial image data 72B corrected in S7, and sets the facial expression of the target person to 6 of "happiness, surprise, anger, fear, sadness, disgust". Estimated from the facial expressions on the street (S8). The emotion determination unit 11 determines the emotion of the target person from the estimation result of the facial expression in S8, and quantifies the determination result (S9).

[Effect of the embodiment]
When the facial expression of the target person is estimated by the facial expression estimation device 1 having the above configuration, the image data obtained by correcting the reference face image data 72A, which has less missing information on the face of the target person, is obtained. Since the facial expression is estimated as compared with 70A, the estimation accuracy of the facial expression is high. By increasing the estimation accuracy of the facial expression, the accuracy of the emotion judgment performed using the estimation result of the facial expression also increases.

Further, by determining the emotion of the target person by the emotion determination device 10, for example, when constructing a dialogue system by AI in the vehicle, it is possible to have a dialogue according to the emotion of the target person. As a result, it is possible to have a dialogue that is close to the feelings of the target person.

[program]
The program in this embodiment may be any program that causes a computer to execute steps S1 to S9 shown in FIG. By installing this program on a computer and executing it, the facial expression estimation device 1, the emotion determination device 10, and the facial expression estimation method according to the present embodiment can be realized. In this case, the computer processor functions as an imaging data acquisition unit 2, a face image data acquisition unit 3, an angle estimation unit 4, a correction unit 5, a facial expression estimation unit 6, and an emotion determination unit 11 to perform processing.

Further, the program in the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer functions as one of the imaging data acquisition unit 2, the face image data acquisition unit 3, the angle estimation unit 4, the correction unit 5, the facial expression estimation unit 6, and the emotion determination unit 11, respectively. May be good.

[Physical configuration]
Here, a computer that realizes the facial expression estimation device and the emotion determination device by executing the program in the present embodiment will be described with reference to FIG. FIG. 13 is a block diagram showing an example of a computer that realizes the facial expression estimation device 1 and the emotion determination device 10 in the present embodiment.

As shown in FIG. 13, the computer 110 includes a CPU 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. Each of these parts is connected to each other via a bus 121 so as to be capable of data communication. The computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to the CPU 111 or in place of the CPU 111.

The CPU 111 expands the programs (codes) of the present embodiment stored in the storage device 113 into the main memory 112 and executes them in a predetermined order to perform various operations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Further, the program in the present embodiment is provided in a state of being stored in a computer-readable recording medium 120. The program in this embodiment may be distributed on the Internet connected via the communication interface 117.

Further, specific examples of the storage device 113 include a semiconductor storage device such as a flash memory in addition to a hard disk drive. The input interface 114 mediates data transmission between the CPU 111 and an input device 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads a program from the recording medium 120, and writes a processing result in the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and another computer.

Specific examples of the recording medium 120 include a general-purpose semiconductor storage device such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), a magnetic recording medium such as a flexible disk, or a CD-. Examples include optical recording media such as ROM (Compact Disk Read Only Memory).

The facial expression estimation device and the emotion determination device in the present embodiment can also be realized by using the hardware corresponding to each part instead of the computer in which the program is installed. Further, the facial expression estimation device and the emotion determination device may be partially realized by a program and the rest by hardware.

A part or all of the above-described embodiments can be expressed by the following descriptions (Appendix 1) to (Appendix 12), but the present invention is not limited to the following description.

(Appendix 1)
An imaging data acquisition unit that acquires image data of the face of the target person captured by the imaging device, and
A face image data acquisition unit that acquires face image data of the target person in normal times, which is stored in advance in a storage device,
An angle estimation unit that estimates the face angle of the target person from the image data acquired by the imaging data acquisition unit, and an angle estimation unit.
A correction unit that corrects the face image data acquired by the face image data acquisition unit based on the estimation result of the angle estimation unit, and a correction unit.
A facial expression estimation unit that estimates the facial expression of the target person by comparing the image data acquired by the imaging data acquisition unit with the facial image data corrected by the correction unit.
A facial expression estimation device, which comprises.

(Appendix 2)
The facial expression estimation device described in Appendix 1
The correction unit
From the face image data acquired by the face image data acquisition unit, the feature points of the face of the target person are detected, the feature points are moved according to the angle estimated by the angle estimation unit, and the feature points are moved in the face image data. Matches the face angle of the image data acquired by the imaging data acquisition unit with the face angle of the image data.
A facial expression estimation device characterized by this.

(Appendix 3)
The facial expression estimation device according to Appendix 1 or Appendix 2.
The normal face image data of the target person is image data captured by the target person facing the surface of the imaging device.
A facial expression estimation device characterized by this.

(Appendix 4)
An imaging data acquisition unit that acquires image data of the face of the target person captured by the imaging device, and
A face image data acquisition unit that acquires face image data of the target person in normal times, which is stored in advance in a storage device,
An angle estimation unit that estimates the face angle of the target person from the image data acquired by the imaging data acquisition unit, and an angle estimation unit.
A correction unit that corrects the face image data acquired by the face image data acquisition unit based on the estimation result of the angle estimation unit, and a correction unit.
A facial expression estimation unit that estimates the facial expression of the target person by comparing the image data acquired by the imaging data acquisition unit with the facial image data corrected by the correction unit.
An emotion determination unit that determines the emotion of the target person based on the estimation result by the facial expression estimation unit, and
An emotional determination device comprising.

(Appendix 5)
The emotional determination device described in Appendix 4,
The correction unit
From the face image data acquired by the face image data acquisition unit, the feature points of the face of the target person are detected, the feature points are moved according to the angle estimated by the angle estimation unit, and the feature points are moved in the face image data. Matches the face angle of the image data acquired by the imaging data acquisition unit with the face angle of the image data.
A table emotion determination device characterized by this.

(Appendix 6)
The emotional determination device according to Appendix 4 or Appendix 5.
The normal face image data of the target person is image data captured by the target person facing the surface of the imaging device.
An emotional judgment device characterized by this.

(Appendix 7)
(A) A step of acquiring image data of the face of the target person captured by the imaging device, and
(B) A step of acquiring the normal face image data of the target person stored in the storage device in advance, and
(C) A step of estimating the face angle of the target person from the image data acquired in the step (a), and
(D) A step of correcting the face image data acquired in the step (b) based on the estimation result in the step (c), and
(E) A step of estimating the facial expression of the target person by comparing the image data acquired in the step (a) with the corrected face image data in the step (d).
A facial expression estimation method, which comprises.

(Appendix 8)
The facial expression estimation method described in Appendix 7
In step (d),
From the image data acquired in the step (a), the feature points of the face of the target person are detected, and the feature points are moved according to the angle estimated in the step (c) to move the face image data. Match the angle of the face inside with the angle of the face of the image data acquired in the step (a).
A facial expression estimation method characterized by this.

(Appendix 9)
The facial expression estimation method according to Appendix 7 or Appendix 8, wherein the facial expression is estimated.
The normal face image data of the target person is image data captured by the target person facing the surface of the imaging device.
A facial expression estimation method characterized by this.

(Appendix 10)
To a computer that acquires image data of the face of the target person captured by the image pickup device
(A) A step of acquiring the normal face image data of the target person stored in advance in the storage device, and
(B) A step of estimating the face angle of the target person from the image data, and
(C) A step of correcting the face image data acquired in the step (a) based on the estimation result in the step (b), and a step of correcting the face image data.
(D) A step of estimating the facial expression of the target person by comparing the image data with the face image data corrected in the step (c).
A program characterized by executing.

(Appendix 11)
The program described in Appendix 10
In step (c),
The feature points of the face of the target person are detected from the image data, the feature points are moved according to the angle estimated in the step (b), and the angle of the face in the face image data is determined by the image. Match the face angle of the data,
A program characterized by that.

(Appendix 12)
The program according to Appendix 10 or Appendix 11,
The normal face image data of the target person is image data captured by the target person facing the surface of the imaging device.
A program characterized by that.

As described above, according to the present invention, there is little loss of facial information of the target person, the reference facial image data is corrected, and the facial expression is estimated by comparing with the image data obtained by imaging. , The facial expression can be estimated accurately.

1: Facial expression estimation device 2: Imaging data acquisition unit 3: Face image data acquisition unit 4: Angle estimation unit 5: Correction unit 6: Facial expression estimation unit 7: Face detection unit 8: Face feature point detection unit 9: Face organ measurement unit 10: Emotion determination device 11: Emotion determination unit 70: Imaging device 70A: Image data 71: Authentication pattern 72: Storage device 72A: Facial expression data

Claims (12)

An imaging data acquisition unit that acquires image data of the face of the target person captured by the imaging device, and
A face image data acquisition unit that acquires face image data of the target person in normal times, which is stored in advance in a storage device,
An angle estimation unit that estimates the face angle of the target person from the image data acquired by the imaging data acquisition unit, and an angle estimation unit.
A correction unit that corrects the face image data acquired by the face image data acquisition unit based on the estimation result of the angle estimation unit, and a correction unit.
A facial expression estimation unit that estimates the facial expression of the target person by comparing the image data acquired by the imaging data acquisition unit with the facial image data corrected by the correction unit.
A facial expression estimation device, which comprises. The facial expression estimation device according to claim 1.
The correction unit
From the face image data acquired by the face image data acquisition unit, the feature points of the face of the target person are detected, the feature points are moved according to the angle estimated by the angle estimation unit, and the feature points are moved in the face image data. Matches the face angle of the image data acquired by the imaging data acquisition unit with the face angle of the image data.
A facial expression estimation device characterized by this. The facial expression estimation device according to claim 1 or 2.
The normal face image data of the target person is image data captured by the target person facing the surface of the imaging device.
A facial expression estimation device characterized by this. An imaging data acquisition unit that acquires image data of the face of the target person captured by the imaging device, and
A face image data acquisition unit that acquires face image data of the target person in normal times, which is stored in advance in a storage device,
An angle estimation unit that estimates the face angle of the target person from the image data acquired by the imaging data acquisition unit, and an angle estimation unit.
A correction unit that corrects the face image data acquired by the face image data acquisition unit based on the estimation result of the angle estimation unit, and a correction unit.
A facial expression estimation unit that estimates the facial expression of the target person by comparing the image data acquired by the imaging data acquisition unit with the facial image data corrected by the correction unit.
An emotion determination unit that determines the emotion of the target person based on the estimation result by the facial expression estimation unit, and
An emotional determination device comprising. The emotion determination device according to claim 4.
The correction unit
From the face image data acquired by the face image data acquisition unit, the feature points of the face of the target person are detected, the feature points are moved according to the angle estimated by the angle estimation unit, and the feature points are moved in the face image data. Matches the face angle of the image data acquired by the imaging data acquisition unit with the face angle of the image data.
An emotional judgment device characterized by this. The emotion determination device according to claim 4 or 5.
The normal face image data of the target person is image data captured by the target person facing the surface of the imaging device.
An emotional judgment device characterized by this. (A) A step of acquiring image data of the face of the target person captured by the imaging device, and
(B) A step of acquiring the normal face image data of the target person stored in the storage device in advance, and
(C) A step of estimating the face angle of the target person from the image data acquired in the step (a), and
(D) A step of correcting the face image data acquired in the step (b) based on the estimation result in the step (c), and
(E) A step of estimating the facial expression of the target person by comparing the image data acquired in the step (a) with the corrected face image data in the step (d).
A facial expression estimation method, which comprises. The facial expression estimation method according to claim 7.
In step (d),
From the image data acquired in the step (a), the feature points of the face of the target person are detected, and the feature points are moved according to the angle estimated in the step (c) to move the face image data. Match the angle of the face inside with the angle of the face of the image data acquired in the step (a).
A facial expression estimation method characterized by this. The facial expression estimation method according to claim 7 or 8.
The normal face image data of the target person is image data captured by the target person facing the surface of the imaging device.
A facial expression estimation method characterized by this. To a computer that acquires image data of the face of the target person captured by the image pickup device
(A) A step of acquiring the normal face image data of the target person stored in advance in the storage device, and
(B) A step of estimating the face angle of the target person from the image data, and
(C) A step of correcting the face image data acquired in the step (a) based on the estimation result in the step (b), and a step of correcting the face image data.
(D) A step of estimating the facial expression of the target person by comparing the image data with the face image data corrected in the step (c).
A program characterized by executing. The program according to claim 10.
In step (c),
The feature points of the face of the target person are detected from the image data, the feature points are moved according to the angle estimated in the step (b), and the angle of the face in the face image data is determined by the image. Match the face angle of the data,
A program characterized by that. The program according to claim 10 or 11.
The normal face image data of the target person is image data captured by the target person facing the surface of the imaging device.
A program characterized by that.

Images