WO2023112606A1

WO2023112606A1 - Face direction determination system

Info

Publication number: WO2023112606A1
Application number: PCT/JP2022/042961
Authority: WO
Inventors: 俊剛関
Original assignee: 矢崎総業株式会社
Priority date: 2021-12-14
Filing date: 2022-11-21
Publication date: 2023-06-22
Also published as: JP7418386B2; JP2023087986A

Abstract

A face direction detection system (1) includes a face video acquisition camera (10), a face feature point detection unit (20), and a face direction detection unit (33). The face video acquisition camera (10) captures an image of the face of a person. The face feature point detection unit (20) detects positions of corners of eyes of the person on the basis of the image of the face of the person captured by the face video acquisition camera (10). The face direction detection unit (33) detects a direction of the face of the person on the basis of the positions of the corners of the eyes detected by the face feature point detection unit (20) and a position of a head axis serving as a reference when the head of the person turns. As a result, the face direction detection system (1) can appropriately determine the direction of the face of a driver.

Description

facial recognition system

The present invention relates to a face orientation determination system.

Conventionally, as a face orientation determination system, for example, Patent Document 1 discloses a driver's face by detecting at least one three-dimensional position of the driver's eyebrows, the corners of the eyes, and the mouth as feature points of the driver's face. A driver monitoring device for estimating orientation is described.

JP 2011-154721 A

By the way, the driver monitoring device described in Patent Literature 1 may not be able to estimate the driver's face orientation if, for example, only part of the facial feature points can be detected when the driver looks aside.

Therefore, the present invention has been made in view of the above, and it is an object of the present invention to provide a face orientation determination system that can appropriately determine the orientation of a person's face.

In order to solve the above-described problems and achieve the object, a face orientation detection system according to the present invention includes an image capturing unit that captures an image of a person's face, and an image of the person's face captured by the image capturing unit. an eye corner detection unit for detecting the position of the corner of the eye of the person; the position of the corner of the eye detected by the corner of the eye detection unit; and a face orientation detection unit that detects the orientation of the person's face based on the above.

The face orientation detection system according to the present invention can determine the orientation of a person's face even when only one corner of the eye can be detected when the person looks aside, and as a result, the orientation of the person's face can be properly determined. can be done.

FIG. 1 is a block diagram showing a configuration example of a face orientation detection system according to an embodiment. FIG. 2 is a diagram showing the range of the head axis according to the embodiment. FIG. 3 is a diagram showing a calculation example of the position of the head axis according to the embodiment. FIG. 4 is a diagram illustrating an example of calculation of the face orientation angle in the vertical direction according to the embodiment. FIG. 5 is a diagram illustrating an example of calculation of the face orientation angle in the vertical direction according to the embodiment. FIG. 6 is a diagram illustrating a calculation example of the face direction angle in the horizontal direction according to the embodiment. FIG. 7 is a diagram illustrating a calculation example of the face direction angle in the horizontal direction according to the embodiment. FIG. 8 is a diagram illustrating an example of determination of left and right facing directions according to the embodiment. FIG. 9 is a diagram showing the positional relationship between the facial image capturing camera and the driver according to the embodiment. FIG. 10 is a diagram illustrating an update example (increase) of parameters according to the embodiment. FIG. 11 is a diagram illustrating an example of parameter update (decrease) according to the embodiment. FIG. 12 is a diagram showing the installation angles of the facial image acquisition camera in the horizontal direction according to the embodiment. FIG. 13 is a diagram showing installation angles of the face image capturing camera in the vertical direction according to the embodiment. FIG. 14 is a diagram illustrating average errors of face direction detection results (angles) according to the embodiment. FIG. 15 is a flowchart showing inattentiveness detection processing according to the embodiment. FIG. 16 is a flowchart showing face orientation detection processing according to the embodiment. FIG. 17 is a flow chart showing processing for calculating the position of the head axis according to the embodiment. FIG. 18 is a flowchart showing parameter update processing according to the embodiment. FIG. 19 is a flowchart showing processing for calculating the position of the head axis according to the first modified example of the embodiment. FIG. 20 is a flow chart showing calculation processing of the position of the head axis according to the second modification of the embodiment. FIG. 21 is a flow chart showing processing for calculating the position of the head axis according to the third modification of the embodiment.

The form (embodiment) for carrying out the present invention will be described in detail with reference to the drawings. The present invention is not limited by the contents described in the following embodiments. In addition, the components described below include those that can be easily assumed by those skilled in the art and those that are substantially the same. Furthermore, the configurations described below can be combined as appropriate. In addition, various omissions, substitutions, or changes in configuration can be made without departing from the gist of the present invention.

[Embodiment]
A face orientation detection system 1 according to an embodiment will be described with reference to the drawings. The face orientation detection system 1 detects the orientation of a person's face. For example, it is installed in a vehicle and detects the orientation of the face of a driver sitting in the driver's seat of the vehicle. The face orientation detection system 1 includes, for example, a face image acquisition camera 10 as an imaging section, a facial feature point detection unit 20 as an eye corner detection section, and a face orientation calculation unit 30, as shown in FIG.

Here, when the driver faces the front, the direction along the left and right corners of the eyes E of the driver is referred to as the left-right direction X, the direction along the up-down direction is referred to as the vertical direction or the up-down direction Y, and the driver faces the front. A direction along the line of sight of the driver is referred to as a depth direction Z. The horizontal direction X, the vertical direction Y, and the depth direction Z intersect each other, and are typically orthogonal.

The face image capturing camera 10 captures the face of the driver. For example, it is mounted on a vehicle and captures the face of the driver seated in the driver's seat of the vehicle. The face image acquisition camera 10 is arranged in front of the driver, for example, in front of the dashboard or the roof of the vehicle, and is provided so that the center of the eyelid of the driver becomes the center of the imaging range. The driver's face is imaged from the The facial image acquisition camera 10 is connected to the facial feature point detection unit 20 and outputs image (moving (video) or still image) data representing the captured driver's face to the facial feature point detection unit 20 .

The facial feature point detection unit 20 detects facial feature points from image data. The facial feature point detection unit 20 detects facial feature points from image data using, for example, known image recognition technology (machine learning) such as Open CV. The facial feature point detection unit 20 detects, for example, the left and right corners of the eyes E, the left and right pupils, etc. as facial feature points from the image data, and detects the positions of these facial parts. Here, the facial feature point detection unit 20 treats information on the position of the detected facial part as coordinate information. The facial feature point detection unit 20 is connected to the face direction calculation unit 30 and outputs information representing the detected positions of the left and right corners of the eye E (information representing the position coordinates of the left and right corners of the eye E) to the face direction calculation unit 30 .

The face orientation calculation unit 30 calculates the orientation of the driver's face, and includes an electronic circuit mainly composed of a well-known microcomputer including a CPU, a ROM that constitutes a memory, a RAM, and an interface. The face orientation calculation unit 30 includes an eye corner distance calculation unit 31, a storage unit 32, a face orientation detection unit 33, and a parameter update unit 34. deal.

The eye corner distance calculation unit 31 calculates the distance d ₀ (see FIG. 3) between the left and right corners of the eye E when the person is facing forward. The outer-eye corner distance calculator 31 is connected to the facial feature point detection unit 20, and converts information representing the positions of the left and right outer corners of the eye E (information representing the position coordinates of the left and right outer corners of the eye E) output from the facial feature point detection unit 20 into Based on this, the distance _d0 between the left and right corners of the eye E (the left corner of the eye LE and the right corner of the eye RE) is calculated, and the center position of the left and right corners of the eye E is calculated. For example, as shown in FIG. 3, the outer corner of eye distance calculation unit 31 calculates the distance d ₀ between the position coordinates (x _LE0 , y _LE0 ) of the left corner of the eye LE and the position coordinates (x _RE0 , y _RE0 ) of the right corner of the eye RE. At the same time, the _central position coordinates (x C0 , y C0 ) between the position coordinates (x _LE0 , y _LE0 ₎ of the left corner of the eye LE and the position coordinates (x _RE0 , y _RE0 ) of the right corner of the eye RE are calculated. The eye corner distance calculator 31 is connected to the face orientation detector 33 and outputs the calculated distance d ₀ and the center position coordinates (x _C0 , y _C0 ) to the face orientation detector 33 .

The storage unit 32 stores information, and includes, for example, volatile memory such as RAM, and nonvolatile memory such as ROM and flash memory. The storage unit 32 stores, for example, conditions and information necessary for various processes in the face orientation calculation unit 30, various programs and applications to be executed in the face orientation calculation unit 30, control data, and the like. The storage unit 32 stores predetermined standard human body dimensions based on a human body dimension database that statistically represents human body dimensions. The storage unit 32 stores, for example, standard human body dimensions as shown in FIG. w, the distance h in the vertical direction Y between the central positions of the left and right corners of the eyes E and the axis G, etc. are stored. That is, the storage unit 32 stores, as standard human body dimensions, a standard distance d between the left and right corners of the eye E, a standard distance w between the center positions of the left and right corners of the eyes E and the axis G, and a standard distance w between the left and right corners of the eyes E. and the position of the axis G (standard head axis P) determined from the standard distance h between the axis G and the axis G.

The face direction detection unit 33 detects the face direction of the driver. The face orientation detection unit 33 detects the orientation of the driver's face, for example, based on the position of the outer corner of the eye E detected by the facial feature point detection unit 20 and the position of the head axis P as a head rotation reference. . Here, the head axis P is a reference portion for turning the driver's head, and is set between the axis G and the ischium J as shown in FIG. In other words, the driver's head pivots about the head axis P. The face orientation detection unit 33 performs personalization processing for adjusting the standard head axis P stored in the storage unit 32 to match the individual. Specifically, the face orientation detection unit 33 detects the position of the predetermined standard head axis P from the face image of the driver captured by the face image acquisition camera 10 with the driver facing forward. The orientation of the driver's face is detected based on the corrected position of the head axis P corrected according to the obtained distance _d0 between the left and right corners of the eyes E of the driver. The face orientation detection unit 33 uses the following equation (1), for example, to determine the position of the head from the position coordinates (x _LE0 , y _LE0 ) of the left corner of the eye LE and the position coordinates (x _RE0 , y _RE0 ) of the right corner of the eye RE. Obtain the X coordinate x _A (x _c0 ) of the axis P. The face orientation detection unit 33 also obtains the Y coordinate y _A of the head axis P using the following equations (2) to (4). For example, the face direction detection unit 33 uses Equation (2) to determine the standard distance d between the left and right corners of the eye E, the position of the corners of the eyes E (for example, the center position of the left and right corners of the eyes E), and the standard head axis P The parameter Ph is obtained from the standard distance _h in the vertical direction Y between and, using equation (3), the corrected distance h ₀ is obtained from the distance d ₀ between the left and right corners of the eye E and the parameter _Ph , and the equation (4 ), the corrected Y coordinate of the head axis P is calculated from the position coordinates (x _LE0 , y _LE0 ) of the left corner of the eye LE, the position coordinates (x _RE0 , y _RE0 ) of the right corner of the eye RE, and the corrected distance h ₀ . Find _yA . That is, the face orientation detection unit 33 personalizes the Y coordinate _yA of the head axis P using the distance _d0 between the left and right corners of the eyes E. In addition, the face orientation detection unit 33 obtains the distance _w0 in the depth direction Z between the central position of the left and right corners of the eyes E and the head axis P using the following equations (5) and (6). The face direction detection unit 33 uses, for example, Equation (5) to determine the standard distance d between the left and right corners of the eyes E, and the standard distance in the depth direction Z between the central position of the left and right corners of the eyes E and the standard head axis P. A parameter _Pw is obtained from , and the corrected distance _w0 is obtained from the distance _d0 between the left and right corners of the eye E and the parameter _Pw using equation (6). That is, the face direction detection unit 33 personalizes the Z coordinate of the head axis P using the distance _d0 between the left and right corners of the eyes E.

The face direction detection unit 33 detects the face direction of the driver based on the personalized head axis P, that is, the head axis P after correction. For example, as shown in FIGS. 4 and 5, the face orientation detection unit 33 detects the distance _h1 in the vertical direction Y between the position coordinates (y _RE1 , z _RE1 ) of the corner of the eye E and the position coordinates of the head axis P. . Based on the detected distance _h1 , the face direction detection unit 33 uses the following equations (7) to (18) to obtain the angle of the face in the vertical direction Y. In the following formulas (7) to (18), "k" represents the hypotenuse of a right-angled triangle, "θ _P0 " represents the angle between the outer corner of the eye E and the head axis P of the face when facing forward, “θ _P1 ” represents the angle of the vertical direction Y of the face, and “θ _P2 ” represents the angle between the depth direction Z and the oblique side k. In this example, the position coordinates of the head axis P are the origin (0, 0). The face orientation detection unit 33 obtains the angle θ _P2 using Equation (7). At this time, the angle θ _P2 is represented by Equation (8) using a trigonometric function. The face orientation detection unit 33 combines equations (7) and (8) to obtain equation (9). Equation (18) can be obtained, and as a result, the angle θ _P1 of the vertical direction Y of the face can be obtained.

The face orientation detection unit 33 detects the position coordinates (x _RE1 , z _RE1 ) of the right corner of the eye RE, as shown in FIGS. 6 and 7, for example. Based on the detected position coordinates (x _RE1 , z _RE1 ) of the right corner of the eye RE, the face orientation detection unit 33 obtains the angle in the horizontal direction X of the face using the following equations (19) to (28). In the following equations (19) to (28), “θ _Y1 ” represents the angle of the left-right direction X of the face, and the range is 0°≦θ _Y1 ≦90°, and “x _c1 ” represents the left-right angle. The center position coordinate (X coordinate) of the outer corner of the eye E is represented, and “d _E ” represents the displacement of the outer corner of the eye E. In this example, the position coordinates of the head axis P are the origin (0, 0). The face orientation detection unit 33 obtains the angle θ _Y1 using Equation (19). At this time, “x _c1 ” in Equation (19) is represented by Equation (20), and Equation (21) is obtained by substituting Equation (20) into Equation (19). In equation (21), the only unknown variable is 'dx'. Equation (21) can be transformed into equation (22) using trigonometric functions, and equation (22) can be transformed into equations (23) and (24). Then, by replacing variables as shown in Equation (25), Equation (26) can be derived, and Equations (27) and (28) can be derived from Equation (26). The face direction detection unit 33 can calculate “dx” from equation (28), and by substituting the calculated “dx” into equation (21), the angle θ _Y1 of the horizontal direction X of the face can be obtained. .

Note that in equation (20), if "dx≧0" and the face is turned to the right, then "x _c1 =x _LE1 +dx".

As shown in FIG. 8, the face direction detection unit 33 detects that the position coordinates (x _RE1 , z _RE1 ) of the right corner of the eye RE are higher than the position coordinates (x _RE0 , z _RE0 ) of the right corner of the eye RE when facing forward. If it is larger, it is determined that the face is facing left. On the other hand, if the position coordinates (x _RE1 , z _RE1 ) of the right corner of the eye RE are smaller than the position coordinates (x _RE0 , z _RE0 ) of the right corner of the eye RE when the face is facing forward, the face orientation detection unit 33 is directed to the right.

Next, the parameter updating unit 34 will be explained. The parameter update unit 34 updates the above-described parameters P _h and P _w according to the relative position (positional deviation) between the position of the face image acquisition camera 10 and the position of the driver, so that the person is positioned relative to the front position. The position of the standard head axis P is corrected in accordance with the relative position of the facial image acquisition camera 10 . Here, due to the structure of the vehicle, the facial image acquisition camera 10 is often arranged at a position shifted from the front of the driver, as shown in FIG. 9, for example. In this case, the face image capturing camera 10 captures an image of the driver's face that is shifted by an angle θa from the front direction M when the driver looks straight ahead. Therefore, the parameter updating unit 34 subtracts the angle θa corresponding to the deviation amount of the face image acquisition camera 10 from the angle of the face detected by the face orientation detecting unit 33 . As a result, theoretically, the angle θa corresponding to the amount of deviation of the facial image capturing camera 10 can be offset. The angle θa corresponding to the deviation amount of the face image acquisition camera 10 is subtracted from the angle of the face detected by the face orientation detection unit 33. may not be zero. Therefore, the parameter updating unit 34 performs processing to bring the absolute value of the value obtained by subtracting the angle θa corresponding to the deviation amount of the face image capturing camera 10 from the angle of the face detected by the face direction detecting unit 33 closer to zero.

For example, as shown in FIG. 10, the parameter update unit 34 increases the parameters P _h and P _w described above, that is, the parameters D1 to D4 step by step. At this time, the face orientation detection unit 33 detects the angle of the face based on the parameter P _h and the parameter P _w that are increased stepwise. As shown in FIG. 10, the parameter update unit 34 acquires a face image from the angle of the face detected by the face orientation detection unit 33 as the parameters P _h and P _w , that is, the parameters D1 to D4 are increased in order. The absolute value of the value obtained by subtracting the angle θa corresponding to the deviation amount of the camera 10 gradually decreases, and the parameter _D3 is the smallest among the parameters D1 to D4. P _w ).

The parameter updating unit 34, as shown in FIG. 11, decreases the parameters P _h and P _w described above, that is, the parameters D1 to D4 step by step. At this time, the face orientation detection unit 33 detects the angle of the face based on the parameter P _h and the parameter P _w that are increased stepwise. As shown in FIG. 11, the parameter update unit 34 acquires a face image from the angle of the face detected by the face orientation detection unit 33 as the parameters P _h and P _w , that is, the parameters D1 to D4 are sequentially decreased. The absolute value of the value obtained by subtracting the angle θa corresponding to the deviation amount of the camera 10 gradually decreases, and the parameter _D3 is the smallest among the parameters D1 to D4. P _w ). The parameter updating unit 34 updates the parameters P _h and P _w so that, for example, as shown in FIG. was "19.6" in the previous model, but after the improvement it was reduced to "4.5". It used to be "7.9", but after the improvement, it has decreased to "3.8". The angle θa (see FIG. 9) corresponding to the amount of deviation of the facial image acquisition camera 10 is the amount of deviation in the horizontal direction X. Usually, the amount of deviation of the facial image acquisition camera 10 is the horizontal direction The deviation amount of the angle θ _CYaw of X and the angle θ _CPitch in the vertical direction Y shown in FIG. 13 is included. When obtaining the angle of the face in the vertical direction Y, the parameter updating unit 34 subtracts the deviation amount of the angle _θCPitch in the vertical direction Y, and when obtaining the angle of the face in the horizontal direction X, the parameter updating unit 34 subtracts Subtract the angle θ _CYaw .

Next, inattentive looking detection processing will be described. FIG. 15 is a flowchart showing inattentiveness detection processing according to the embodiment. The face direction detection system 1 includes a line-of-sight detection unit 40 that detects the driver's line of sight. The face image capturing camera 10 captures an image of the driver's face from the front of the vehicle and outputs image data representing the captured face of the driver to the face feature point detection unit 20 . The facial feature point detection unit 20 detects a face from image data (step S1), detects the positions of the left and right corners of the eyes E, the left and right pupils, etc. as feature points of the detected face, and represents the positions of the left and right corners of the eyes E. Information (information representing the positional coordinates of the left and right corners of the eyes E) is output to the face direction calculation unit 30, and information representing the positions of the left and right pupils is output to the line of sight detection section 40 (step S2). The face orientation calculation unit 30 detects the orientation of the face based on the information representing the positions of the left and right corners of the eyes E output from the facial feature point detection unit 20 (step S3; detection processing shown in the flowchart of FIG. 16 to be described later). . The line-of-sight detection unit 40 detects the direction of the line of sight based on the information representing the positions of the left and right pupils output from the facial feature point detection unit 20 (step S4). The inattentive-looking determination processing unit (not shown) determines whether the driver is looking aside based on the face direction detected by the face direction calculation unit 30 and the line-of-sight direction detected by the line-of-sight detection unit 40 (step S5). . If it is determined that the driver is looking aside, the inattentive determination processing unit notifies that the driver is looking aside by voice or the like (step S6). Do not notify.

Next, face orientation detection processing will be described. FIG. 16 is a flowchart showing face orientation detection processing according to the embodiment. In the face orientation detection system 1, the eye corner distance calculation unit 31 calculates the left and right corners of the eye based on the information representing the positions of the left and right corners of the eye E (information representing the position coordinates of the left and right corners of the eye E) output from the facial feature point detection unit 20. The distance d ₀ of the outer corner of the eye E is calculated, and the position coordinates of the center of the left and right corners of the eye E are calculated (step T1). (Step T2). The face orientation detection unit 33 acquires the X coordinate of the head axis P in step T2. In addition, the face orientation detection unit 33 uses the above equation (2) to determine the standard distance d between the left and right corners of the eyes E, and the vertical direction Y between the center position of the left and right corners of the eyes E and the standard head axis P. The initial parameter P _h is obtained from the standard distance h, and using equation (5), the standard distance d between the left and right outer corners of the eye E, and the depth direction between the central position of the left and right outer corners of the eye E and the standard head axis P An initial parameter _Pw is obtained from the standard distance w of Z (step T3). Then, the face direction detection unit 33 uses equation (3) to obtain the corrected distance h ₀ from the distance d ₀ between the left and right corners of the eyes E and the parameter P _h , and uses equation (6) to obtain the corrected distance h 0 . A corrected distance w ₀ is obtained from the distance d ₀ of the outer corner of the eye E and the parameter P _w , and the vertical direction Y of the face is calculated based on the corrected head axis P determined based on the initial distance h ₀ and the distance w ₀ . angle (formulas (7) to (18)) and the angle of the horizontal direction X of the face (formulas (19) to (28)) are obtained (step T4). The parameter update unit 34 acquires predetermined angles (angle θ _CPitch , angle θ _CYaw ) of the face image acquisition camera 10 (step T5), and the initial face angle obtained by the face direction detection unit 33 is output. (step T6). Then, the parameter update unit 34 obtains the absolute value of a value obtained by subtracting the angle (angle θ _CPitch , angle θ _CYaw ) of the face image acquisition camera 10 from the initial face angle obtained by the face direction detection unit 33, and obtains the initial face angle. parameter P _h and parameter P _w are increased or decreased and updated (steps T7 and T9). The face direction detection unit 33 uses equation (3) to obtain the corrected distance h 0 from the distance d ₀ between the left and right corners of the eyes E and the updated parameter P _h , and uses equation (6) to obtain the corrected distance h ₀ . The corrected distance w _{0 is obtained from the distance d 0} _of the outer corner of the eye E and the updated parameter P _w , and based on the corrected head axis P determined based on the updated distance h ₀ and distance w ₀ , The angle of the vertical direction Y of the face (formulas (7) to (18)) and the angle of the horizontal direction X of the face (formulas (19) to (28)) are obtained (step T10), and the orientation of the face is output. (Step T11).

Next, calculation processing for the position of the head axis P will be described. FIG. 17 is a flow chart showing calculation processing of the position of the head axis P according to the embodiment. The facial image acquisition camera 10 images the driver's face while the driver's face is facing forward (steps U1 and U2). The facial feature point detection unit 20 detects a face from the image data captured by the facial image acquisition camera 10 (step U3), and detects the left and right corners of the eyes E, the left and right pupils, etc. as feature points of the face. The position of the part of the face is detected (step U4). Based on information representing the positions of the left and right corners of the eye E (information representing the position coordinates of the left and right corners of the eye E) output from the facial feature point detection unit 20, the outer corner of the eye distance calculator 31 calculates the distance d ₀ between the left and right corners of the eye E. is calculated, and the central positions of the left and right corners of the eyes E are calculated (step U5). The face orientation detection unit 33 detects the position of the predetermined standard head axis P from the left and right sides of the driver obtained from the image of the driver's face taken by the face image acquisition camera 10 with the driver facing the front. The corrected position of the head axis P is calculated according to the distance _d0 of the outer corner of the eye E (step U6).

Next, parameter update processing will be described. FIG. 18 is a flowchart showing parameter update processing according to the embodiment. The face orientation detection unit 33 inputs the initial parameters P _h and P _w (step L1). Then, the face direction detection unit 33 obtains the corrected distance w0 from the distance _d0 between the left and right corners of the eye E and the parameter _Pw , and the corrected distance _w0 from the distance _d0 between the left and right corners of the eye E and the parameter _Ph . _h0 is obtained, and the initial face angle of the driver is calculated based on the corrected head axis P (step L2). The parameter update unit 34 updates the initial difference between the initial face angle calculated by the face direction detection unit 33 and the angle θa (angle θ _CPitch , angle θ _CYaw ) corresponding to the deviation amount of the face image capturing camera 10. Calculate (step L3). The parameter update unit 34 increases the parameter P _h and the parameter P _w (step L4), and the face orientation detection unit 33 calculates the corrected distance w from the distance d ₀ between the left and right corners of the eyes E and the increased parameter P _w ₀ is obtained, the corrected distance h ₀ is obtained from the distance d ₀ between the left and right corners of the eyes E and the increased parameter P _h , and the angle of the driver's face is calculated based on the corrected head axis P ( step L5). Again, the parameter update unit 34 calculates the difference between the face angle calculated by the face direction detection unit 33 and the angle θa (angle θ _CPitch , angle θ _CYaw ) corresponding to the deviation amount of the face image acquisition camera 10. (Step L6). If the calculated difference is smaller than the initial value (initial difference) (step L7; Yes), the parameter updating unit 34 increases the parameters P _h and P _w (step L8). The face direction detection unit 33 obtains the corrected distance w0 from the distance _d0 between the left and right corners of the eyes E and the increased parameter _Pw , and calculates the corrected distance _w0 from the distance _d0 between the left and right corners of the eyes E and the increased parameter _Ph . The corrected distance _h0 is obtained, and the angle of the driver's face is calculated based on the corrected head axis P (step L9). The parameter update unit 34 calculates the difference between the angle of the face calculated by the face direction detection unit 33 and the angle θa (angle θ _CPitch , angle θ _CYaw ) corresponding to the deviation amount of the face image acquisition camera 10 (step L10). If the calculated difference is larger than the previous difference (step L11; Yes), the parameter updating unit 34 sets the previous parameters P _h and P _w as updated values (step L12), and terminates the process. On the other hand, if the calculated difference is smaller than the previous difference (step L11; No), the parameter updating unit 34 returns to step L8 and increases the parameter P _h and parameter P _w .

In step L7, when the calculated difference is larger than the initial value (initial difference) (step L7; No), the parameter updating unit 34 reduces the parameters P _h and P _w (step L13). The face orientation detection unit 33 obtains the corrected distance w0 from the distance _d0 between the left and right corners of the eyes E and the reduced parameter _Pw , and calculates the corrected distance _w0 from the distance _d0 between the left and right corners of the eyes E and the reduced parameter _Ph . The corrected distance _h0 is obtained, and the angle of the driver's face is calculated based on the corrected head axis P (step L14). The parameter update unit 34 calculates the difference between the angle of the face calculated by the face direction detection unit 33 and the angle θa (angle θ _CPitch , angle θ _CYaw ) corresponding to the deviation amount of the face image acquisition camera 10 (step L15). If the calculated difference is greater than the previous difference (step L16; Yes), the parameter updating unit 34 sets the previous parameters P _h and P _w as update values (step L17), and terminates the process. On the other hand, if the calculated difference is smaller than the previous difference (step L16; No), the parameter updating unit 34 returns to step L13 and decreases the parameter P _h and parameter P _w .

As described above, the face orientation detection system 1 according to the embodiment includes the face image acquisition camera 10, the facial feature point detection unit 20, and the face orientation detection section 33. A face image capturing camera 10 captures an image of a person's face. The facial feature point detection unit 20 detects the positions of the corners of the eyes E of the person based on the image of the person's face taken by the face image acquisition camera 10 . The face orientation detection unit 33 detects the orientation of the person's face based on the position of the corner of the eye E detected by the facial feature point detection unit 20 and the position of the head axis P that serves as a reference when turning the person's head. to detect

With this configuration, the face orientation detection system 1 can determine the orientation of the person's face even if only one corner of the eye can be detected when the person looks aside, or even if the face is partially covered by an object such as a mask. can do. In addition, the face direction detection system 1 does not detect the face direction using a three-dimensional rotation matrix or the like using a large number of facial feature points as in the conventional art, but detects the face direction based on two-dimensional coordinate data. Since the orientation is detected, an increase in the amount of calculation can be suppressed, and the processing load can be reduced. As a result, the face orientation determination system can properly detect the orientation of the person's face.

In the face orientation detection system 1, the face orientation detection unit 33 detects the orientation of the person's face in the vertical direction Y based on the distance _h1 between the position of the head axis P and the position of the corner of the eye E in the vertical direction Y. do. With this configuration, the face direction detection system 1 can appropriately detect the face direction of the person in the vertical direction Y while suppressing an increase in the amount of calculation.

In the face orientation detection system 1, the face orientation detection unit 33 detects the position of the head axis P and the position of the outer corner of the eye E (for example, the position coordinates of the right outer corner of the eye RE (x _RE1 , z _RE1 )), the orientation of the person's face in the horizontal direction X is detected. With this configuration, the face orientation detection system 1 can appropriately detect the orientation of the person's face in the left-right direction X while suppressing an increase in the amount of calculation.

In the face orientation detection system 1, the face orientation detection unit 33 captures a predetermined standard position of the head axis P as the position of the head axis P with the face image acquisition camera 10 while the person faces the front. The orientation of the person's face is detected based on the position of the corrected head axis P corrected according to the distance _d0 between the left and right corners of the eyes E of the person obtained from the image of the person's face. With this configuration, the face direction detection system 1 can correct the head axis P according to individual differences of the person, so that the detection accuracy of the person's face direction can be improved.

In the face orientation detection system 1, the face orientation detection unit 33 determines the distance h between the position of the head axis P and the position of the outer corner of the eye E in the vertical direction Y, according to the distance _d0 between the left and right corners of the eye E of the person. Then, based on the corrected position of the head axis P obtained by correcting the distance w between the position of the head axis P in the depth direction Z intersecting the vertical direction Y and the central positions of the left and right corners of the eyes E, the human face is determined. to detect the orientation of With this configuration, the face direction detection system 1 can correct the head axis P along the vertical direction Y and the depth direction Z according to individual differences of the person, so that the detection accuracy of the person's face direction is improved. be able to.

In the face orientation detection system 1, the face orientation detection unit 33 sets a predetermined standard position of the head axis P as the position of the head axis P according to the relative position of the face image acquisition camera 10 with respect to the front position of the person. The orientation of the person's face is detected based on the corrected position of the head axis P corrected by . With this configuration, the face orientation detection system 1 can correct an error in the installation accuracy of the face image acquisition camera 10, etc., so that it is possible to suppress a decrease in accuracy in detecting the face orientation of a person.

In the face direction detection system 1, the distance h between the position of the head axis P and the position of the corner of the eye E in the vertical direction Y and the distance h A parameter updating unit 34 is further provided for correcting the distance w between the position of the head axis P in the depth direction Z intersecting Y and the central positions of the left and right corners of the eyes E. With this configuration, the face direction detection system 1 can correct an error in mounting accuracy of the face image acquisition camera 10 in the vertical direction Y and the depth direction Z, thereby suppressing a decrease in accuracy in detecting the face direction of a person. can do.

[Modification]
Next, modifications of the embodiment will be described. In addition, in the modified example, the same reference numerals are given to the same constituent elements as in the embodiment, and detailed description thereof will be omitted. FIG. 19 is a flow chart showing the calculation processing of the position of the head axis P according to the first modified example of the embodiment. The calculation processing of the position of the head axis P according to the first modification differs from the calculation processing of the position of the head axis P according to the embodiment in that unique information is stored by face authentication. In FIG. 19, the face image capturing camera 10 captures an image of the driver's face while the driver's face is facing forward (steps V1 and V2). The face feature point detection unit 20 detects a face from the image data captured by the face image acquisition camera 10 (step V3), detects the left and right corners of the eyes E, the left and right pupils, etc. as feature points of the face, and detects these features. The positions of facial parts are detected (step V4). Based on information representing the positions of the left and right corners of the eye E (information representing the position coordinates of the left and right corners of the eye E) output from the facial feature point detection unit 20, the outer corner of the eye distance calculator 31 calculates the distance d ₀ between the left and right corners of the eye E. is calculated, and the central positions of the left and right corners of the eyes E are calculated (step V5). The face orientation detection unit 33 detects the position of the predetermined standard head axis P from the left and right sides of the driver obtained from the image of the driver's face taken by the face image acquisition camera 10 with the driver facing the front. The corrected position of the head axis P is calculated according to the distance _d0 of the outer corner of the eye E (step V6). A face authentication unit (not shown) authenticates the driver's face (step V7), and stores the face image, the position of the head axis P, the seat position, etc. in the storage unit 32 (step V8).

FIG. 20 is a flow chart showing the calculation processing of the position of the head axis P according to the second modified example of the embodiment. The face orientation detection system 1 includes an acceleration sensor 50 that detects the acceleration of the vehicle, and a front determination section 60 that determines whether the driver is facing forward. In FIG. 20, the face direction detection system 1 detects the acceleration of the vehicle using the acceleration sensor 50 (step W1). The acceleration sensor 50 outputs the detected acceleration to the front determination section 60 . When the acceleration sensor 50 detects the acceleration of the vehicle (step W2; Yes), the front determination unit 60 determines that the vehicle is moving forward and the driver is facing forward. When the front determination unit 60 determines that the driver faces the front, the face image capturing camera 10 captures the face of the driver while the driver faces the front (step W3). The face feature point detection unit 20 detects a face from the image data captured by the face image acquisition camera 10 (step W4), detects the left and right corners of the eyes E, the left and right pupils, etc. as feature points of the face, and detects these features. The positions of facial parts are detected (step W5). Based on information representing the positions of the left and right corners of the eye E (information representing the position coordinates of the left and right corners of the eye E) output from the facial feature point detection unit 20, the outer corner of the eye distance calculator 31 calculates the distance d ₀ between the left and right corners of the eye E. is calculated, and the central positions of the left and right corners of the eyes E are calculated (step W6). The face orientation detection unit 33 detects the position of the predetermined standard head axis P from the left and right sides of the driver obtained from the image of the driver's face taken by the face image acquisition camera 10 with the driver facing the front. Then, the corrected position of the head axis P is calculated according to the distance _d0 of the outer corner of the eye E (step W7). If the acceleration sensor 50 does not detect acceleration (step W2; No), the face direction detection system 1 returns to step W1 and detects acceleration again. As described above, the face direction detection system 1 includes the acceleration sensor 50 that detects the acceleration of the vehicle, and the front determination unit that determines whether the driver is facing forward based on the acceleration of the vehicle detected by the acceleration sensor 50. 60. The face image capturing camera 10 captures an image of the driver's face based on the determination result based on the acceleration of the vehicle determined by the front determination unit 60 . With this configuration, the face orientation detection system 1 can automatically determine that the driver is facing forward based on the acceleration of the vehicle, and capture an image of the front face of the driver.

FIG. 21 is a flow chart showing the calculation processing of the position of the head axis P according to the third modified example of the embodiment. In FIG. 21, the face direction detection system 1 detects the line of sight of the driver using the line of sight detection unit 40 (step F1). The line of sight detection unit 40 outputs the detected line of sight of the driver to the front determination unit 60 . When the line of sight detected by the line of sight detection unit 40 is facing the front (step F2; Yes), the front determination unit 60 determines that the driver is facing the front. When the front determination unit 60 determines that the driver faces the front, the face image acquisition camera 10 captures the face of the driver while the driver faces the front (step F3). The facial feature point detection unit 20 detects a face from the image data captured by the face video acquisition camera 10 (step F4), detects the left and right corners of the eyes E, the left and right pupils, etc. as feature points of the face, and The positions of facial parts are detected (step F5). Based on information representing the positions of the left and right corners of the eye E (information representing the position coordinates of the left and right corners of the eye E) output from the facial feature point detection unit 20, the outer corner of the eye distance calculator 31 calculates the distance d ₀ between the left and right corners of the eye E. is calculated, and the central positions of the left and right corners of the eyes E are calculated (step F6). The face orientation detection unit 33 detects the position of the predetermined standard head axis P from the left and right sides of the driver obtained from the image of the driver's face taken by the face image acquisition camera 10 with the driver facing the front. The corrected position of the head axis P is calculated according to the distance _d0 of the outer corner of the eye E (step F7). When the line of sight detected by the line of sight detection unit 40 is not facing the front (step F2; No), the face orientation detection system 1 returns to step F1 and detects the line of sight again. As described above, the face orientation detection system 1 includes the line-of-sight detection unit 40 that detects the driver's line of sight, and determines whether the driver is facing forward based on the driver's line of sight detected by the line-of-sight detection unit 40. A front determination unit 60 is further provided. The face image acquisition camera 10 captures an image of the driver's face based on the determination result based on the line of sight of the driver determined by the front determination unit 60 . With this configuration, the face orientation detection system 1 can automatically determine that the driver is facing forward based on the line of sight of the driver, and capture an image of the front face of the driver.

In the above description, the face orientation detection unit 33 performs personalization processing for correcting the standard head axis P stored in the storage unit 32 according to the individual. The face direction may be detected based on the standard head axis P without performing the detection.

The parameter update unit 34 performs parameter update processing for correcting the standard head axis P stored in the storage unit 32 in accordance with the relative position of the face image acquisition camera 10, but is not limited to this. The face direction may be detected based on the standard head axis P without performing parameter update processing.

1 Face direction detection system 10 Face video acquisition camera (imaging unit)
20 facial feature point detection unit (eye corner detection unit)
33 face orientation detection unit 34 parameter update unit 40 line of sight detection unit 50 acceleration sensor E corner of eye P head axis (based on head rotation)
Y up-down direction (vertical direction)
X Horizontal direction Z Depth direction h, h ₁ , d ₀ , w Distance

Claims

an imaging unit that captures an image of a person's face;
an eye corner detection unit that detects the positions of the corners of the eyes of the person based on the image of the person's face captured by the imaging unit;
Face orientation detection for detecting the orientation of the person's face based on the position of the outer corner of the eye detected by the outer corner of the eye detection unit and a head rotation reference position serving as a reference for turning the head of the person. A face orientation detection system comprising:
2. The face according to claim 1, wherein the face orientation detection unit detects the orientation of the person's face in the vertical direction based on the distance between the position of the head rotation reference and the position of the corner of the eye in the vertical direction. Orientation detection system.
3. The face orientation detection unit detects the orientation of the person's face in the left-right direction based on the position of the head rotation reference and the position of the corner of the eye in the left-right direction intersecting the vertical direction. The described face orientation detection system.
The face direction detection unit detects a predetermined standard head rotation reference position as the head rotation reference position of the person captured by the imaging unit while the person faces the front. 4. The direction of the face of the person is detected based on the position of the head rotation reference after correction corrected according to the distance between the left and right corners of the eyes of the person obtained from the face image. 2. The face direction detection system according to claim 1.
The face orientation detection unit detects the distance between the position of the head rotation reference in the vertical direction and the position of the outer corner of the eye, and the depth intersecting the vertical direction, according to the distance between the outer corners of the eyes on the left and right sides of the person. Detecting the orientation of the face of the person based on the position of the head rotation reference after the correction of the distance between the position of the head rotation reference in the direction and the center positions of the left and right corners of the eyes. Item 5. The face orientation detection system according to item 4.
The face direction detection unit corrects a predetermined standard head rotation reference position as the head rotation reference position according to the relative position of the imaging unit with respect to the front position of the person. The face orientation detection system according to any one of claims 1 to 5, wherein the face orientation of the person is detected based on the position of the head rotation reference.
According to the relative position of the imaging unit with respect to the front position of the person, the distance between the position of the head rotation reference in the vertical direction and the position of the corner of the eye, and the head in the depth direction intersecting the vertical direction 7. The face orientation detection system according to claim 6, further comprising a parameter updating unit that corrects a distance between a position of a partial rotation reference and a center position of the left and right corners of the eyes.
an acceleration sensor that detects the acceleration of the vehicle;
a front determination unit that determines whether the person is facing forward based on the acceleration of the vehicle detected by the acceleration sensor;
The face orientation detection system according to any one of claims 1 to 7, wherein the imaging section images the face of the person according to the determination result based on the acceleration of the vehicle determined by the front determination section.
a line-of-sight detection unit that detects the line of sight of the person;
a front determination unit that determines whether the person faces the front based on the person's line of sight detected by the line of sight detection unit;
The face orientation detection system according to any one of claims 1 to 8, wherein the imaging unit images the face of the person according to the determination result based on the line of sight of the person determined by the front determination unit.