JP2016115120A - Opened/closed eye determination device and opened/closed eye determination method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To further accurately make an opened/closed eye determination of a driver.SOLUTION: An opened/closed eye determination device comprises: a storage unit which stores a three-dimensional face model in which a three-dimensional face shape of a subject, positions of eyes of the subject and change information are registered, the change information having the state of shape change of eyes from an eye standard state to a narrow eye state of narrowing eyes due to a face expression change, and a shape parameter associated with each other; a check unit which checks a captured image of a driver captured by an imaging device from the three-dimensional face shape to identify the positions of eyes; a shape parameter determination unit which makes the shape parameter change and identifies a shape parameter corresponding to the state of eyes in the captured image; a degree-of-opened/closed eye calculation unit which calculates the degree of opened/closed eye indicating the eye opening state in the captured image; and an opened/closed eye determination unit which determines the opened/closed eye of the driver, even when the degree of opened/closed eye shows a closed eye state, on the basis of whether or not the identified shape parameter indicates the narrow eye state.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an eye opening determination device and an eye opening determination method.

  In recent years, development of a face detection technique for detecting the position and orientation of a face included in a captured still image or moving image and the state of facial parts such as eyes and mouth has been promoted. For example, in a vehicle, it is possible to perform open / closed eye determination from the driver's face detection result, detect doze driving, etc., and perform predetermined processing such as warning.

  Conventionally, in order to detect a drowsy driving or the like, in addition to detecting the state of the driver's open / closed eyes, a technique using estimated information such as a nap driving from a driving behavior such as vehicle motion is known (for example, Patent Documents). 1). Also known is a technique for detecting the driver's open / closed eye state using both the distance between the upper and lower eyelids (opening degree) of the driver's eyes obtained from the face detection result and the shape of the eyelid, and detecting a drowsy driving or the like. (For example, refer to Patent Document 2).

JP 2008-165348 A JP 2004-192551 A

  However, in the prior art, when detecting the state of the driver's open / closed eyes, the influence of disturbance is large and it is difficult to accurately determine the open / closed eyes. In addition, since the open / closed eye determination is performed on the two-dimensional image, the appearance of the eyelid shape on the two-dimensional image changes according to the relative angular relationship between the face and the camera, and the open / closed eye determination is accurate. Is difficult to do.

  Accordingly, one of the objects of the present invention is to provide an eye opening determination device and an eye opening determination method that can perform more accurate determination of the opening / closing eyes of a driver.

  The open / close eye determination device according to the present invention includes a three-dimensional face shape of a subject, a position of the eye of the subject, and a standard state of the eye from a standard state of the eye to a state of narrowing the eye with a change in facial expression of the face. A storage unit that stores a change information in which a state of change in eye shape and a shape parameter are associated, a registered three-dimensional face model, and a driver's captured image captured by the imaging device, A matching unit that identifies the position of the eye by collating with the original face shape, a shape parameter determining unit that identifies the shape parameter corresponding to the eye state in the captured image by changing the shape parameter, Whether or not the eye opening degree calculation unit that calculates the eye opening degree indicating the eye opening state in the captured image, and whether or not the specified shape parameter indicates the state of the fine eye even when the eye opening degree indicates the eye closing state Base Te, with a, and open-eye or closed-eye determination unit which determines the opening and closing the eyes of the driver. With the configuration described above, erroneous determination can be avoided and the driver's opening / closing eye can be determined more accurately.

  Further, in the open / closed eye determination device according to the present invention, the detailed state includes a smile state, and the open / closed eye determination unit determines whether or not the eye opening degree is within a range indicating the closed eye state, When the eye opening degree is within a range indicating a closed eye state, it is determined whether or not the specified shape parameter is within a range indicating a smile, and when the shape parameter is within a range indicating a smile, The driver determines that the eye is open. With the above-described configuration, it is possible to avoid erroneous determination and determine the driver's eyes more accurately.

  In the open / closed eye determination device of the present invention, the open / closed eye determination unit may be configured such that the degree of eye opening is within a range indicating a closed eye, and the specified shape parameter is outside a range indicating a smile. Determines that the driver is in an eye-closed state. With the above configuration, erroneous determination can be avoided and the driver's eyes can be determined more accurately.

  In addition, the open / closed eye determination device of the present invention further includes a processing unit that performs an operation of prompting the driver to warn when it is determined that the driver is in an eye-closed state. With the above configuration, the driver can be prevented from falling asleep.

  Further, the open / closed eye determination device of the present invention is based on the captured image of the subject, the three-dimensional face structure data including the three-dimensional face shape and the eye position, the state of the eye shape change, and the A measurement unit that obtains the change information associated with a shape parameter; and a model generation unit that registers the measured three-dimensional face structure data and the change information in the storage unit as the three-dimensional face model. And further provided. With the configuration described above, erroneous determination can be avoided and the driver's opening / closing eye can be determined more accurately.

  The opening / closing eye determination method of the present invention is an eye opening determination method executed by an opening / closing eye determination device, and is an eye opening determination method executed by an opening / closing eye determination device, wherein the opening / closing eye determination device The three-dimensional face shape, the eye position of the subject, the state of the eye shape change from the standard state of the eye to the state of narrowing the eye with the facial expression change, and the shape parameters A storage unit that stores a three-dimensional face model in which the associated change information is registered, and compares the captured image of the driver imaged by the imaging device with the three-dimensional face shape, Specifying a position, changing the shape parameter, specifying a shape parameter corresponding to the eye state in the captured image, calculating an eye opening degree indicating the eye opening state in the captured image, and calculating the eye opening degree Indicates a closed eye condition Oite also the shape parameters identified based on whether shows the state of the minutia, and determines the opening and closing the eyes of the driver. With the configuration described above, erroneous determination can be avoided and the driver's opening / closing eye can be determined more accurately.

FIG. 1 is a perspective view showing a state in which a part of a compartment of a vehicle according to the present embodiment is seen through. FIG. 2 is a diagram illustrating an example of the arrangement of the imaging devices according to the present embodiment. FIG. 3 is a block diagram illustrating an example of an open / closed eye determination system according to the present embodiment. FIG. 4 is a block diagram showing a functional configuration of the ECU of the present embodiment. FIG. 5 is a schematic diagram for explaining an example of the three-dimensional face model of the present embodiment. FIG. 6 is a schematic diagram illustrating an example of change information registered in the three-dimensional face model of the present embodiment. FIG. 7 is a flowchart illustrating an example of a procedure for generating a three-dimensional face model in the present embodiment. FIG. 8 is a flowchart illustrating an example of the procedure of the open / closed eye determination process according to the present embodiment. FIG. 9 is a schematic diagram showing a template T created for the face model M in the three-dimensional face model in the present embodiment.

  Hereinafter, an example in which the open / closed eye determination device of the present embodiment is mounted on the vehicle 1 will be described.

  In the present embodiment, the vehicle 1 may be, for example, an automobile (an internal combustion engine automobile) using an internal combustion engine (engine, not shown) as a drive source, or an electric motor (motor, not shown) as a drive source. The vehicle may be an automobile (electric vehicle, fuel cell vehicle, etc.), or an automobile (hybrid vehicle) using both of them as drive sources. The vehicle 1 can be mounted with various transmissions, and various devices (systems, components, etc.) necessary for driving the internal combustion engine and the electric motor. In addition, the method, number, layout, and the like of the device related to driving of the wheels 3 in the vehicle 1 can be variously set.

  As shown in FIG. 1, the vehicle body 2 of the vehicle 1 constitutes a passenger compartment 2a in which a driver (not shown) gets. A steering section 4 and the like are provided in the passenger compartment 2a so as to face the driver's seat 2b as a passenger. In the present embodiment, as an example, the steering unit 4 is a steering wheel that protrudes from a dashboard (instrument panel) 12.

  As shown in FIG. 1, in the present embodiment, as an example, the vehicle 1 is a four-wheeled vehicle (four-wheeled vehicle), and includes two left and right front wheels 3F and two right and left rear wheels 3R. Furthermore, in the present embodiment, these four wheels 3 are configured so that any of them can be steered (turnable).

  Further, a monitor device 11 is provided in the vehicle width direction of the dashboard 12 in the passenger compartment 2a, that is, in the center in the left-right direction. The monitor device 11 is provided with a display device and an audio output device. The display device is, for example, an LCD (Liquid Crystal Display), an OELD (Organic Electroluminescent Display), or the like. The audio output device is, for example, a speaker. The display device is covered with a transparent operation input unit such as a touch panel, for example. The occupant can visually recognize an image displayed on the display screen of the display device via the operation input unit. In addition, the occupant can execute an operation input by operating the operation input unit by touching, pushing, or moving the operation input unit with a finger or the like at a position corresponding to the image displayed on the display screen of the display device.

  As shown in FIG. 2, an imaging device 201 is installed in the handle column 202. The imaging device 201 is, for example, a CCD (Charge Coupled Device) camera or the like. In the imaging apparatus 201, the viewing angle and the posture are adjusted so that the face of the driver 302 sitting on the seat 2b is positioned at the center of the visual field. The imaging device 201 sequentially captures the face of the driver 302 and sequentially outputs image data regarding images obtained by the capturing.

  Next, an open / close eye determination system having the open / close eye determination device in the vehicle 1 according to the present embodiment will be described. FIG. 3 is a block diagram illustrating an example of the configuration of the open / closed eye determination system 100 according to the present embodiment. As illustrated in FIG. 3, in the open / close eye determination system 100, in addition to the ECU 14, the monitor device 11, the steering system 13, the distance measuring units 16 and 17, the brake system 18, the steering angle sensor 19, the accelerator sensor 20, A shift sensor 21, a wheel speed sensor 22, and the like are electrically connected via an in-vehicle network 23 as an electric communication line. The in-vehicle network 23 is configured, for example, as a CAN (Controller Area Network). The ECU 14 can control the steering system 13, the brake system 18, and the like by sending a control signal through the in-vehicle network 23. Further, the ECU 14 detects the torque sensor 13b, the brake sensor 18b, the rudder angle sensor 19, the distance measuring unit 16, the distance measuring unit 17, the accelerator sensor 20, the shift sensor 21, the wheel speed sensor 22, and the like via the in-vehicle network 23. Results, operation signals from the operation input unit, and the like can be received. Here, the ECU 14 is an example of an open / closed eye determination device.

  The ECU 14 includes, for example, a CPU 14a (Central Processing Unit), a ROM 14b (Read Only Memory), a RAM 14c (Random Access Memory), a display control unit 14d, an audio control unit 14e, an SSD 14f (Solid State Drive), a flash memory, and the like. ing. The CPU 14a controls the entire vehicle 1. The CPU 14a is installed in a non-volatile storage device such as the ROM 14b, can read the stored program, and can execute arithmetic processing according to the program. The RAM 14c temporarily stores various types of data used in computations by the CPU 14a. The display control unit 14d mainly executes image processing using the image data obtained by the imaging unit 15, synthesis of image data displayed on the display device, etc., among the arithmetic processing in the ECU 14. In addition, the voice control unit 14e mainly performs processing of voice data output from the voice output device among the calculation processes in the ECU 14. The SSD 14f is a rewritable nonvolatile storage unit that can store data even when the ECU 14 is powered off. The CPU 14a, the ROM 14b, the RAM 14c, and the like can be integrated in the same package. Further, the ECU 14 may have a configuration in which another logical operation processor such as a DSP (Digital Signal Processor), a logical circuit, or the like is used instead of the CPU 14a. Further, an HDD (Hard Disk Drive) may be provided instead of the SSD 14f, and the SSD 14f and the HDD may be provided separately from the ECU 14.

  The configuration, arrangement, electrical connection form, and the like of the various sensors and actuators described above are examples, and can be set (changed) in various ways.

  FIG. 4 is a block diagram showing a functional configuration of the ECU 14 of the present embodiment. As shown in FIG. 4, the ECU 14 includes an input unit 401, a face data measurement unit 402, a model generation unit 403, a collation unit 404, a shape parameter determination unit 405, an eye opening degree calculation unit 406, and an open / closed eye A determination unit 407, a processing unit 408, and a three-dimensional face model 410 are mainly provided. Each configuration excluding the three-dimensional face model 410 shown in FIG. 4 is realized by the CPU 14a configured as the ECU 14 executing a program stored in the ROM 14b. Note that these configurations may be realized by hardware.

  The three-dimensional face model 410 is stored in a storage medium such as the SSD 14f. The three-dimensional face model 410 is a statistical face shape model. The average three-dimensional face shape of the subject, the positions of the facial parts such as the subject's eyes, mouth, and nose, and the change information of the subject's eye shape, Is registered. As an example, the 3D face model 410 may use CLM (Constrained Local Model), AAM (Active Appearance Model), or ASM (Active Shape Model), but is not limited thereto.

  The ECU 14 of the present embodiment uses the three-dimensional face model 410 to detect the direction of the driver's face and the position of the driver's eyes while tracking the driver's face. The eye opening degree, which is the distance, is determined, the shape parameter corresponding to the eye shape is specified, and the driver narrows the eye according to the facial expression change such as a smile based on the eye opening degree and the shape parameter. It is determined whether or not there is an eye open. Details will be described below.

  FIG. 5 is a schematic diagram for explaining an example of the three-dimensional face model 410 of the present embodiment. In FIG. 5, an exemplary facial model M is shown. The model M shows a three-dimensional shape of an average face from face shapes obtained from 100 subjects, for example, and includes a plurality of feature points P each representing a predetermined face part. The feature point P is represented by coordinates with an arbitrary point as the origin. In the example shown in FIG. 5, feature points P representing the eyebrows, eyes, nose, mouth, and contour are shown, but the model M may include feature points P different from those shown in FIG. 5.

  Further, change information is registered in the three-dimensional face model 410. The change information is data in which a step change in the shape of the driver's eyes associated with a change in the facial expression or movement of the driver is associated with a shape parameter for each step. Specifically, as a change state, the standard state is a state in which the face is faceless and the front faces and the eyes are open, and the stage of the shape of the eye from this standard state until the expression changes and smiles Shape parameters such as gradual changes, gradual changes in eye shape from normal to closed eyes, and gradual changes in eye shape from standard to downward (downward view) Are registered in the three-dimensional face model 410. Here, the smile is an example of a fine state in which the eyes become narrower as the facial expression changes.

  FIG. 6 is a schematic diagram illustrating an example of change information registered in the three-dimensional face model 410 of the present embodiment. In the example shown in FIG. 6, the stepwise change in the shape of the eye from the standard eye state to the closing of the eye is associated with the shape parameter 1 that takes a different value for each step. Further, the stepwise change in the shape of the eye from the standard eye state to the smile is associated with the shape parameter 2 that takes a different value for each step. Further, a stepwise change in the shape of the eye from the standard eye state to the downward view is associated with a shape parameter 3 having a different value for each step. Each of the shape parameters 1, 2, 3 decreases in a stepwise manner from the standard eye state toward the changed eye shape state.

  However, the present invention is not limited to this, and the shape parameter may be configured so that the value increases as the state changes. Alternatively, the shape parameter may be configured so as to change differently depending on the type of change, such as a value increasing or decreasing.

  The change in the shape of the eye registered as the change information is not limited to the example shown in FIG. 6. In addition, the eye shape is gradually changed from the standard eye state until yawning. Changes, step-by-step changes in eye shape from standard eye condition to eye-closed state, and step-by-step changes in eye shape from standard eye state to eye-open state It can be configured to be registered in the three-dimensional face model 410 in association with the parameter.

  The generation of the three-dimensional face model 410 is performed as follows by the input unit 401, the face data measurement unit 402, and the model generation unit 403 shown in FIG. FIG. 7 is a flowchart illustrating an example of a procedure for generating the three-dimensional face model 410 according to this embodiment.

  First, the administrator or the like images the faces of, for example, 100 subjects with a 3D scanner or the like (S11). In S11, each of the large number of subjects is imaged by a 3D scanner or the like by performing various changing operations such as a standard face state, a smile, a closed eye state, and a downward state. To do. Note that the number of subjects is not limited to 100.

  The input unit 401 inputs such captured images. Then, the face data measurement unit 402 extracts facial features of a plurality of subjects, feature points of facial parts such as eyes and mouths from the input captured image, and calculates an average facial shape, position of facial parts, etc. Is measured (S12). Here, the face data measurement unit 402 is an example of a measurement unit.

  Further, in S12, the face data measurement unit 402, for example, from the captured images of the various changes of the subject imaged in S11, for example, non-patent document “Automatic eye region structure search by Active Appearance Model search Takeshi Moriyama Takeo Kanade Jeffrey F. Cohn Shinji Ozawa “Symposium on Recognition and Understanding of Images (MIRU2006)” in accordance with the procedure for creating a three-dimensional face model, using a statistical data analysis method such as principal component analysis to step the shape of the eye Shape parameters corresponding to the state of various changes are extracted.

Specifically, when a vector in which the coordinates of each point constituting the outline of the eye in the captured image are arranged is represented by x, x represents the shape. The face data measurement unit 402 averages all the captured images from the standard eye state to the changed state to obtain an average shape x _av . The face data measurement unit 402 performs principal component analysis on the deviation of each x from x _av from the standard eye state to the changed state to obtain an eigenvector P _s . At this time, x is expressed by the following equation (1).

x = x _av + P _s b _s (1)
Here, b _s is a principal component score vector, which is called a shape parameter. That is, the face data measuring unit 402 obtains the shape parameter b _s from the equation (1).

  Then, the face data measuring unit 402 generates change information as illustrated in FIG. 6 by associating the state of the step change of the eye shape with the shape parameter.

  The model generation unit 403 generates an average face shape, which is measured 3D face structure data, the position of the face part, and the change information as the 3D face model 410 (S13). As a result, an average face model is registered and generated in the three-dimensional face model 410.

  Next, the opening / closing eye determination process of this embodiment will be described. FIG. 8 is a flowchart illustrating an example of the procedure of the open / closed eye determination process according to the present embodiment. First, the imaging device 201 images the driver's face (S31). Imaging is performed periodically at regular time intervals. The captured image of the driver is input to the verification unit 404.

  The matching unit 404 matches the two-dimensional captured image of the driver with the three-dimensional face structure data constituting the three-dimensional face model 410 (S32). That is, the matching unit 404 performs model fitting and model tracking. As a result, the matching unit 404 identifies (estimates) the direction of the driver's face, and further searches for and identifies (estimates) the eye position on the three-dimensional face model 410.

  Here, in the present embodiment, model fitting uses an average face model created by the procedure of FIG. 7 as an initial state for a three-dimensional face model 410 that is a statistical face shape model, and features of the model. By placing P at each part of the face of the captured image, a model M that approximates the face is generated.

  In the present embodiment, in model tracking, after the model M is generated in model fitting, the model M is continuously adapted to match the face in the driver's captured image that is periodically imaged in S31. Let In this embodiment, the model is tracked by using a template.

  FIG. 9 is a schematic diagram showing a template T created for a face model M in the three-dimensional face model 410 in the present embodiment. The template T has a predetermined area including the feature point P of the model M in the image. For example, the template T includes a region including a feature point P representing an eyebrow, a region including a feature point P representing an eye, a region including a feature point P representing a nose, a region including a feature point P representing a mouth, and a feature representing a contour. A region including the point P is included. Each region of the template T corresponds to one or more feature points P and is associated with the coordinates of the feature points P. That is, if the position of the area of the template T in the image is determined, the coordinates of the feature point P corresponding to the area can be calculated.

  In the tracking of the model according to the present embodiment, the position of each area of the template T in the image is determined, and the angle, position, and size of the model M in the image are determined using the position of each area. Then, by applying these determined angles, positions and sizes to the model M, the model M can be adapted to the image. The position and number of regions of the template T can be arbitrarily selected as long as the model can be tracked.

  Returning to FIG. 8, since the position of the eye is specified by the three-dimensional face model adapted in S32, the shape parameter determination unit 405 creates a three-dimensional eye model. Then, the shape parameter determination unit 405 adapts the two-dimensional captured image to the three-dimensional eye model (S33). That is, the shape parameter determination unit 405 changes the shape parameter, and collates the eye shape corresponding to the shape parameter with the eye shape of the captured image. The shape parameter determination unit 405 specifies a shape parameter corresponding to the shape of the eye in the captured image from the three-dimensional face model 410.

  Next, the eye opening degree calculation unit 406 calculates the eye opening degree of the driver in the captured image (S34). The degree of eye opening indicates the state of eye opening of the driver, and in the present embodiment, is the distance between the upper and lower eyelids. As a method for calculating the degree of eye opening by the eye opening degree calculation unit 406, a known method such as calculation from pixels of a captured image is used. In the present embodiment, the distance between the upper and lower eyelids is used as the eye opening degree, but the present invention is not limited to this as long as the eye opening state is indicated.

  The open / close eye determination unit 407 determines whether the driver is in an open state or a closed eye state based on the shape parameter specified in S33 and the eye opening degree calculated in S34. Specifically, the following processing is performed.

  The open / close eye determination unit 407 determines whether or not the eye opening degree calculated in S34 is equal to or less than a predetermined closing threshold (S35). Here, the closing threshold is the maximum distance between the upper and lower eyelids that indicates a state in which the eyes are closed. Therefore, the eye opening degree being equal to or smaller than the closing threshold means that the eyes may be closed. The closing threshold is predetermined and stored in the ROM 14b, the SSD 14f, or the like.

  When the degree of eye opening is larger than the closing threshold (S35: No), the open / close eye determination unit 407 determines that the driver's eyes are in an open state (S38). Then, the process ends.

  On the other hand, when the eye opening degree is equal to or smaller than the closing threshold value in S35 (S35: Yes), the open / closed eye determination unit 407 determines whether or not the shape parameter specified in S33 is equal to or smaller than the predetermined smile threshold value. (S36). Here, the smile threshold value is the maximum value of the shape parameter of the eye shape when smiling. Therefore, when the shape parameter is in the range equal to or smaller than the smile threshold, it means that the eye shape is the range of the eye shape when smiling. The smile threshold is determined in advance and stored in the ROM 14b, the SSD 14f, or the like. Note that the smile threshold value may be determined for each individual. Moreover, you may comprise so that a smile threshold value may be changed according to a condition.

  In the present embodiment, when the shape parameter is equal to or smaller than the smile threshold, the eye shape is set as the smile range, but as described above with reference to FIG. 6, as the standard eye shape approaches the smile, This is because the shape parameter of the present embodiment is set to be small. Therefore, depending on how the shape parameter is set, the method of determining the smile threshold value that becomes the boundary of the smile range differs.

  If the shape parameter is equal to or smaller than the smile threshold value in S36 (S36: Yes), the open / close eye determination unit 407 determines that the driver is smiling (S40). That is, in the determination of S35, since the degree of eye opening is equal to or less than the closing threshold and the distance between the upper and lower eyelids is small, the shape parameter is within the range of smiles. This distance is small, and it is determined that the eye is open. Thereby, a driver's smile and an open eye state can be detected more correctly. Then, the process ends.

  On the other hand, when the shape parameter is larger than the smile threshold value in S36 (S36: No), the open / close eye determination unit 407 determines that the driver is in the closed state (S37). That is, in the determination of S35, since the degree of eye opening is equal to or less than the closing threshold, the distance between the upper and lower eyelids is small, and the shape parameter is not within the range of smile, the opening / closing eye determination unit 407 causes the driver to close his eyes. Judge that As a result, the driver's closed eye state can be detected more accurately. In this case, since there is a high possibility that the driver is taking a nap, the processing unit 408 outputs an alarm from the audio output device of the monitor device 11 (S39). In this embodiment, an alarm is output. However, the present invention is not limited to this as long as it performs an operation that urges the driver to wake up from a dozing state.

  As described above, in this embodiment, the shape parameter corresponding to the change in the shape of the eyes is registered in the three-dimensional face model 410, and the eyes of the driver are determined using this shape parameter in addition to the degree of eye opening. Therefore, it is possible to estimate (specify) a smile or the like that does not depend on the relative position or angle between the face and the imaging device 201. For this reason, according to the present embodiment, it is possible to avoid misjudgment and more accurately determine the eyes of the driver.

  In the present embodiment, the shape of the eyes of the driver's face is estimated (specified) using not only a two-dimensional captured image but also a three-dimensional face model. Less affected by relative face position and face orientation. For this reason, according to the present embodiment, the shape of the eyes can be estimated more accurately, and thereby the driver's open / closed eyes can be determined more accurately.

  Further, in this embodiment, since the captured image is fitted to the three-dimensional face model, the fitting is performed simultaneously on the upper, lower, left, and right eyelids, compared to the conventional technique that detects the curve independently with the upper, lower, left, and right eyelids. Thus, a smile can be detected robustly against noise in the captured image.

  In the present embodiment, when it is determined that the driver is in the closed eye state, the processing unit 408 outputs an alarm from the audio output device of the monitor device 11 as an example of an operation that prompts the driver to warn. Therefore, it is possible to prevent the driver from falling asleep.

  In this embodiment, a smile is given as an example where the eyes become narrower as the facial expression changes and a smile is given as an example, and changes in the shape of the eye from the standard eye state to smile are registered in the 3D face model 410 together with the shape parameters. Then, the smile is determined in the open / closed eye determination, but the three-dimensional face model 410, the shape parameter determining unit 405, the open / closed eye are determined so as to perform a determination other than a smile as a fine state in which the eyes become narrower as the facial expression changes. The determination unit 407 and the like may be configured.

  For example, in the three-dimensional face model 410, the stepwise change in the shape of the eye from when the standard eye state is yawned as the fine eye state is registered together with the shape parameter, and the yawning is determined. A three-dimensional face model 410, a shape parameter determination unit 405, an open / close eye determination unit 407, and the like can be configured.

  In addition, in the three-dimensional face model 410, a stepwise change in the shape of the eye from the standard eye state to the state where the eye is focused as the above-mentioned fine state is registered together with the shape parameter, and The three-dimensional face model 410, the shape parameter determination unit 405, the open / closed eye determination unit 407, and the like may be configured so as to determine.

  In addition, the shape parameters are extracted from the above-mentioned non-patent document “Automatic Extraction of Eye Region Structure by Active Appearance Model Search Takeshi Moriyama Takeo Kanade Jeffrey F. Cohn Shinji Ozawa“ Image Recognition and Understanding Symposium (MIRU2006) ”” and non-patent document “ Costen, N., Cootes, TF, Taylor, CJ: Compensating for ensemble-specificity effects when building facial models.Image and Vision Computing 20, 673−682 The three-dimensional face model 410 may be configured to be classified and registered.

  Further, in the present embodiment, smile determination is performed in order to accurately determine the driver's opening / closing eye, but the present invention is not limited to this. The three-dimensional face model 410, the shape parameter determination unit 405, the open / closed eye determination unit 407, and the like may be configured so that smile detection according to the method of the present embodiment is used in human interaction such as interaction with an interface.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

1 Vehicle 14 ECU
201 imaging device 401 input unit 402 face data measurement unit 403 model generation unit 404 collation unit 405 shape parameter determination unit 406 eye opening degree calculation unit 407 opening / closing eye determination unit 408 processing unit 410 three-dimensional face model

Claims (6)

The three-dimensional face shape of the subject, the eye position of the subject, and the state and shape of the eye shape change from the standard state of the eye to the state of narrowing the eye with the facial expression change Change information associated with the parameters, a storage unit that stores the registered three-dimensional face model,
A collation unit that collates a captured image of a driver imaged by an imaging device with the three-dimensional face shape and identifies the position of the eye;
A shape parameter determination unit that changes the shape parameter to identify a shape parameter corresponding to the eye state in the captured image;
An eye opening degree calculation unit for calculating an eye opening degree indicating the eye opening state of the eye in the captured image;
Even in the case where the degree of eye opening indicates a closed eye state, an opening / closing eye determination unit that determines whether the driver's opening / closing eye is based on whether the specified shape parameter indicates the detailed state,
An open / close eye determination device. The detailed state includes a smiling state,
The open / close eye determination unit determines whether or not the eye opening degree is within a range indicating an eye-closed state. When the eye opening degree is within a range indicating an eye-closed state, the specified shape parameter is the smile It is determined whether or not the driver is in an open state when the shape parameter is in a range indicating the smile.
The open / close eye determination device according to claim 1. The open / close eye determination unit is in a case where the degree of eye opening is within a range indicating closed eyes, and the driver is in an eye closed state when the specified shape parameter is outside the range indicating the smile. To determine,
The open / close eye determination device according to claim 2. When it is determined that the driver is in an eye-closed state, a processing unit that performs an operation for prompting the driver to warn,
The open / close eye determination device according to claim 3, further comprising: The change information in which the three-dimensional face structure data including the three-dimensional face shape and the eye position, the change state of the eye shape, and the shape parameter are associated with each other based on the captured image of the subject. And a measurement unit for
A model generation unit that registers the measured three-dimensional face structure data and the change information in the storage unit as the three-dimensional face model;
The open / close eye determination device according to any one of claims 1 to 4, further comprising: An eye opening determination method executed by an opening / closing eye determination device,
The open / close eye determination device includes: a three-dimensional face shape of a subject, a position of the subject's eyes, and a shape of the eye from a standard state of the eye to a fine state of narrowing the eye with a change in facial expression of the face A storage unit for storing a three-dimensional face model in which change information in which a state of change and a shape parameter are associated is registered,
A driver's captured image captured by the imaging device is compared with the three-dimensional face shape to identify the position of the eyes,
Change the shape parameter to identify the shape parameter corresponding to the eye state in the captured image,
Calculating an eye opening degree indicating the eye opening state in the captured image;
Even in the case where the degree of eye opening indicates a closed eye state, the eyes of the driver are determined based on whether or not the specified shape parameter indicates the state of the fine line,
Open / closed eye determination method.

Images