WO2017159215A1

WO2017159215A1 - Information processing device and information processing method

Info

Publication number: WO2017159215A1
Application number: PCT/JP2017/006046
Authority: WO
Inventors: 雄大中村; 内藤　正博
Original assignee: 三菱電機株式会社
Priority date: 2016-03-18
Filing date: 2017-02-20
Publication date: 2017-09-21
Also published as: JP2019082743A

Abstract

The present invention is provided with: a brightness-change detecting unit (113) that estimates the coordinates of a first endpoint existing on a first boundary between an eye and an upper eyelid and the coordinates of a second endpoint existing on a second boundary between the eye and a lower eyelid by detecting the magnitudes of variations in brightness between pixels along the vertical direction in an eye presence region identified from a face image; an eyelid- reference-position estimating unit (114) that estimates the coordinates of the inner corner of the eye and the coordinates of the outer corner of the eye in the eye presence region; an approximate-curve generating unit (115) that selects a plurality of coordinates from a first selected region formed on a straight line passing the first endpoint and the second endpoint in the eye presence region and that generates a plurality of approximate curves from the individual coordinates in the plurality of selected coordinates, the coordinates of the inner corner of the eye, and the coordinates of the outer corner of the eye; and a shape determining unit (116) that selects an approximate curve that is most suitable as the first boundary from among the plurality of generated approximate curves.

Description

Information processing apparatus and information processing method

The present invention relates to an information processing apparatus and an information processing method, and more particularly to an information processing apparatus and an information processing method for determining the shape of an eyelid from a face image.

In recent years, accidents caused by the falling asleep of car and train drivers are regarded as a problem. On the other hand, it is effective to detect the degree of opening of the driver's eyes (hereinafter referred to as the degree of eye opening) from the camera image and monitor the dozing. Conventionally, a technique has been proposed in which the position of an eye is detected from a driver's face image captured by a camera, the position and shape of the eyelid is estimated by analyzing bright and dark edges near the eye, and the degree of eye opening is determined. . For example, the apparatus described in Patent Document 1 extracts horizontal edges and vertical edges of a face image, estimates the eyelid position from the positional relationship, and determines the degree of eye opening. Further, the apparatus described in Patent Document 2 extracts an edge from a face image and approximates the eyelid with a curve to accurately determine the degree of eye opening even when wearing glasses. The device described in Patent Document 3 extracts a white-eye region from a face image and realizes an eye opening degree robust to makeup such as eye shadow.

Japanese Patent No. 4137969 Japanese Patent No. 4825737 Japanese Patent No. 4623044

However, due to cost, processing amount, and environmental influences (dark environment), it is difficult to extract edges and white-eye areas when the face image has low image quality.

In addition, it is effective to use near infrared light that cannot be seen by human eyes as auxiliary light in a dark environment. However, when near-infrared light is used, it is difficult to distinguish between a white-eye area and a black-eye area.
Further, when the image quality is low, an error occurs in the eye detection position. For example, in the case of low image quality, some of the eye positions are detected by using changes in eyebrow and eye brightness, but are limited to those indicating the presence area of the eye, such as the positions of the corners of the eyes and the eyes. It is difficult to detect accurately.

Therefore, the present invention has been made in view of such circumstances, and an object of the present invention is to detect the eyelid shape with high accuracy even for a low-quality face image or a face image using near infrared light. Is to be able to do it.

An information processing apparatus according to a first aspect of the present invention includes an eye presence area specifying unit that specifies an eye presence area, which is an area where an eye exists, from a person's face image, and between pixels in the vertical direction of the eye presence area. By detecting the magnitude of the change in brightness of the eye, the coordinates of the first end point on the first boundary line between the eye and the upper eyelid and the second boundary line between the eye and the lower eyelid In the eye presence region, a light / dark change detection unit that estimates the coordinates of the second end point, an eyelid reference position estimation unit that estimates the coordinates of the eye's eye corner and the eye's eye corner in the eye presence region, A plurality of coordinates are selected from a straight line passing through the first end point and the second end point, and a plurality of first approximate curves are generated from each of the selected plurality of coordinates, the eye coordinates and the eye corner coordinates. An approximate curve generation unit that performs the plurality of first approximate songs A shape determining unit that selects the most suitable first approximate curve as the first boundary line and determines the shape of the upper eyelid from the selected first approximate curve, and the light / dark change detecting unit includes: A first selection area formed on a straight line passing through the first end point and the second end point and including the first end point in the eye presence area based on a change in brightness between the pixels; The approximate curve generation unit selects the plurality of coordinates from the first selection region.

An information processing apparatus according to a second aspect of the present invention provides an eye presence area specifying unit that specifies an eye presence area, which is an area where an eye exists, from a person's face image, and between pixels in the vertical direction of the eye presence area. By detecting the magnitude of the change in brightness of the eye, the coordinates of the first end point on the first boundary line between the eye and the upper eyelid and the second boundary line between the eye and the lower eyelid In the eye presence region, a light / dark change detection unit that estimates the coordinates of the second end point, an eyelid reference position estimation unit that estimates the coordinates of the eye's eye corner and the eye's eye corner in the eye presence region, A plurality of coordinates are selected from a straight line passing through the first end point and the second end point, and a plurality of first approximate curves are generated from each of the selected plurality of coordinates, the eye coordinates and the eye corner coordinates. An approximate curve generation unit and the plurality of first approximate curves In the eye presence region, an approximate curve selection unit that determines the coordinates of a fixed end point that has determined the highest point in the first boundary line by selecting the most suitable first approximate curve as the first boundary line; Selecting a plurality of first selection coordinates from a straight line in the vertical direction passing through the coordinates of the eye and a plurality of second selection coordinates from a straight line in the vertical direction passing through the coordinates of the corner of the eye, and the plurality of first selection coordinates Each of the plurality of second selected coordinates and a correction curve generation unit that generates a plurality of correction curves that are approximate curves from the coordinates of the determined end point, and the first boundary from the plurality of correction curves A shape determining unit that selects a most suitable correction curve as a line and determines the shape of the upper eyelid from the selected correction curve.

The information processing method according to the first aspect of the present invention specifies an eye presence area, which is an area where an eye exists, from a human face image, and changes in brightness between pixels in the vertical direction of the eye presence area. By detecting the size, the coordinates of the first end point on the first boundary line between the eye and the upper eyelid and the coordinates of the second end point on the second boundary line between the eye and the lower eyelid are obtained. Estimating the coordinates of the eye's eyes and the coordinates of the outer corners of the eye in the eye presence region, and forming the eye presence region on a straight line passing through the first end point and the second end point, A first selection area including a first end point is specified based on a change in brightness between the pixels, a plurality of coordinates are selected from the first selection area, and each of the plurality of selected coordinates is A plurality of first approximate curves are generated from the coordinates of the eyes and the coordinates of the corners of the eyes. From said plurality of first approximation curve, and select the most suitable first approximation curve as said first boundary line, and determines the shape of the upper eyelid from the first approximation curve is the selected.

The information processing method according to the second aspect of the present invention specifies an eye presence area, which is an area where an eye exists, from a human face image, and changes in brightness between pixels in the vertical direction of the eye presence area. By detecting the size, the coordinates of the first end point on the first boundary line between the eye and the upper eyelid and the coordinates of the second end point on the second boundary line between the eye and the lower eyelid are obtained. Estimating the coordinates of the top of the eye and the coordinates of the outer corner of the eye in the eye presence region, and in the eye presence region, a plurality of coordinates from a straight line passing through the first end point and the second end point A plurality of first approximate curves are generated from each of the selected plurality of coordinates, the coordinates of the top of the eye, and the coordinates of the corners of the eyes, and the first boundary line is most preferably generated from the plurality of first approximate curves. By selecting a suitable first approximate curve, the first boundary line Determining the coordinates of the final endpoint that has determined the uppermost point, and selecting a plurality of first selection coordinates from a straight line extending in the vertical direction passing through the coordinates of the eye in the eye presence area, and in the eye presence area, A plurality of second selection coordinates are selected from a straight line extending in the vertical direction passing through the coordinates of the plurality of first selection coordinates, each of the plurality of second selection coordinates, and the coordinates of the determined end point, Generating a plurality of correction curves as approximate curves, selecting a correction curve most suitable as the first boundary line from the plurality of correction curves, and determining the shape of the upper eyelid from the selected correction curve; It is characterized by.

According to one aspect of the present invention, by generating a plurality of approximate curves based on a point with a large change in brightness, comparing each curve, and determining a boundary line between the eye and the eyelid, Even with a low-quality face image or a face image using near-infrared light, the eyelid shape can be detected with high accuracy.

3 is a block diagram schematically showing a configuration of an eye opening degree detection apparatus according to Embodiments 1 to 3. FIG. (A) And (B) is the schematic for demonstrating the face image imaged with the low-resolution camera. FIG. 3 is a schematic diagram for explaining an eye presence area in the first embodiment. (A) And (B) is the schematic which shows the filter used by the light-and-dark change detection part in Embodiment 1. FIG. In Embodiment 1, it is the schematic which shows the example by which the coordinate was specified on the double eyelid. In Embodiment 1, it is the schematic which shows the method of producing | generating several approximate curves with respect to an upper eyelid. 4 is a flowchart showing processing in the eye opening degree detection apparatus according to the first embodiment. (A) And (B) is the schematic which shows the calculation method of the area | region in Embodiment 2. FIG. FIG. 10 is a block diagram schematically showing a configuration of an eye opening degree detection device according to a fourth embodiment. FIG. 10 is a schematic diagram illustrating an example of an image near the eye when the driver is facing in a lateral direction in the fourth embodiment. FIG. 10 is a block diagram schematically showing a configuration of an eye opening degree detection device according to a fifth embodiment. In Embodiment 5, it is the schematic for demonstrating the production | generation of the correction | amendment curve of an upper eyelid. It is a block diagram which shows roughly the structure of the dozing degree detection apparatus which concerns on Embodiment 6. FIG. (A) and (B) are schematic diagrams illustrating an example of a hardware configuration of an eye-opening degree detection apparatus according to Embodiments 1 to 5. FIG.

Embodiment 1 FIG.
FIG. 1 is a block diagram schematically showing a configuration of an eye opening degree detection apparatus 100 as an information processing apparatus according to the first embodiment.
The eye opening degree detection device 100 includes an eyelid shape detection device 110 and an eye opening degree calculation unit 130.

FIG. 2A shows a face image FI captured by a low-resolution camera.
The face image FI includes an upper eyelid elu, a lower eyelid elb, a double eyelid els, and eyebrows ebr. FIG. 2B is a schematic diagram showing the positions of the upper eyelid elu, the lower eyelid elb, the double eyelid els, and the eyebrows ebr in the face image FI.
As shown in FIG. 2A, in the face image FI captured by the low-resolution camera, it is difficult to recognize as an edge even though the brightness change of brightness can be confirmed.
The eyelid shape detection device 110 detects the eyelid shape with high accuracy even for a low-quality face image FI.

The eyelid shape detection device 110 illustrated in FIG. 1 includes an eye presence region specifying unit 111, a normalization processing unit 112, a light / dark change detection unit 113, an eyelid reference position estimation unit 114, and an approximate curve generation unit 115. And a shape determining unit 116.

The eye presence area specifying unit 111 is provided with low-resolution face image data Im representing a face image that is a face image of a person (hereinafter referred to as a driver for simplification of description) to be an eye opening degree detection target, From the face image represented by the face image data Im, an eye presence area that is an area where the driver's eyes exist is specified. Then, the eye presence region specifying unit 111 generates eye presence region data Dm that is data indicating the specified eye presence region. Then, the eye presence area specifying unit 111 provides the eye presence area data Dm to the normalization processing unit 112 together with the face image data Im.

Specifically, the eye presence area specifying unit 111 is given face image data Im representing a driver's face image, and the left eye, right eye, or both eyes of the driver are selected from the face image. An eye presence area is specified, and eye presence area data Dm indicating the specified eye presence area is generated.

The face image data Im is, for example, color image data or gray scale image data captured by an RGB camera, a gray scale camera, or an infrared light camera. In the face image data Im, the upper face (head, forehead) of the driver is shown above the face image, and the lower face (mouth, chin) is shown below the face image.
In the face image data Im, the upper left point of the image is the origin of the image coordinate system, the right direction of the origin is the x coordinate direction, and the downward direction of the origin is the y coordinate direction.
The left eye of the driver is the eye located on the left side of the face when facing the face image, and the right eye of the driver is the eye located on the right side of the face when facing the face image. is there. Hereinafter, in order to simplify the description, it is assumed that the eye presence area data Dm indicates the presence area of the right eye of the driver.

The eye presence area will be described with reference to FIG.
The eye presence area is an area where each eye in the face image is considered to exist. Specifically, as shown in FIG. 3, the eye presence region Dr is a region including the corner of the eye eout, the eye ein, the end point etop of the upper eyelid, and the end point ebot of the lower eyelid.
Here, the eye presence region is a region that satisfies the following first condition and second condition.
The first condition is that when both eyes, eyebrows, nose and mouth are face parts, only one eye is included and the other parts are not included. However, the eyebrows may be partially included.
The second condition is that when the eye presence area is a rectangular area, the center of gravity Mg of the rectangular area is included in an area surrounded by the corner of the eye eout, the eye ein, the end point etop of the upper eyelid, and the end point ebot of the lower eyelid. It is that.

FIG. 3 shows an example in which the eye presence area Dr is expressed as a rectangular area. For example, when the eye presence region Dr is expressed as a rectangular region, the eye presence region data Dm is data represented by the coordinates of the center of gravity Mg of the rectangular region and the width Mw and height Mh of the rectangular region.
The eye presence area data Dm may be a combination of the coordinates of the upper left point of the rectangular area and the coordinates of the lower right point. Alternatively, when the eye existing area is a circular area, the eye existing area data Dm may be a combination of the coordinates of the center point of the circular area and the radius.
Hereinafter, in order to simplify the description, the eye presence area Dr is a rectangular area, and the coordinates (Mgx, Mgy) of the center of gravity Mg of the rectangular area and the width Mw and height Mh of the rectangular area are the eye presence area data Dm. .

The eye presence region Dr may be set based on the position of the nostril, for example, as shown in Patent Document 1, by detecting a nostril that is a characteristic part that is easy to detect in the face. Further, the eye presence area may be set by detecting the approximate positions of the corners of the eyes and the eyes by statistical processing using AAM (Active Appearance Model). In the present embodiment, the method for setting the eye presence region Dr is not particularly limited.

The normalization processing unit 112 shown in FIG. 1 is given eye presence area data Dm and face image data Im, and from the face image represented by the face image data Im, the eye indicated by the eye presence area data Dm. Cut out the image of the existing area. Then, the normalization processing unit 112 converts the clipped image into a normalized image having a predetermined size. Then, the normalization processing unit 112 generates normalized image data In representing the normalized image. Further, the normalization processing unit 112 converts the normalized image data In and the normalized eye presence region data Dn indicating the eye presence region in the normalized image data In, the light / dark change detection unit 113 and the eyelid reference position estimation unit 114. To give.

When the normalization processing unit 112 converts the image size, first, an image of the eye existing area is cut out, and then an enlargement or reduction process is performed. For example, if the width of the rectangle of the eye presence region is Mw, the height is Mh, the width of the normalized image is Nw, and the height is Nh, the enlargement or reduction magnification is Nw / Mw times in the x-axis direction , Nh / Mh times in the y-axis direction. At this time, if the ratio between the width Mw and the height Mh and the ratio between the width Nw and the height Nh are set in advance, the aspect ratio does not change between the image before normalization and the image after normalization. Therefore, it is desirable to set Mw = αMh (α is a constant) and Nw = αNh. The value of α is preferably α = 2.5, for example. Further, the width and height of the image after normalization are preferably Nw = 20 and Nh = 50, for example.

For the enlargement or reduction of the image, for example, bilinear interpolation, bilateral interpolation, or nearest neighbor interpolation may be used, and is not particularly limited in the present embodiment. Here, it is assumed that enlargement or reduction processing is performed using bilinear interpolation.

Further, since the normalized image indicated by the normalized image data In is obtained by cutting out the eye existing area of the original image and converting the image size, the normalized eye existing area data Dn is the center of the normalized image. It is defined by coordinates Ng, width Nw, and height Nh. The center coordinate Ng is a center coordinate on the normalized image and is defined by Ng (Ngx, Ngy) = (Nw / 2, Nh / 2).

The light / dark change detection unit 113 calculates light / dark change data En by calculating the magnitude of the light / dark change in the vertical direction of the face image with respect to the normalized image represented by the normalized image data In. Then, the light / dark change detection unit 113 supplies the light / dark change data En to the eyelid reference position estimation unit 114 and the approximate curve generation unit 115 together with the normalized image data In.

For example, the light / dark change detection unit 113 detects the magnitude of the change in brightness between pixels in the y-axis direction of the normalized image represented by the normalized image data In, so that the driver's eyes and the upper eyelid are detected. The coordinates of the first end point on the boundary line (first boundary line) between and the coordinates of the second end point on the boundary line (second boundary line) between the driver's eyes and the lower eyelid presume. In the first embodiment, the brightness change detection unit 113 uses the coordinates of the pixel with the greatest degree of brightness decrease from the upper pixel as the coordinates of the first end point in the normalized image, and the brightness from the lower pixel. Let the coordinates of the pixel with the greatest degree of decrease be the coordinates of the second end point. Then, the light / dark change detection unit 113 generates light / dark change data En indicating the estimated coordinates of the first end point and the coordinates of the second end point, and supplies the data to the reference position estimation unit 114 and the approximate curve generation unit 115 which are the eyelids. give.

In order to extract the change in light and dark, the light and dark change detection unit 113 uses, for example, the filters shown in FIGS.
The 5 × 5 filter Ebf shown in FIG. 4A is an example of a filter that extracts a point where light and dark changes from dark to bright in the y-axis direction. The filter Ebf is a second filter that extracts a change in brightness near the lower eyelid.
Assuming that the center position (second target pixel) in the filter Ebf is the position “3” shown in FIG. 4A, the filter Ebf is set with the point (nx, ny) on the normalized image as the center position. When applied, the evaluation value Ebv of the filter Ebf is calculated by the following equation (1). Note that In (x, y) indicates a luminance value at coordinates (x, y) in the normalized image data In.

The filter Ebf has a large one-item area (fourth area) (5 × 4 = 20 pixels) and a small two-item area (third area) in equation (1) (5 × 1 = 5 pixels). Min). This is because when the change in brightness around the lower eyelid is observed, the area below the lower eyelid (in the y-axis direction) is an area where the change in brightness is relatively small and the change due to eye opening and closing is small. Become. On the other hand, the region above the lower eyelid (opposite to the y-axis direction) is affected by the presence of the pupil and the opening / closing of the eye, and is a region where the brightness changes greatly. That is, the filter Ebf is suitable for stably detecting the brightness of the lower eyelid irrespective of the eye state.

The 5 × 5 filter Etf shown in FIG. 4B is an example of a filter that extracts a point where light and dark changes from light to dark in the y-axis direction. The filter Etf is a first filter that extracts a change in brightness near the upper eyelid.
If the center position (first pixel of interest) in the filter Etf is the position of “23” shown in FIG. 4B, the filter Etf is set with the point (nx, ny) on the normalized image as the center position. When applied, the evaluation value Etv of the filter Etf is calculated by the following equation (2).

The filter Etf is a filter obtained by inverting the top and bottom of the filter Ebf so that the area of one item (second area) on the right side of equation (2) is large and the area of two items (first area) is small. Designed to. This is the same as in the case of the lower eyelid described above, and when observing the change in brightness around the upper eyelid, the region above the upper eyelid (the side opposite to the y-axis direction) is relatively bright and dark. This is a region where the change is small and the change due to opening and closing of the eye is small. On the other hand, the region on the lower side of the upper eyelid (in the y-axis direction) is a region where the brightness changes greatly due to the influence of pupil movement and eye opening / closing. Therefore, the filter Etf is suitable for detecting the brightness of the upper eyelid stably regardless of the eye state.

The light / dark change detection unit 113 applies the above-described two types of filters to each pixel of the normalized image data In, whereby an evaluation value Etv indicating the degree of light / dark change of the two types of luminance values in each pixel and the evaluation The value Ebv is obtained. Here, the evaluation value Etv indicates the magnitude of change from light to dark in the y-axis direction from light to dark, that is, the degree of decrease in brightness in the y-axis direction. The evaluation value Ebv is the lightness and darkness in the y-axis direction. The magnitude of the change from dark to light, that is, the degree of decrease in brightness in the direction opposite to the y-axis direction.

The coordinates of the point having the maximum value in the evaluation value Etv of the light-dark change are set to Ent (Entx, Enty), and the coordinates of the point having the maximum value in the evaluation value Ebv are set to Enb (Enbx, Enby). The brightness change detection unit 113 estimates the coordinate Ent as the coordinate of the first end point that is the uppermost point of the boundary line between the driver's eye and the upper eyelid, and sets the coordinate Enb as the driver's eye and the lower eyelid. Is estimated to be the coordinates of the second end point that is the lowest point of the boundary line between the two, and light-dark change data En indicating the coordinates of these two points is given to the eyelid reference position estimating unit 114 and the approximate curve generating unit 115 .

Here, the coordinate Ent and the coordinate Enb are coordinates indicating a likely position as an end point of the upper eyelid and an end point of the lower eyelid in the brightness change included in the eye presence region Dr. However, with respect to the coordinate Ent, the coordinate point may deviate upward from the upper eyelid due to the influence of makeup such as double eyelids on the upper eyelid, false eyelashes or eye shadow, or bangs. .

FIG. 5 is a schematic diagram illustrating an example in which the coordinate Ent is located on the double eyelid els.
For example, in FIG. 5, the evaluation value Etv of the filter may take the maximum value at a point on the double eyelid els instead of a point on the upper eyelid elu. Since the processing in this case is handled by the shape determination unit 116, it will be described later.
In the following description, it is assumed that the coordinate Ent is a point on the double eyelid els.

The eyelid reference position estimation unit 114 shown in FIG. 1 is given the normalized image data In, the normalized eye presence area data Dn, and the light / dark change data En, and at the position of the reference point existing on the eyelid. At least two eyelid reference positions are calculated. For example, the eyelid reference position estimation unit 114 estimates the coordinates of the eyes and the coordinates of the eyes in the normalized image represented by the normalized image data In. Then, the eyelid reference position estimating unit 114 generates coordinate data Fn indicating the coordinates of the reference point on the normalized image. Then, the eyelid reference position estimation unit 114 gives the coordinate data Fn to the approximate curve generation unit 115.

The point serving as the eyelid reference position is preferably, for example, a point corresponding to the corner of the eye and the eye. When the face image captures the front face of the driver, it can be estimated that the positions of the corners of the eyes and the eyes on the y-axis are substantially the same as the lower end of the lower eyelid. Therefore, it is possible to estimate the y-coordinates of the corners of the eyes and the eyes from the y-coordinate (Enby) of the coordinates Enb in the light / dark change data En.

FIG. 5 shows an example of the positional relationship of each point when the y-coordinate of the corner of the eye and the top of the eye is determined from the y-coordinate of the coordinate Enb.
For example, the coordinates of the corner of the eye are coordinates Eno (Enox, Enoy), the coordinates of the head of the eye are coordinates Eni (Enix, Eniy), and the x coordinate of the right end of the eye existing area in the normalized image indicated by the normalized eye existing area data Dn is Assuming Endx, the coordinates of the eye head Eni = (0, Enby) and the coordinates of the eye corner Eno = (Endx, Enby). Here, since the width of the normalized image is the width Nw, the coordinates of the corner of the eye Eno = (Nw, Enby).
Then, the eyelid reference position estimation unit 114 estimates the estimated coordinates Eno of the eye corners and the estimated coordinates Eni of the eyes as the coordinates of the eyelid reference position, and generates coordinate data Fn indicating these coordinates.

The approximate curve generation unit 115 shown in FIG. 1 generates a plurality of approximate curves for the eyelid from the normalized image data In, the light / dark change data En, and the eyelid reference position data Fn. Parameter data Gn including a set of parameters indicating each of the curves is given to the shape determining unit 116.

For example, the approximate curve generation unit 115 is selected from a straight line passing through the coordinates Ent of the first end point and the coordinates Enb of the second end point indicated by the light / dark change data En in the normalized image represented by the normalized image data In. Further, a plurality of approximate curves (first approximate curves) are generated from each of the plurality of coordinates, from the coordinates Eno of the corner of the eye and the coordinates Eni of the eye indicated by the eyelid reference position data Fn. In the first embodiment, the approximate curve generation unit 115 selects a plurality of coordinates from a line segment between the coordinates Ent of the first end point and the coordinates Enb of the second end point. Then, the approximate curve generation unit 115 gives parameter data Gn including a set of parameters indicating each of the plurality of approximate curves to the shape determination unit 116. The curve used for approximation is, for example, a quadratic or higher curve.
In the present embodiment, it is assumed that the y-coordinate of the estimated coordinate Eno of the corner of the eye and the estimated coordinate Eni of the eye is equal to the coordinate Enb of the second end point, and therefore the curve Cb of the lower eyelid is a straight line. Therefore, in the following description, generation of an approximate curve of the upper eyelid will be described.

FIG. 6 is a schematic diagram illustrating a method of generating a plurality of approximate curves for the upper eyelid.
In general, the approximate curve of the upper eyelid is a curve that passes through the eyelid reference points (coordinates of the corner of the eye and the eye) located on the eyelid and the end point etop of the upper eyelid. That is, if the eyelid reference point and the upper eyelid end point etop are correctly estimated, the approximate curve can be accurately obtained.
However, as described above, the coordinates Ent specified by the change in brightness to detect the upper eyelid are not necessarily located on the upper eyelid, but may be located on the eye shadow or the double eyelid. There is. Here, for the sake of explanation, it is assumed that the coordinates Ent are located on the double eyelid.
Therefore, the approximate curve generation unit 115 generates a plurality of approximate curves Cu (i) shown in FIG. 6 as curves that are candidates for approximate curves that pass through the eyelid reference point and the end point etop of the upper eyelid. Here, i = 1, 2, 3,..., N. N is the number of candidate approximate curves, and is an integer of 2 or more.

Each approximate curve is defined using the three points of the coordinates Cupa (i), the coordinates Cupb, and the coordinates Cupc.
The coordinates Cupa (i) are selected for each curve with respect to the generated curves. For example, a plurality of coordinates Cupa (i) are selected from points on a straight line l passing through the coordinates Enb and the coordinates Ent. Specifically, N points on the straight line l that satisfy Entity ≦ y ≦ Enby are selected and set as a point group P. Then, each point P (i) (iε1, 2,..., N) included in the point group P is set as a coordinate Cupa (i). In addition, what is necessary is just to select each point P (i) so that it may become equal intervals, for example.

As for the coordinates Cupb and the coordinates Cupc, common coordinates are selected among a plurality of generated curves. For example, the coordinates Cupb and the coordinates Cupc are set based on the eyelid reference position data Fn. In this case, the approximate curve generation unit 115 sets coordinates Cupb = coordinate Eni and coordinates Cupc = coordinate Eno.

The x and y coordinates of the coordinates Cupa (i) are (Cupa (i) x, Cupa (i) y), the x and y coordinates of the coordinates Cupb are (Cupbx, Cupby), and the coordinates of Cupc The x coordinate and y coordinate are (Cupcx, Cupcy). Note that Cupbx <Cupa (i) x <Cupcx.

Since the eyelid shape may be asymmetrical depending on the face orientation of the face image input to the device, the approximate curve is the curve Cul (i) of the portion where the x coordinate is smaller than the coordinate Cupa (i). The large curve Cur (i) is set separately, and two curves are set as one curve Cu (i).
For example, a quadratic function is used as the approximate curve. In order to define a quadratic function, for example, a point that is a vertex of the quadratic function and a point through which the quadratic function passes may be specified. In the following description, a quadratic function is described as an approximate curve.

The curve Cul (i) has a coordinate Cupa (i) as a vertex and a coordinate Cupb as a point passing through other than the vertex. When the quadratic function indicating the curve Cul (i) is represented by the following equation (3), each parameter can be obtained by the following equation (4).

Similarly, the curve Cur (i) has a coordinate Cupa (i) as a vertex and a point passing through the coordinate Cupc other than the vertex. When the quadratic function indicating the curve Cur (i) is the following equation (5), each parameter can be obtained by the following equation (6).

Parameters indicating the plurality of curves Cu (i) and the curve Cb generated as described above are given to the shape determining unit 116 as parameter data Gn.
The parameter data Gn includes a set of parameters for each curve. For example, the parameter data Gn includes a “quadratic function” that is a type of curve, “N upper eyelids and one lower eyelid” that are the number of curves, eyelid reference points (coordinates Eno of eye corners and coordinates Eni of eye eyes), and It includes a coordinate Enb, as well as a set of parameters for each curve. In the above example, the set of parameters is the parameters abl (i), bbl (i), cbl (i), acr (i), bcr (i) and ccr (i) of the upper eyelid approximate curve, and This is the eyelid linear parameter. The linear parameter of the lower eyelid is the y coordinate value of one of the eyelid reference point (coordinate of the eye corner Eno and the coordinate of the eye head Eni) and the coordinate Enb. Since it is assumed that the y coordinate of the estimated coordinate Eno of the eye corner and the estimated coordinate Eni of the eye head is equal to the y coordinate of the coordinate Enb, the straight line of the lower eyelid is a straight line parallel to the x axis. It is only necessary that the y-coordinate value of any one of (coordinates Eno of the corner of the eye and coordinates Eni of the top of the eye) and the coordinate Enb is included as a straight line parameter.

The shape determining unit 116 is given parameter data Gn including parameters of a plurality of approximate curves, evaluates pixel values on the approximate curve represented by each parameter, and calculates a shape parameter Hn representing the shape of the upper eyelid and the lower eyelid. . Then, the shape determination unit 116 gives the shape parameter Hn to the eye opening degree calculation unit 130.
For example, the shape determining unit 116 selects the most suitable approximate curve as a boundary line between the driver's eye and the upper eyelid from a plurality of approximate curves indicated by the parameter data Gn, thereby increasing the selected approximate curve. Determine the shape of the eyelid. In the first embodiment, the selected approximate curve is determined as the shape of the upper eyelid. In addition, the shape determining unit 116 determines the shape of the lower eyelid based on the eyelid reference point (coordinate of the eye corner Eno and the coordinate of the eye head Eni) and the coordinate Enb of the second end point, which are linear parameters of the lower eyelid included in the parameter data Gn. To decide. In the first embodiment, the shape of the lower eyelid is determined from the line segment between the coordinates of the corner of the eye Eno and the coordinates of the eye head Eni. Here, the line segment between the coordinate Eno of the corner of the eye and the coordinate Eni of the eye is determined as the shape of the lower eyelid. Then, the shape determination unit 116 calculates a parameter indicating the shape of the driver's eye from the determined shape of the upper eyelid and the shape of the lower eyelid, and calculates a shape parameter Hn indicating the calculated parameter as the degree of eye opening. Part 130.

The shape parameter Hn is preferably, for example, a value ep (= eh / ew) of a ratio between the distance ew between the corners of the eyes and the eyes in the normalized image and the distance eh between the end point etop of the upper eyelid and the end point ebot of the lower eyelid. Alternatively, the shape parameter Hn may be a parameter that specifies one approximate curve parameter for each of the upper eyelid and the lower eyelid. Further, the shape parameter Hn may include the coordinates of the eyelid reference point, the coordinates of the upper eyelid end point etop, and the coordinates of the lower eyelid end point ebot, instead of the approximate curve parameters. In addition, the shape parameter Hn may represent one curvature of each of the approximate curves of the upper eyelid and the lower eyelid.
In the following description, the shape parameter Hn is a ratio value ep between the distance ew between the corners of the eyes and the eyes and the distance eh between the end point etop of the upper eyelid and the end point ebot of the lower eyelid.

In order to specify the shape of the eyelid, it is necessary to select an approximate curve that is likely to be an eyelid from a plurality of approximate curves, one for the upper eyelid and one for the lower eyelid. In the present embodiment, since the curve of the boundary line between the eye and the lower eyelid has been uniquely set, one curve is selected from a plurality of approximate curves generated for the upper eyelid.

When selecting one approximate curve from a plurality of approximate curves generated for the upper eyelid, an approximate curve that passes on the boundary between the eye and the upper eyelid, and an approximate curve that passes over the eye shadow or the double eyelid The difference is explained.
Near the upper eyelid, there is a step with the eyeball and the hairline of the eyelashes, so the luminance value of the pixel on the boundary line between the eye and the upper eyelid is compared with the luminance value of the pixel on the eye shadow or double eyelid Tend to be smaller. This tendency is the same when comparing the upper and lower eyelids. The luminance value of the pixel on the boundary line between the eye and the upper eyelid is the luminance value of the pixel on the boundary line between the eye and the lower eyelid. There is a tendency to be smaller than the value.

Using this tendency, the shape determining unit 116 obtains the average value of the luminance values of the pixels on each approximate curve among the multiple approximate curves generated for the upper eyelid, and obtains the curve having the lowest average value. By selecting, the most suitable approximate curve can be selected as the boundary line between the eye and the upper eyelid.
As described above, the shape determination unit 116 selects a curve Cu (iu) that is likely to be the upper eyelid from the approximate curves, and sets the coordinates Cupa (iu) as the end point etop of the upper eyelid. The end point ebot of the lower eyelid is set to the end point of the lower eyelid curve Cb, that is, the coordinate Enb.

The shape determining unit 116 uses the distance value ep = eh / ew between the distance ew = Nw between the corners of the eyes and the eyes and the distance eh = Nd between the end points of the upper eyelid and the lower eyelid as the shape parameter Hn. This is given to the calculation unit 130. The value Nd can be obtained from the following equation (7).

The eye opening degree calculation unit 130 compares the given shape parameter Hn with the reference shape parameter Hs stored in the memory 131 in advance, and calculates the eye opening degree Kn of the driver.
Here, the degree of eye opening Kn indicates the degree of opening of the driver's eyes. For example, the degree of opening of the eyes in a normal state is preferably represented by a value from 0% to 100%, assuming that the degree of eye opening is 100%.

In the memory 131, as a reference shape parameter Hs, a ratio value eps = ehs / ews between the distance between the eye corners and the eye distance ews in the steady state of the driver and the distance ehs between the end point etop of the upper eyelid and the end point ebot of the lower eyelid. keeping.
The eye opening degree calculation unit 130 compares the reference shape parameter Hs and the given shape parameter Hn, and calculates the eye opening degree Kn by the following equation (8).
Kn = 100 × ep / eps (8)

In addition, although the eyelid shape detection apparatus 110 in Embodiment 1 assumes that the boundary line between the eye and the lower eyelid is a straight line, it may be assumed to be a curved line. In that case, the eyelid reference position estimation unit 114 calculates the coordinates of the corner of the eye and the eye using, for example, the y coordinate Ngy of the center coordinate Ng of the normalized image instead of the coordinate Enb. Alternatively, in the calculation of the eye presence area, when the eye presence area specifying unit 111 uses the above-described AAM, the coordinates of the feature points of the corner of the eye and the eye can be obtained. Therefore, the eyelid reference position estimation unit 114 determines the coordinates. Used as the coordinates of the corner of the eye and the eye.
Further, when the approximate curve generation unit 115 selects a plurality of points through which the approximate curve passes from the straight line l, the upper eyelid is selected in the range of Enty ≦ y ≦ Eny, and the lower eyelid is selected from Anyy ≦ y ≦ Enby. By selecting, a plurality of approximate curves for the upper eyelid and a plurality of approximate curves for the lower eyelid are generated.
Then, the shape determining unit 116 determines the most suitable approximate curve as the boundary line between the eye and the upper eyelid from the plurality of approximate curves for the upper eyelid in the same manner as described above, From the approximate curve, an approximate curve most suitable as a boundary line between the eye and the upper eyelid may be determined.
However, since the lower eyelid is less likely to have a difference between the curves with respect to the evaluation value of the luminance value on the approximate curve compared to the upper eyelid, the shape determining unit 116 shows a curve passing through the coordinates Eni, the coordinates Eno, and the coordinates Enb. It may be selected as an approximate curve.

In the eyelid shape detection apparatus 110 according to the first embodiment, the light / dark change detection unit 113 performs the filter processing for extracting the light / dark change for all the pixels included in the eye presence region. There is no need to implement it.
For example, the light / dark change detection unit 113 may perform the filtering process only on the points included in the normalized image that satisfy the conditions of x = Ngx, Ngy−Nh / 4 ≦ y ≦ Ngy + Nh / 4. When the face image given to the eyelid shape detection device 110 is a frontal face, the upper and lower end points etop and ebot of the eyelid are basically straight lines passing through the center point between the corner of the eye and the eye and parallel to the y axis. It can be assumed that it is located above. Therefore, the brightness change detection unit 113 can obtain the estimated coordinates Ent and Enb of the upper and lower eyelid points simply by searching for a point that satisfies the above conditions. In this way, the processing time can be shortened by limiting the number of pixels on which the filter processing is performed.

In addition, the pixel to be filtered is not limited to a straight line that passes through the center point between the corner of the eye and the top of the eye and is parallel to the y-axis, but is within a range of Ngx−Nw / 4 ≦ x ≦ Ngx + Nw / 4. In this case, a plurality of straight lines parallel to the y-axis may be selected, and the light / dark change detection unit 113 may perform filter processing on pixels on the straight lines. By performing filter processing on a plurality of straight lines, even when the input face image is not a front image, the processing time can be shortened while accurately calculating the coordinates Ent and Enb.

FIG. 7 is a flowchart showing processing in the eye opening degree detection apparatus 100 according to the first embodiment.
The flowchart illustrated in FIG. 7 is started, for example, when the face image data Im is input to the eye opening degree detection device 100.

The eye presence region specifying unit 111 specifies the eye presence region from the face image represented by the face image data Im, and generates eye presence region data Dm indicating the specified eye presence region (S10). Then, the eye presence region specifying unit 111 provides the generated eye presence region data Dm to the normalization processing unit 112 together with the face image data Im.

The normalization processing unit 112 cuts out the eye presence area from the face image represented by the face image data Im based on the eye presence area data Dm, and converts the cut out image into a predetermined image size. Normalized image data In representing the normalized image after conversion is generated, and normalized eye presence region data Dn indicating the eye presence region in the normalized image data In is generated (S11). Then, the normalization processing unit 112 gives the normalized image data In and the normalized eye presence region data Dn to the light / dark change detection unit 113.

The light / dark change detection unit 113 calculates the magnitude of the light / dark change in the vertical direction (y-coordinate direction) of the normalized image data In, and generates light / dark change data En (S12). Then, the light / dark change detection unit 113 gives the light / dark change data En to the eyelid reference position estimation unit 114, and gives the light / dark change data En and the normalized image data In to the approximate curve generation unit 115.

The eyelid reference position estimation unit 114 calculates at least two reference point positions existing on the eyelid based on the light / dark change data En, and obtains coordinate data Fn indicating coordinates of the calculated reference points on the normalized image. Generate (S13). Then, the eyelid reference position estimation unit 114 gives the coordinate data Fn to the approximate curve generation unit 115.

The approximate curve generation unit 115 generates a plurality of approximate curves for the eyelid from the normalized image data In, the light / dark change data En, and the eyelid reference position data Fn, and includes parameters of each of the plurality of approximate curves. Parameter data Gn is generated (S14). And it outputs to the shape determination part 116.

The shape determining unit 116 calculates feature amounts of pixel values on a plurality of approximate curves indicated by a plurality of parameters included in the parameter data Gn (S15).

The shape determination unit 116 compares and evaluates the feature values of the approximate curves, and selects the approximate curve that seems to be the most eyelid (S16).

The shape determining unit 116 calculates a shape parameter Hn representing the shape of the upper eyelid and the lower eyelid from the selected approximate curve (S17). Then, the shape determination unit 116 gives the calculated shape parameter Hn to the eye opening degree calculation unit 130.

The eye opening degree calculation unit 130 compares the shape parameter Hn with the reference shape parameter Hs stored in advance in the memory 131, and calculates the driver's eye opening degree Kn (S18).

The eyelid shape detection apparatus 110 according to the first embodiment configured as described above estimates the eyelid reference position from a point with a large change in brightness, and is an area defined by the point with a large change in brightness and the estimated reference position. Detecting the shape of the eyelid accurately even when edges cannot be detected in low-quality face images by finding multiple approximate curves from the points included in the image and comparing and evaluating the feature values of the pixel values on the multiple approximate curves can do.

The eyelid shape detection apparatus 110 according to the first embodiment configured as described above opens even a low-quality face image by using a vertically asymmetric filter to extract changes in brightness of the eyelid. The position of the eyelid can be estimated regardless of the eye state such as being present or closed. For this reason, the shape of the eyelid can be detected with high accuracy.

The eyelid shape detection apparatus 110 according to the first embodiment configured as described above accurately sets an eyelid shape approximate curve separately on the left and right sides with respect to the reference point, so that the eyelid shape can be accurately detected even for the left and right asymmetric eyelid shapes. The eyelid shape can be detected.

The eyelid shape detection apparatus 110 according to Embodiment 1 configured as described above compares and evaluates the average value of the luminance values of the pixels with respect to a plurality of generated approximate curves, and the average value of the luminance values is the highest. By selecting a small curve as an approximate curve that passes through the upper eyelid, even if the person in the face image has an eye shadow, or the person in the face image has a double eyelid, the eyelid shape can be accurately determined. Can be detected.

The eye-opening degree detection apparatus 100 according to Embodiment 1 configured as described above detects an eyelid shape after converting a high-quality face image into a low-quality face image even for a high-quality face image. Thus, the eyelid shape can be accurately detected with a small amount of processing.

Since the eye-opening degree detection apparatus 100 according to Embodiment 1 configured as described above can accurately detect an eyelid shape with respect to a low-quality face image, the degree of freedom regarding the camera installation position is high. For example, even if the camera is placed at a position away from the driver, such as near the display of a car navigation system, and the face image has low image quality, the eye-opening degree detection device 100 can detect the eyelid shape with high accuracy. In addition, a camera equipped with a wide-angle lens (for example, 90 ° to 150 °) is used to simultaneously capture the faces of the driver and the passenger on the front passenger seat. The eye opening degree detection device 100 can detect the eyelid shape with high accuracy. Furthermore, even when a camera is placed near the driver, such as an A-pillar and a cluster meter, and a high-quality face image is acquired, the eye-opening degree detection device 100 converts the eyelid shape after converting it to a low-quality face image. By detecting the eyelid shape, the eyelid shape can be detected accurately with a small amount of processing.

The eye-opening degree detection apparatus 100 according to Embodiment 1 configured as described above estimates the eyelid reference position from a point with a large change in brightness, and is an area defined by the point with a large change in brightness and the estimated reference position. Detecting the shape of the eyelid accurately even when edges cannot be detected in low-quality face images by finding multiple approximate curves from the points included in the image and comparing and evaluating the feature values of the pixel values on the multiple approximate curves can do. Furthermore, the eye-opening degree detection device 100 can accurately detect the degree of eye opening for a low-quality face image by comparing the shape of the detected eyelid with a shape registered in advance.
In the above embodiment, a straight line connecting the first end point, which is the highest point of the first boundary line, and the second end point, which is the lowest point of the second boundary line, is used. The straight line may be shifted to some extent in the x-axis direction. This allowable “some degree” refers to a range of about one-fifth of the distance between the corners of the eyes and the corners of the eyes, for example, centering on the center point between the corners of the corners of the eyes.

Embodiment 2. FIG.
As shown in FIG. 1, the eye opening degree detection apparatus 200 according to Embodiment 2 includes an eyelid shape detection apparatus 210 and an eye opening degree calculation unit 130.
The eye opening degree detection apparatus 200 according to the second embodiment is configured in the same manner as the eye opening degree detection apparatus 100 according to the first embodiment except for the eyelid shape detection apparatus 210.

The eyelid shape detection apparatus 210 according to the second embodiment includes an eye presence region specifying unit 111, a normalization processing unit 112, a light / dark change detection unit 213, an eyelid reference position estimation unit 114, an approximate curve generation unit 215, and a shape. A determination unit 216.
The eyelid shape detection device 210 according to the second embodiment is configured in the same manner as the eyelid shape detection device 110 according to the first embodiment, except for the light / dark change detection unit 213, the approximate curve generation unit 215, and the shape determination unit 216.

When generating an approximated curve, eye opening degree detection apparatus 100 according to Embodiment 1 selects a plurality of points from the points satisfying Enby ≦ y ≦ Enty on line l connecting coordinates Ent and Enb. An approximate curve is generated. The eye-opening degree detection apparatus 200 according to Embodiment 2 reduces the processing time by limiting the range of points used for the approximate curve.

The light / dark change detection unit 213 generates the light / dark change data En by calculating the magnitude of the change in the vertical brightness of the face image with respect to the normalized image data In. Then, the light / dark change detection unit 213 supplies the light / dark change data En to the eyelid reference position estimation unit 114 and the approximate curve generation unit 215 together with the normalized image data In.
The light / dark change data En includes the coordinate data of the coordinates Ent and the coordinate Enb, the region where the light / dark change (bright to dark) is large (first selection region) Rnt, and the region where the light / dark change (dark to light) is large (second selection). Area) Rnb.

The region Rnt and the region Rnb are determined according to the evaluation value of the filter. Specifically, the light / dark change detection unit 213 uses the threshold value THtf for the filter Etf and the threshold value THbf for the filter Ebf to set a region where the evaluation value of the filter Etf is equal to or greater than the threshold value THtf as the region Rnt, and the evaluation value of the filter Ebf is the threshold value. A region equal to or greater than THbf is defined as a region Rnb. At this time, the region Rnt and the region Rnb are pixels on a straight line l connecting the coordinate Ent of the point where the light / dark change (bright to dark) is maximum and the coordinate Enb of the point where the light / dark change (dark to light) is the maximum. Ask for.

8A and 8B are schematic diagrams illustrating a method for calculating the region Rnt.
FIG. 8A shows the evaluation value of the filter Etf and the straight line 1 in each image near the coordinate Ent.
Here, the threshold value THtf = 100, and the pixels belonging to the region Rnt in which the evaluation value of the filter Etf is equal to or greater than the threshold value THtf are hatched or cross-hatched.

The light / dark change detection unit 213 starts from the coordinate Ent and first scans on the straight line l in the y-axis direction (downward in the figure). Then, as illustrated in FIG. 8B, the light / dark change detection unit 213 detects the pixel scanned immediately before the pixel for which the evaluation value of the filter Etf is less than the threshold value THtf for the first time as the pixel Rntb at the lower end of the region Rnt. To do. In the example shown in FIG. 8B, when the y-axis direction is scanned along the straight line 1 from the coordinate Ent, the evaluation values of the filter Etf are obtained in the order of 150, 124 and 46. At this time, the threshold value THtf is not reached until the evaluation value has shifted from 124 to 46. Therefore, the light / dark change detection unit 213 sets the pixel having the evaluation value 124 of the filter Etf as the pixel Rntb at the lower end of the region Rnt.

Subsequently, the light / dark change detection unit 213 scans on the straight line l in the reverse direction of the y-axis (upward in the figure). Then, the light / dark change detection unit 213 sets a pixel scanned immediately before a pixel whose evaluation value of the filter Etf is lower than the threshold value THtf for the first time as a pixel Rntt at the upper end of the region Rnt. In the example of FIG. 8, when the scanning is performed in the y-axis reverse direction along the straight line 1 from the coordinate Ent, the evaluation values of the filter Etf are obtained as 100, 90, and 31, respectively. At this time, the threshold value Thhtf is not reached until the output value has shifted from 100 to 90. Therefore, a pixel having an evaluation value of 100 for the filter Etf is set as a pixel Rntt at the upper end of the region Rnb.

Similarly, the light / dark change detection unit 213 obtains a pixel Rnbt at the upper end and a pixel Rnbb at the lower end of the region Rnb based on the coordinates Enb and the threshold value THbf.

It is assumed that the region Rnt is indicated by, for example, the y coordinate of the coordinates corresponding to the pixel Rntt and the pixel Rntb. Similarly, the region Rnb is indicated by, for example, the y coordinate of the coordinates corresponding to the pixel Rnbt and the pixel Rnbb.

The threshold value THtf and the threshold value THbf for defining the region Rnt and the region Rnb may be defined in advance as constants, or may be obtained using the maximum value and the minimum value of the evaluation values of the filter Etf and the filter Ebf. Good.
For example, the threshold value THtf and the threshold value THbf may be obtained from the following equations (9) and (10).
THtf = α × (FntMax−FntMin) + FntMin (9)
THbf = β × (FnbMax−FnbMin) + FnbMin
(10)
Here, the value FntMax is the maximum value of the evaluation value when the pixel in the eye presence region is processed by the filter Etf. The value FntMin is its minimum value.
Similarly, the value FnbMax is the maximum evaluation value when the pixel in the eye presence region is processed by the filter Ebf. The value FnbMin is its minimum value.
Further, α and β are coefficients for determining whether a change in brightness is large if the value is greater than what percentage of the maximum evaluation value of each filter, and is assumed to be predetermined. For example, it is assumed that α = β = 0.5 is set.

The approximate curve generation unit 215 generates a plurality of approximate curves for the eyelid from the normalized image data In, the light / dark change data En, and the eyelid reference position data Fn, and sets parameters indicating each of the plurality of approximate curves. Parameter data Gn including the set is given to the shape determining unit 216.

When selecting the coordinates Cupa (i), the approximate curve generation unit 215 generates an approximate curve by selecting from the region Rnt for the upper eyelid and from the region Rnb for the lower eyelid. By generating the approximate curve so as to pass through the region where the change in brightness is large as described above, the approximate curve generation unit 215 can reduce the number of approximate curves as compared with the first embodiment, and the processing time can be reduced. Can be shortened.

As in the first embodiment, the shape determining unit 216 determines the most suitable approximate curve as a boundary line between the eye and the upper eyelid from a plurality of approximate curves (first approximate curve) corresponding to the upper eyelid. select.
In addition, the shape determining unit 216 selects the most suitable approximate curve as a boundary line between the eye and the lower eyelid from a plurality of approximate curves (second approximate curve) corresponding to the lower eyelid. The method of generating and selecting the approximate curve is the same as in the case of the upper eyelid.
Then, from the determined shape of the upper eyelid and the shape of the lower eyelid, the shape determining unit 216 determines the distance ew = Nw between the corner of the eye and the eye, the end points of the upper eyelid and the end points of the lower eyelid, as in the first embodiment. The ratio value ep = eh / ew to the distance eh = Nd is given to the eye opening degree calculation unit 130 as the shape parameter Hn.

Since the eyelid shape detection apparatus 210 according to the second embodiment generates an approximate curve so as to pass through a region having a large change in brightness, the shape of the eyelid can be detected in a short processing time.

Embodiment 3 FIG.
As shown in FIG. 1, an eye opening degree detection apparatus 300 according to Embodiment 3 includes an eyelid shape detection apparatus 310 and an eye opening degree calculation unit 130.
The eye opening degree detection apparatus 300 according to the third embodiment is configured in the same manner as the eye opening degree detection apparatus 100 according to the first embodiment, except for the eyelid shape detection apparatus 310.

The eyelid shape detection apparatus 310 according to the third embodiment includes an eye presence region specifying unit 111, a normalization processing unit 112, a light / dark change detection unit 313, an eyelid reference position estimation unit 114, an approximate curve generation unit 115, a shape And a determination unit 116.
The eyelid shape detection device 310 in the third embodiment is configured in the same manner as the eyelid shape detection device 110 in the first embodiment, except for the light / dark change detection unit 313.

The eye-opening degree detection apparatus 100 according to Embodiment 1 obtains two points (bright to dark, dark to bright) of points having a large light / dark change in the eye presence region, and generates light / dark change data En. In this case, since the filtering process is necessary for all the pixels in the eye presence region, the processing load is large. The eye-opening degree detection apparatus 200 according to Embodiment 2 sets a plurality of straight lines in the eye presence region, and obtains two points having a large contrast change on each straight line. Then, the eye opening degree detection device 200 calculates the distance between the two points obtained on each straight line, and generates the light / dark change data En indicating the coordinates of the two points having the largest distance. Thereby, the pixels to be filtered can be limited to pixels on a straight line, and eyelid shape detection robust to changes in eyelid shape due to face orientation changes and individual differences can be performed.

The light / dark change detection unit 313 generates the light / dark change data En by calculating the magnitude of the change in brightness in the vertical direction of the normalized image represented by the normalized image data In. For example, the brightness change detection unit 313 sets a plurality of predetermined straight lines extending in the vertical direction in the normalized image, and reduces brightness from the upper pixel on each straight line included in the plurality of straight lines. The distance between the coordinates of the first candidate pixel having the largest degree of the brightness and the coordinates of the second candidate pixel having the greatest degree of brightness reduction from the lower pixel is calculated. Then, the light / dark change detection unit 313 identifies the coordinates of the first candidate pixel and the second candidate pixel having the largest calculated distance, and the coordinates of the identified first candidate pixel are the coordinates of the first end point Ent, The coordinates of the specified second candidate pixel are set as the coordinates Enb of the second end point.
Then, the light / dark change detecting unit 313 supplies the light / dark change data En to the eyelid reference position estimating unit 114 and the approximate curve generating unit 115 together with the normalized image data In.

The light / dark change detection unit 313 sets a plurality of straight lines on the normalized image represented by the normalized image data In. Here, in order to simplify the description, a case where three straight lines parallel to the y-axis are set will be described as an example. The straight line l (1) is x = Ngx−Nw / 4, the straight line l (2) is x = Ngx, and the straight line l (3) is x = Ngx + Nw / 4.

First, the light / dark change detection unit 313 executes processing of the filter Etf and the filter Ebf along each straight line. Next, the point where the evaluation value of the filter becomes maximum is obtained for each straight line. Here, the coordinates of the point at which the evaluation value of the filter Etf is maximum on the straight line l (k) (kε {1, 2, 3}) are the coordinates of the first candidate pixel Ent (k) and the evaluation value of the filter Ebf. The coordinate of the point at which is the maximum is defined as the coordinate Enf (k) of the second candidate pixel.

Next, the brightness change detection unit 313 calculates a distance d (k) between the coordinates Ent (k) and the coordinates Enb (k) on each straight line. For the distance, for example, the Euclidean distance is used. Here, since each straight line is parallel to the y-axis, d (k) = Enby (k) −Enty (k). Here, the value Enby (k) is the y coordinate of the coordinate Enb (k), and the value Enty (k) is the y coordinate of the coordinate Ent (k).

The light / dark change detecting unit 313 selects the maximum distance from the obtained distances d (k) (k corresponding to the selected distance is K), and coordinates Ent (K) and Enb ( Brightness / darkness change data En indicating K) is generated.

In the third embodiment, the number of straight lines is set to three, but the number is not particularly limited. Further, although a straight line parallel to the y-axis is selected as the straight line, the present invention is not limited to this, and an arbitrary straight line may be set.

Since the eyelid shape detection apparatus 310 according to Embodiment 3 configured as described above can limit the pixels to be filtered to pixels on a straight line, the processing time can be shortened. Further, by selecting the two points where the distance between the two kinds of points with the large change in brightness is the largest, two points close to the upper end point and the lower end point of the eyelid can be selected. The shape of the approximate curve to be generated is a shape close to the original eyelid shape. For this reason, the eyelid shape can be detected with higher accuracy.

Embodiment 4 FIG.
FIG. 9 is a block diagram schematically showing the configuration of the eye opening degree detection apparatus 400 according to the fourth embodiment.
The eye opening degree detection device 400 includes an eyelid shape detection device 410 and an eye opening degree calculation unit 130. The eye opening degree detection apparatus 400 according to the fourth embodiment is configured in the same manner as the eye opening degree detection apparatus 100 according to the first embodiment except for the eyelid shape detection apparatus 410.

The eyelid shape detection device 410 according to the fourth embodiment includes an eye presence region specifying unit 111, a normalization processing unit 112, a light / dark change detection unit 413, an eyelid reference position estimation unit 114, an approximate curve generation unit 115, a shape, And a determination unit 116.
The eyelid shape detection device 410 according to the fourth embodiment is configured in the same manner as the eyelid shape detection device 110 according to the first embodiment except for the light / dark change detection unit 413. Further, the face direction information Hd indicating the direction of the driver's face is given from the outside to the light / dark change detection unit 413 in the fourth embodiment.

The eye-opening degree detection apparatus 300 according to Embodiment 3 calculates the distance between two types of points with a large change in brightness in a plurality of straight lines on the normalized image, and uses the two points on the straight line with the longest distance. The pixels to which the filter process is applied are limited. The eye-opening degree detection apparatus 400 according to Embodiment 4 uses the face orientation information Hd to identify a straight line that should be used for extraction of a light / dark change from a plurality of straight lines, thereby further reducing processing time. I do.

FIG. 10 is a schematic diagram illustrating an example of an image near the eye when the driver is facing sideways.
Unlike the example of the image near the eye shown in FIG. 2 when the driver is facing the front, the coordinates of the upper end point of the eyelid to be detected when the driver is facing the lateral direction There is a tendency that the Ent and the coordinate Enb of the lower end point are closer to the left side of the eye presence region. Such a bias in the lateral direction (x-axis direction) of the coordinate Ent of the upper end point and the coordinate Enb of the lower end point of the eyelid is mainly caused by a change in the face direction in the horizontal direction. Therefore, the light / dark change detection unit 413 determines a straight line from which the light / dark change is extracted based on the face orientation information Hd, thereby filtering a plurality of straight lines like the light / dark change detecting unit 313 in the third embodiment. There is no need to perform processing.

The light / dark change detection unit 413 generates light / dark change data En by calculating the magnitude of the change in brightness in the vertical direction of the normalized image represented by the normalized image data In. For example, the light / dark change detection unit 413 sets a plurality of predetermined straight lines extending in the vertical direction in the normalized image, and selects one straight line from the plurality of straight lines based on the orientation of the person's face. Then, the light / dark change detection unit 413 sets, on the selected straight line, the coordinates of the pixel with the greatest degree of brightness reduction from the upper pixel as the first endpoint coordinate Ent, and the brightness of the lower pixel. The coordinate of the pixel with the greatest degree of decrease is taken as the coordinate Enb of the second end point.
Then, the light / dark change detection unit 413 supplies the light / dark change data En to the eyelid reference position estimating unit 114 and the approximate curve generating unit 115 together with the normalized image data In.

The light / dark change detection unit 413 sets a plurality of straight lines in the normalized image represented by the normalized image data In. Here, in order to simplify the description, a case where three straight lines parallel to the y-axis are set will be described as an example. The straight line l (1) is x = Ngx−Nw / 4, the straight line l (2) is x = Ngx, and the straight line l (3) is x = Ngx + Nw / 4.

Then, the light / dark change detection unit 413 selects a straight line to be filtered based on the horizontal face orientation information Hd. Here, the face direction information Hd is the direction of the driver's face, the front direction is 0 degree, the right direction is a positive rotation angle, and the left direction is a negative direction. It shall be shown with the rotation angle. FIG. 10 shows a state in which it faces rightward, that is, in a positive direction.

For example, the light / dark change detection unit 413 provides a threshold THfdp and a threshold THfdn (THfdp> THfdn) with respect to the face direction (rotation angle) indicated by the face direction information Hd, and a straight line l (if Hd ≧ THfdp) 1) is selected. If THfdp <Hd <THfdn, the straight line l (2) is selected, and if Hd ≦ Thfdn, the straight line l (3) is selected.

Then, the light / dark change detection unit 413 performs filter processing on the selected straight line, and generates light / dark change data En indicating the coordinates of the two types of large light-dark changes (bright to dark, dark to light).

The eyelid shape detection apparatus 410 according to the fourth embodiment can select a straight line that should be used for extraction of light and dark changes from a plurality of straight lines by using the face orientation information Hd, and therefore the number of times of filter processing is reduced. The processing time can be shortened.

Embodiment 5 FIG.
FIG. 11 is a block diagram schematically showing a configuration of eye opening degree detection apparatus 500 according to the fifth embodiment.
The eye opening degree detection device 500 includes an eyelid shape detection device 510 and an eye opening degree calculation unit 130. Eye opening degree detection apparatus 500 according to Embodiment 5 is configured in the same manner as eye opening degree detection apparatus 100 according to Embodiment 1 except for eyelid shape detection apparatus 510.

The eyelid shape detection apparatus 510 according to the fifth embodiment includes an eye presence region specifying unit 111, a normalization processing unit 112, a light / dark change detection unit 113, an eyelid reference position estimation unit 114, an approximate curve generation unit 115, an approximation A curve selection unit 517, a correction curve generation unit 518, and a shape determination unit 516 are provided.
The eyelid shape detection apparatus 510 according to the fifth embodiment is the same as the first embodiment except that the processing by the shape determination unit 516 and the approximate curve selection unit 517 and the correction curve generation unit 518 are further provided. It is comprised similarly to the eyelid shape detection apparatus 110 in FIG.

The approximate curve selection unit 517 evaluates the pixel value on the approximate curve represented by each parameter from the parameter data Gn including the parameters of the plurality of approximate curves, by the same processing as the shape determination unit 116 in the first embodiment. The coordinates of the end point etop of the eyelid and the end point ebot of the lower eyelid are specified. Then, the approximate curve selection unit 517 generates the shape parameter data On indicating the coordinates Cupa (iu) of the end point etop of the upper eyelid, the coordinates Enb of the end point ebot of the lower eyelid, the coordinates Eno of the eye corners, and the coordinates Eni of the eye head, This is given to the correction curve generation unit 518. Here, the end point etop of the upper eyelid is also referred to as a fixed end point.

The correction curve generation unit 518 generates a plurality of correction curves that are approximate curves for correcting the positions of the corners of the eyes and the eyes from the given shape parameter data On, and sets each parameter of the generated plurality of correction curves. The correction curve parameter data Pn that is included is provided to the shape determination unit 516.

The operation of the correction curve generation unit 518 will be described in detail.
FIG. 12 is a schematic diagram for explaining generation of a correction curve for the upper eyelid.
The approximate curve selection unit 517 calculates the coordinates Cupa (iu) of the end point etop of the upper eyelid and the coordinates Enb of the end point ebot of the lower eyelid. However, the coordinates Eni of the eyelid and the coordinate Eno of the eyelid are estimated based on the coordinate Enb. Value, and lacks accuracy. Therefore, the correction curve generation unit 518 generates a correction curve that is an approximate curve for correcting the coordinate Eno and the coordinate Eno.

Each correction curve is defined using three points of coordinates Pupa, Pupb (i), and Pupc (i) passing through the curve. The coordinate Pupb (i) is also referred to as a first selected coordinate, and the coordinate Pupc (i) is also referred to as a second selected coordinate.
The coordinate Pupa is a common point among a plurality of generated curves. Here, as the coordinates Pupa, the coordinates Cupa (iu) of the end point etop of the upper eyelid are used.
The coordinates Pupb (i) and the coordinates Pupc (i) are selected for each curve with respect to a plurality of generated curves. Note that the x and y coordinates of the coordinates Pupa are (Pupax, Pupay) = (Cupa (iu) x, Cupa (iu) y). The x and y coordinates of the coordinates Pupb (i) are (Pupb (i) x, Pupb (i) y). The x and y coordinates of the coordinate Pupc (i) are (Pupc (i) x, Pupc (i) y). Note that Pupbx (i) <Pupax <Pupcx (i).

A plurality of coordinates Pupb (i) are selected from points on the straight line l (1) passing through the coordinates Eni. Specifically, M (M is an integer of 2 or more) points satisfying Any-mgin ≦ y ≦ Eny + mgin are selected on the straight line l (1), and the selected point is defined as a point group Pin. Then, each point Pin (i) (iε1, 2,..., M) included in the point group Pin is set as Pupb (i). The value mgin is a predetermined constant. The straight line l (1) is a straight line represented by x = Enix, for example. Note that the points Pin (i) may be selected so as to be equally spaced.

A plurality of coordinates Pupc (i) are selected from points on the straight line l (3) passing through the coordinates Eno. Specifically, M points that satisfy Enoy−mgout ≦ y ≦ Enoy + mgout are selected on the straight line l (3), and the selected point is set as a point group Pout. Then, each point Pout (i) (i∈1, 2,..., M) included in the point group Pout is set as Pout (i). The value mgout is a predetermined constant. The straight line l (3) is a straight line represented by x = Enox, for example. In addition, each point Pout (i) should just be selected so that it may become equal intervals.

The coordinate Pupa is set using the coordinate value of the end point etop, and the x coordinate Pupax = etopx of the coordinate Pupa and its y coordinate Pupay = etopy.

The eyelid shape may be asymmetrical depending on the face orientation of the face image input to the device. Therefore, one correction curve Pu (i) has two curves, a curve Pu (i) where the x coordinate is smaller than the x coordinate of the coordinate Pupa and a curve Pur (i) where the x coordinate is large. Consists of.
For example, a quadratic function is used as the correction curve. For example, the quadratic function can be uniquely defined by specifying a point that is a vertex of the quadratic function and a point through which the quadratic function passes. In the following description, a quadratic function is described as a correction curve.

The curve Pul (i) has a coordinate Pupa as a vertex and a coordinate Pupb (i) as a point passing through other than the vertex.
When the quadratic function indicating the curve Pul (i) is the following equation (11), each parameter can be obtained by the following equation (12).

Similarly, the curve Pur (i) has a coordinate Pupa as a vertex and a coordinate Pupc (i) as a point passing through other than the vertex.
When the quadratic function indicating the curve Pur (i) is represented by the following equation (13), each parameter can be obtained by the following equation (14).

The parameters indicating the plurality of curves Pu (i) and the curve Cb generated as described above are given to the shape determination unit 516 as the correction curve parameter data Pn.
The correction curve parameter data Pn includes a set of parameters for each curve. For example, the correction curve parameter data Pn includes a “quadratic function” that is a type of curve, “M upper eyelid and one lower eyelid” that are the number of curves, eyelid reference points (coordinates Eno of eye corners and coordinates Eni of eye eyes) ) And coordinates Enb, and a set of parameters for each curve. In the above example, the set of parameters is the upper eyelid correction curve parameters abl (i), bbl (i), cbl (i), acr (i), bcr (i) and ccr (i), and This is the eyelid linear parameter. The linear parameter of the lower eyelid is the y coordinate value of one of the eyelid reference point (coordinate of the eye corner Eno and the coordinate of the eye head Eni) and the coordinate Enb.

The shape determining unit 516 shown in FIG. 11 is given correction curve parameter data Pn including a plurality of correction curve parameters, evaluates pixel values on the correction curve represented by each parameter, and determines the upper eyelid and lower eyelid. A shape parameter Hn representing the shape is calculated. Then, the shape determination unit 516 gives the shape parameter Hn to the eye opening degree calculation unit 130.

In order to specify the shape of the eyelid, it is necessary to select a correction curve that is likely to be an eyelid from a plurality of correction curves given as inputs, one for each of the upper eyelid and the lower eyelid. In the present embodiment, since the lower eyelid curve is uniquely set, one correction most suitable as a boundary line between the eye and the upper eyelid is obtained from a plurality of correction curves generated for the upper eyelid. A curve is selected.

Similar to the shape determining unit 116 in the first embodiment, the shape determining unit 516 obtains the average value of the luminance values of the pixels on the correction curve in each of the plurality of correction curves, and selects the curve having the lowest average value. This selects the correction curve that passes through the upper eyelid.
As described above, the shape determining unit 516 selects the most likely curve Pu (iu) as the upper eyelid from among the correction curves, and uses the coordinates Pupb (iu) as the coordinates of the head and Pupc (iu) as the coordinates of the corner of the eye. .

Here, the x-coordinate value of the eye coordinates Pupb (iu) is the x-coordinate value of the estimated eye coordinates Eni, and the y-coordinate value of the eye coordinates Pupb (iu) is the correction curve Pu (iu). Can be obtained by substituting the value of the x coordinate of the estimated coordinate Eni of the eye.
Further, the x coordinate value of the corner coordinates Pupc (iu) is the x coordinate value of the estimated corner coordinates Eno, and the y coordinate value of the corner coordinates Pupb (iu) is represented by the correction curve Pu (iu). It can be obtained by substituting the value of the x coordinate of the estimated coordinate Eno of the corner of the eye.
Further, the value of the x coordinate of the coordinate Pupa is the parameter bcr (i) or the parameter bbl (i), and the value of the y coordinate of the coordinate Pupa is the parameter cbl (i) or the parameter ccr (i).

Then, the shape determining unit 516 uses the value ep = eh / ew, which is the ratio of the distance ew = Lw between the corner of the eye and the corner of the eye and the distance eh = Ld between the end point of the upper eyelid and the end point of the lower eyelid, as the shape parameter Hn. This is given to the degree calculation unit 130. The value Lw can be obtained by the following equation (15). The value Ld can be obtained from the following equation (16).

As described above, in the eyelid shape detection apparatus 510 according to Embodiment 5, a plurality of correction curves are generated again from the end points of the upper eyelid and the end points of the lower eyelid calculated by using the approximate curve, and the positions of the corners of the eyes and the eyes are corrected. This makes it possible to detect the eyelid shape with high accuracy.

Embodiment 6 FIG.
FIG. 13 is a block diagram schematically showing the configuration of the dozing level detection apparatus 600 according to the sixth embodiment.
The dozing level detection device 600 includes eye opening degree detection devices 100 to 500 and a dozing level calculation unit 601.
The dozing level detection device 600 receives face image data Im continuously at regular intervals and outputs a driver's dozing level Qn. The face image data Im is given, for example, 30 frames per second, and the dozing level Qn is updated and output every time the face image data Im is newly input.
The eye opening degree detection devices 100 to 500 may be any one of the eye opening degree detection devices 100 to 500 described in the first to fifth embodiments. The degree of eye opening Kn calculated by any one of the eye opening degree detection devices 100 to 500 described in the first to fifth embodiments is given to the dozing level calculation unit 601.

The dozing level is a value indicating the degree of sleepiness. For example, the degree of drowsiness represented by the Karolinska Sleepiness Scale (KSS) indicates the degree of sleepiness by a numerical value represented by 9 levels from 1 (very awakening) to 9 (very sleepy). In addition, the degree of sleepiness may be indicated by a numerical value of 10 levels obtained by adding 10 (sleeping) to the 9 levels. In the following description, as an example, the dozing level is considered based on a numerical value of 10 levels, and is defined as a continuous value rather than a stepwise value. For example, numerical values expressed using decimal points such as 1.1 and 5.4 are also included. It should be noted that the numerical value of the dozing level is not limited to the above-described 9 levels or 10 levels. For example, in the facial expression determination method, the level of dozing level is indicated by a numerical value of 1 to 5.

The dozing level calculation unit 601 is provided with the eye opening degree Kn, and calculates the driver's dozing level Qn from the eye opening degree Kn for the past Nq frames.
For example, the dozing level calculation unit 601 calculates PERCLOS (Percentage of Eyelid Closure) for the past Nq frames, and calculates the dozing level. PERCLOS is expressed, for example, as a percentage of the number Nqc of frames Cf in the closed eye state included in the past Nq frames, and is calculated as PERCLOS = Nqc / Nq.
Here, the closed eye state is a state in which, for example, a frame whose eye opening degree Kn is equal to or less than the threshold value Knh is continuously observed for Nqch. For example, Nq = 5,400, Knh = 50, and Nqch = 5 are used.

The dozing level Qn is calculated by the following equation (17).
Qn = fq (PERCLOS) (17)

In the equation (17), f is an arbitrary function with PERCLOS as a variable, and for example, a linear function, a quadratic function, or a nonlinear function using a neural network is used. Here, since it is assumed that the dozing level is indicated by a numerical value of 1 to 10, any function that can calculate a numerical value of 1 to 10 can be used as f. For example, a function represented by the following formula (18) or (19) can be used as the function f that can calculate a numerical value of 1 to 10.

However, sigmoid (a, x) is a sigmoid function and is defined by the following equation (20).
Sigmaid (a, x) = 1 / (1 + exp (−ax)) (20)

Each parameter (α1 to αNN, β, etc.) in equation (18) or equation (19) is statistically calculated from past experimental data. For example, in an experiment for measuring the degree of drowsiness, by evaluating the degree of dozing using the Karolinska sleepiness scale and simultaneously measuring PERCLOS, the relationship between the degree of drowsiness and PERCLOS can be acquired. Based on the data acquired in this way, a parameter for calculating the degree of doze from the PERCLOS is specified, and the specified parameter may be applied to the above equation (18) or (19).

A neural network is a technique for modeling the correspondence between input variables and results as a function. If a neural network is used, the correspondence between PERCLOS and the degree of dozing obtained from experimental data can be acquired as a function. That is, a function that outputs a likely dozing level with respect to PERCLOS obtained as an input is obtained.
However, the same data as the past data is not always obtained in the actual usage scene. In this case, an output other than 1 to 10 may be obtained. On the other hand, as shown in the above equation (18), when a value larger than 10 is obtained, the output is set to 10, and when a value smaller than 0 is obtained, This can be handled by setting the output to 0.

Also, PERCLOS for the past Nq frame is PERCLOS (1), PERCLOS for the past 2 × Nq to Nq frames is PERCLOS (2),... ) And the dozing level Qn may be calculated by the following equation (21).

Here, PERCLOS in the equation (21) will be described. For example, assume that the current frame is F (k), Nq is 900, and NN = 10.
In this case, PERCLOS calculated for F (k) to F (k-899) is PERCLOS (1).
Also, PERCLOS calculated for F (k−900) to F (k−1799) becomes PERCLOS (2).
Also, PERCLOS calculated for F (k-1800) to F (k-2699) becomes PERCLOS (3).
Similarly, PERCLOS calculated for F (k-8100) to F (k-8999) is PERCLOS (10).

In the above description, PERCLOS is a variable of fq, and the degree of drowsiness Qn is calculated. However, from the degree of eye opening Kn, arbitrary features such as the interval at which the closed eye state occurs, the blink frequency, and the blink interval are extracted, It may be a variable of fq.

The driver may be alerted or alerted by using the dozing level output by the dozing level detecting device 600 configured as described above.

By configuring as described above, it is possible to accurately estimate the degree of drowsiness of the driver using a highly accurate eye opening degree detection result.

Part or all of the eye opening degree detection devices 100 to 500 and the drowsiness level detection device 600 described above are stored in the memory 150 and the memory 150 as shown in FIG. 14A, for example. And a processor 151 such as a CPU (Central Processing Unit) for executing the program. Such a program may be provided through a network, or may be provided by being recorded on a recording medium.

Further, as shown in FIG. 14B, for example, as shown in FIG. 14B, a part of or all of the eye-opening degree detection devices 100 to 500 and the drowsiness detection device 600 may be a single circuit, a decoding circuit, a programmed processor, It can also be configured by a processing circuit 152 such as a programmed processor, ASIC (Application Specific Integrated Circuits), or FPGA (Field Programmable Gate Array).

In the first to fifth embodiments described above, the eye opening degree Kn is used as the output of the eye opening degree detecting devices 100 to 500. However, the present invention is not limited to such an example. For example, the shape parameter Hn may be used as the output. In such a case, the eyelid shape detection devices 110 to 510 function as information processing devices.
In the sixth embodiment, the dozing level detection device 600 that outputs the dozing level Qn functions as an information processing device.

100, 200, 300, 400, 500 Eye opening degree detection device, 600 Dozing degree detection device, 601 Dozing degree calculation unit, 110, 210, 310, 410, 510 Eyelid shape detection device, 130 Eye opening degree calculation unit, 111 Eye presence region Identification unit, 112 Normalization processing unit, 113, 213, 313, 413 Light / dark change detection unit, 114 Eyelid reference position estimation unit, 115, 215 Approximation curve generation unit, 116, 216, 516 Shape determination unit, 517 Approximation curve selection unit , 518 correction curve generation unit, 150 memory, 151 processor, 152 processing circuit.

Claims

An eye-existing area specifying unit that specifies an eye-existing area, which is an area where the eye exists, from a human face image;
By detecting the magnitude of the change in brightness between pixels in the vertical direction of the eye presence region, the coordinates of the first end point on the first boundary line between the eye and the upper eyelid, and the eye and the lower eyelid A light / dark change detection unit that estimates the coordinates of the second end point on the second boundary line between
An eyelid reference position estimator that estimates the coordinates of the eye's eye corner and the eye's eye corner in the eye presence region;
In the eye presence region, a plurality of coordinates are selected from a straight line passing through the first end point and the second end point, and a plurality of coordinates are selected from each of the selected plurality of coordinates, the eye coordinates and the eye corner coordinates. An approximate curve generation unit for generating the first approximate curve of
A shape determining unit that selects the most suitable first approximate curve as the first boundary line from the plurality of first approximate curves, and determines the shape of the upper eyelid from the selected first approximate curve; ,
The brightness change detection unit is formed on a straight line passing through the first end point and the second end point in the eye presence region, and changes in brightness between the pixels in the first selection region including the first end point. Based on
The information processing apparatus, wherein the approximate curve generation unit selects the plurality of coordinates from the first selection region.
In the eye presence region, the brightness change detection unit uses the coordinates of the pixel with the highest degree of brightness reduction from the upper pixel as the coordinates of the first end point, and the degree of brightness reduction from the lower pixel. The information processing apparatus according to claim 1, wherein a coordinate of a pixel having the largest value is a coordinate of the second end point.
The brightness change detection unit sets a plurality of predetermined straight lines extending in the vertical direction in the eye presence region, and reduces brightness from the upper pixel on each straight line included in the plurality of straight lines. The distance between the coordinates of the first candidate pixel having the largest degree of the brightness and the coordinates of the second candidate pixel having the largest degree of brightness reduction from the lower pixel is calculated, and the first candidate having the largest calculated distance is calculated. The information processing apparatus according to claim 1, wherein the coordinates of the pixel and the coordinates of the second candidate pixel are the coordinates of the first end point and the coordinates of the second end point, respectively.
The brightness change detection unit sets a plurality of predetermined straight lines extending in the vertical direction in the eye presence region, and selects one straight line from the plurality of straight lines based on the orientation of the person's face. On the selected straight line, the coordinate of the pixel having the greatest degree of brightness decrease from the upper pixel is set as the coordinate of the first end point, and the pixel having the greatest degree of brightness decrease from the lower pixel The information processing apparatus according to claim 1, wherein the coordinates are the coordinates of the second end point.
The brightness change detecting unit uses a first filter for estimating the coordinates of the first end point and a second filter for estimating the coordinates of the second end point,
The first filter is configured to calculate an average value of luminance values of pixels included in a first region including a plurality of pixels in the lateral direction of the first target pixel with respect to the first target pixel in the eye presence region. Subtract from the average value of the luminance values of the pixels included in the second area that touches the first area and has the same number of pixels in the horizontal direction as the first area and has a larger number of pixels in the vertical direction than the first area. Is calculated as an evaluation value of the first target pixel, and the calculated evaluation value is set as a degree of decrease in brightness from the upper pixel.
In the eye presence region, the second filter calculates an average value of luminance values of pixels included in a third region including a plurality of pixels in the horizontal direction of the second target pixel with the second target pixel as a center. Subtract from the average value of the luminance values of the pixels included in the fourth region that is in contact with the third region and has the same number of pixels in the horizontal direction as the third region and has a larger number of pixels in the vertical direction than the third region. The calculated value as the evaluation value of the second pixel of interest, and the calculated evaluation value as the degree of decrease in brightness from the lower pixel,
The information processing apparatus according to any one of claims 2 to 4, wherein:
The eyelid reference position estimation unit uses the value of the axis in the vertical direction of the coordinates of the second end point and the value of the axis in the horizontal direction of one end in the horizontal direction of the eye presence region as the coordinates of the eye head, The value of the axis in the vertical direction of the coordinates of the second end point and the value of the axis in the horizontal direction of the other end in the horizontal direction of the eye presence region are the coordinates of the corner of the eye,
The information processing apparatus according to any one of claims 1 to 5, wherein the shape determining unit determines the shape of the lower eyelid from a line segment between the coordinates of the eyes and the coordinates of the corners of the eyes.
The approximate curve generation unit selects the plurality of coordinates from a line segment between the first end point and the second end point in the eye presence region. Information processing device.
The light / dark change detection unit includes the first end point of coordinates on a straight line passing through the first end point and the second end point in the eye presence region, and an evaluation value of the first filter is predetermined. The information processing apparatus according to claim 5, wherein the first selection area including coordinates that are equal to or greater than a value is specified.
The light / dark change detection unit includes the second end point of coordinates on a straight line passing through the first end point and the second end point in the eye presence region, and an evaluation value of the second filter is predetermined. Specify a second selection area consisting of coordinates greater than or equal to the value,
The approximate curve generation unit selects a plurality of coordinates from the second selection area, and each of a plurality of second coordinates is selected from the plurality of coordinates selected from the second selection area, the coordinates of the eyes and the coordinates of the corner of the eye. Generate an approximate curve,
The shape determining unit selects a second approximate curve most suitable as the second boundary line from the plurality of second approximate curves, and determines the shape of the lower eyelid from the selected second approximate curve. The information processing apparatus according to claim 8.
The shape determination unit determines a second approximate curve having the smallest average value of luminance values of pixels on the second approximate curve in the eye presence region among the plurality of second approximate curves as the second boundary line. The information processing apparatus according to claim 9, wherein the information processing apparatus is determined to be the most suitable.
The first approximate curve passes through the coordinates of the eye, and a quadratic function having one of the selected coordinates as a vertex, and passing through the coordinates of the corner of the eye, the one coordinate as a vertex. The information processing apparatus according to any one of claims 1 to 10, wherein the information processing apparatus is specified by a quadratic function.
The shape determining unit determines a first approximate curve having the smallest average value of luminance values of pixels on the first approximate curve in the eye presence region among the plurality of first approximate curves as the first boundary line. The information processing apparatus according to any one of claims 1 to 11, wherein the information processing apparatus is determined to be most suitable as the information processing apparatus.
An eye-existing area specifying unit that specifies an eye-existing area, which is an area where the eye exists, from a human face image;
By detecting the magnitude of the change in brightness between pixels in the vertical direction of the eye presence region, the coordinates of the first end point on the first boundary line between the eye and the upper eyelid, and the eye and the lower eyelid A light / dark change detection unit that estimates the coordinates of the second end point on the second boundary line between
An eyelid reference position estimator that estimates the coordinates of the eye's eye corner and the eye's eye corner in the eye presence region;
In the eye presence region, a plurality of coordinates are selected from a straight line passing through the first end point and the second end point, and each of the selected plurality of coordinates, from the coordinates of the eyes and the coordinates of the eye corners, An approximate curve generator for generating a first approximate curve;
An approximate curve selection unit that determines the coordinates of the determined end point that determines the highest point on the first boundary line by selecting the most suitable first approximate curve as the first boundary line from the plurality of first approximate curves. When,
In the eye presence region, a plurality of first selection coordinates are selected from a straight line in the vertical direction passing through the coordinates of the eyes, and a plurality of second selection coordinates are selected from a straight line in the vertical direction passing through the coordinates of the corners of the eyes. A correction curve generation unit that generates a plurality of correction curves that are approximate curves from each of the first selection coordinates, each of the plurality of second selection coordinates, and the coordinates of the determined end point;
A shape determining unit that selects the most suitable correction curve as the first boundary line from the plurality of correction curves and determines the shape of the upper eyelid from the selected correction curve. Processing equipment.
An eye opening degree calculation unit,
The shape determining unit calculates a shape parameter indicating the shape of the eye from the determined shape of the upper eyelid, the coordinates of the eyes, the coordinates of the corner of the eye, and the second end point,
The information processing apparatus according to claim 1, wherein the eye opening degree calculation unit calculates an eye opening degree that is an opening degree of the eye based on the shape parameter.
The information processing apparatus according to claim 14, further comprising a drowsiness level calculation unit that calculates a drowsiness level that is a degree of drowsiness of the person based on the eye open degree.
From the person's face image, specify the eye presence area that is the area where the eye exists,
By detecting the magnitude of the change in brightness between pixels in the vertical direction of the eye presence region, the coordinates of the first end point on the first boundary line between the eye and the upper eyelid, and the eye and the lower eyelid Estimate the coordinates of the second end point on the second boundary between and
Estimating the coordinates of the eye's eyes and the corners of the eyes in the eye presence region,
In the eye presence region, a first selection region that is formed on a straight line passing through the first end point and the second end point and includes the first end point is specified based on a change in brightness between the pixels,
Selecting a plurality of coordinates from the first selection area;
Generating a plurality of first approximate curves from each of the plurality of selected coordinates, the coordinates of the eyes and the coordinates of the corners of the eyes;
A first approximate curve that is most suitable as the first boundary line is selected from the plurality of first approximate curves, and the shape of the upper eyelid is determined from the selected first approximate curve. Method.
From the person's face image, specify the eye presence area that is the area where the eye exists,
By detecting the magnitude of the change in brightness between pixels in the vertical direction of the eye presence region, the coordinates of the first end point on the first boundary line between the eye and the upper eyelid, and the eye and the lower eyelid Estimate the coordinates of the second end point on the second boundary between and
Estimating the coordinates of the eye's eyes and the corners of the eyes in the eye presence region,
In the eye presence region, select a plurality of coordinates from a straight line passing through the first end point and the second end point,
Generating a plurality of first approximate curves from each of the plurality of selected coordinates, the coordinates of the eyes and the coordinates of the corners of the eyes;
By selecting the most suitable first approximate curve as the first boundary line from the plurality of first approximate curves, the coordinates of the fixed end point that has determined the highest point on the first boundary line are determined,
In the eye presence region, a plurality of first selection coordinates are selected from a straight line extending in the vertical direction passing through the coordinates of the eye head,
In the eye presence region, select a plurality of second selection coordinates from a straight line extending in the vertical direction passing through the coordinates of the corner of the eye,
From each of the plurality of first selection coordinates, each of the plurality of second selection coordinates, and the coordinates of the fixed end point, a plurality of correction curves that are approximate curves are generated,
An information processing method comprising: selecting a correction curve most suitable as the first boundary line from the plurality of correction curves and determining the shape of the upper eyelid from the selected correction curve.