JPH0944680A - Eye zone extracting device for moving image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an eye zone extracting device for moving image with which erroneous extraction at the time of blinking can be reduced, the motion of face or eye with time can be dealt with and an eye zone can be efficiently extracted in a comparatively short time. SOLUTION: The image of the face of a detection object person 1 is time sequentially and continuously photographed by a photographing means 10, the moving image is provided, and the eye zone is extracted by an extracting means 22 in the search area set to a current image by a search area setting means 21. Especially, the search area setting means 21 sets any one of the position of search area, size, number and shape corresponding to the extracted result of eye zone in the image in the past at least. Besides, the extracting means 22 judges the opening/closing state of eyes.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image eye area extracting apparatus for extracting an eye area from a moving image obtained by time-sequentially capturing the face of a person to be detected.

[0002]

2. Description of the Related Art As a conventional device of this type, for example, in the device disclosed in Japanese Patent Laid-Open No. 4-174309,
When extracting the eye region from the face image, the extraction range is limited within one face image. For example, the image of the face of the person to be detected is continuously photographed by the photographing means, the one still image is binarized, and then the existence range of the eyes is limited to be the search area. The eye area is extracted with.

The method of setting the search area of this apparatus will be described in more detail below. One still image obtained by photographing the face of the person to be detected is binarized into white / black. The number of continuous white pixels is calculated by scanning at least one of the left and right sides from the center of the binary image after this binarization processing. In this case, white indicates the face, and black indicates the background and each part of the face, such as eyebrows and eyes. Next, the end point of the portion where the number of white pixels is maximum in the horizontal direction is specified as the end of the face, and the horizontal position of the eye search area is determined from the general face width. Then, in the search area determined only in the horizontal direction, the lower end of the black pixel area from the bottom in the vertical direction, that is, the lower end of the eye area is detected, and based on this and the general vertical size of the eye, Determine the vertical position of the eye search area. In this device, the eye region is extracted from the search area thus determined.

The eye open / closed state is determined from the shape of the eye region extracted by the eye region extracting device, and the result is applied to determine the state of a person, for example, the state of dozing. In the case of determining a state such as dozing, it is necessary to continuously extract eye regions from images obtained in a time series from the photographing means, that is, moving images, and determine the open / closed state of the eyes. In this case, for each of the plurality of images obtained in a time series from the photographing means,
The search area was determined as described above, and the eye area was extracted.

Further, a retina reflection image is extracted as an eye region,
There was also a device for determining the open / closed state of the eyes. For example, it is shown in JP-A-6-261863 and JP-A-6-270711, and a simplified configuration thereof is shown in FIG. In the figure, 11 is an LED light source, 12 is a half mirror, and the irradiation axis of the illumination light from the LED light source 11 and the camera 10 are shown.
They are installed so that their optical axes are almost coaxial. 13 is L
It is an LED drive circuit that sends a current to the ED light source 11. In the device configured as described above, the camera 10 detects the person 1 to be detected.
When the face is photographed, the reflected image from the retina of the eyeball shines brightly and can be photographed. This retina reflection image is extracted as a pupil region, the open / closed state of the eyes is determined, and this is used for detecting the line of sight and detecting the awakening level.

[0006]

Since the conventional eye area extracting device is constructed as described above, when the eye area is continuously extracted from the moving image, it is continuously obtained from the photographing means in time series. For each of the plurality of images, the search region is determined by searching the pixels of the entire image, and the eye region is extracted. Therefore, there is a problem that it takes time to extract the eye region.

In the case of a moving image, the eyes are opened by blinking,
The extraction target is changing, such as closed eyes or half-closed eyes. As a result, in the conventional device, the search area including the eye area can be correctly set when the eyes are opened, but the eye area is not included when the eyes are closed, and a search area including another black area, for example, eyebrows is erroneously set. Become.

Furthermore, the face of the person to be detected is not still stationary, but is constantly moving a little, and the position and size are fixed with respect to a plurality of images obtained continuously in time series. There is a problem that it is difficult to extract the eye region by the search area.

Further, when the person wearing the eyeglasses is the detection target person, the eyeglasses are photographed in the obtained image in the vicinity of the eye region to be extracted, and for this reason, a search area that does not include the eye region to be extracted is set. Sometimes I set it.

Further, the elapsed time from the previously extracted image to the current image is different depending on whether the eye region is continuously extracted in the moving image or not, and the eye moving range must be different during this time. is there. For this reason, if the position and size of the search area are always constant, the eye area may not be included in the search area, or unnecessary parts such as eyebrows may be included in addition to the eye area.

FIG. 18 is an explanatory view showing a photographed image of the vicinity of the eye photographed by the photographing device shown in FIG. It is assumed that the detection target person 1 wears glasses. In the figure, 31 is eyeglasses, 32 is eyes, 33 is a retina reflection image, 34 is a frame reflection image, and 35 is a lens reflection image. As shown in this figure, when the detection target person 1 wears spectacles, not only the retina reflection image 33 but also the lens reflection image 3 of the spectacles.
Since the image 5 and the frame reflection image 34 are also photographed, these reflection images 34 and 35 and the retina reflection image 33 must be identified and extracted, which makes extraction processing difficult. For example, when the search area is set to the area A, the retinal reflection image 33 and the lens reflection image 35 and the frame reflection image 34 are similar to each other, and thus the lens reflection image 35 and the frame of the glasses included in the search area A are included. There is a high possibility that the reflection image 34 will be extracted as a retina reflection image. Thus, it is very difficult to set the optimum search area.

The present invention has been made in order to solve the above-mentioned conventional problems, and it is possible to prevent erroneous extraction of an eye region from a moving image and cope with temporal movement of the face and eyes. It is an object of the present invention to provide a moving image eye area extracting device that can perform eye area extraction with high accuracy in a relatively short time.

It is another object of the present invention to provide a moving image eye area extracting apparatus capable of preventing the eye area from being erroneously extracted even for a detection target person wearing eyeglasses.

[0014]

According to a first aspect of the present invention, a moving image eye area extracting apparatus obtains a moving image by continuously taking images of the face of a detection target person in time series. Means, a search area setting means for setting a search area for a current image among the images captured in time series, and an extracting means for extracting an eye area in the search area. At least one of the position, size, number, and shape of is set variable according to the extraction result of the eye region in the past image.

According to a second aspect of the present invention, there is provided a moving image eye area extracting device, which is a photographing means for obtaining a moving image by continuously photographing the images of the face of the person to be detected in time series.
A search area setting means for setting a search area with respect to the current image among the images captured in time series, and an extraction means for extracting an eye area in the search area and determining the open / closed state of the eyes,
The search area setting means is such that at least one of the position, size, number, and shape of the search area can be set and changed according to the eye open / closed determination result in the past image.

According to a third aspect of the present invention, in addition to the first or second aspect, the moving image eye area extracting device has a search area setting means that determines whether or not the eye area is extracted in the immediately preceding image. The size of the search area for the current image is variable according to the duration of the presence / absence of extraction in the immediately preceding image to the past image.

According to a fourth aspect of the present invention, in addition to the first or second or third aspect of the present invention, the moving image eye area extracting device extracts a moving image from a past eye area extracted by the extracting means. The position estimation means for calculating the movement vector of the eye area and estimating the position of the eye area in the current image from the movement vector is provided, and the search area setting means is based on the estimation result of the position estimation means. The position of the search area is set.

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiment 1. A moving image eye area extracting apparatus according to Embodiment 1 of the present invention will be described. In this embodiment,
As an actual eye model, for example, width 4 cm, height 2 cm
Assuming the eye model, the size of the search area is set based on the eye model. The term “blinking” refers to a state in which the eyes are closed due to blinking or the eyes are closed halfway. FIG. 1 is a block diagram showing a moving image eye area extracting apparatus according to the first embodiment.
In the figure, 1 is a face of a detection target person, 10 is a photographing means for obtaining a moving image by continuously photographing the face image of the detection target person 1 in time series, for example, a camera, 20 is a camera 10.
Image memory for temporarily storing the image signal output from 2
Reference numeral 1 is a search area setting means for setting a search area for the current face image of the detection target person 1. For example, the search areas for extracting the left and right eye areas are independently set for the left and right eyes. A search area setting circuit, 22 is an extraction unit that extracts an eye area in the search area set by the search area setting circuit 21 and determines the open / closed state of the eye, and is, for example, a feature extraction circuit. It should be noted that the determination of the open / closed state of the eyes may be performed with two types of eyes open and eyes closed, and how long the eyes are closed may be determined.

The processing according to the first embodiment will be described below. FIG. 2 is a flowchart showing the processing according to the first embodiment. After the apparatus is started in step S101, the search area setting circuit 21 sets an initial search area having the entire image as a search area (step S102). In step S103, the camera 10 captures a face image of the person to be detected 1 and inputs it to the image memory 20 for storage. Next, the feature extraction circuit 22 uses the image memory 2 to store the current image just captured.
Input from 0, and the search area setting circuit 21 inputs the initial search area as the search area for the current image. Then, for example, the image in this search area is binarized into white / black, and the search is performed to extract the left and right eye areas (step S104).
If this search area is set appropriately, in the binarized image, the eyes are black pixels in the white pixels of the face. Therefore, the eye region can be easily extracted by detecting the black pixel. Then, in step S105,
The open / closed state of the eye is determined based on the shape of the extracted eye region. However, if the eye region cannot be extracted, it is considered that the eye is closed. After the determination, in step S106, the search area for the next image is reset based on the center coordinate value of the eye area. The processing from step S103 to step S106 is repeated at intervals of 30 ms, for example.

FIG. 3 is an explanatory diagram for explaining the method of setting the search area in the search area setting circuit 21 according to the first embodiment, and the method of setting the search area will be described in detail based on this figure. Here, the current time of the process is T and the time-series time interval is Δt. Further, the image is represented by a two-dimensional XY coordinate system. Also, one image is 25.6c near the face.
The image is taken in an area of m × 25.6 cm, and the number of pixels corresponding to the image is 512 × 512. The right-eye region can be extracted at the immediately preceding shooting time (T-Δt), and its center coordinate value is set to (X1, Y1). In this case, at time T, for example, a search area in the range of: width direction: X1-50 <x <X1 + 50 height direction: Y1-30 <y <Y1 + 30 is set. However, the values of 50 and 30 in this formula represent the number of pixels of the width and height of the search area on the image, and are determined as follows in this embodiment, for example. Assuming an eye model with a width of 4 cm and a height of 2 cm, the width and height are determined so that the search area is larger than the eye model. As a guide, it may be set to about 1 to 5 times as large as the eye model.
m, height 3 cm. In one image in this embodiment, 1 cm on the face is 2 on the captured image.
This corresponds to 0 pixel, which is a search area having a width of 100 pixels and a height of 60 pixels on the image. The same applies to the search area for the left eye.

As in this embodiment, the extraction time can be shortened by setting the search area in the current image with the eye area extracted immediately before as the center and extracting the eye area within this search area. Also, when the eyeblink is closed or halfway closed, the number of black pixels in the eye area is reduced, but by using the extraction position when the eye is opened before the eyeblink, the search area can be set in a range that does not include eyebrows. Therefore, erroneous extraction can be prevented. Further, the open / closed state of the eyes can be accurately determined from the shape of the extracted eye region and the like.

Embodiment 2 FIG. In this embodiment, the case where the detection target person 1 wears glasses will be described. The configuration and main processing of the moving image eye area extraction device are the same as in the first embodiment.
Is the same as In this embodiment, for example, an eye model having a width of 4 cm and a height of 2 cm is assumed, and an eyeglass model having a lens having a width of 6 cm and a height of 4 cm is assumed, and the size of the search area is based on the eye model and the eyeglass model. Is set.
When the entire photographed image is binarized, there are black pixels and white pixels for the frame of the glasses, and it is very difficult to extract the eye region. Furthermore, when the eyes are closed or when the eyes are closed halfway, for example, eyebrows or spectacle frames included in the search area are extracted as eye regions, and there is a high possibility that the extraction may be erroneous. Therefore, the open / closed state of the eyes cannot be correctly determined. On the other hand, in this embodiment, a small search area is set as the search area so as to exclude the spectacle frame to facilitate extraction of the eye area. Further, the open / closed state of the eyes can be accurately determined from this eye region.

A method of setting the search area in the search area setting circuit 21 will be described. In the captured image of the same size as in the first embodiment, the right eye region can be extracted at the immediately preceding capturing time (T-Δt), and its center coordinate value is (X1, Y1). In this case, at time T, for example, a search area in the range of: width direction: X1-45 <x <X1 + 45 height direction: Y1-30 <y <Y1 + 30 is set. However, the values of 45 and 30 in this formula represent the number of pixels in the width and height of the search area on the image, and are determined as follows in this embodiment, for example. The eye model is 4 cm wide and 2 cm high,
Moreover, an eyeglass model having a lens with a width of 6 cm and a height of 4 cm is assumed. Then, assuming that the eye region is located at the center of the spectacle lens, the width and height are determined so that the search region is larger than the eye model and smaller than the spectacle lens. As a guide, above eye model, 4/5 of eyeglass lens model
It may be set to about twice or less, for example, a width of 4.5 cm,
The search area is 3 cm high. This is a search area having a size of 90 pixels in width and 60 pixels in height on the image. If this search area is set larger than necessary, eyebrows, eyeglasses, and the like are likely to be mixed together, and erroneous extraction is likely to occur particularly when the eyes are closed or when the eyes are closed halfway. Therefore, it is set to an appropriate size.

As in this embodiment, by setting a search area of an appropriate size in consideration of the eyeglass model centering on the eye area that has been extracted immediately before, and extracting the eye area within this search area, the extraction time Can be shortened. Further, when the eyes are closed or when the eyes are closed halfway, the extraction position when the eyes are opened can be used to set the search area in a range that does not include the eyebrows, eyeglass frames, etc., and erroneous extraction can be prevented. Further, even if the eyeglasses are worn, the open / closed state of the eyes can be accurately determined at all times.

Embodiment 3. When the detection target person 1 blinks and the closed eye or the halfway closed eye is photographed, the eye region cannot be extracted from the image. From this, it can be considered that the detection target person 1 has the eyes closed in the image in which the eye region cannot be extracted. In this embodiment, a process in the case where the eye region cannot be extracted from the immediately previous image and the eye is regarded as closed is described. FIG. 4 is a block diagram showing the configuration of the moving image eye area extracting apparatus according to the third embodiment.
In the figure, 23 is an extraction result storage memory that stores the extraction result of the feature extraction circuit 22, and the search area setting circuit 21 in this embodiment is the extraction result storage memory 23.
The past extraction result stored in is checked and the search area is set.

The processing according to the third embodiment will be described below. FIG. 5 is a flowchart showing the processing according to the third embodiment. Step S101 to Step S105
Is the same as in the first embodiment. After extracting the eye region in step S104, the open / closed state of the eye is determined from the shape of the eye region in step S105. Then, in step S107, the coordinate value of the eye area is stored in the extraction result storage memory 23.
If the eye region cannot be extracted in step S104, it is determined that the eye is closed and the extraction result is stored. After that, in step S108, the search area setting circuit 21 checks the extraction result of the past eye area from the extraction result storage memory 23 and resets the search area for the next image. Step S10
The process from 3 to step S108 is repeated at intervals of, for example, 30 ms.

Hereinafter, a method of setting the search area in the search area setting circuit 21 will be described in detail with reference to FIG. In this embodiment, when the eye region can be continuously extracted in time, a small search area is set, and when the eye region cannot be temporarily extracted due to a blink or the like and the eye is regarded as closed eyes, If a large search area is set and it is considered that the eyes cannot be extracted for a predetermined time or longer and the eyes are closed, a larger search area is set. And
When the eye area can be extracted again, return to the small search area.

FIG. 6 is an explanatory diagram for explaining a search area setting method in the search area setting circuit 21 according to the third embodiment. When the right-eye area can be extracted at the immediately preceding shooting time (T-Δt), as in the first embodiment, based on the extracted center coordinate values (X1, Y1), the width direction: X1-50 <x <X1 + 50. Height direction: A search area in the range of Y1-30 <y <Y1 + 30 is set. The values of 30 pixels and 50 pixels are set based on the eye model described in the first embodiment.

As shown in FIG. 6A, when the right eye region cannot be extracted at the immediately preceding photographing time (T-Δt) and the eye is regarded as closed, within a predetermined time before that (T in FIG. 6A).
Using the coordinate values (X2, Y2) when extracted at −2Δt), a search area larger than the search area when extracted at the immediately preceding shooting time (T−Δt) is set. For example, a search area in the range of: width direction: X2-100 <x <X2 + 100 height direction: Y2-60 <y <Y2 + 60 is set. However, 100 and 6
A value of 0 is 2 in the search area when the previous eye area could be extracted.
Just decided to double the size, 100 and 6
It need not be zero. If the eye area cannot be extracted in the immediately preceding image and the eye is regarded as closed, the eye can move in the imaging time interval of two, and therefore the distance may be taken into consideration for the determination. The same applies to the search area for the left eye.

Further, as shown in FIG. 6 (b), a predetermined time, for example, 30 minutes before the immediately preceding photographing time (T-Δt).
When the right eye region cannot be continuously extracted for 0 ms and it is considered that the eye is closed, the entire image is set as the search area. The same applies to the search area for the left eye.

Here, 100 pixels × 60 pixels, 200 pixels
An eye area extraction device in which search areas of three sizes of pixels × 120 pixels and the entire image (512 pixels × 512 pixels) are provided in advance, and one search area is selected and set from these in accordance with past extraction results. However, it may be configured to be set in the search areas of two sizes. In addition, a small search area is set according to the time when the eye area is continuously extracted, or a large search area is set according to the time when the eye area cannot be continuously extracted and the eye is regarded as closed. You may set the search area of a step.

Further, in FIGS. 6A and 6B, when only one of the left and right eye regions can be extracted at the immediately preceding photographing time (T-Δt), the extraction is performed as shown in FIG. The position of the search area that could not be extracted may be determined using the coordinate values (X1, Y1) of the created eye area. For example, in the width direction: (X1 + m1) -100 <x <(X1 + m
1) +100 Height direction: (Y1 + m2) -60 <y <(Y1 + m
2) Set the search area in the range of +60. However, m1 and m2 may be constants, and approximate intervals between the left and right eyes on the image may be set. For example, in FIG. 7, m1 = 150 in the width direction and m2 = 0 in the height direction. The size of the search area is shown in FIG.
Is the same as For the eye area that was extracted at the immediately preceding shooting time (T-Δt), the search area is set in the same manner as in the first embodiment.

In this embodiment, the position of the search area is determined according to the extraction position of the past eye area, and the size of the search area is changed according to the extraction result of the past eye area. Even if you can not extract the eye area and consider it as closed eyes,
By utilizing the position of the eye region extracted before that, the search area can be set in a range not including the eyebrows, the eyeglass frames, etc., and erroneous extraction of the eyebrows, eyeglasses, etc. can be reduced. Further, in such moving image processing, the longer the duration of time when the eye region cannot be extracted before the current image and the eye is regarded as closed eyes, the more difficult it is to specify the position of the current eye region. On the other hand, in this embodiment, when the eye area cannot be extracted and the eye is regarded as closed, the search area is increased, and therefore the eye area is extracted by allowing the temporal movement of the face and eyes even after the blink. You can
Furthermore, the open / closed state of the eyes can always be accurately determined from the shape of the extracted eye region.

Embodiment 4 A moving image eye area extracting apparatus according to Embodiment 4 of the present invention will be described. The configuration and processing of the eye area extraction device are the same as those in the third embodiment, and this embodiment shows another example when setting the search area.

FIG. 8 is an explanatory view for explaining a search area setting process in the search area setting circuit 21 according to the fourth embodiment, and the method for setting the search area will be described in detail with reference to the drawing. As shown in FIG. 8A, the immediately preceding shooting time (T-Δ
When the right eye region can be extracted in t), the width direction: X1-50 <x <X1 + 50 height direction: Y1-30 based on the extracted center coordinate values (X1, Y1) as in the first embodiment. A search area in the range of <y <Y1 + 30 is set. The same applies to the left eye. That is, the search areas are set independently for the left and right eyes.

As shown in FIG. 8B, when the right eye region cannot be extracted at the immediately preceding photographing time (T-Δt) and the eye is regarded as closed, within a predetermined time before that (T-2Δ in FIG. 8).
A plurality of search areas, in this case, five search areas are set by using the coordinate values (X2, Y2) when the extraction is possible in t). For example, width direction: X2-50 <x <X2 + 50 height direction: search area in the range of Y2-30 <y <Y2 + 30, and width direction: X2-150 <x <X2-50 height direction: Y2-30 < Search area in the range of y <Y2 + 30, and width direction: X2 + 50 <x <X2 + 150 Height direction: Search area in the range of Y2-30 <y <Y2 + 30, and width direction: X2-50 <x <X2 + 50 Height direction A search area in the range of Y2-90 <y <Y2-30 and a width direction: X2-50 <x <X2 + 50 in the height direction: a search area in the range of Y2 + 30 <y <Y2 + 90. The same applies to the left eye. Of course, as in the third embodiment, the immediately preceding shooting time (T-Δ
When only one of the left and right eye regions can be extracted in t), the coordinate value (X1,
A plurality of search areas may be set using Y1). Also,
Here, an example is shown in which a plurality of search areas having the same size as in the case of being extracted are set, but a plurality of search areas having different sizes may be set.

As shown in FIG. 8C, when the right eye region cannot be continuously extracted for a predetermined time, for example, 300 ms before the immediately preceding photographing time (T-Δt), it is considered that the entire image is closed. Is set as the search area. The same applies to the search area for the left eye.

In the eye area extraction device according to this embodiment, the position of the search area is determined according to the extraction position of the past eye area, and the number of search areas is changed according to the extraction result of the past eye area. It is possible to reduce erroneous extraction of eyebrows, eyeglasses, and the like, and allow temporal movement of the face and eyes even after the blink to allow accurate extraction. In particular, when extracting the eye region, there is a high possibility in the vertical and horizontal directions considering the movement from the time (T-2Δt). Therefore, when the eye area cannot be extracted on the immediately preceding screen and it is considered that the eye is closed, the eye area can be efficiently extracted by setting a plurality of search areas vertically and horizontally. Furthermore, the open / closed state of the eyes can always be accurately determined from the shape of the extracted eye region.

Embodiment 5 A moving image eye area extracting apparatus according to Embodiment 5 of the present invention will be described. The configuration and processing of the eye area extraction device are the same as those in the third embodiment, and this embodiment shows another example when setting the search area.

FIG. 9 is an explanatory view for explaining a search area setting method in the search area setting circuit 21 according to the fifth embodiment, and the search area setting method will be described in detail with reference to the drawing. As shown in FIG. 9A, the immediately preceding shooting time (T-Δ
When the right eye region can be extracted in t), based on the extracted center coordinate values (X1, Y1), the width direction: X1-50 <x <X1 + 50 Height direction: Y1-50 <y <Y1 + 50 Set a square search area. The same applies to the left eye.

As shown in FIG. 9B, when the right eye region cannot be extracted at the immediately preceding photographing time (T-Δt) and the eye is regarded as closed, within a predetermined time before that (T-2Δ in FIG. 9).
The search area of a horizontally long rectangle is set by using the coordinate values (X2, Y2) when extracted in t). For example, a search area in the range of: width direction: X2-100 <x <X2 + 100 height direction: Y2-60 <y <Y2 + 60 is set. The same applies to the left eye.

As shown in FIG. 9C, when the right eye region cannot be continuously extracted for a predetermined time, for example, 300 ms before the immediately preceding photographing time (T-Δt), the entire image is regarded as closed. Is set as the search area. The same applies to the search area for the left eye.

In this embodiment, the position of the search area is determined according to the extraction position of the past eye area, and the shape of the search area is changed according to the extraction result of the past eye area. In addition to reducing erroneous extraction, it is possible to accurately extract the eye region by allowing temporal movement of the face and eyes even after the blink. Especially in the case of extraction of the eye region, there is a high possibility of moving laterally by turning the neck. Therefore, when the eye area cannot be extracted on the immediately preceding screen and it is considered that the eye is closed, it is considered that the eye area is moving in the lateral direction, and the eye area can be efficiently extracted by lengthening the search area horizontally. Furthermore, the open / closed state of the eyes can always be accurately determined from the shape of the extracted eye region.

Embodiment 6 FIG. A moving image eye area extracting apparatus according to Embodiment 6 of the present invention will be described. FIG.
FIG. 14 is a block diagram showing a moving image eye area extracting apparatus according to Embodiment 6 of the present invention. In the figure, 21 is a search area setting circuit that sets a search area for the current face image of the detection target person 1, 22 is an eye area within the search area set by the search area setting circuit 21, and the open / closed state of the eyes. A reference numeral 23 is a feature extraction circuit for determining whether or not the extraction result storage memory stores the extraction result of the feature extraction circuit 22. Further, 24 calculates the movement vector of the position of the eye area on the image from the past extraction result stored in the extraction result storage memory 23, and calculates the movement vector of the position of the eye area on the image from the calculated movement vector and the past position. It is a position estimation circuit that estimates the position of the area. The search area setting circuit 21 in this embodiment sets the search area based on the estimation result of the position estimation circuit 24 and the past extraction result stored in the extraction result storage memory 23.

The extraction processing according to the sixth embodiment will be described below. FIG. 11 is a flowchart showing the processing according to the sixth embodiment. After the apparatus is started in step S101, the search area setting circuit 21 sets an initial search area having the entire image as a search area (step S102). In step S 103, the face image of the person to be detected 1 is captured by the camera 10 and stored in the image memory 20. Next, the feature extraction circuit 22 inputs the current image just shot from the image memory 20, and the search region setting circuit 21 inputs the initial search region as the search region for the current image. Then, the image within this search area is binarized into white / black, and the left and right eye areas are extracted by searching (step S104). If this search area is set appropriately, in the binarized image, the eyes are black pixels in the white pixels of the face. Therefore, the eye region can be easily extracted by detecting the black pixel. Then, in step S105, the open / closed state of the eyes is determined from the shape of the extracted eye region, and step S1
At 07, the coordinate value of the eye area is stored in the extraction result storage memory 23. If the eye region cannot be extracted, it is considered that the eye region is closed, and the extraction result is stored. This step S10
1 to step S107 are the same processes as in the third embodiment.

After that, in step S109, the position estimation circuit 24 checks the past extraction result of the eye area from the extraction result storage memory 23 and calculates the movement vector of the position of the eye area. However, in the past, it is assumed that the eye has not moved in the time zone for the image in which the eye region cannot be extracted and the eye is regarded as closed. Then, the position of the eye region in the current image is estimated from the calculated movement vector of the eye region and the past extraction position. The search area setting circuit 21 resets the search area for the next image in step S110 based on the position estimated by the position estimating circuit 24 and the extraction result stored in the extraction result storage memory 23. S10
The process from 3 to step S110 is repeated at intervals of 30 ms, for example.

The method of setting the search area will be described in detail below with reference to FIG. For example, the current time of the process is T
When the coordinate value of the right eye region extracted at time (T-2Δt) is (X2, Y2) and the coordinate value of the right eye region extracted at time (T-Δt) is (X1, Y1), The movement vector from time (T-2Δt) to time (T-Δt) can be calculated as (XV, YV) = (X1, Y1)-(X2, Y2). Using the movement vector (XV, YV), the estimated coordinate value (Xt, Yt) of the right eye region at the current time T is calculated. For example, it is estimated by (Xt, Yt) = (X1, Y1) + (XV, YV). However, at time (T-Δt),
When the right eye region cannot be extracted and the eye is regarded as closed, the movement vector is (XV, YV) = (0, 0). The same applies to the left eye region.

The search area setting circuit 21 sets the search area around the estimated position. For example, based on the estimated center coordinate value (Xt, Yt), a range of width direction: Xt-50 <x <Xt + 50 height direction: Yt-30 <y <Yt + 30 is set as a search area. The same applies to the left eye. The size of this search area can be determined by any one of the first to fifth embodiments based on the past extraction result of the eye area.

In the eye area extracting apparatus according to this embodiment, an appropriate search area can be set by deciding the position of the search area based on the past extraction position of the eye area and the movement vector of the eye area calculated thereby. . In particular, there is an effect that the temporal movements of the face and eyes are allowed and false extraction can be reduced.
Further, the open / closed state of the eyes can be accurately determined from the shape of the extracted eye region and the like. The feature extraction circuit 2
When the eye area is extracted from the search area in 1, the position estimation circuit 2
The estimated position of the eye region estimated in 4 may be used. That is,
It suffices that a position closer to the position estimated by the position estimation circuit 21 is extracted as an eye region.

Embodiment 7 FIG. A moving image eye area extracting apparatus according to Embodiment 7 of the present invention will be described. FIG.
9 is a flowchart showing a process relating to a moving image eye area extracting device according to a seventh embodiment of the present invention, and the configuration of the moving image eye area extracting device according to the present embodiment is the same as that of the sixth embodiment. The eye area extraction processing in this embodiment will be described with reference to FIG.

After the device is started in step S101,
The search area setting circuit 21 sets an initial search area having the entire image as the search area (step S102). Step S
In 103, the camera 10 captures a face image of the detection target person 1 and stores it in the image memory 20. Next, the feature extraction circuit 2
2 is step S201, step S104, step S
From 202 to step S205, the eye area is extracted and the open / closed state of the eye is determined. That is, after the counter (count) is initially set to 0 in step S201, the current image just shot is input from the image memory 20, and the search area setting circuit 2
From 1 enter the search area for the current image. Then, the image within this search area is binarized into white / black, and the left and right eye areas are extracted by searching (step S104). If this search area is set appropriately, in the binarized image, the eyes are black pixels in the white pixels of the face. Therefore, the eye region can be easily extracted by detecting the black pixel.

However, in step S202, it is determined whether or not the eye region has been extracted, and if it cannot be extracted,
In step S203, it is confirmed that the value of the counter (count) is smaller than a predetermined number of times, for example, five times, and the search area is reset in step S204. For example, the position of the search area is moved near the previous search area. And step S
At 205, the counter (count) is incremented by 1 and updated,
In step S104, the eye area is extracted again from the current image. The search area is reset until the eye area can be extracted within a predetermined number of times, and the current image is searched.

After extracting the eye area, the open / closed state of the eye is determined in step S105, and the coordinate value of the eye area is stored in the extraction result storage memory 23 in step S107. On the contrary, if the eye region cannot be extracted even after repeating the predetermined number of times, it is considered that the eye region is closed, and the extraction result is stored.

Thereafter, as in the sixth embodiment, step S
At 109, the position estimation circuit 24 checks the past extraction result of the eye area from the extraction result storage memory 23 and calculates the movement vector of the position of the eye area. However, in the past, the time zone for images that could not be extracted and were considered to be closed eyes is
Suppose your eyes did not move. Then, the position of the eye region in the current image is estimated from the calculated movement vector of the eye region and the past extraction position. Search area setting circuit 21
Is based on the position estimated by the position estimation circuit 24 and the extraction result stored in the extraction result storage memory 23, and in step S110, the search area is reset for the next image.
The processing from step S103 to step S110 is repeated at intervals of 30 ms, for example.

In this embodiment, since the search area is changed and the search is performed a plurality of times in one image, the movement of the eye area from the immediately preceding image is large to some extent.
Even if the eye region cannot be extracted by the method of setting the search area as in the first embodiment, the eye region can be further searched and extracted. Therefore, the eye open / closed state can be accurately determined, and a highly reliable moving image eye area extraction device can be obtained. Note that this processing may be added to the third embodiment.

Embodiment 8 FIG. A moving image eye area extracting apparatus according to Embodiment 8 of the present invention will be described. This embodiment relates to a device that extracts a retina reflection image of a detection target person and determines the open / closed state of eyes, and particularly, an extraction process when the detection target person wears glasses. FIG.
FIG. 19 is a block diagram showing a moving image eye area extracting device according to an eighth embodiment. In the figure, 11 is an LED light source,
A half mirror 12 is installed so that the irradiation axis of the illumination light from the LED light source 11 and the optical axis of the camera 10 are substantially coaxial. 13 is an LE that sends a current to the LED light source 11.
It is a D drive circuit. The same reference numerals as those in the sixth embodiment indicate the same or corresponding portions.

The extraction processing of this embodiment will be described below. FIG. 15 is a flowchart showing the eye area extraction processing. After the apparatus is started in step S101, the search area setting circuit 21 sets an initial search area having the entire image as a search area (step S102). Step S103
Then, the face image of the detection target person 1 is captured by the camera 10 by the illumination light from the LED light source 11 and stored in the image memory 20. Next, the feature extraction circuit 22 inputs the currently captured current image from the image memory 20, and the search region setting circuit 21 inputs the search region for the current image. Then, the image in the search area is binarized into white / black with an appropriate binarization threshold value so that the face becomes black pixels, and left and right retina reflection images are extracted (step S301). If this search area is set appropriately, in the image after binarization, the portion of the retina reflection image is white pixels in the black pixels of the face. Therefore, the retinal reflection image can be easily extracted by detecting the white pixels. When the retina reflection image can be extracted in step S105, it is determined that the eye is open, and the coordinate value of the retina reflection image is stored in the extraction result storage memory 23 in step S107. When the retinal reflection image cannot be extracted, it is determined that the eyes are closed and the result is stored.

Thereafter, in step S302, the position estimation circuit 24 checks the extraction result of the past retinal reflection image from the extraction result storage memory 23 and calculates the movement vector of the position of the retinal reflection image. However, in the past, it is assumed that the retinal reflex image did not move during the time period when it could not be extracted and was regarded as closed eyes. Then, the position of the retinal reflection image in the current image is estimated from the calculated movement vector of the retinal reflection image and the past extraction position. Based on the position estimated by the position estimation circuit 24 and the extraction result stored in the extraction result storage memory 23, the search area setting circuit 21 determines in step S303
Reset the search area for the next image. For example, the processing from step S103 to step S303 is performed for 30 ms.
Repeat at intervals of.

When a person to be detected wears glasses as shown in FIG. 16, when a retina reflection image is photographed, unnecessary images such as a lens reflection image and a frame reflection image are photographed together with the retina reflection image. In this embodiment, the search area is set to the area indicated by B in the figure, and is set so that the lens reflection image 35 and the frame reflection image 34 are not included. Therefore, the retina reflection image can be easily extracted, and the open / closed state of the eyes can be accurately determined.

The method of setting the search area according to this embodiment is the same as that of the sixth embodiment. For example, when the current time of the process is T, the coordinate values of the right retina reflection image extracted at time (T-2Δt) are (X2, Y2), and the right retina extracted at time (T-Δt). Set the coordinate value of the reflection image to (X
1, Y1), the time (T-2Δt) to the time (T
The movement vector to −Δt) can be calculated as (XV, YV) = (X1, Y1) − (X2, Y2). Using this movement vector (XV, YV), the estimated coordinate value (Xt, Yt) of the right retina reflection image at the current time T is calculated. For example, it is estimated by (Xt, Yt) = (X1, Y1) + (XV, YV). However, at time (T-Δt),
When the right retinal reflection image cannot be extracted and the eye is regarded as closed, the movement vector is (XV, YV) = (0, 0). The same applies to the left retina reflection image.

The search area setting circuit 21 sets the search area around the estimated position. For example, the size of the search area is set by the same method as in the third embodiment based on the past extraction result of the retina reflection image. Last shooting time (T-Δ
When the right retinal reflection image can be extracted in t), based on the extracted center coordinate values (X1, Y1), the width direction: X1-20 <x <X1 + 20 The height direction: Y1-20 <y <Y1 + 20 Set the search area for the square of the range. The value of 20 pixels is set as follows.

As a model of the retina reflection image, a general pupil size, that is, a circle having a diameter of 0.8 cm is assumed, and an eyeglass model having a lens having a width of 6 cm and a height of 4 cm is assumed. Further, assuming that the retinal reflection image is located near the center of the spectacle lens, the width and height of the search area are determined so that the search area is larger than the retinal reflection image model and smaller than the spectacle lens. As a guide, it is determined so that the search area is larger than the retinal reflection image model and 1/5 to 4/5 the size of the spectacle lens model. For example, the search area has a width of 2 cm and a height of 2 cm on the face. Assuming that 1 cm on the face corresponds to 20 pixels on the captured image, the search area has a width of 40 pixels and a height of 40 pixels on the image. If the person to be detected does not wear glasses, the search area may be set within the range of about 1 to 10 times the retinal reflection image model. The same applies to the search area of the left retina reflection image.

When the right retinal reflection image cannot be extracted at the immediately preceding photographing time (T-Δt) and it is considered that the eye is closed, the coordinate value when it can be extracted within a predetermined time before that, for example, at time (T-2Δt). By using (X2, Y2), a search area larger than the search area when it can be extracted is set. For example, a search area in the range of: width direction: X2-40 <x <X2 + 40 height direction: Y2-40 <y <Y2 + 40 is set. However, the value of 40 pixels is
The size is determined to be twice the size of the search area when the immediately preceding retinal reflection image can be extracted, and the size is not limited to this. When the immediately preceding retinal reflection image cannot be extracted and the eye is regarded as a closed eye, the retinal reflection image may move in two photographing time intervals, and therefore the distance may be taken into consideration for the determination. The same applies to the search area of the left retina reflection image.

Further, when the right retinal reflection image cannot be extracted for a predetermined time from the immediately preceding photographing time (T-Δt) to the past side, for example, 300 ms continuously, and the eye is regarded as closed, the retinal reflection image extracted previously is detected. Regardless of the position, the entire image is set as the search area. The same applies to the left retina reflection image. After setting the search area, the feature extraction circuit 22 distinguishes the retinal reflection image from the spectacle reflection image and extracts it.

In the eye area extracting apparatus according to this embodiment, the position of the search area is determined based on the extraction position of the past retinal reflection image and the movement vector of the retinal reflection image calculated thereby, so that the eye area is included in the search area. The effect is to minimize the reflected image of the glasses and reduce erroneous extraction. Moreover, temporal movement of the face and eyes can be allowed to some extent. In particular, even when the detection target wears spectacles and not only the retinal reflection image in the captured image that extracts the retinal reflection image but also the reflection image by the lens or frame of the spectacles, the retina is accurately measured. The reflection image can be extracted. Furthermore, the open / closed state of the eyes can be correctly determined from the extraction result.

When the feature extraction circuit 21 extracts an eye region in the search area, the estimated position of the eye region estimated by the position estimation circuit 24 may be used. That is, it is only necessary to extract the eye area closer to the position estimated by the position estimating circuit 21 as the eye area.

In the first to seventh embodiments,
Although the eye area extracting device for extracting the eye area and determining the open / closed state of the eye is shown, the pupil area may be extracted as the eye area as shown in the eighth embodiment. Further, the image capturing means according to the first to eighth embodiments extracts the eye region from a moving image continuously captured at time intervals of 30 ms, and determines the open / closed state of the eye. The shooting intervals are not limited to this, and may not be even intervals. In addition, the determination of the open / closed state of the eyes in the extraction means according to the first to eighth embodiments is as follows.
Not only two types of eyes open and eyes closed, but how long the eyes are closed may be determined. Further, although the extraction means according to the first to eighth embodiments determines the open / closed state of the eyes, the present invention is not limited to this, and the configuration is such that only the eye region is extracted, and the shape and position of the extracted eye region are changed. You may use it.

[0068]

As described above, according to the first aspect of the invention, a photographing means for obtaining a moving image by continuously photographing the face image of the person to be detected in time series, the time series. A search area setting means for setting a search area with respect to the current image of the captured images, and an extracting means for extracting an eye area in the search area are provided, and the search area setting means includes a position, a size of the search area,
Since at least one of the number and the shape is variable according to the extraction result of the eye region in the past image,
A search area capable of efficiently extracting an eye area in a moving image can be set, thereby preventing an erroneous extraction of the eye area and accurately extracting the eye area in a relatively short time. There is an effect to be obtained.

According to the second aspect of the invention, the photographing means for obtaining a moving image by continuously photographing the face image of the person to be detected in time series, among the images photographed in time series, The search area setting means for setting a search area for the current image of, and extraction means for extracting the eye area in the search area to determine the open / closed state of the eye, the search area setting means is a position of the search area, At least one of size, number, and shape
Since the setting is variable according to the eye open / closed determination result in the past image, it is possible to set the search area that can efficiently extract the eye area in the moving image, and thus the eye open / closed state can be correctly determined. There is an effect that an eye area extraction device for a moving image that can accurately extract an eye area in a relatively short time can be obtained.

According to the third aspect of the invention, in addition to the first or second aspect, the search area setting means determines whether or not the eye region is extracted in the immediately preceding image, and the images in the images from the immediately preceding to the past. Since the size of the search area for the current image is variable according to the duration of the presence / absence of extraction, in addition to the effect of claim 1 or claim 2, the temporal change of the face and eyes in the moving image There is an effect that a search area can be set according to the motion, and a highly reliable moving image eye area extracting device can be obtained.

According to the invention of claim 4, in addition to claim 1, 2 or 3, the movement vector of the eye area in the moving image is calculated from the past eye area extracted by the extracting means. Then, the position estimation means for estimating the position of the eye region in the current image from the movement vector is provided,
Since the search area setting means sets the position of the search area in the current image based on the estimation result of the position estimating means,
In addition to the effects of claim 1 or claim 2 or claim 3,
Further, there is an effect that it is possible to cope with temporal movements of faces and eyes in a moving image, extract an eye region in a relatively short time, and obtain a highly reliable eye region extracting device for a moving image.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a moving image eye area extracting device according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing a process according to the first embodiment.

FIG. 3 is an explanatory diagram illustrating a search area setting process in the search area setting circuit according to the first embodiment.

FIG. 4 is a block diagram showing the configuration of a moving image eye area extracting device according to a third embodiment of the present invention.

FIG. 5 is a flowchart showing a process according to the third embodiment.

FIG. 6 is an explanatory diagram illustrating a search area setting process in a search area setting circuit according to the third embodiment.

FIG. 7 is an explanatory diagram illustrating a search area setting process in a search area setting circuit according to the third embodiment.

FIG. 8 is an explanatory diagram illustrating a search area setting process in a search area setting circuit according to the fourth embodiment of the present invention.

FIG. 9 is an explanatory diagram illustrating a search area setting process in a search area setting circuit according to the fifth embodiment of the present invention.

FIG. 10 is a block diagram showing a configuration of a moving image eye area extracting device according to a sixth embodiment of the present invention.

FIG. 11 is a flowchart showing a process according to the sixth embodiment.

FIG. 12 is an explanatory diagram illustrating a search area setting process in the search area setting circuit according to the sixth embodiment.

FIG. 13 is a flowchart showing a process according to the seventh embodiment of the present invention.

FIG. 14 is a block diagram showing a configuration of a moving image eye area extracting device according to an eighth embodiment of the present invention.

FIG. 15 is a flowchart showing a process according to the eighth embodiment.

FIG. 16 is an explanatory diagram illustrating a captured image of a detection target person according to the eighth embodiment.

FIG. 17 is a configuration diagram of a conventional device for capturing a retina reflection image.

FIG. 18 is an explanatory diagram illustrating a captured image of a person to be detected according to a conventional device.

[Explanation of symbols]

1 detection target person, 10 imaging means, 21 search area setting means, 22 feature extraction means, 24 position estimation means.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical indication location // B60R 21/00 630 B60R 21/00 630 (72) Inventor Minoru Nishida Two Marunouchi, Chiyoda-ku, Tokyo Chome 2-3, Sanryo Electric Co., Ltd.

Claims (4)

[Claims] 1. A photographing means for obtaining a moving image by continuously photographing images of a face of a detection target person in time series, and a search area for a current image of the images photographed in time series. And search means for extracting an eye area within the search area, wherein the search area setting means sets at least one of the position, size, number, and shape of the search area. A moving image eye area extracting device, wherein setting is variable according to a result of extracting the eye area in a past image. 2. A photographing means for obtaining a moving image by continuously photographing images of a face of a person to be detected in time series, a search area for a current image of the images photographed in time series. A search area setting means for setting the search area, and an extraction means for extracting the eye area in the search area to determine the open / closed state of the eyes, wherein the search area setting means includes the position, size, number, and An eye area extraction device for a moving image, wherein at least one of the shapes is set variable according to the eye open / closed determination result in a past image. 3. The search area setting means sets the size of the search area for the current image in accordance with the presence / absence of extraction of the eye area in the immediately preceding image and the duration of the presence / absence of the extraction in the immediately preceding image. 3. The moving image eye area extracting device according to claim 1, wherein the height is variable. 4. From the past eye region extracted by the extraction means,
The moving image includes a position estimating means for calculating a movement vector of the eye area and estimating the position of the eye area in the current image from the movement vector, and the search area setting means is based on the estimation result of the position estimating means. The eye area extraction device for a moving image according to claim 1, 2 or 3, wherein a position of a search area in the current image is set.