JP2002282210A - Method and apparatus for detecting visual axis - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a device for detecting the visual axis capable of detecting the visual axis appropriately in a wide interior space without having the device worn by a subject of the detection. SOLUTION: Each personal computer 14 for a camera judges whether the face on the front view is being caught in a prescribed angle range based on the estimated angle of the face, and after normalization of the size of the pupil region in the eye region in the image data satisfying the condition for the judgement, the center of the eye and the center of the pupil are calculated, and the quantity of divergence between them is calculated. After that, a main personal computer 16 compares the plural quantities of divergence as the results of the operation and determines that the visual axis is directed to a video camera with the smallest quantity of divergence based on the comparison.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gaze detection method and apparatus.

[0002]

2. Description of the Related Art Conventionally, a person's request is sensed from information obtained by sensing a person, such as a line of sight and movement of a person, and a surrounding environment constructed by object sensing.
It has been proposed to provide a service suitable for the intention of the person. In order to realize these, it is important to sense a person and its surrounding environment, and to know what the person is seeing and what kind of operation is being performed. At this time, the line-of-sight information is one of information that the person is paying attention to or information that is indispensable for estimating the intention and situation of the person.

[0003] The following are known as eye gaze detection methods for detecting the eye gaze. That is, a goggle-type line-of-sight detection device having a line-of-sight detection light source is mounted on the head of the detection target person, and the light source irradiates the eye with infrared light. Then, a light receiving sensor provided in the eye gaze detecting device receives the reflected light reflected by the eyes (pupil and cornea), and detects the eye gaze based on the reflected light.

[0004]

However, the above-described gaze detection method has a problem that a goggle type device must be mounted on the head (eye), which is very troublesome. In addition, since the goggle-type gaze detection device is usually wiredly connected to a control computer or the like that performs predetermined processing and control based on the detected gaze, the movement range is restricted, and it cannot be used in a large indoor space or the like. There was a problem.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and has as its object to provide a gaze detection device capable of suitably detecting a gaze even in a large indoor space without mounting the device on a person to be detected. It is an object of the present invention to provide a method and an apparatus thereof.

[0006]

According to a first aspect of the present invention, there is provided an imaging apparatus for detecting an object to be detected from a predetermined point, and detecting image data from image data captured by the imaging unit. A face direction estimating means for detecting a face area of the target person and estimating a face direction based on the detected face area;
Determining means for determining whether the face direction estimated by the face direction estimating means is a frontal face including a predetermined angle range, and detecting an eye area of the frontal face in the image data determined by the determining means to be a frontal face Eye area detecting means, a distance measuring means for measuring a distance between a first predetermined part and a second predetermined part in the eye area detected by the eye area detecting means, and a distance measured by the distance measuring means Gaze estimating means for estimating the gaze based on the gist.

According to a second aspect of the present invention, in the first aspect, a pupil detecting means for detecting a pupil center in the eye region is further provided, and the first predetermined portion is a pupil center.

According to a third aspect of the present invention, in the first or second aspect, the apparatus further comprises pupil detection means for detecting a pupil center in the eye area, and the second predetermined portion is a pupil center. Is the gist.

According to a fourth aspect of the present invention, in the first or second aspect, the apparatus further comprises a center of gravity detecting means for detecting a position of a center of gravity in the eye region, wherein the second predetermined portion is a center of gravity of the eye region. Is the gist.

[0010] According to a fifth aspect of the present invention, an image capturing step of capturing an image of a person to be detected from a predetermined point, a face area of the person to be detected is detected from the captured image data, and a face orientation is detected based on the detected face area. A face direction estimating step for estimating, a determining step of determining whether the estimated face direction is a front face including a predetermined angle range, and an eye area of the front face in the image data determined to be the front face. An eye region detection process for detecting, and a distance measurement process for measuring a distance between a first predetermined portion and a second predetermined portion in the detected eye region in the detected eye region;
A gaze estimation process for estimating the gaze based on the measured distance is provided.

The invention according to claim 6 is the invention according to claim 5, further comprising a pupil detection step of detecting a pupil center in the eye area, wherein the first predetermined portion is a pupil center.

According to a seventh aspect of the present invention, in the fifth or sixth aspect, the method further includes a pupil detection step of detecting a pupil center in the eye region, wherein the second predetermined portion is a pupil center. Is the gist.

According to an eighth aspect of the present invention, in the fifth or sixth aspect, the method further includes a center of gravity detecting step of detecting a center of gravity position in the eye region, wherein the second predetermined portion is a center of gravity of the eye region. Is the gist.

According to the first aspect of the present invention, the face direction is estimated by the face direction estimating means from the image data picked up by the image pick-up means from the predetermined point, and the face direction is determined by the judging means in the predetermined angle range. It is determined whether or not the face is a front face. Then, the distance measuring means measures the distance between the first predetermined part and the second predetermined part in the eye area in the image data determined as the frontal face, and the gaze is detected by the gaze estimating means based on the distance. You.

According to the second aspect of the present invention, the line of sight is estimated by the line of sight estimating means using the pupil center. According to the third aspect of the present invention, the gaze is estimated by the gaze estimating means using the pupil center in addition to the pupil center.

According to the fourth aspect of the present invention, the line of sight is estimated by the line of sight estimating means using the center of gravity of the eye region in addition to the center of the pupil. According to the fifth aspect of the present invention, a face orientation is estimated in a face orientation estimation process from image data captured in an imaging process from a predetermined point, and it is determined whether or not the face orientation is a front face including a predetermined angle range in a determination process. Is determined. Then, the distance between the first predetermined part and the second predetermined part in the eye area in the image data determined to be the front face in the distance measurement step is measured, and the line of sight is detected based on the distance in the line of sight estimation step.

According to the sixth aspect of the present invention, the sight line is estimated in the sight line estimation process using the pupil center. According to the invention of claim 7, the gaze is estimated in the gaze estimation process using the pupil center in addition to the pupil center.

According to the invention of claim 8, the line of sight is estimated in the line of sight estimation process using the center of gravity of the eye region in addition to the center of the pupil.

[0019]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment embodying a gaze detecting apparatus according to the present invention will be described below with reference to FIGS.

The gaze detecting device 10 according to the present embodiment, when controlling a plurality of electric devices 17 (for example, a television, an audio system, an air conditioner, etc.) to turn on / off, etc. This is for giving a corresponding command signal to the device 17.

For example, when the television is turned off (or turned on) and a line of sight is turned in the direction in which the television is arranged, the line of sight is detected and an on signal (or off signal) is detected as a command signal. ).

The eye-gaze detecting device 10 includes a plurality of (four in the present embodiment) video cameras (CCD cameras) 11 as imaging means, a camera personal computer 14, and a main personal computer 1.
6 and so on. The video camera 11 is arranged at the same place where a plurality of electric devices 17 (for example, television, audio, air conditioner, etc.) are arranged. In this embodiment, the camera personal computer 14 is a face direction estimating unit,
The main personal computer 16 corresponds to the determining means, the eye area detecting means, the pupil detecting means, and the pupil detecting means, and the main personal computer 16 corresponds to the gaze estimating means. The video cameras 11 are arranged at arbitrary positions, and the position of each video camera 11 corresponds to a predetermined point.

A camera personal computer 14 is connected to each video camera 11. Camera PC 1
4, each frame (image data) photographed by the video camera 11 is a video image color image (640).
× 480).

The camera personal computer 14 is the main personal computer 1
The main personal computer 16 communicates with each camera personal computer 14 by socket communication via Ethernet (registered trademark). In addition, a network time server system is used. The main personal computer 16 is set as a time server, and the time of each camera personal computer 14 is adjusted to the main personal computer 16. Also, the main personal computer 16
It is electrically connected to each of the electric devices 17 (for example, a television, an audio, an air conditioner, and the like), and performs on / off control according to a gaze detection result of the gaze detection device 10. That is, a command signal corresponding to the current state of the electric device 17 is output. For example, when the electric device 17 is on, an off signal is output as a command signal, and when it is off, an on signal is output as a command signal. It should be noted that the main personal computer 16 and each electric device 17 may be controlled by infrared rays without being connected by wire, that is, may be controlled by wireless.

(Operation) The operation of the visual line detection device 10 of the present embodiment will be described below. First, the gaze detection device 10
An outline of gaze detection performed by the user will be described.

Each video camera 11 captures an image of the person H to be detected and inputs the captured image to the personal computer 14 for each camera. Each camera personal computer 14 captures an image from the video camera 11, subsequently performs skin color region extraction and face orientation estimation, determines whether or not the face orientation estimation result satisfies a predetermined condition, and sets image data that satisfies the condition. The eye area 32 is detected from. Then, from the detected eye region 32, the size of the pupil is normalized, and the center part of the pupil (pupil center) C1 and the position of the pupil in the pupil (pupil center) C2 are calculated (see FIG. 9). The distance between them is calculated (measured). The camera personal computer 14 transmits the calculation result of the distance to the main personal computer 16, and the main personal computer 16 compares the magnitude of the distance to determine which of the plurality of video cameras 11 is sending the line of sight. That is, the line of sight is detected.

The details will be described below with reference to the flowchart of FIG. This flowchart is started when a start request signal is transmitted from the main personal computer 16 to the camera personal computer 14. Then, the processing of S1 to S11 is repeatedly performed until an end request signal is transmitted from the main personal computer 16 to the camera personal computer 14.

In step (hereinafter abbreviated as “S”) 1, first, the camera personal computer 14
It is determined whether or not to capture an image from. That is, in the present embodiment, capture of an image from the video camera 11 is performed at predetermined intervals (for example, 0.3 seconds), and each camera personal computer 14 determines whether or not that time is reached. . If it is determined that it is time to capture an image (S1 is YES), each camera personal computer 14 captures an image from the video camera 11 (S2). On the other hand, when the camera personal computer 14 determines that it is not time to capture an image (S1).
Is NO), this determination is repeated. Since the time of each camera personal computer 14 is set to the time of the main personal computer 16, each camera personal computer 14 captures an image at the same time.

(Face area detection) Personal computer 14 for each camera
Performs face area detection after capturing a frame (image data, for example, see FIG. 3) from the video camera 11. The face area detection uses a method based on a known skin color reference value using color information. In this embodiment, the CIE L * u * v color system, which is one of the uniform perceived color spaces, is used.

First, a two-dimensional color histogram based on U and V coordinate values is obtained from the input image data over the entire area of the image, and a peak value (the maximum frequency) within a predetermined effective skin color range is determined. The skin color reference value is used. A threshold value is determined by applying a known discriminant analysis method to the color difference from the reference value,
Based on the threshold value, the image is binarized into a skin color area and other areas (see FIG. 4). In the present embodiment, it is assumed that there is only one detection target person H. Therefore, when a plurality of skin color areas are detected, each camera personal computer 14 determines the maximum area as the face area 31 (S3). . That is, the number of pixels (area) is determined in the plurality of extracted skin color areas, and the maximum area Sma is calculated.
The area x is a face area 31. In the following description, the U and V coordinate values are referred to as UV values or U
Value, V value.

(Estimation of Face Direction) Next, in S 4, each camera personal computer 14 estimates a face direction based on image data obtained from the corresponding video camera 11.

In the present embodiment, the face orientation is estimated by a method of creating a face orientation discrimination space by linear discriminant analysis of the results of the four-way surface feature extraction. In the four-direction surface feature extraction, a vector field in each of four directions (vertical, horizontal, diagonally right 45 degrees, diagonally left 45 degrees) is obtained for each pixel based on the gradient of the gray value of the image data, and an edge image divided for each direction is obtained. . The obtained edge image becomes a shaded image having directionality.

Specifically, a prewitt filter process as a differential filter is performed from the image data input in S3 using a Prewitt operator, and horizontal (horizontal),
Edge images in four directions of vertical (vertical), rising 45 degrees (45 degrees to the right), and falling 45 degrees (45 degrees to the left) are generated. These edge images are hereinafter referred to as direction planes. Next, the images of these four directions are normalized by the face area 31 and the resolution is reduced to 8 × 8, and the gray value of the pixel in each direction is extracted as a feature value (hereinafter, referred to as a feature vector). I do.

Since the resolution of this feature vector is reduced after being divided into four directional planes, the edge information is held at a higher resolution than when the resolution of the input image is directly reduced. As a result, it is hard to be affected by the displacement and the shape change, and the processing cost can be reduced and the processing speed can be increased.

Next, each camera personal computer 14 performs a linear discriminant analysis. The linear discriminant analysis is for discriminating to which class the extracted feature amount (feature vector: xi) belongs, and the variance within the class is small, and the average feature vectors of each class are separated from each other. By configuring such a discrimination space, a high discrimination power can be obtained. FIG. 5 is a conceptual diagram showing classes related to discriminant analysis.

In this embodiment, the coefficient matrix A based on the learning data is stored in advance in a storage device (not shown) of each camera personal computer 14. The learning data is data based on image data obtained by imaging a plurality of detection target persons H. That is, as shown in FIG. 7, 1 is radially arranged at equal angular intervals (22.5 degrees in the present embodiment) so that the optical axis is directed toward the center of the room.
Image data obtained from sixteen directions is obtained by the six video cameras 11, and face area detection and face area
The feature vector x is obtained by performing the four-way surface feature extraction in Step 1.

X = {x1, x2,..., X256} Note that instead of using 16 video cameras 11,
For example, one video camera 11 may be used to capture an image each time the detection target person H rotates at equal angles around the center of the room, and the image data at that time may be used as the learning data.

A coefficient matrix A that linearly maps from the feature vector x to the feature vector y (= Ax) in the discriminant space has been obtained, and each class (in this embodiment, 22.5 16 classes corresponding to the video cameras 11 arranged at intervals of degrees are generated, and the average feature vector yj of the classes is calculated. And
The data of the coefficient matrix A and the average feature vector yj of each class are stored in the storage device of each camera personal computer 14 in advance.

In this embodiment, the class number j is
0, 22.5, 45, 67.5, 90, 112.5, 1
35, 157.5, 180, -157.5, -135,
-112.5, -90, -67.5, -45, -22.
It is a value of 16 which is an equal difference of 5. As shown in FIG. 7, each class number (numerical value) coincides with an angle between the optical axis (camera direction) of the video camera 11 of the camera personal computer 14 and the relative face direction (relative face direction). FIG. 7 shows the arrangement of the video cameras 11 arranged in 16 directions at intervals of 22.5 degrees around the detection target H. When the detection target H is imaged from each camera, the video cameras 11 correspond to image data obtained from each camera. This shows the content of class assignment. In the figure, for example, a class -22.5 is assigned to image data obtained by imaging the detection target person H from a camera to which -22.5 is assigned. In the present embodiment, the case where the front face is imaged is the class number 0 degrees related to the relative face direction. In addition,
"-" Indicates an angle in the counterclockwise direction from the optical axis of the video camera 11 in FIG.

In the linear discriminant analysis for identifying the unknown data, the feature vector xi relating to the four-directional surface features extracted from the unknown data is mapped based on the coefficient matrix A, and the feature vector yi (= Axi) is obtained. Generate Next, a distance (hereinafter referred to as a square distance) Dij which is a square of the Euclidean distance between the generated feature vector yi and the average feature vector yj of each class is calculated by the following equation (1). Pattern recognition is performed by determining the class in which the riding distance Dij has the minimum value (see FIG. 6). afterwards,
The camera direction (the direction in which the optical axis γ of the video camera 11 is oriented; see FIG. 1) and the relative face are calculated by the following equation (2) using the class corresponding to the values of the lower three square distances Dij including the minimum value. Angle F with the direction (relative face direction to the optical axis γ) β
Is estimated. Note that i is omitted from Dj in FIG. 6 and corresponds to Dij in the present specification.

Dij = | yi-yj | ² (1)

[0042]

(Equation 1) In Expression (2), i indicates a class number, and in the present embodiment, n = 3 is assumed. Therefore, the class number corresponding to the lower three square distances Dij including the minimum value is:
It is assigned to i in order from the class number corresponding to the minimum value.
θ indicates the relative angle of the face direction in each class (the angle formed by the relative face direction with respect to the camera direction = class number). In Equation (2), j is omitted from the square distance Dij.

(Face Direction Determination) In step S5, each camera personal computer 14 uses the result of the face direction estimation performed in step S4 to set the estimated face direction in the relative face direction to a predetermined angle (this embodiment). It is determined whether it is within the range of ± 20 degrees in the embodiment. If the angle is within the predetermined angle (S5: YES), the process proceeds to S6. In the present embodiment, the condition of whether or not the estimated angle is within a predetermined angle (for example, ± 20 degrees) may be referred to as a predetermined condition.

At this time, since the video cameras 11 are not arranged at regular intervals, the image data in which the angle F of the relative face direction is within a predetermined angle (± 20 degrees), in other words, the predetermined conditions described above are satisfied. The number of image data to be satisfied is not limited to one.
Therefore, in the present embodiment, there are two cameras 11 that have imaged the frontal face with the relative face direction angle F within a predetermined angle, and the image data captured by the video camera 11A and the video camera 11B are candidates for which the eyes are directed, That is, a predetermined condition is satisfied,
The following description will be continued assuming that the eye area has been determined as an eye area detection target described later. The camera personal computer 14 that has determined that the estimated face orientation angle F does not satisfy the predetermined condition (NO in S5) does not perform the following steps for the current image data, and ends this flowchart. .

(Line-of-sight detection) The outline of the following S6 to S10 will be described. The video camera 11A and the video camera 11B
The camera personal computer 14 detects the eye area 32 from the face area 31 (see FIG. 9). And pupil area 3
5, the size of the pupil region 35 is normalized, and the pupil region 36 is further detected therefrom.
And the pupil center C2 are calculated, and the distance between both positions is calculated (measured). Then, the calculation result of the distance is transmitted to the main personal computer 16. The main personal computer 16 detects (estimates) the line of sight by comparing the distance calculation results received from the camera personal computers 14 of the video cameras 11A and 11B. In the present embodiment, the pupil center C1 corresponds to a first predetermined part, and the pupil center C2 corresponds to a second predetermined part.

(Detection of Eye Area) In S6, first, the camera personal computer 14 recalculates the skin color reference value for the image data and extracts the skin color area. Of the extracted skin color regions, the largest region is determined to be the face region 31.

The camera personal computer 14 has its face area 31
, An eye region 32 and a mouth region are detected by a template matching method using a four-way surface feature and a color difference surface feature.

By the way, if the eye area 32 and the mouth area are detected in S6 in the image data processed by using this flowchart immediately before the current image data, based on the previous detection result. The face area 31 obtained this time is deleted by a predetermined area, and the face area 31 is set as a search range narrowed by the predetermined area. The search range is used for the current image data, and the eye region 32 and the mouth region are detected by the template matching method. When the eye region 32 and the mouth region are not detected in the search range as a result of performing the template matching, the detection of both regions is performed again on the face region 31.

Here, the template matching method will be described. According to this method, a four-directional surface feature (directional surface) and a color difference surface feature based on U and V coordinate values are extracted from the obtained image data by the above-described four-directional surface feature extraction, and obtained by skin color region extraction. The similarity is calculated for the skin color area (face area 31) or the search range using the right eye, left eye, and mouth templates.

The color difference plane feature indicates a difference between the U value and the V value from the skin color reference value. In addition, the template means that a plurality of images of the right eye, the left eye, and the mouth are prepared in advance, the image data obtained by extracting the four-way surface features and the color difference surface features is reduced at a predetermined ratio, and the width is set to a predetermined pixel (for example, 32 Pixels) and normalize the size. Regarding the four-directional surface features, the edge direction information is decomposed into four directions, and the four-directional surface features and the color difference surface features are smoothed by a Gaussian filter, and each image data is converted to an 8 × 8 resolution. It was done. This template is stored in a storage device (not shown).

Then, the similarity a of the four-way plane feature between the template T and the image data (input image) I is calculated by the following equation (3), and the similarity b of the chrominance plane feature is calculated by the following equation (4). ).

[0052]

(Equation 2) In the expressions (3) and (4), I indicates an input image, and T indicates a template. i and j are values of 1 to m and 1 to n,
It corresponds to a template of m × n pixels and an input image. (X, y) indicates the upper left coordinates of the input image. Also,
(4) where Tu and Tv are the UV values of the template, Iu,
Iv indicates the UV value of the image data, and Umax and Vmax
Indicates the maximum range of the V value. In this embodiment, the CIE L * u * v color system is used, and in this CIELUV color system,
To speed up processing and save storage space, Uma
x = 256 and Vmax = 256.

Next, based on the similarities a and b calculated by the equations (3) and (4), the final similarity c is calculated by the following equation (5). c = Wa × a + Wb × b (5) In the equation (5), Wa and Wb are predetermined constants to be multiplied by the similarities a and b as weightings, and Wa + W
b = 1 is satisfied. In this embodiment, Wa =
Wb is set to 0.5.

Based on the calculation result, a portion where the similarity c is equal to or larger than a predetermined threshold value is set as an eye candidate region. The input image (image data) is given upper left coordinates in advance, and the positional relationship between the eyes and the mouth can be grasped based on the coordinates. Therefore, based on the coordinates, for example, the eye satisfies the rough positional relationship (coordinate position) between the eyes and the mouth, such as the right eye and the left eye, and the combination having the highest similarity c is determined as the eye area. 32 and the mouth area.
As a result, the eye region 32 is detected in the face region 31.

(Pupil Detection) Next, in S7, the camera personal computer 14 performs pupil detection for detecting the center C1 of the pupil from the detected eye area 32. In the present embodiment, pupil detection and pupil detection described below are performed on one of the eye regions 32 (for example, the right eye) among the eye regions 32 detected in S6.

First, a chroma histogram of an eye area image is created, and a known discriminant analysis method is applied to convert the face area 31 into an eye area 32 and a skin area (an area other than the eye area 32 of the face area). To separate. Generally, the saturation of the skin area is high and the eye area 3
The saturation of 2 is low. For this reason, this separation process utilizes its characteristics. Next, a luminance histogram of the eye area image is created, and a known discriminant analysis method is applied to divide the separated eye area 32 into a pupil area 35 and a white eye area 34.

Thereafter, based on the detection result of the pupil region 35, the pupil region 35 is reduced or enlarged and normalized to a predetermined size. Then, circular interpolation is performed on the pupil region 35.
At this time, as described above, in the pupil region 35 obtained by applying the discriminant analysis method to each of the chroma value histogram and the luminance histogram and dividing the pupil region 35, as shown in FIG. The existence of 35a is considered. At this time, when the grayscale value of the image is usually represented by 8 bits, the grayscale value 0 is black and the grayscale value 256 is white. Therefore, a horizontal projection histogram is created for an area having a gray value of 0 (black) in the area division result (see FIG. 8B), and as shown in the upper part of the histogram in the vertical axis direction, an extreme peak is obtained. Is deleted based on a preset threshold. That is, the portion of the shade 35a due to the eyelid appears as a peak on the histogram, and by deleting it,
As shown in FIG. 8C, only the pupil region 35 is extracted. Note that, in the present embodiment, the direction of the vertical axis corresponds to FIGS.
(C) and FIG. 9, the vertical direction is shown, and the horizontal axis direction is
8 (a) to 8 (c) and FIG. 9 show the left and right directions.

Next, with respect to the eye region 32, the white eye region 34
The contour (edge) is extracted by generating an edge image of the pupil region 35 shown in FIG. afterwards,
A circle equation of the pupil region 35 is obtained by using a well-known Hough transform for the point group forming the contour. As a result, the pupil center C1 is detected from the circular equation (see FIG. 9).

(Pupil Detection) Next, in S8, the camera personal computer 14 performs pupil detection for detecting the center C2 of the pupil from the detected pupil area 35. At this time, since the pupil region 36 is very small, in the image data that has been detected up to the pupil region 34, it is not possible to discriminate the difference in shading between the pupil and the iris and perform edge extraction. C
2 cannot be detected. Therefore, the video cameras 11A, 1
1B is zoomed up, and as shown in FIG.
2 is acquired.

Then, the pupil region 35 (iris) and the pupil region 3
The contour (edge) is extracted by generating an edge image of the pupil region 36 using the Prewitt operator using the difference in shading of No. 6. Thereafter, the size of the pupil is estimated based on the size of the pupil (for example, 1/3 to 1/1 of the pupil).
5) Using the estimation result, a circle equation of the pupil region 36 is obtained by a well-known Hough transform for the point group forming the contour. At this time, since various things are projected on the pupil, when the Prewitt operator extracts an edge of the pupil region 36, a contour (edge) other than the pupil region 36 may be detected. For this reason, the pupil center C1
The detection accuracy of the pupil region 36 is increased by using only the edges detected in the vicinity. Then, the pupil center C2 is detected from the circular equation (see FIG. 9).

(Gaze determination (camera determination)) Then, S9
In FIG. 9, as shown in FIG. 9, the camera personal computer 14 calculates (measures) a distance between the calculated pupil center C1 and the pupil center C2, that is, a shift amount of the pupil center C2 with respect to the pupil center C1. . Then, each camera personal computer 14 transmits the result of the calculated shift amount to the main personal computer 16. Since the time of each camera personal computer 14 is adjusted to the time of the main personal computer 16, the shift amount transmitted from each camera personal computer 14 is calculated from the image data captured at the same time.

At S10, the main personal computer 16
The displacement amount received from the camera personal computer 14 of the video camera 11A and the camera personal computer 1 of the video camera 11B
Then, the video camera to which the line of sight is pointed is determined by comparing the deviation amount received from No. 4. At this time, the one with the smaller displacement is regarded as the video camera to which the line of sight is directed. When the line of sight is determined, the main personal computer 16 outputs a command signal to the electric device 17 corresponding to the video camera to which the line of sight is directed (S11). The line of sight is thus detected.

Therefore, according to the above embodiment, the following effects can be obtained. (1) In the above embodiment, the camera personal computer 14 determines whether the image data captures a frontal face within a predetermined angle range based on the estimated face orientation angle, and determines whether the image data satisfy the condition. After normalizing the size of the pupil region 35 in the eye region 32, the pupil center C1 and the pupil center C2 are calculated, and the amount of deviation between both positions is calculated. Then, the main personal computer 16 compares the respective shift amounts corresponding to the respective video cameras 11A and 11B, and performs a gaze estimation that the gaze is directed to the video camera 11A having the smallest shift amount. For this reason, unlike the related art, it is possible to preferably detect the line of sight even in a large indoor space without mounting the device on the head. Further, even when there are a plurality of video cameras for imaging the frontal face, it is possible to accurately estimate the camera to which the line of sight is directed by comparing the deviation amounts.

(2) In the above embodiment, pupil detection is performed after acquiring image data in which the eye region 32 is enlarged. For this reason, the difference between the pupil and the iris can be reliably determined, and pupil detection can be suitably performed.

(3) In the above embodiment, pupil detection for detecting a line of sight is performed by extracting a contour (edge) using image data captured by the zoomed-up video camera 11 using a Prewitt operator. Furthermore, the point group was realized by performing Hough transform. Therefore, for example, unlike the case where a light source is provided in each video camera 11 and each light source irradiates infrared light to detect the pupil center C2 based on the reflected light reflected from the pupil region 35 (pupil), There is no problem that external light is disturbed and jumps and infrared light becomes noise, and the pupil center C2 can be easily obtained.
Can be detected.

(4) In the above embodiment, in order to detect the line of sight, the pupil center C1 is detected, and further the pupil center C2 is detected. Then, in the final determination of the line of sight, it was determined which video camera the line of sight was directed to based on the amount of deviation between the pupil center C1 and the pupil center C2. Therefore, the eye area 3
Unlike the case where the line of sight is detected based on the amount of displacement between the other parts in 2, the direction of the line of sight can be detected most accurately.

(5) In the above embodiment, the pupil is detected by using only the edges detected near the pupil center C1.
The circular equation of the pupil region 36 is obtained by the Hough transform. Usually, the pupil is often located near the pupil center C1, so that the detection accuracy of the pupil region 36 can be improved.

The above embodiment may be modified as follows. In the above embodiment, pupil detection may be performed by the following method. That is, the video camera 11 is provided with a light source for irradiating infrared light. When infrared light is used, the pupil region appears white. At this time, the range with high luminance corresponds to the pupil region 36, and the range with low luminance corresponds to the iris region. Then, the pupil region 36 (high-brightness (bright) range) is detected by binarization based on the threshold. Then, the center of gravity of the pupil region 36 is calculated, and the center of gravity is set as the pupil center C2. Also at this time, the zoom-up by the video camera 11 is performed in order to appropriately capture the pupil region 36.

In this case, the timing at which the pupil region 35 is irradiated with infrared light is controlled by the main personal computer 16. That is, in S4, the angle F of the relative face direction estimated by each camera personal computer 14 is input to the main personal computer 16, and it is determined whether the angle F satisfies a predetermined condition (the process of S5). 16 is performed. Then, the main personal computer 16 outputs a control signal to the camera personal computer 14 that satisfies the predetermined condition, and the predetermined video cameras 11A and 11B sequentially emit infrared light from each light source, and the camera 1
S6 ~ on the camera personal computer 14 corresponding to 1A, 11B
The process of S9 is performed again. The main PC 16
For the camera personal computer 14 for which no control signal was output from, the processing after S6 is not performed for the current image data.

Even in this case, each video camera 11
Since the infrared light whose timing is controlled is emitted from the light sources A and 11B, the infrared light is not disturbed and jumps, and the infrared light does not become noise, so that the pupil center C2 can be easily detected. Further, the pupil region can be clearly distinguished within the pupil region by the infrared light. In this case, the main personal computer 16
Corresponds to the determination means.

Also, even when infrared light is used, the pupil area 36 may be detected by the edge extraction and Hough transform using the Prewitt operator to detect the pupil center C2. In the above-described embodiment, the final determination of the line of sight is performed based on the amount of deviation between the pupil center C1 and the pupil center C2, but other parts in the pupil region 35 are used instead of the pupil center C1 or the pupil center C2. Then, the shift amount may be obtained.

In the above embodiment, the main personal computer 16
Although the communication with the camera personal computer 14 is performed by socket communication via Ethernet, the communication may be performed by wireless radio waves.

In the above embodiment, the calculation of the circular equation of the pupil region 35 is performed by the Hough transform, but it may be performed by the following method. That is, four points are sampled from a group of points constituting the contour extracted using the Prewitt operator by a known four-point sampling method. Then, using the four points, a circle equation of the pupil region 35 is obtained by a known least square method.

In the above embodiment, the pupil detection and the pupil detection in S7 and S8 are performed on one of the eye regions 32 detected in S6.
This may be performed on both the right and left eye regions 32. In this case, the average value of the shift amounts calculated in each eye region 32 is calculated, and the value is used as the shift amount of each image data.
Be compared. In this way, it is possible to perform the gaze detection with higher accuracy than in the case of calculating the shift amount for one eye.

In the above embodiment, the line of sight is detected at the pupil center C
Although it was performed based on the amount of deviation between 1 and the pupil center C2, the center of gravity C3 of the eye region 32 may be used instead of the pupil center C2 as shown in FIG. In this case, after the eye region 32 is detected in S6, the eye region 32 is enlarged or reduced to normalize it to a predetermined size, and the center of gravity C3 is obtained for the normalized eye region 32. Then, in S9, the amount of deviation between the pupil center C1 and the center of gravity C3 of the eye region 32 is calculated, and the line of sight is estimated. In this way, it is possible to detect the line of sight without having to zoom up the video camera 11 as compared with the case where the pupil center C2 is used. That is, the amount of deviation can be obtained by simple calculation from low-resolution image data in which a pupil cannot be detected. In this case, S8 becomes unnecessary.

In the above embodiment, the pupil centers C1 and pupil centers C1 calculated by the camera personal computers 14 are determined on the assumption that the plurality of video cameras 11 image the front face within a predetermined angle.
The main PC 16 compares the amount of deviation from the value of
Although the line of sight has been detected, the line of sight may be detected not by comparing the amounts of deviation but by comparing with a threshold value. That is, for example, 1
Camera personal computer 14 corresponding to one video camera 11
If only the personal computer 16 determines that the angle F of the relative face direction is within a predetermined angle, the main personal computer 16
4 is compared with a preset threshold value. When the threshold value is exceeded, the detection target person H
Is looking at the video camera 11.

Even in such a case, it is possible to preferably perform the gaze detection. Further, even when the displacement amounts are transmitted from the plurality of camera personal computers 14 to the main personal computer 16, it is possible to detect the line of sight by comparing each displacement amount with a threshold value. In the above embodiment, a plurality of video cameras 11 are installed, but one video camera 11 may be installed.

Next, technical ideas that can be grasped from the above-described embodiment and other examples will be described below together with their effects. (1) In the eye-gaze detecting device according to any one of claims 1 to 4, there are a plurality of the predetermined points, and when the judging means judges that a plurality of image data is a frontal face, the gaze is detected. The estimating means is a gaze detecting device that detects a gaze by comparing distances measured by the distance measuring means. In this way, even when the detection target person is imaged from a plurality of points, it is possible to preferably perform the gaze detection.

(2) In the eye-gaze detecting device according to claim 3, the pupil detection by the pupil detecting means is performed based on image data obtained by enlarging and capturing the eye area of the detection target by the imaging means. . In this way, pupil detection can be easily realized.

[0080]

As described above in detail, according to the first aspect of the present invention, it is possible to detect a line of sight suitably even in a large indoor space without mounting the apparatus on a detection target person.

According to the invention of claim 2, in addition to the effect of the invention of claim 1, by setting the first predetermined portion as the pupil center, the gaze estimating means can suitably realize the gaze estimation. According to the third aspect of the invention, in addition to the effects of the first or second aspect, the gaze estimating means can accurately estimate the gaze by setting the second predetermined portion as the center of the pupil.

According to the fourth aspect of the present invention, in addition to the effects of the first or second aspect of the present invention, the line-of-sight estimating means uses the second predetermined portion as the center of gravity of the eye region so that the line-of-sight estimating means Therefore, it is possible to suitably realize the gaze estimation.

According to the fifth aspect of the invention, the same effect as that of the first aspect is obtained. According to the invention of claim 6, in addition to the effect of the invention of claim 5, by setting the second predetermined portion as the center of the pupil, it is possible to suitably realize the gaze estimation in the gaze estimation process.

According to the seventh aspect of the present invention, in addition to the effects of the fifth or sixth aspect of the present invention, by setting the first predetermined portion as the center of the pupil, the gaze can be accurately estimated in the gaze estimation process. .

According to the eighth aspect of the present invention, in addition to the effect of the fifth or sixth aspect, the second predetermined portion is set as the center of gravity of the eye region, so that low-resolution image data can be obtained in the line-of-sight estimation process. Therefore, it is possible to suitably realize the gaze estimation.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a gaze detection device according to an embodiment of the present invention.

FIG. 2 is also a flowchart.

FIG. 3 is an explanatory diagram of image data captured by a video camera.

FIG. 4 is an explanatory diagram of image data extracted on the basis of skin color.

FIG. 5 is a conceptual diagram showing classes related to discriminant analysis.

FIG. 6 is a conceptual diagram of pattern recognition.

FIG. 7 is an explanatory diagram of learning data acquisition for an angle between an optical axis (camera direction) of a video camera and a relative face direction.

FIGS. 8A and 8C are explanatory diagrams showing pupil detection, and FIGS.
FIG. 4 is an explanatory diagram showing a horizontal projection histogram in pupil detection.

FIG. 9 is an explanatory diagram showing an eye area.

FIG. 10 is an explanatory diagram showing an eye region according to another embodiment.

[Explanation of symbols]

H: person to be detected, C1: pupil center, C2: pupil center, 11
... video camera (imaging means), 14 ... camera personal computer (face direction estimating means, determining means, eye area detecting means, pupil detecting means, pupil detecting means), 16 ... main personal computer (gaze estimating means), 31 ... face area , 32 ... eye area.

Continued on the front page F term (reference) 5B057 AA20 BA02 BA13 CA01 CA08 CA13 CA16 CE06 CE09 DA08 DA20 DB03 DB06 DB09 DC03 DC06 DC08 DC16 DC32 5L096 AA02 AA09 BA08 CA05 EA35 FA34 FA37 FA60 FA67 JA03 JA09

Claims (8)

[Claims] An image pickup means for picking up an image of a detection target person from a predetermined point, and a face for detecting a face area of the detection target person from image data picked up by the image pickup means and estimating a face direction based on the detected face area. Direction estimating means; determining means for determining whether or not the face direction estimated by the face direction estimating means is a frontal face including a predetermined angle range; and a front face in the image data determined by the determining means to be a frontal face. An eye area detecting means for detecting an eye area of the face; a distance measuring means for measuring a distance between a first predetermined part and a second predetermined part in the eye area detected by the eye area detecting means; A gaze estimating unit for estimating a gaze based on the distance measured by the measuring unit; 2. The eye-gaze detecting device according to claim 1, further comprising pupil detecting means for detecting a pupil center in the eye area, wherein the first predetermined portion is a pupil center. 3. The eye gaze detection according to claim 1, further comprising a pupil detection unit that detects a pupil center in the eye area, wherein the second predetermined part is a pupil center. apparatus. 4. The eye gaze detection according to claim 1, further comprising a center of gravity detecting means for detecting a position of a center of gravity in the eye region, wherein the second predetermined portion is a center of gravity of the eye region. apparatus. 5. An imaging step of imaging a detection target person from a predetermined point, a face direction estimation step of detecting a face area of the detection target person from captured image data and estimating a face direction based on the detected face area. A determination step of determining whether the estimated face orientation is a front face including a predetermined angle range; an eye area detection step of detecting an eye area of the front face in the image data determined to be a front face; In the detected eye area, a distance measurement step for measuring a distance between the first predetermined part and the second predetermined part in the same eye area, and a gaze estimation step for estimating a gaze based on the measured distance are provided. A featured gaze detection method. 6. The gaze detection method according to claim 5, further comprising a pupil detection step of detecting a pupil center in the eye area, wherein the first predetermined portion is a pupil center. 7. The eye gaze detection according to claim 5, further comprising a pupil detection step of detecting a pupil center in the eye region, wherein the second predetermined portion is a pupil center. Method. 8. The eye gaze detection according to claim 5, further comprising a center-of-gravity detecting step of detecting a center-of-gravity position in the eye region, wherein the second predetermined portion is a center of gravity of the eye region. apparatus.