KR100372933B1 - Apparatus and method for three dimensional position detection of eye gazing point - Google Patents

Abstract

The present invention relates to an apparatus and a method for calculating a three-dimensional coordinate value of a gaze point.

When the operator of the apparatus of the present invention gazes at a point, three-dimensional coordinate values of various points of each part of the eye are obtained from the reference point set in the apparatus of the present invention. Using these three-dimensional coordinate values, the three-dimensional position coordinate values of the gaze direction viewed by each eye and the pupil center point of the eye are calculated. Using this calculated value, the eye gaze direction is a straight line, and a gaze straight line passing through the eye center point of the eye is obtained for each eye. When the coordinates of the intersections of the gaze straight lines are found, the coordinate values become the three-dimensional coordinate values of the gaze point gaze at by the operator. In this way, a three-dimensional coordinate value of a gaze point is obtained, which is based on a point set in the apparatus of the present invention or a point at which a relative position is known from the point, and the value is applied to an automatic orientation or aiming of the director. Can be.

Description

Apparatus and method for three dimensional position detection of eye gazing point}

The present invention relates to the calculation of three-dimensional coordinate values of gaze points using images of two eyes.

The technical field to which the present invention belongs is the field of eye gaze calculation based on an image and an interface between a human and a machine.

Conventionally, the eye gaze direction was found by analyzing the shape of the pupil or the relative position of the pupil on the face, without directly obtaining and using three-dimensional position coordinate values of each point of the eye from the image of the eye. Therefore, prior knowledge of the characteristics and shape of the operator's eyes, the shape of the face or the operator's gaze, the relationship between the image acquisition device and the eye position is required, and the gaze direction calculation algorithm is complicated. In addition, since the calculated value may come out differently in various individual situations, there was a disadvantage in that correction is required for each situation.

However, in the present invention and the method, since three-dimensional position coordinate values of each point of the eye are directly obtained from the image of the eye, the three-dimensional coordinate values of the gaze point are searched spatially geometrically using the values, so that the characteristics and shape of the operator pupil No prior knowledge of the shape of the face, the gaze of the operator, the positional relationship between the image acquisition device and the eye is required, and the gaze direction calculation algorithm is simple. In addition, the calculated value is not influenced by various individual situations, and thus no need for correction is required.

Conventionally, when it is necessary to precisely aim or aim a target such as a projectile or a shooting device as a target point in space, an auxiliary device such as a scale and a scale is installed on the director, and the operator directly sees the target. They mainly used to move or aim and aim.

However, using the method and apparatus of the present invention, the operator of the present invention is based on one point set within the apparatus of the present invention or one point that can know the relative position from the point, as long as the operator merely looks at the target point. The three-dimensional coordinate value up to the pupil center of each eye of the eye is extracted, and the gaze direction of each eye is calculated, and the three-dimensional coordinate value of the gaze point of the operator of the device of the present invention can be obtained using these values. Therefore, it can be used to direct or aim the directing object automatically, move any object to the gaze point position, or mark the gaze point position. It can be used. This is a technical problem to be achieved by the present invention.

1 is a process flow diagram illustrating a method of the present invention.

2 is a name diagram of each part of the eye

3 is an explanatory diagram of a coordinate system and a gaze point position vector

The apparatus of the present invention "the three-dimensional coordinate value calculation device of the gaze point"

When the operator gazes at a point with both eyes, the gaze point is obtained by calculating a three-dimensional spatial coordinate value based on a point set in the present invention or a point at which a relative position can be known from the point. Device.

The eyes can be divided into white and black, and the white is composed of the sclera (14) and the black is composed of the iris (15) and the pupil (pupil) 16, which is the center hole of the iris. Can be. In explaining the structure and operation of the present invention, the boundary between the white and the black is called the boundary line 13 of the sclera and the iris, and the circle formed by the pupil is called the boundary line 12 of the pupil. In addition, after a circular disc (disc) including the boundary between the sclera and the iris, it is defined as the iris disc (iris disc). Similarly, after imagining the circular plate including the boundary line of the pupil, it is defined as a pupil disc. There is no difference in using either the center point of the iris disc or the center point of the dong park version to obtain the gaze straight line. Therefore, the center point of the iris disc or the center point of the dong park plate is defined as a pupil center 11.

The apparatus of the present invention comprises an "image acquisition unit" configured by arranging a camera or an image sensor in order to acquire an image of left and right eyes including the sclera and iris of the eye and a part or all of the pupil;

From the images obtained by the image acquisition unit, the three-dimensional coordinate values of the points on the boundary line between the left and right eye sclera and the iris, or the points on the pupil boundary line, or a relative position from the point can be known. The "three-dimensional coordinate value calculation part of the eye" which is calculated based on one point which exists, and (the three-dimensional coordinate value acquisition part of the eye combined with the image acquisition part and the three-dimensional coordinate value calculation part of the eye.)

A "gaze straight line calculator" for calculating a gazing line gaze by each eye with respect to the reference point by using three-dimensional coordinate values of the respective point points of the eye calculated by the eye three-dimensional coordinate value acquisition unit;

It is characterized in that it consists of a "three-dimensional coordinate value calculation unit of gaze point" to find the intersection point of the gaze straight line of each eye obtained by the gaze straight line calculation unit to determine this point as the three-dimensional coordinate value of the gaze point with respect to the reference point.

In relation to the apparatus of the present invention, when the operator gazes at a point with both eyes, the coordinate point of the gaze point is set as a reference point based on a point set in the apparatus of the present invention or a point at which a relative position is known from the point. To get the value,

An image acquiring step of acquiring an image of left and right eyes using an image acquiring means;

From the images obtained in the image acquisition step, the three-dimensional coordinate values of the points on the boundary line between the sclera and the iris of the left and right eyes, and the points on the boundary line of the pupil are determined at one point or relative position from the point of the present invention. A three-dimensional coordinate value obtaining step (10) of the eye, which comprises a three-dimensional coordinate value calculating step of the eye to be calculated based on one possible point;

A "looking straight line calculating step" 20 for calculating a gaze straight line gazed by each eye with respect to the reference point using the three-dimensional coordinate values of the points acquired in the step of acquiring the three-dimensional coordinate values of the eye;

It consists of a "three-dimensional coordinate calculation step of gaze point" (30) to find the intersection of the gaze straight line of each eye obtained in the gaze straight line calculation step and set this point as the three-dimensional coordinate value of the gaze point with respect to the reference point.

Here, a specific method for the three-dimensional coordinate value acquisition step 10 of the eye, which is the first step, is as follows.

"Obtaining three-dimensional coordinate values of the eye" (10)

An image of a left and right eye including a part or all of the sclera and the iris and the pupil of the eye is obtained by using a camera or an image sensor fixed so that the relative position does not change from the reference point set in the apparatus of the present invention.

From the obtained left and right eye images, three-dimensional coordinate values of points on the boundary line of the sclera and iris of the left and right eyes or points on the boundary line of the pupil are directly obtained from the reference point set in the apparatus.

As an example, in the images of the left and right eyes obtained from two or more cameras or image sensors, points on the boundary line of the sclera and iris of the left and right eyes or points on the boundary line of the pupil are found. Then, a coordinate value on a two-dimensional image coordinate system of a matching point for each of the points between the images of the left and right eyes is found.

Next, a triangulation method using binocular disparity of a camera or an image sensor with respect to the same point of each of the points, and a point set in the apparatus of the present invention or a relative position can be known from the point. Using one point as a reference point, three-dimensional coordinate values of the points are obtained.

Specific methods for the second step, “line of sight gaze” 20 are as follows.

The plane passing through the center point of the pupil and the gaze point is defined as a gazing plane. Obtaining two gaze planes defined in this way, and determining the intersection of the two gaze planes and defining this intersection as a gaze straight line.

Or calculating a direction in which the eye gazes; calculating a three-dimensional coordinate value of the pupil center point of the eye; and obtaining a gaze straight line passing through the eye center point of the eye with the eye gaze direction as a straight line. do.

First, the method of obtaining the two gaze planes described above is

At least three points are randomly selected on the boundary between the sclera and the iris, and two non-parallel lines are formed by connecting two random points, and vertical bisectors for each line are gaze planes. Or, at least three points are randomly selected on the boundary line of the pupil, and two non-parallel line segments formed by any two points among them are created, and vertical bisectors for each line segment are gaze planes.

And how to determine the straight line

The intersection of the two gaze planes obtained as above is obtained.

Calculating the direction the eye stares

At least three points on the boundary line between the sclera of the eye and the iris are randomly selected to create two non-parallel vectors that follow any two of them. Then, the two vectors are cross-productd so that the direction is determined to the outside of the eye, and the direction of the vector obtained as the gaze direction is determined.

Alternatively, any three of the points on the boundary line of the pupil are selected to create two non-parallel vectors that are generated by connecting any two of the points. Then, the two vectors are multiplied by the vector so that the direction is determined to the outside of the eye, and the direction of the vector derived from this is determined as the gaze direction.

The three-dimensional coordinate calculation step of the pupil center point is

At least three points are randomly selected on the boundary between the sclera and the iris, and two non-parallel segments formed by connecting two random points are formed on the iris disc. Draw and find the intersection of them. The three-dimensional coordinate value of the intersection may be determined as the three-dimensional coordinate value of the pupil center point with respect to the reference point.

Alternatively, at least three points are randomly selected on the boundary line of the pupil, and two non-parallel lines formed by connecting two random points among them are formed, and each vertical bisector is formed on the plane of the plane, which includes the boundary line of the pupil. Draw and find the intersection of them. The three-dimensional coordinate value of the intersection may be determined as the three-dimensional coordinate value of the pupil center point with respect to the reference point.

Referring to the third step, the three-dimensional coordinate value calculation step 30 of the gaze point in detail as follows.

When the operator gazes at a point with both eyes, to find the coordinate value in the three-dimensional space of the gaze point, the intersection point of the gaze straight lines of each eye obtained in the gaze straight line calculation step is found in the three-dimensional coordinate system, and the three-dimensional coordinate value of the intersection point is obtained. What is necessary is just to set it as the three-dimensional coordinate value of the gaze point with respect to the said reference point.

The three-dimensional coordinate value of the gaze point obtained here is a gaze point with respect to the optical axis of the eye. In practice, however, the optical axis of the eye and the visual axis of the eye are distorted at a fixed angle, so a simple correction can be made using this angle to find the actual gaze point.

Then, using the coordinate values calculated in the three-dimensional coordinate calculation step of the gaze point, the directing body provided with one point set in the apparatus of the present invention or one point whose relative position can be known from the point, is a gaze point. Can be oriented in a direction.

Using the coordinate values calculated in the three-dimensional coordinate calculation step 30 of the gaze point, an object located with reference to a point set in the apparatus of the present invention or a point where a relative position can be known from the point. It can be moved to a location or marked at the gaze point.

Instead of acquiring the three-dimensional coordinates of the points on the boundary between the sclera and the iris of the left and right eyes or the points of the pupil's boundary, at least three feature points are displayed on the contact lens and worn on each eye. It is easy to find the three-dimensional coordinates of the feature points needed to find a straight line.

This calculation method is described mathematically as follows.

An origin of a three-dimensional world coordinate system for obtaining a three-dimensional coordinate value of a gaze point is set at one point of the apparatus.

Figure 3 shows the setting of the coordinate system for mathematical analysis and the location of the pupil center point of the eye, the gaze direction vector of the eye, and the gaze point. Here, all positional coordinates or positional vectors used for the development of the equations are expanded with respect to the three-dimensional reference coordinate system (X, Y, Z), and (X _G , Y _G , Z _G ) is the coordinate system of the directing body. E _L is the center point of the left pupil, E _R is the center point of the right pupil, and T is the gaze point.

First, the way to calculate the eye gaze straight line,

The gaze straight line of the eye is a straight line passing through the center of the pupil and having the same direction as the gazing vector of the eye. Select any three points P ₁ , P ₂ , P ₃ on the iris circle 13 forming the boundary between the iris and the sclera to find the gaze line, or the pupil circle forming the boundary of the pupil. Choose any three points P ₁ , P ₂ , and P ₃ on (12). Then, find a plane S _1. that vertically bisects the line connecting P ₁ and P ₃ and S ₁ becomes the gaze plane containing the center of the pupil and the gaze point. Also, if we find a plane S ₂ that vertically bisects the line connecting P ₂ and P _3, then S ₂ is another gaze plane, and when we find the intersection where S ₁ and S ₂ meet, the intersection is the straight line L. .

The process of finding the gaze straight line L can be described in the following steps.

The step of obtaining a unit vector in a direction perpendicular to an iris disc, which is a virtual plane including an iris circumference, or a pupil disc, which is an imaginary plane including a dongjuk,

The three-dimensional position vectors of the three points P ₁ , P ₂ , and P ₃ are respectively In this case, the unit vector perpendicular to the iris disc or the park plate is given by the following equation.

here, Is the unit vector from point P ₃ to point P ₁ Is the unit vector from point P ₃ to point P ₂ .

The second step is to obtain the plane S ₁ which vertically bisects the line connecting points P ₁ and P ₃ and the plane S ₂ which vertically bisects the line connecting P ₂ and P ₃ as two gaze planes.

Position vector representing any point on plane S ₁ that vertically bisects the line connecting points P ₁ and P ₃ Is given by

Where α ₁ and β ₁ are real numbers with arbitrary values.

Position vector representing any point on plane S ₂ that vertically bisects the line connecting points P ₂ and P ₃ Is given by

Where α ₂ and β ₂ are real numbers with arbitrary values.

The third step of obtaining the intersection of the plane S ₁ and the plane S ₂ , that is, the gaze straight line,

In order to find the intersection of two planes S ₁ and S _2, the values of α _1, α _2, β _{1, and} β ₂ satisfying the following vector equation should be obtained.

Vector from equation (4) Since only the bay is perpendicular to the iris disc or the park, and the remaining vector terms are parallel to the iris disc or the park, the equation (4)

and,

This must be true.

If the solutions of α _{1 and} α ₂ satisfying equation (6) are α _{1C and} α _{2C ,} respectively, and β ₁ = β ₂ = β , the intersection of plane S ₁ and plane S ₂ , that is, the arbitrary The position vector representing the point Is given by

here, Is

Vector any three points on the iris circumference as given by P _1, P _2, if you select the P ₃ is provided with a position vector of the iris disc center point, any three points on the pupil circumference P _1, P _2, P ₃ If is selected, it becomes the position vector of the center point of the park.

Secondly, the method of calculating the eye gaze,

The eye gaze direction may be approximated perpendicularly to the iris disc which is the plane containing the iris circumference or the dong park which is the plane including the dong park. The error due to the approximation can simply be corrected during the gaze point calculation process. Therefore, each three-dimensional position vector of three random points on the iris circumference Or three-dimensional position vectors of three random points Is a unit vector representing the eye's gaze direction. Is obtained as in the above formula (1).

Third, the three-dimensional coordinate calculation method of the pupil center point,

As described above, the center point of the pupil is obtained as shown in Equation (8) through three steps of obtaining the gaze straight line by finding the intersection of the gaze plane. However, when describing the method for obtaining only the position of the pupil center point separately, selecting any three points P _1, P _2, P ₃ to the cavity circumference, and on the pupil disc while bisector perpendicular to a line segment connecting the P ₁ and P ₃ The intersection of the straight line / ₁ and the straight line / _{2 on} the park plate becomes the center point of the park plate while vertically dividing the line connecting P ₁ and P ₂ and P ₃ . In the equation, the position vectors representing arbitrary points on the straight line / ₁ and the straight line / ₂ are respectively

Is given by here, Is the unit vector from point P ₃ to point P ₁ Is the unit vector from point P ₃ to point P ₂ . Position vector representing the center point of the same park as the intersection of the two straight lines Is an equation

By α _{1C and} α _{2C which} are solutions of α _{1 and} α _{2 to} satisfy,

Is given by

If any _{three points} P ₁ , P ₂ , and P ₃ on the iris circumference are selected, the position vector obtained by the above method Represents the center point of the iris disc. The center point of the dong park version and the center point of the iris disc thus obtained are the center points of the pupil.

Fourth, the method of calculating the three-dimensional coordinates of the gazing point of the eye,

Each of the target candidate's 3D position vectors of the center of the left pupil and the center of the right pupil The gaze direction unit vectors of the left and right eyes, respectively Let's say. Position vector representing the point on the gaze line connecting the center point of the left pupil and the gaze point T And a position vector representing a point on the gaze straight line connecting the center point of the right pupil and the gaze point T. Is given by

Where k ₁ and k ₂ are arbitrary real numbers. Since the intersection of the above two straight lines becomes the gaze point, to find the position of the gaze point

We need to find the values of k ₁ and k ₂ that satisfy. For k _1c and k _2c , the solutions of k _{1 and} k ₂ satisfying equation (15), the position vector of the gaze point is

Given by

Fifth, the method of directing the director in the direction of gaze point,

Referring to FIG. 3, the position vector of the gaze point based on the directing coordinate system Is

The position vector of the gaze point with respect to the reference coordinate system It can be found simply by using

Becomes here, Is the position vector of the origin of the orientation coordinate system with respect to the reference coordinate system. And, the angle of rotation of the directing body for directing the gaze point Are each

Is calculated.

Sixth, a method of moving an object or mark to a gaze point position or marking a gaze point position,

The position vector of the gaze point computed relative to the reference coordinate system Move the object to the three-dimensional coordinate value of, or Just mark the three-dimensional coordinate value of.

As described above, in the present invention, unlike the conventional method, the operator of the apparatus of the present invention only needs to stare at the target point. As a reference point, a three-dimensional coordinate value for the gaze point is extracted.

The feature of this device is that it is not necessary to know in advance the positional relationship between the device of the present invention and the eye. In addition, the apparatus of the present invention does not require prior knowledge of eye features, facial features, gaze habits, and the like.

That is, the apparatus of the present invention directly extracts three-dimensional coordinate values of each point of the eye, and these values are calculated based on a point set in the apparatus of the present invention or a point at which the relative position is known from the point. Therefore, the apparatus and the operator of the present invention can obtain the three-dimensional coordinates of the gaze point at any position in space.

In addition, since the method using the image obtained from the camera or the image sensor, it is not necessary to attach additional devices to the eye or separate signal or control wiring.

By attaching the directing body to the apparatus of the present invention and allowing the operator to stare at the target point, the operator can automatically direct the directing object to the gaze point.

Examples of specific applications of the present invention include automatically directing or aiming a directing object such as a searchlight, a lighting lamp, a headlight, an indicator light, a laser, a camera, a microphone, a propellant, a projectile, and a shooting device to a gaze point gazeed by an operator. Can be. Also, any object may be moved to a gaze point position or a mark may be placed on the gaze point position. One application is to automatically move the cursor to the gaze point on the computer monitor the operator is looking at. As a result, it can be utilized to control various devices by eye gaze in various fields.

Claims (14)

Points on the boundary line 13 between the sclera 14 and the iris 15 of the left and right eyes, based on a point set in the apparatus of the present invention or a point at which a relative position is known from the point. A "three-dimensional coordinate value acquisition part of the eye" for acquiring three-dimensional coordinate values of points on the boundary line 12 of the pupil 16, A "looking straight line calculator" for calculating a gazing line for each eye by using a three-dimensional coordinate value of points calculated by the three-dimensional coordinate value obtaining unit of the eye with respect to the reference point; Finds the intersections of the gaze straight lines of the eyes obtained by the gaze straight line calculating unit, and sets this point as a three-dimensional coordinate value of the gaze point with respect to the reference point. "Three-dimensional coordinate value calculation device of the gaze point" configured. The method of claim 1, The three-dimensional coordinate value acquisition unit of the eye, An image acquisition unit for acquiring an image of left and right eyes including a part or all of the sclera, the iris and the pupil of the eye by using a camera or an image sensor fixed so that the relative position does not change from the reference point set in the apparatus of the present invention; Using the obtained left and right eye images, the three-dimensional coordinate value calculation unit of the eye that directly obtains the three-dimensional coordinate values of the points on the boundary line of the sclera and iris of the left and right eyes or the point of the pupil boundary from the reference point set in the present invention device A device characterized in that it is made. Points on the boundary line 13 between the sclera 14 and the iris 15 of the left and right eyes, or the pupil 16, based on a point set in the apparatus of the present invention or a point at which a relative position is known from the point. "Obtaining three-dimensional coordinate values of an eye" 10 for obtaining three-dimensional coordinate values of points on the boundary line 12 of A “stare straight line calculation step” 20 for calculating a gaze straight line gaze by each eye using the three-dimensional coordinate values of the points calculated by the three-dimensional coordinate value calculator of the eye with respect to the reference point; To find the intersection of the gaze straight lines of the eyes obtained by the gaze straight line calculating unit, and set this point as the three-dimensional coordinate value of the gaze point with respect to the reference point. "Three-dimensional coordinate value calculation method of gaze point" characterized in that consisting of. The method of claim 3, wherein "Obtaining three-dimensional coordinate values of the eye" (10) An image acquiring step of acquiring an image of left and right eyes including a part or all of the sclera and the iris and the pupil of the eye by using a camera or an image sensor fixed so that the relative position does not change from a reference point set in the apparatus of the present invention; And calculating a three-dimensional coordinate value of the eye by using the acquired left and right eye images to directly obtain three-dimensional coordinate values of points on the boundary line of the sclera and iris of the left and right eyes and points on the boundary line of the pupil from the reference point set in the present invention. Characterized in that the method. The method of claim 3, wherein The step of calculating the gaze straight line (20) comprises the steps of obtaining two gaze planes passing through the center of the pupil and the gaze point, and determining the intersection of the two planes to define the gaze straight line. The method of claim 5, Finding the two gaze planes, Randomly selecting at least three points on the boundary between the sclera and the iris, and creating two non-parallel segments that connect any two of the points, and creating a vertical bisector for each segment. How to feature. Or randomly selecting at least three points on the boundary line of the pupil and creating two non-parallel segments that are continuous from any two of the points and creating a vertical bisector for each line segment. How to. The method of claim 3, wherein "Straight line calculation step" (20), Calculating a direction in which the eye gazes; calculating a three-dimensional coordinate of the pupil center point of the eye; and obtaining a gaze straight line passing through the eye center point of the eye with the direction of the eye gazing in a straight line direction. Way. The method of claim 7, wherein Calculating the direction the eye stares Randomly selecting at least three points on the border between the sclera of the eye and the iris, Create two non-parallel vectors that connect any two of the points, cross-product the two vectors so that they are oriented outward of the eye, and look at the direction of the vector The method comprising the step of determining. Or optionally selecting at least three points on the boundary of the pupil, Creating two non-parallel vectors resulting from any two of the points, and multiplying the two vectors so that the direction is outward of the eye, and setting the direction of the vector as the gaze direction. Characterized in that. The method of claim 7, wherein The three-dimensional coordinate calculation step of the pupil center point is In a three-dimensional coordinate system based on a point set in the present invention or a point at which a relative position is known from the point, Select at least three points on the boundary between the sclera and the iris, create two non-parallel segments that connect any two of them, and for each line, create a vertical bisector on the plane containing the iris boundary. Drawing an intersection of the same and determining the intersection with the three-dimensional coordinates of the pupil center point with respect to the reference point. Alternatively, at least three points are randomly selected on the boundary line of the pupil, and two non-parallel lines are formed by connecting any two points among them, and a vertical bisector is drawn on the plane including the boundary line of the pupil for each line segment. Obtaining an intersection thereof and defining the intersection point in three-dimensional coordinates of the pupil center point with respect to the reference point. The method of claim 3, wherein Three-dimensional coordinate calculation step 30 of the gaze point, The point of intersection of the gaze straight lines of each eye obtained in the gaze straight line calculating step 20 is one point set in the apparatus of the present invention or a point from which the relative position can be known. The method characterized in that. The method of claim 10, How to calculate the three-dimensional coordinate value of the gaze point by correcting the direction of the gaze straight line by considering the distorted angle between the optical axis of the eye and the visual axis of the eye . The method of claim 3, wherein Using the coordinate values calculated in the three-dimensional coordinate calculation step 30 of the gaze point, the gaze director installed on the basis of one point set in the apparatus of the present invention or a point at which a relative position can be known from the point, How to orient in the direction of points. The method of claim 3, wherein Using the coordinate values calculated in the three-dimensional coordinate calculation step 30 of the gaze point, an object located with reference to a point set in the apparatus of the present invention or a point where a relative position can be known from the point. How to move to a location or mark a gaze point location. Instead of acquiring the three-dimensional coordinates of the points between the sclera and iris of the left and right eyes or the points of the pupil's border, at least three feature points are displayed on the contact lens and worn on each eye, A method of using a contact lens to find the three-dimensional coordinates of a feature point necessary to find a gaze straight line.