JP2012146199A - Sight line position detection device, sight line position detection method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily and reliably detect what a user is looking at based upon a sight line image of the user.SOLUTION: A SIFT feature point Sof a sight line camera image and a SIFT feature point Tof an image to be visually recognized are extracted, and corresponding points of the extracted SIFT feature points S, Tare derived. Then a Delaunay triangle is formed which includes, as a vertex, "the SIFT feature point Sof the sight line camera image" as a corresponding point. Further, three points in the vicinity of a sight line position of an operator are selected as feature points A, B, and C in the vicinity of the sight line position out of vertexes of the Delaunay triangle. Furthermore, "a point P' of the sight line position of the operator in the image to be visually recognized" is determined from coordinates of "SIFT feature points A', B', and C' in the image to be visually recognized" corresponding to "the feature points A, B, and C in the vicinity of the sight line position" so that the same positional relation of "a point P of the sight line position of the operator in the sight line camera image" when viewed from "the feature points A, B, and C in the vicinity of the sight line position"is obtained.

Description

  The present invention relates to a line-of-sight position detection apparatus, a line-of-sight position detection method, and a computer program, and is particularly suitable for use in detecting a line-of-sight position (actual point of view) of a user.

  In recent years, in the manufacturing industry such as the steel industry, it has been promoted to use information processing technology for improving production efficiency, ensuring safety, reducing environmental burden, and the like. As one of them, an action that a skilled person relies on his / her own experience and intuition and which is not described in a workflow or the like can be realized by analyzing gaze measurement data. If the knowledge about the actions that the skilled person is taking based on such experience and intuition (this "knowledge" is called "implicit knowledge") can be clearly indicated, the skilled person does not need to educate directly. , The tacit knowledge can be passed on to unskilled persons.

  Under such a background, there is a technique described in Non-Patent Document 1 as a technique for analyzing gaze measurement data. In Non-Patent Document 1, the position of the field camera is obtained from the field camera image. In Non-Patent Document 1, in principle, if the three-dimensional coordinates of the measurement target are known, it is possible to identify what the user was looking at.

Yuji Kobashi et al., “Three-dimensional gaze point estimation using natural feature points for head-mounted gaze measurement device”, IEICE Technical Report-Human Information Processing, Vol109, No.261, p.5- 10,2009 D.G.Lowe, "Object recognition from local scale-invariant features", International Conference on Computer Vision, Corfu, Greece (September 1999), p.1150-1157 Sugihara Atsuyoshi, “Mathematical model of Nawabari -Introduction to mathematical engineering from Voronoi diagram”, Kyoritsu Shuppan (2009)

However, in the technique described in Non-Patent Document 1, it is difficult to increase the accuracy of measurement, and there is a restriction that the three-dimensional coordinates of the measurement target are known. For this reason, the technique described in Non-Patent Document 1 is not suitable for practical use. For this reason, at present, what the user is looking at is identified by visual confirmation of the user.
The present invention has been made in view of such problems, and it is an object of the present invention to easily and reliably automatically detect what the user is looking at based on the user's visual field image. And

  The line-of-sight position detection device of the present invention is a two-dimensional image captured by a visual target image acquisition unit that acquires a visual target image that is a two-dimensional image of a region that is to be viewed by a user, and an imaging unit that the user is wearing. Field-of-view image acquisition means for associating the field-of-view image acquisition means for acquiring a field-of-view image as an image; A visual target image feature point extracting unit that extracts a feature point that is a middle point, a visual target image acquired by the visual target image acquisition unit, and a visual field image acquired by the visual field image acquisition unit , Field image feature point extraction means for extracting feature points that are points in the field image, field image feature points extracted by the field image feature point extraction means, and feature points of the field image On the other hand, the feature points of the visual target image extracted by the visual recognition object image feature point extracting means corresponding to each other in feature quantities are extracted as corresponding points, and the corresponding points are extracted as corresponding points by the corresponding point extracting means. A line of sight that extracts, from the feature points of the extracted field-of-view image, three feature points that constitute a triangle that internally includes the user's line-of-sight position in the field-of-view image acquired by the field-of-view image acquisition means. A visual point for deriving a position in the visual target image in which the position in the visual image corresponds to the position of the user's visual line in the visual field image as a user's visual line position in the visual target image. A target image upper line-of-sight position deriving unit, and the visual target image upper line-of-sight position deriving unit is a seat determined from the feature points near the three line-of-sight positions. The same coordinates when viewed from the coordinate system determined from the coordinates of the user's line-of-sight position in the visual field image and the three visual point image feature points corresponding to the three line-of-sight position neighboring feature points when viewed from the system Is derived as the user's line-of-sight position in the visual recognition target image.

  The gaze position detection method of the present invention includes a visual target image acquisition step of acquiring a visual target image that is a two-dimensional image of a region that is to be visually recognized by a user, and a two-dimensional image captured by an imaging means that the user is wearing. A visual field image acquisition step for associating a visual field image acquisition step of acquiring a visual field image that is an image, the visual recognition target image acquired in the visual recognition target image acquisition step, and the visual field image acquired by the visual field image acquisition unit A visual target image feature point extraction step for extracting a feature point that is a middle point, a visual recognition target image acquired by the visual recognition target image acquisition unit, and a visual field image acquired by the visual field image acquisition step A field image feature point extracting step for extracting a feature point that is a point in the field image, a field image feature point extracted by the field image feature point extraction step, and a feature point of the field image On the other hand, the feature points of the visual target image extracted by the visual recognition target image feature point extraction step corresponding to each other in feature quantities are extracted as corresponding points, and the corresponding points are extracted as corresponding points by the corresponding point extraction step. A line of sight that extracts, from the feature points of the extracted field-of-view image, three feature points that constitute a triangle that internally includes the user's line-of-sight position in the field-of-view image acquired by the field-of-view image acquisition step A position-neighboring feature point extracting step and a visual recognition for deriving a position in the visual recognition target image corresponding to a user's visual position in the visual field image as the user's visual position in the visual recognition target image. A target image upper line-of-sight position deriving step, and the visual target image upper line-of-sight position deriving step is a seat determined from the three line-of-sight position neighboring feature points. The same coordinates when viewed from the coordinate system determined from the coordinates of the user's line-of-sight position in the visual field image and the three visual point image feature points corresponding to the three line-of-sight position neighboring feature points when viewed from the system Is derived as the user's line-of-sight position in the visual recognition target image.

  A computer program according to the present invention causes a computer to execute each step of the eye gaze position detecting method.

  According to the present invention, the corresponding point between the feature point of the visual field image and the feature point of the visual target image is extracted, and the user's line-of-sight position in the visual field image is extracted from the feature points of the visual field image extracted as the corresponding point. Select three points that make up the vertices of the triangle included in. Then, the three points (feature points of the three field-of-view images) are extracted as gaze position neighboring feature points. From the coordinates of the user's line-of-sight position in the visual field image and the feature points of the three visual target images corresponding to the three line-of-sight position neighboring feature points when viewed from the coordinate system determined from the three line-of-sight position neighboring feature points A position having the same coordinates when viewed from a fixed coordinate system is derived as a user's line-of-sight position in the visual recognition target image. Therefore, what the user is looking at can be easily and reliably automatically detected based on the user's visual field image.

It is a figure which shows embodiment of this invention and shows an example of a functional structure of a gaze position detection apparatus. It is a figure (photograph) which shows embodiment of the present invention and shows notionally an example of a SIFT feature point of a visual field camera image, and a SIFT feature point of a visual recognition object image. It is a figure (photograph) which shows an embodiment of the present invention, and shows an example of a pair of corresponding points after removing a pair of corresponding points corresponding to an incorrect correspondence. It is a figure which shows embodiment of this invention and shows an example of the Delaunay figure obtained by the example of the Voronoi diagram and the Delaunay triangulation. It is a figure which shows embodiment of this invention and shows an example of the point of the operator's gaze position in the visual recognition target image corresponding to the point of the operator's gaze position in the visual field camera image. It is a figure (photograph) which shows an embodiment of the present invention and shows an example of a point of a worker's look position in a visual recognition object picture. It is a flowchart which shows embodiment of this invention and demonstrates an example of operation | movement of a gaze position detection apparatus. It is a flowchart which shows embodiment of this invention and demonstrates the detail of FIG.7 S706. It is a flowchart which shows embodiment of this invention and demonstrates the detail of step S707 of FIG. It is a flowchart which shows embodiment of this invention and demonstrates the detail of step S708 of FIG. It is a flowchart which shows embodiment of this invention and demonstrates the detail of FIG.7 S709.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating an example of a functional configuration of a visual line position detection device. In the present embodiment, a two-dimensional image of a region to be viewed by an operator (user) is obtained in advance with a digital camera (in the following description, “a two-dimensional image of a region to be viewed by an operator”). Are referred to as “visual recognition target images” as necessary). In addition, a head-mounted view camera is mounted on the operator's head, and a two-dimensional image obtained by imaging the worker's view range is obtained with the head-mounted view camera (in the following description, “worker The “two-dimensional image obtained by imaging the visual field range” is referred to as a “visual field camera image” as necessary). The line-of-sight position detection apparatus 100 inputs the visual target image and the visual field image, and the “visual target in the visual field image (the gaze position of the worker in the visual field image)” and the position on the image correspond to each other. It has a function of detecting the “position on the image” as the line-of-sight position on the image to be viewed.
Hereinafter, details of functions of the line-of-sight position detection apparatus 100 will be described. The hardware of the line-of-sight position detection apparatus 100 can be realized by using, for example, an information processing apparatus including a CPU, ROM, RAM, HDD, and various interfaces.

(Image acquisition unit 101)
The image acquisition unit 101 captures a field-of-view camera image (moving image) captured by a head-mounted field-of-view camera worn by the operator on the head. The head-mounted field-of-view camera captures an operator's field-of-view range and can be realized by a known technique, and therefore the details thereof are omitted here. In the following description, the “head-mounted view camera” is abbreviated as “view camera” as necessary.
In addition, the image acquisition unit 101 captures a visual recognition target image captured by a digital camera.
The image acquisition unit 101 receives, for example, a signal of a field-of-view camera image obtained by a field-of-view camera using an image interface or a signal of an image to be viewed obtained by a digital camera, and the CPU stores the image signal in a VRAM or the like. This can be achieved.

(Feature point extraction unit 102)
The feature point extraction unit 102 extracts feature points (feature amount vectors) of the “field-of-view camera image and viewing target image” acquired by the image acquisition unit 101. Here, the feature point extraction unit 102 extracts the feature points of each frame of the visual field camera image. In the present embodiment, Scale-Invariant Feature Transform (SIFT) is used for extracting feature points of an image. SIFT is a technique that determines the pixel of interest from the image as a feature point and calculates the feature value from the luminance distribution in the area around the feature point. For this reason, it is mainly used for applications such as image retrieval for retrieving the same image from a database. Since the SIFT algorithm can be realized by a known technique as described in Non-Patent Document 2 and the like, detailed description thereof is omitted here.
FIG. 2 is a diagram (photograph) conceptually showing an example of the SIFT feature point (FIG. 2A) of the visual field camera image and the SIFT feature point (FIG. 2B) of the visual recognition target image. FIG. 2 shows an example in which the worker is looking at a monitor that displays the operating state of the blast furnace. In FIG. 2, SIFT feature points are indicated by white circles.
The SIFT feature point S ⁱ of the visual field camera image is represented by the following equation (1), and the SIFT feature point T ^j of the visual recognition target image is represented by the following equation (2). That is, both SIFT feature points S ⁱ and T ^j are vectors having 128-dimensional elements.

In the equation (1), i is 1, 2,... N (N is the number of SIFT feature points of the field-of-view camera image). In the equation (2), j is 1, 2,... M (M is the number of SIFT feature points of the visual recognition target image).
In the feature point extraction unit 102, for example, the CPU reads out the data of the visual field camera image and the visual recognition target image from the VRAM or the like, extracts the SIFT feature points by the SIFT algorithm, and extracts the extracted SIFT feature point information. This can be realized by storing in a RAM or the like.

(Feature point matching unit 103)
The feature point matching unit 103 calculates an error between the feature quantity of the SIFT feature point S ⁱ of the visual field camera image and the feature quantity of the SIFT feature point T ^j of the visual recognition target image as shown in the following equation (3). Derived as Euclidean distance d (i, j).

Next, the feature point matching unit 103 searches for the SIFT feature point T ^{j (i, 0)} of the visual target image having the closest Euclidean distance d (i, j) to the SIFT feature point S ⁱ of the visual field camera image. Further, the feature point matching unit 103 searches for the SIFT feature point T ^{j (i, 1)} of the visual target image whose Euclidean distance d (i, j) is the second closest to the SIFT feature point S ⁱ of the visual field camera image. .
Next, the feature point matching unit 103 performs the Euclidean distance d (i, j (i, 0)) between the SIFT feature point S ⁱ of the visual field camera image and the SIFT feature point T ^{j (i, 0)} of the viewing target image. And the Euclidean distance d (i, j (i, 1)) between the SIFT feature point S ⁱ of the field-of-view camera image and the SIFT feature point T ^{j (i, 1)} of the image to be viewed. 4) It is determined whether or not the expression is satisfied.

When the expression (4) is satisfied as a result of this determination, the feature point matching unit 103 uses the SIFT feature point S ⁱ of the visual field camera image and the SIFT feature point T ^{j (i, 0) of} the viewing target image as the feature amount. Are set as corresponding points corresponding to each other. On the other hand, if the equation (4) is not satisfied, the SIFT feature point S ⁱ of the visual field camera image and the SIFT feature point T ^{j (i, 0) of} the visual recognition target image are set as corresponding points whose feature amounts correspond to each other. Not. As described above, in the present embodiment, the field-of-view camera image is only used when the difference between the Euclidean distances d (i, j (i, 0)) and d (i, j (i, 1)) is larger than a predetermined value. The SIFT feature point S ⁱ and the SIFT feature point T ^{j (i, 0) of} the visual recognition target image are set as corresponding points. In addition, “0.5” in the formula (4) is an example, and may be a value other than “0.5”.

The feature point matching unit 103 derives the above Euclidean distance d (i, j ⁾ , searches for the SIFT feature points T ^{j (i, 0)} and T ^{j (i, 1)} , determines the equation (4), and deals with them. Points are set for all SIFT feature points S ⁱ of the field-of-view camera image.
After that, the feature point matching unit 103 has a plurality of corresponding points (a plurality of “SIFT feature points T ^{j (i, 0)} ” of the view target image ⁾ corresponding to the same “SIFT feature points S ⁱ of the field-of-view camera image”. It is determined whether it has been set. When a plurality of corresponding points are set as a result of this determination, the feature point matching unit 103 selects a pair of corresponding points having the closest Euclidean distance d (i, j) from the pair of corresponding points, Remove other pairs of corresponding points.

In the feature point matching unit 103, for example, the CPU reads the information of the SIFT feature point S ⁱ of the field-of-view camera image and the information of the SIFT feature point T ^j of the visual target image from the RAM or the like, as described above. obtains a SIFT feature point S ⁱ of view camera image, SIFT feature point S ⁱ and the feature quantity of the field of view camera image and a SIFT feature point T ^j of the visual target image corresponding to each other ^{(i, 0)} as the corresponding points This can be realized by storing information on the obtained corresponding points in a RAM or the like.

(Incorrect response removal unit 104)
The erroneous correspondence removing unit 104 corresponds to a pair of corresponding points obtained by the feature point matching unit 103 (SIFT feature points S ⁱ of the field camera image, SIFT feature points S ⁱ of the field camera image, and feature amounts mutually. Four SIFT feature points T ^{j (i, 0)} ) of the image to be viewed are selected at random.
Next, the incorrect correspondence removing unit 104 uses the coordinates on the image of the four selected pairs of corresponding points to display the “parameter a ₁ for projective transformation a ₁ expressed by the following formulas (5) and (6). ˜a ₈ ”is derived.

In Expressions (5) and (6), (x ′, y ′) is coordinates on the visual field target image, and (x, y) is coordinates on the visual field camera image.
Next, the erroneous correspondence removing unit 104 selects a pair of corresponding points that are not used for deriving the parameters a _{1 to} a ₈ for projective transformation. Then, the erroneous correspondence removing unit 104 sets the coordinates (x, y) of the “SIFT feature point S ⁱ of the visual field camera image”, which is one of the selected pair of corresponding points, on the visual field camera image, and parameters a _{1 to} a ₈ . Is substituted into equations (5) and (6) to perform projective transformation, and the coordinates (x ′, y ′) on the image to be viewed are derived. The erroneous correspondence removal unit 104 then derives the “coordinate (x ′, y ′) on the viewing target image” and the “SIFT feature point T ^{j (i, 0 of the} viewing target image), which is the other of the selected corresponding points. ^)] , A projection error that is an error from the coordinates (x ′, y ′) on the visual recognition target image is derived. Then, the erroneous correspondence removing unit 104 adds 1 to the variable C when the obtained projection error is equal to or less than the threshold value. On the other hand, when the obtained projection error exceeds the threshold value, the variable C does not change.
The erroneous correspondence removing unit 104 derives a projection error and compares the projection error with a threshold value for all of the pairs of corresponding points that are not used to derive the parameters a _{1 to} a ₈ for projective transformation. Addition of variable C is performed.

The erroneous correspondence removing unit 104 performs the above processing (selection of corresponding point pairs, derivation of parameters a _{1 to} a ₈ , derivation of projection error, comparison between projection error and threshold, and addition of variable C) in four ways. Performed for all combinations of “corresponding point pairs”. Thus, the same number of variables C as all combinations of the four “pairs of corresponding points” are obtained.
Then, erroneous corresponding removal unit 104 selects the maximum value C _max of the variable obtained C, and set the parameters a ₁ ~a ₈ derived in order to obtain the maximum value C _max as the optimal projective transformation parameter.

Next, the incorrect correspondence removing unit 104 selects one pair of corresponding points obtained by the feature point matching unit 103.
Next, the erroneous correspondence removal unit 104 selects the coordinates (x, y) of the “SIFT feature point S ⁱ of the visual field camera image” that is one of the selected pair of corresponding points and the optimal projective transformation parameter a _1. ˜a ₈ is substituted into the equations (5) and (6) to perform projective transformation, and the coordinates (x ′, y ′) on the visual recognition target image are derived. The erroneous correspondence removal unit 104 then derives the “coordinate (x ′, y ′) on the viewing target image” and the “SIFT feature point T ^{j (i, 0 of the} viewing target image), which is the other of the selected corresponding points. ^)] , A projection error that is an error from the coordinates (x ′, y ′) on the visual recognition target image is derived. Then, if the calculated projection error exceeds the threshold value, the erroneous correspondence removing unit 104 removes the selected corresponding point pair as a pair of erroneous corresponding points.
The miscorresponding removal unit 104 obtained the above selection of the corresponding point pair, the derivation of the projection error, the comparison between the projection error and the threshold value, and the removal of the corresponding point pair by the feature point matching unit 103. Repeat for all pairs of corresponding points.
FIG. 3 shows a pair of corresponding points (SIFT feature point S ⁱ of the field-of-view camera image and SIFT feature point S ⁱ of the field-of-view camera image and the feature amount after the corresponding pair of corresponding points corresponding to each other is removed. It is a figure (photograph) which shows notionally an example of SIFT feature point Tj ^{(i, 0)} ) of the visual recognition target image to be performed. FIG. 3 shows an example in which 96 pairs of corresponding points between the SIFT feature point S ⁱ of the visual field camera image 301 and the SIFT feature point T ^{j (i, 0)} of the viewing target image 302 are obtained. ing. In FIG. 3, both ends of the white straight line are corresponding points, but this straight line is shown for convenience of explanation and is not displayed in an actual image.

  For example, if the corresponding correspondence point is read from the RAM or the like and the corresponding correspondence point is determined to be an erroneous correspondence point as described above, the erroneous correspondence removal unit 104 reads the corresponding point information. This can be realized by erasing information from the RAM or the like.

(Gaze position vicinity feature point extraction unit 105)
Gaze position near the feature point extracting unit 105 selects all SIFT feature point S ⁱ of the field of view camera image is set as a corresponding point.
Next, the line-of-sight position feature point extraction unit 105 performs a Delaunay triangulation (Delaunay Triangulation) on a triangle whose vertex is a coordinate (x, y) of the SIFT feature point S ⁱ of the field-of-view camera image on the field-of-view camera image. (This triangle is called a Delaunay triangle).
For this purpose, the line-of-sight position neighboring feature point extraction unit 105 creates a Voronoi Diagram. The Voronoi diagram means a space divided by a Voronoi region V (p _i ) that satisfies the following expression (7).

Here, it is assumed that a point set P = [p ₁ , p ₂ ,..., P _m ] composed of m points (Voronoi Generator) is given. In the equation (7), R ⁿ represents an n-dimensional real number space. D (p, p _i ) and D (p, p _j ) represent Euclidean distances between a point p and points (Voronoi generators) p _i , p _j . That is, equation (7) represents that the point (Voronoi generator) p _i that is closest to the Euclidean distance from the arbitrary point p in the Voronoi region V (p _i ) is the point (Voronoi generator) p _i. A set of such points is used as a Voronoi region.
In such a Voronoi diagram, a diagram in which adjacent points (Voronoi generators) are connected to each other is called Delaunay triangulation. When the Delaunay triangulation is performed in this way, a Delaunay triangle is created so that the minimum interior angle of the triangle is maximized (the triangle is not elongated as much as possible). In addition, the Delaunay triangulation has a property (circumscribed circle characteristic) that the vertices of other Delaunay triangles are not included in the circle circumscribing any created Delaunay triangle. Note that the Delaunay triangulation algorithm can be realized by a known technique as described in Non-Patent Document 3 and the like, and therefore, detailed description thereof is omitted here.

FIG. 4 is a diagram illustrating an example of a Voronoi diagram (FIG. 4A) and an example of a Delaunay triangle obtained by Delaunay triangulation (FIG. 4B).
In FIG. 4, points of coordinates (x, y) on the field-of-view camera image of the SIFT feature point S ⁱ of the field-of-view camera image are Voronoi generators p _{1 to} p ₈ .
When the Delaunay triangle is obtained as described above, the line-of-sight position neighboring feature point extraction unit 105 detects the worker's line-of-sight position in the visual field camera image, and whether the detected line-of-sight position of the worker is inside the Delaunay triangle. Determine whether or not. As a result of this determination, if the operator's line-of-sight position is inside the Delaunay triangle, the line-of-sight position neighboring feature point extraction unit 105 forms “SIFT feature points S ⁱ of the field-of-view camera image” that constitute the three vertices of the Delaunay triangle. Are set as eye-gaze position neighboring feature points A, B, and C (A, B, and C represent coordinates in the visual field camera image).

On the other hand, when the worker's line-of-sight position is not inside the Delaunay triangle, the line-of-sight position feature point extraction unit 105 configures the “field camera” that constitutes both end points of the Delaunay triangle side that is closest to the worker's line-of-sight position. selecting a SIFT feature point S ⁱ "of the image (the two the SIFT feature point S ⁱ of view camera image a, and B. these a, B denote the coordinates in the field of view camera image) .
Next, the line-of-sight position feature point extraction unit 105 selects the feature point S ⁱ having the closest Euclidean distance to the worker's line-of-sight position among the SIFT feature points S ⁱ of the field-of-view camera image. Select from other than SIFT feature points A and B of image (“SIFT feature point S ⁱ of field-of-view camera image” selected here is C. C represents the coordinates in the field-of-view camera image) .
Next, the line-of-sight position feature point extraction unit 105 determines whether or not the angle (∠ACB) based on the selected “SIFT feature points A, B, and C of the visual field camera image” satisfies the following expression (8). To do.
30 ° <∠ACB <150 ° (8)
Expression (8) is for making the triangle having the vertexes of “the coordinate points A, B, and C in the field-of-view camera image” of the SIFT feature point S ⁱ of the field-of-view camera image as thin as possible. Therefore, the lower limit and upper limit of ∠ACB are not limited to 30 ° and 150 °, respectively, and the lower limit and the upper limit of ∠ACB can be determined within a range not departing from this gist.

As a result of this determination, when the expression (8) is satisfied, the eye-gaze position neighboring feature point extraction unit 105 selects the “viewing target whose feature quantity corresponds to the selected“ SIFT feature points A, B, and C of the visual field camera image ”. SIFT feature point T ^{j (i, 0)} ”of the image is selected (“ SIFT feature point T ^{j (i, 0)} ”of the image to be viewed is set to A ′, B ′, and C ′, respectively ⁾ . A ′, B ′, and C ′ represent coordinates in the visual recognition target image).
Next, the line-of-sight position feature point extraction unit 105 has the same sign as the outer product of the two-dimensional vector shown in the following expression (9) and the sign of the outer product of the two-dimensional vector shown in the following expression (10). It is determined whether or not. As a result of this determination, if the signs of these outer products are the same, the eye-gaze position feature point extraction unit 105 converts the SIFT feature points A, B, and C of the selected visual field camera image to the eye-gaze position feature points A, B, C.

On the other hand, when the signs of these outer products are not the same, the positional relationship between the vertices of the triangle formed by the SIFT feature points A, B, and C of the field-of-view camera image and the SIFT feature points A ′ and B ′ of the viewing target image , C ′ is different from the positional relationship of the vertices of the triangle. This occurs due to image distortion and the like. Therefore, in such a case, the line-of-sight position near feature point extraction unit 105, among the SIFT feature point S ⁱ of the field of view camera image, the Euclidean distance is next closest feature point S ⁱ of the working line of sight position, selected Select from other than “SIFT feature points A and B of two field-of-view camera images” (“SIFT feature point S ⁱ of field-of-view camera image” selected here is C. C represents the coordinates in the field-of-view camera image. To represent). And the process mentioned above is performed repeatedly.
In addition, when the expression (8) is not satisfied, the line-of-sight position feature point extraction unit 105 determines points other than “coordinate points A and B in the field-of-view camera image” of “SIFT feature points S ⁱ of two field-of-view camera images”. It is determined whether or not all are selected.
As a result of this determination, if all the points are selected but there are no points that satisfy the conditions of the expressions (8), (9), and (10) that are the conditions for selecting the point C, gaze position near the feature point extracting unit 105, among the SIFT feature point S ⁱ selected far field of view camera image, the Euclidean distance is closest feature point S ⁱ of the working line of sight position, selected "two views Reselect from other than the SIFT feature points A and B of the camera image (the “SIFT feature point S ⁱ of the visual field camera image” selected here is C. Also, C represents the coordinates in the visual field camera image. To do). Then, the line-of-sight position feature point extraction unit 105 selects the line-of-sight camera point SIFT feature points A, B, and C as the line-of-sight position feature points A, B, and C. To do.

On the other hand, when not all the points are selected, the eye-gaze position neighboring feature point extraction unit 105 extracts the feature point S having the next closest Euclidean distance to the operator's eye-gaze position among the SIFT feature points S ⁱ of the visual field camera image. ⁱ is selected from other than the selected “SIFT feature points A and B of the two field-of-view camera images” (“SIFT feature point S ⁱ of the field-of-view camera image” selected here is C. C is the field of view) Representing coordinates in the camera image). The line-of-sight position feature point extraction unit 105 and the line-of-sight position feature point extraction unit 105 perform the above-described equation (8) based on the selected “SIFT feature points A, B, and C of the visual field camera image”. Determine again.
In this way, the feature point in the vicinity of the line-of-sight position when the line-of-sight position of the worker is not inside the Delaunay triangle is determined.

In the line-of-sight position feature point extraction unit 105, for example, the CPU reads information on corresponding points from the RAM or the like, and as described above, the three “field-of-view cameras” in the vicinity of the line-of-sight position of the worker This can be realized by setting the SIFT feature point S ⁱ ”of the image as the line-of-sight position vicinity feature points A, B, and C, and storing the line-of-sight position vicinity feature points A, B, and C in a RAM or the like.

(Viewing target image upper line-of-sight position deriving unit 106)
FIG. 5 is a diagram illustrating an example of “a point P of the operator's line-of-sight position in the field-of-view camera image” and “a point P ′ of the worker's line-of-sight position in the image to be visually recognized” corresponding to the positions on the image. is there.
In the left diagram of FIG. 5, an “oblique coordinate system with the point A of the line-of-sight position vicinity feature points A, B, and C as the origin and the line segments AB and AC as axes obtained by the line-of-sight position feature point extraction unit 105. The point P of the operator's line-of-sight position in the field-of-view camera image is represented by the following equation (11): When this equation (11) is transformed, the parameters s and t are expressed by the following (12 In the equations (11) and (12), A, B, C, and P represent coordinates in the field-of-view camera image, respectively.

  The visual target image upper line-of-sight position deriving unit 106 derives parameters s and t from the line-of-sight position neighboring feature points A, B, and C obtained by the line-of-sight position neighboring feature point extracting unit 105 using Equation (12). . Then, the visual target image upper line-of-sight position deriving unit 106 selects “SIFT feature points A ′, B ′, C ′ of the visual target image” corresponding to the feature amounts corresponding to the “visual point vicinity feature points A, B, C”. Using the parameters s and t, “the point P ′ of the operator's line-of-sight position in the image to be viewed” is derived as the line-of-sight position on the image to be viewed by the following equation (13) (the right diagram in FIG. 5 is shown). reference).

  In the present embodiment, by calculating the equations (12) and (13), the operator's view in the field-of-view camera image when viewed from the coordinate system determined from the “line-of-sight position neighboring feature points A, B, C”. Coordinates determined from “the coordinates of the point P of the line-of-sight position” and “SIFT feature points A ′, B ′, C ′ of the image to be viewed” corresponding to the “feature point vicinity feature points A, B, C” corresponding to each other. When viewed from the system, the position having the same coordinates is obtained as “a point P ′ of the operator's line-of-sight position in the image to be viewed”.

FIG. 6 is a diagram (photograph) conceptually illustrating an example of the point P ′ of the operator's line-of-sight position in the visual recognition target image.
Point u _a left view of FIG. _{6, u b, u c,} p e , respectively, the line-of-sight position neighbor feature points A, B, C, corresponding to the point P of the operator of the line-of-sight position in the field of view camera image. On the other hand, the point U _a of the right view in FIG. _{6, U b, U c,} P e , respectively, A'point gazed object image, B', C', point operator gaze location in viewing the target image P Corresponds to '.

  The visual target image upper line-of-sight position deriving unit 106 includes information on the line-of-sight position feature points A, B, and C, information on the point P of the worker's line-of-sight position in the visual field camera image, and line-of-sight position vicinity feature points A, B, Information on the points A ′, B ′, and C ′ of the visual recognition target image corresponding to C and the feature amount are read from the RAM or the like, and the point of the operator's line-of-sight position in the visual recognition target image as described above. This can be realized by deriving P ′ and storing information on the point P ′ of the operator's line-of-sight position in the image to be viewed in a RAM or the like.

(Viewing target image upper line-of-sight position display unit 107)
The visual target image upper line-of-sight position display unit 107 uses the information of “the point P ′ of the worker's visual line position in the visual target image” derived by the visual target image upper line-of-sight position deriving unit 106 as a display device such as a liquid crystal display. indicate. For example, the visual target image upper line-of-sight position display unit 107 displays a mark indicating the point ^Pe as shown in the right diagram of FIG. 6 (white in the right diagram of FIG. 6) on the visual target image shown in FIG. X) can be displayed on the display device.
In the visual target image upper line-of-sight position display unit 107, for example, the CPU reads information on the point P ′ of the visual line position of the worker in the visual target image from the RAM or the like, and generates display data for displaying the information. This can be realized by outputting the generated display data to a display device.

(Operation flowchart of the line-of-sight position detection apparatus 100)
Next, an example of the operation of the eye gaze position detecting device 100 will be described with reference to the flowchart of FIG.
First, in step S701, the image acquisition unit 101 acquires a visual recognition target image.
Next, in step S702, the image acquisition unit 101 acquires a visual field camera image.
Next, in step S703, the feature point extraction unit 102 extracts SIFT feature points of the viewing target image from the viewing target image acquired in step S701.
Next, in step S704, the feature point extraction unit 102 selects one frame of the visual field camera image acquired in step S702 in the acquisition order.

Next, in step S705, the feature point extraction unit 102 extracts SIFT feature points of the visual field camera image from the frame of the visual field camera image acquired in step S702.
Next, in step S706, the feature point matching unit 103, a SIFT feature point S ⁱ on the viewing target image, SIFT feature points of the viewing target image SIFT feature point S ⁱ and the feature quantity of the field of view camera image correspond to each other A feature point matching process for obtaining T ^{j (i, 0)} as a corresponding point is performed. Details of the feature point matching process will be described later with reference to FIG.
Next, in step S <b> 707, the incorrect correspondence removing unit 104 performs an erroneous correspondence removing process for removing the pair of erroneous corresponding points from the pair of corresponding points obtained in step S <b> 706. Details of the error handling removal process will be described later with reference to FIG.

Next, in step S708, the line-of-sight position feature point extraction unit 105 selects the field of view among the “SIFT feature points S ⁱ of the field-of-view camera image” that constitute the corresponding points after the erroneous correspondence is removed in step S707. A line-of-sight position feature point extraction process is performed to extract three “SIFT feature points S ⁱ of the visual field camera image” in the vicinity of the worker's line-of-sight position in the camera image. Details of the gaze position vicinity feature point extraction process will be described later with reference to FIG.
Next, in step S709, the visual target image upper line-of-sight position deriving unit 106 performs a visual target image upper line-of-sight position derivation process for deriving a point P ′ of the operator's line-of-sight position in the visual target image. The details of the visual target image upper line-of-sight position derivation process will be described later with reference to FIG.

Next, in step S 710, the visual target image upper line-of-sight position display unit 107 displays the information of “the point P ′ of the worker's visual line position in the visual target image” derived in step S 709 on the display device. Display.
Next, in step S711, the feature point extraction unit 102 determines whether all frames of the visual field camera image acquired in step S702 have been acquired. As a result of this determination, if all the frames of the visual field camera image have not been acquired, the process returns to step S704. On the other hand, when all the frames of the visual field camera image have been acquired, the processing according to the flowchart of FIG.

Next, details of the feature point matching process in step S706 of FIG. 7 will be described with reference to the flowchart of FIG.
First, in step S801, the feature point matching unit 103, a SIFT feature point S ⁱ of view camera image unselected, selects one from among those extracted in step S705.
Next, in step S802, the feature point matching unit 103 sets the feature quantity of the SIFT feature point S ⁱ of the visual field camera image selected in step S705 and the SIFT feature point T ^j of the visual target image extracted in step S703. These Euclidean distances d (i, j) are derived using equation (3) as an error from each of the feature quantities.
Next, in step S803, the feature point matching unit 103 determines the SIFT feature point T ^{j (i, 0) of the} viewing target image having the smallest Euclidean distance d (i, j) from the SIFT feature point S ⁱ of the visual field camera image. Explore.
Next, in step S804, the feature point matching unit 103 determines the SIFT feature point T ^{j (i, i, of the} visual target image having the second smallest Euclidean distance d (i, j) from the SIFT feature point S ⁱ of the visual field camera image ^. Search for ¹⁾ .

Next, in step S805, the feature point matching unit 103 performs the Euclidean distance d (i, j (i ⁾ between the SIFT feature point S ⁱ of the visual field camera image and the SIFT feature point T ^{j (i, 0)} of the viewing target image. , 0)) and the Euclidean distance d (i, j (i, 1)) between the SIFT feature point S ⁱ of the visual field camera image and the SIFT feature point T ^{j (i, 1)} of the image to be viewed. , (4) is satisfied.
As a result of this determination, if the expression (4) is not satisfied, the SIFT feature point S ⁱ of the visual field camera image and the SIFT feature point T ^{j (i, 0) of} the viewing target image are not used as corresponding points. S806 is omitted, and the process proceeds to step S807 described later.

On the other hand, if the expression (4) is satisfied, the process proceeds to step S806. In step S806, the feature point matching unit 103 matches the SIFT feature points S ⁱ of the field-of-view camera image and the SIFT feature points T ^{j (i, 0) of the} image to be viewed with corresponding feature amounts. Set as.
Next, in step S807, the feature point matching unit 103 determines whether all the SIFT feature points S ⁱ of the visual field camera image extracted in step S705 have been selected. If all the SIFT feature points S ⁱ of the visual field camera image have not been selected as a result of this determination, the process returns to step S801.

On the other hand, if all the SIFT feature points S ⁱ of the visual field camera image have been selected, the process proceeds to step S808. In step S808, a plurality of corresponding points (a plurality of “SIFT feature points T ^{j (i, 0)} ” of the viewing target image) are set for the same “SIFT feature point S ⁱ of the visual field camera image”. Determine whether or not. As a result of this determination, if a plurality of corresponding points are set, the process proceeds to step S809.
In step S809, the feature point matching unit 103 selects a pair of corresponding points having the closest Euclidean distance d (i, j) from the corresponding points, and removes the other corresponding points. Thus, the SIFT feature point S ⁱ of the field of view camera image, SIFT feature point T ^{j (i, 0)} of the visual target image SIFT feature point S ⁱ and the feature quantity corresponding to each other of the viewing camera images and a pair 1 relationship. Then, the process proceeds to step S707 in FIG.
On the other hand, when a plurality of corresponding points are not set, it is not necessary to remove the corresponding points, so step S809 is omitted and the process proceeds to step S707 in FIG.

Next, details of the erroneous correspondence removal processing in step S707 of FIG. 7 will be described with reference to the flowchart of FIG.
First, in step S901, the erroneous correspondence removing unit 104 determines the four corresponding point pairs (SIFT feature points S ⁱ of the field camera image and the field camera image obtained in step S706 of FIG. 7 (flowchart of FIG. 8)). SIFT feature points S ⁱ and SIFT feature points T ^{j (i, 0)} ) of the image to be viewed whose feature values correspond to each other, and a pair of unselected corresponding points is selected.
Next, in step S902, the erroneous correspondence removing unit 104 uses the coordinates on the image of the four corresponding point pairs selected in step S901 to display “projection transformation of formula (5)” and (6). Parameters a _{1 to} a ₈ ”are derived.

Next, in step S903, the erroneous correspondence removal unit 104 is different from the pair of corresponding points not used for deriving the parameters a _{1 to} a ₈ for projective transformation (the pair of corresponding points selected in step S901). A pair of corresponding points) which is an unselected pair of corresponding points.
Next, in step S904, the erroneous correspondence removing unit 104 determines the coordinates (x, y) on the field camera image of the “SIFT feature point S ⁱ of the field camera image” which is one of the pair of corresponding points selected in step S903. And parameters a _{1 to} a ₈ are substituted into the equations (5) and (6) to perform projective transformation, and the coordinates (x ′, y ′) on the image to be viewed are derived. Then, the erroneous correspondence removing unit 104 selects the derived “coordinate (x ′, y ′) on the viewing target image” and “SIFT feature point T of the viewing target image, which is the other of the selected corresponding points selected in step S903”. ^A projection error, which is an error between the coordinates (x ′, y ′) on the visual recognition target image of ^{j (i, 0)} ”, is derived.

Next, in step S905, the erroneous correspondence removing unit 104 determines whether or not the projection error derived in step S904 is equal to or less than a preset threshold value. If the result of this determination is that the projection error is not less than or equal to the threshold, the “projection transformation parameters a _{1 to} a ₈ ” derived in step S 902 are not correct according to the projection error derived in step S 904. Therefore, the variable C is not changed. Therefore, step S906 is omitted and the process proceeds to step S907 described later.
On the other hand, if the projection error is less than or equal to the threshold value, the process proceeds to step S906. In step S906, the incorrect correspondence removing unit 104 adds 1 to the variable C.

Next, in step S907, the erroneous correspondence removing unit 104 is different from the pair of corresponding points not used for deriving the parameters a _{1 to} a ₈ for projective transformation (the pair of corresponding points selected in step S901). It is determined whether or not all pairs of corresponding points have been selected. As a result of this determination, if not all corresponding pairs of corresponding points not used for deriving the parameters a _{1 to} a ₈ for projective transformation are selected, the process returns to step S903.
On the other hand, if all corresponding pairs of corresponding points that are not used for deriving the parameters a _{1 to} a ₈ for projective transformation are selected, the process proceeds to step S908. In step S908, the incorrect correspondence removing unit 104 determines whether all combinations of the four corresponding point pairs obtained in step S706 of FIG. 7 (the flowchart of FIG. 8) have been selected. As a result of the determination, if all the combinations of the four corresponding points are not selected, the process returns to step S901. On the other hand, when all the combinations of the four corresponding points are selected, the process proceeds to step S909. In the case of proceeding to step S909, the same number of variables C as all combinations of the four “corresponding point pairs” are obtained.

When the process proceeds to step S909, erroneous corresponding removal unit 104, resulting selects the maximum value C _max of the variable C has a maximum value C _max was derived to obtain the parameters a ₁ ~a ₈ optimal projective transformation parameters Set as.
Next, in step S910, the incorrect correspondence removal unit 104 selects one pair of unselected corresponding points from the pair of corresponding points obtained in step S706 (the flowchart in FIG. 8).
Next, in step S911, the erroneous correspondence removing unit 104 determines the coordinates (x, y) on the field camera image of the “SIFT feature point S ⁱ of the field camera image” which is one of the pair of corresponding points selected in step S910. And the optimal projection transformation parameters a _{1 to} a ₈ are substituted into the equations (5) and (6) to perform the projection transformation, and the coordinates (x ′, y ′) on the visual recognition target image are derived. The erroneous correspondence removal unit 104 then derives the “coordinate (x ′, y ′) on the viewing target image” and the “SIFT feature point T ^{j (i, 0 of the} viewing target image), which is the other of the selected corresponding points. ^)] , A projection error that is an error from the coordinates (x ′, y ′) on the visual recognition target image is derived.

Next, the incorrect correspondence removing unit 104 determines whether or not the projection error derived in step S911 is equal to or less than a preset threshold value. If the result of this determination is that the projection error is less than or equal to the threshold value, the corresponding point selected in step S910 is not a miscorresponding corresponding point, so step S913 is omitted and processing proceeds to step S914 described later.
On the other hand, if the projection error is not less than or equal to the threshold value, the process proceeds to step S913. In step S913, the incorrect correspondence removing unit 104 removes the pair of corresponding points selected in step S910 as a pair of corresponding points corresponding to the wrong correspondence.
Next, in step S914, the incorrect correspondence removing unit 104 determines whether all the pairs of corresponding points obtained in step S706 (the flowchart in FIG. 8) have been selected. As a result of the determination, if not all the corresponding point pairs have been selected, the process returns to step S910. On the other hand, when all the pairs of corresponding points have been selected, the process proceeds to step S708 in FIG.

Next, details of the line-of-sight position feature point extraction process in step S708 of FIG. 7 will be described with reference to the flowchart of FIG.
First, in step S1001, the line-of-sight position feature point extraction unit 105 sets the SIFT feature points of the field-of-view camera image set as the corresponding points from which the incorrect correspondences are removed in step S707 of FIG. 7 (the flowchart of FIG. 9). all the S ⁱ to select.
Next, in step S1002, the line-of-sight position neighboring feature point extraction unit 105 uses the Delaunay having the coordinates (x, y) on the field camera image of the “SIFT feature point S ⁱ of the field camera image” selected in step S1001 as a vertex. A triangle is created (see FIG. 4B).

Next, in step S1003, the line-of-sight position feature point extraction unit 105 detects the line-of-sight position of the worker in the visual field camera image, and determines whether the detected line-of-sight position of the worker is inside the Delaunay triangle. . As a result of the determination, if the operator's line-of-sight position is inside the Delaunay triangle, the process proceeds to step S1004.
In step S1004, the line-of-sight position feature point extraction unit 105 sets “SIFT feature points S ⁱ of the visual field camera image” constituting the three vertices of the Delaunay triangle as line-of-sight position feature points A, B, and C. To do. Then, the process proceeds to step S709 in FIG.
On the other hand, if the operator's line-of-sight position is not inside the Delaunay triangle, the process proceeds to step S1005. In step S1005, the line-of-sight position feature point extraction unit 105 selects “SIFT characteristic points A and B of the field camera image” that constitute both end points of the Delaunay triangle whose Euclidean distance is closest to the line-of-sight position of the operator. select.
Next, in step S1006, the line-of-sight position feature point extraction unit 105 performs a step of selecting a feature point C having the closest Euclidean distance to the operator's line-of-sight position among unselected SIFT feature points S ⁱ of the visual field camera image. Selection is made from other than “SIFT feature points A and B of two field-of-view camera images” selected in S1005.
Next, in step S1007, the line-of-sight position feature point extraction unit 105 selects “SIFT feature points A and B of the field camera image” selected in step S1005 and “SIFT feature point C of the field camera image selected in step S1006”. It is determined whether or not the angle based on (∠ACB) satisfies the expression (8).

If the result of this determination is that the expression (8) is not satisfied, the routine proceeds to step S1011 described later. On the other hand, if the expression (8) is satisfied, the process proceeds to step S1008.
In step S1008, the line-of-sight position feature point extraction unit 105 selects “SIFT feature points A and B of the visual field camera image” selected in step S1005 and “SIFT feature point C of the visual field camera image” selected in step S1006. And “SIFT feature points A ′, B ′, C ′ of the image to be viewed” corresponding to each other.
Next, in step S1009, the line-of-sight position neighboring feature point extraction unit 105 has the same sign as the outer product of the two-dimensional vector shown in Expression (9) and the sign of the outer product of the two-dimensional vector shown in Expression (10). It is determined whether or not. As a result of this determination, if the signs of these outer products are not the same, the positional relationship between the vertices of the triangle formed by the SIFT feature points A, B, and C of the field-of-view camera image and the SIFT feature point A ′ of the viewing target image , B ′ and C ′ are different from each other in the positional relationship of the vertices of the triangle. Therefore, the SIFT feature points A, B, and C of the field-of-view camera image cannot be set as feature points near the line-of-sight position. Therefore, the process returns to step S1006.

On the other hand, if the signs of these outer products are the same, the process proceeds to step S1010. In step S1010, the line-of-sight position feature point extraction unit 105 selects the “SIFT feature points A and B of the field camera image” selected in step S1005 and the latest “SIFT feature points of the field camera image selected in step S1006”. C ”is set as the feature point A, B, C near the line-of-sight position. Then, the process proceeds to step S709 in FIG.
As described above, if the expression (8) is not satisfied in step S1007, the process proceeds to step S1011. In step S1011, the line-of-sight position feature point extraction unit 105 selects all the points other than the “SIFT feature points A and B of the two field camera images” selected in step S1005 as the SIFT feature points C of the field camera image. Determine whether or not. If all the points other than the SIFT feature points A and B of the two field-of-view camera images have not been selected as a result of this determination, the process returns to step S1006.

On the other hand, if all points other than the SIFT feature points A and B of the two field-of-view camera images have been selected, the process proceeds to step S1012. In step S1012, the line-of-sight position feature point extraction unit 105 has the shortest Euclidean distance from the line-of-sight position of the worker among the SIFT feature points S ⁱ of the visual field camera image. The feature point C is selected again from other than the “SIFT feature points A and B of the two field-of-view camera images” selected in step S1005. Then, the process proceeds to step S1010. When the process proceeds from step S1012 to step S1010, the line-of-sight position feature point extraction unit 105 selects the “SIFT feature points A and B of the visual field camera image” selected in step S1005 and the “SIFT of the visual field camera image” selected in step S1012. The feature point C ”is set as the feature point A, B, C near the line-of-sight position. Then, the process proceeds to step S709 in FIG.

Next, details of the line-of-sight position feature point extraction process in step S708 of FIG. 7 will be described with reference to the flowchart of FIG.
First, in step S1101 of FIG. 11, the visual target image upper line-of-sight position deriving unit 106 determines based on the line-of-sight position feature points A, B, and C obtained in step S708 (flow chart of FIG. 10) of FIG. The parameters s and t are derived from the equation (12). As shown in the equations (11) and (12), the parameters s and t are the side AB having the feature point A near the line-of-sight position as the origin and the feature points A and B near the line-of-sight position, and the line of sight This is a parameter for representing the point P (coordinate in the field-of-view camera image) of the line-of-sight position in the oblique coordinate system with the vicinity of the position vicinity feature points A and C as the sides AC.

  Next, in step S1102, the visual target image upper line-of-sight position deriving unit 106 selects “SIFT feature points A ′ and B of the visual target image corresponding to the characteristic points A, B and C near the visual line position”. By using “′, C ′” and the parameters s, t derived in step S1101, “the point P ′ of the operator's line-of-sight position in the visual recognition target image” is set as the visual recognition target image upper line-of-sight position. (See the right figure of FIG. 5). Then, the process proceeds to step S710 in FIG.

(Summary)
As described above, in the present embodiment, first, the three-dimensional space that is the target of eye gaze position detection is converted into a visual object image (two-dimensional plane) photographed with a digital camera or the like, and the actual eye gaze position detection processing is performed. Limited to two-dimensional planes. Then, the SIFT feature point S ⁱ of the field-of-view camera image that is an image of the mobile gaze measurement camera and the SIFT feature point T ^{j of the} image to be viewed are extracted, and the corresponding points of the extracted SIFT feature points S ⁱ and T ^j are determined. To derive. Next, a Delaunay triangle whose vertex is the coordinate (x, y) on the field camera image of the “SIFT feature point S ⁱ of the field camera image” which is the corresponding point is formed. Next, three points in the vicinity of the operator's line-of-sight position are selected from the vertices of the Delaunay triangle formed as line-of-sight position vicinity feature points A, B, and C. Next, the “near gaze position neighborhood” is the same as the positional relationship of “the point P of the gaze position of the worker in the visual field camera image” when viewed from the selected “gaze point neighboring feature points A, B, C”. From the coordinates on the image of “SIFT feature points A ′, B ′, C ′ of the viewing target image” whose feature amounts correspond to the feature points A, B, C ”, the“ visual line position of the operator in the viewing target image ” Of point P ′ ”. Therefore, the feature points of the two-dimensional visual field camera image and the two-dimensional visual recognition target image can be associated with each other, and the visual point position neighboring feature point A surrounding the point P of the worker's visual line position in the visual field camera image. , B, and C can be prevented from being extremely crushed (one of the inner angles of the triangle is extremely small). Therefore, what is viewed in the actual three-dimensional space does not require the three-dimensional coordinates, and without performing the three-dimensional projective transformation, the point P ′ of the operator's line-of-sight position in the image to be viewed can be easily obtained. It can be obtained with high accuracy by calculation processing. Therefore, the point P ′ of the operator's line-of-sight position in the visual recognition target image can be obtained without being affected by the error of the projective transformation parameter. Further, the point P ′ of the operator's line-of-sight position in the visual recognition target image can be accurately obtained without performing “preprocessing for removing distortion of the image” for reducing the error in the parameters of the projective transformation. As described above, in this embodiment, the correspondence between the visual field camera image that the worker is viewing and the visual target image that is obtained in advance as a region that is the visual target of the worker can be obtained. It is possible to easily and reliably automatically detect whether or not the user is watching.
Further, in the present embodiment, the feature points A, B, and C in the vicinity of the line-of-sight position are extracted after removing the corresponding points corresponding to the erroneous correspondence. It can be determined even more accurately.

(Modification)
In this embodiment, in addition to being applicable to a general image, there is a feature that it has high robustness. Therefore, SIFT is used for extracting feature points of a visual field camera image and a visual recognition target image. . However, if feature points can be extracted, the method is not limited to SIFT.
Further, in the present embodiment, “corresponding point pair selection, derivation of parameters a _{1 to} a ₈ , derivation of projection error, comparison of projection error and threshold value, and addition of variable C” by erroneous correspondence removing unit 104 ( Steps S901 to S908) are performed for all combinations of the four “pairs of corresponding points”. However, this is not always necessary. For example, as a result of performing these processes on all combinations of four “corresponding point pairs”, the number of times of performing these processes is smaller than the number of all combinations of four “corresponding point pairs”. Even when it is confirmed that the same optimal projective transformation parameters as those obtained by performing these processes for all the combinations of the four “pairs of corresponding points” can be obtained, Processing may be performed.

Further, in this embodiment, the line-of-sight position feature point extraction unit 105 performs a Delaunay triangulation method on a triangle whose vertex is the coordinate (x, y) on the field-of-view camera image of the SIFT feature point S ⁱ of the field-of-view camera image. (Delaunay Triangulation), and when the operator's line-of-sight position is inside the Delaunay triangle, the “SIFT feature points S ⁱ of the visual field camera image” constituting the three vertices of the Delaunay triangle Points A, B, and C were set. However, this is not always necessary. For example, it may be as follows. First, all SIFT feature points S ⁱ of the visual field camera image set as corresponding points are selected. Next, among the SIFT feature points S ⁱ of the field-of-view camera image, the SIFT feature points S ⁱ are selected in order of increasing distance from the user's line-of-sight position, and three triangles are formed that include the line-of-sight position inside. Feature points are extracted as feature points near the line-of-sight position. Here, among the three feature points selected in the order of the shortest distance from the user's line-of-sight position, if a triangle that includes the line-of-sight position is not formed, the feature point with the next closest distance is extracted and the line-of-sight is extracted. Three feature points that can form a triangle including the position inside are extracted as feature points near the line-of-sight position.

Further, in the present embodiment, the erroneous correspondence removing unit 104 performs projective conversion from the visual field camera image to the visual recognition target image (step S902). However, you may make it perform the projective transformation from a visual recognition target image to a visual field camera image.
Further, the imaging device may not be a head-mounted visual field camera as long as the visual field range of the worker can be captured and the worker's line-of-sight position in the captured image can be detected.

(Correspondence with claims)
In the present embodiment, for example, by using the image acquisition unit 101, a visual target image acquisition unit and a visual field image acquisition unit are realized. In the present embodiment, for example, by using the feature point extraction unit 102, the visual target image feature point extraction unit and the visual field image feature point extraction unit are realized.
In the present embodiment, for example, the corresponding point extraction unit is realized by using the feature point matching unit 103. Specifically, for example, the derivation unit is realized by the feature point matching unit 103 performing the process of step S802. In addition, for example, the feature point matching unit 103 performs the processing in steps S803 and S804, thereby realizing a search unit. In addition, for example, the feature point matching unit 103 performs the processing of steps S805 and S806, thereby realizing an extraction unit. Here, when the relationship of the expression (4) is satisfied, the difference between the error of the feature point of the viewing target image with the smallest error and the error of the feature point of the viewing target image with the second smallest error is the threshold value. It is an example in the case of larger than.
Further, in the present embodiment, for example, the gaze position vicinity feature point extraction unit is realized by using the gaze position vicinity feature point extraction unit 105. In the present embodiment, for example, the visual target image upper line-of-sight position deriving unit 106 is realized by using the visual target image upper visual line position deriving unit 106.
Further, in the present embodiment, for example, the corresponding point removing unit is realized by using the erroneous correspondence removing unit 104. Here, for example, the erroneous correspondence removing unit 104 performs the processes of steps S901 and S902 to derive parameters for projective transformation between the points of the visual field image and the points of the visual recognition target image. . In addition, for example, the incorrect correspondence removing unit 104 performs the processes of steps S910 to S912 and S914, so that the position on the image obtained by projective transformation of one of the corresponding points using the derived parameter and the corresponding correspondence. Comparing the position of the point on the other image with respect to each of the corresponding points extracted by the corresponding point extracting means is realized.

The embodiment of the present invention described above can be realized by a computer executing a program. Further, a means for supplying the program to the computer, for example, a computer-readable recording medium such as a CD-ROM recording such a program, or a transmission medium for transmitting such a program may be applied as an embodiment of the present invention. it can. A program product such as a computer-readable recording medium that records the program can also be applied as an embodiment of the present invention. The programs, computer-readable recording media, transmission media, and program products are included in the scope of the present invention.
In addition, the embodiments of the present invention described above are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. Is. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 100 Eye-gaze position detection apparatus 101 Image acquisition part 102 Feature point extraction part 103 Feature point matching part 104 Incorrect correspondence removal part 105 Eye-gaze position vicinity feature point extraction part 106 Visual object image upper eye-gaze position derivation part 107 Visual object image upper eye-gaze position display part

Claims (11)

A visual target image acquisition means for acquiring a visual target image that is a two-dimensional image of an area to be visually recognized by the user;
A field-of-view image acquisition means for acquiring a field-of-view image that is a two-dimensional image captured by an imaging means worn by a user;
A visual target image feature that extracts a feature point that is a point in the visual recognition target image for associating the visual recognition target image acquired by the visual recognition target image acquisition unit with the visual field image acquired by the visual field image acquisition unit. Point extraction means;
Field-of-view image feature point extraction for extracting feature points that are points in the field-of-view image for associating the field-of-view image acquired by the field-of-view image acquisition unit with the field-of-view image acquired by the field-of-view image acquisition unit Means,
The feature points of the visual field image extracted by the visual field image feature point extracting unit and the visual target image extracted by the visual target image feature point extracting unit corresponding to the feature points of the feature points of the visual field image correspond to each other. Corresponding point extracting means for extracting feature points as corresponding points;
From the feature points of the visual field image extracted as the corresponding points by the corresponding point extraction unit, three characteristic points constituting a triangle that internally includes the user's line-of-sight position in the visual field image acquired by the visual field image acquisition unit, Eye gaze position neighborhood feature point extracting means for extracting gaze position neighborhood feature points;
A visual target image upper line-of-sight position deriving unit for deriving a position in the visual recognition target image corresponding to a user's visual line position in the visual field image as a user's visual line position in the visual recognition target image; Have
The visual target image upper line-of-sight position deriving means calculates the coordinates of the user's line-of-sight position in the visual field image and the three line-of-sight position neighboring feature points when viewed from a coordinate system determined from the three characteristic points near the line-of-sight position. A line-of-sight position detection apparatus, wherein a position having the same coordinates when viewed from a coordinate system determined from feature points of three corresponding visual target images is derived as a user's line-of-sight position in the visual target image.   The line-of-sight position neighboring feature point extracting unit performs Delaunay triangulation on the feature points of the visual field image extracted as the corresponding points by the corresponding point extracting unit, and the Delaunay triangle obtained by performing the Delaunay triangulation is performed. The feature points of the three visual field images constituting the vertices of the Delaunay triangle in which the visual line position of the user in the visual field image is included are extracted as the visual line position neighboring feature points. The line-of-sight position detection apparatus according to 1.   Based on the coordinates on the image of the corresponding point extracted by the corresponding point extracting means, a parameter for projective transformation between the point of the visual field image and the point of the visual recognition target image is derived, and the derived parameter is used. The position on the image obtained by projective transformation of one of the corresponding points and the position on the other image of the corresponding point are compared for each of the corresponding points extracted by the corresponding point extracting unit, The line-of-sight position detection apparatus according to claim 1, further comprising: a corresponding point removing unit that removes the corresponding point based on the result. The corresponding point extracting unit includes a feature amount of the feature point of the visual field image extracted by the visual field image feature point extraction unit, and a feature amount of the feature point of the visual target image extracted by the visual target image feature point extracting unit. Derivation means for deriving the error of
The error derived by the deriving unit is the characteristic point of the visual field image extracted by the visual field image feature point extracting unit from the characteristic points of the visual target image extracted by the visual recognition target image feature point extracting unit. Search means for searching for the feature point of the viewing target image that is the smallest and the feature point of the viewing target image that is the second smallest in error.
When the difference between the error of the feature point of the viewing target image with the smallest error and the error of the feature point of the viewing target image with the second smallest error is larger than a threshold value, The line-of-sight position detection according to any one of claims 1 to 3, further comprising: extraction means for extracting, as the corresponding point, a feature point of the visual target image with the smallest error. apparatus. The visual target image feature point extraction unit extracts the feature point of the visual target image acquired by the visual target image acquisition unit by SIFT (Scale-Invariant Feature Transform),
5. The visual field image feature point extraction unit extracts a feature point of the visual field image acquired by the visual field image acquisition unit by SIFT (Scale-Invariant Feature Transform). The line-of-sight position detection apparatus according to the item. A visual recognition target image acquisition step of acquiring a visual recognition target image that is a two-dimensional image of an area to be visually recognized by the user;
A field-of-view image acquisition step of acquiring a field-of-view image that is a two-dimensional image captured by the imaging means worn by the user;
A visual target image feature that extracts a feature point that is a point in the visual recognition target image for associating the visual recognition target image acquired in the visual recognition target image acquisition step with the visual field image acquired by the visual field image acquisition unit. A point extraction process;
Field-of-view image feature point extraction for extracting feature points that are points in the field-of-view image for associating the field-of-view image acquired by the field-of-view image acquisition means with the field-of-view image acquired by the field-of-view image acquisition step Process,
The feature points of the visual field image extracted by the visual field image feature point extracting step, the feature points of the visual field image extracted by the visual field image feature point extracting step, and the feature amounts corresponding to the feature points of the visual field image. A corresponding point extracting step of extracting feature points as corresponding points;
From the feature points of the visual field image extracted as the corresponding points by the corresponding point extraction step, three characteristic points constituting a triangle that internally includes the user's line-of-sight position in the visual field image acquired by the visual field image acquisition step, A line-of-sight position vicinity feature point extracting step for extracting as a line-of-sight position vicinity feature point;
A visual target image upper line-of-sight position deriving step of deriving a position in the visual recognition target image corresponding to a user's visual line position in the visual field image as a user's visual line position in the visual recognition target image; Have
The visual target image upper line-of-sight position deriving step uses the coordinates of the user's line-of-sight position in the visual field image and the three line-of-sight position neighboring feature points when viewed from a coordinate system determined from the three characteristic points near the line-of-sight position. A line-of-sight position detection method, wherein a position having the same coordinates when viewed from a coordinate system determined from feature points of three corresponding visual target images is derived as a user's line-of-sight position in the visual target image.   The line-of-sight position feature point extraction step performs Delaunay triangulation on the feature points of the visual field image extracted as the corresponding points by the corresponding point extraction step, and the Delaunay triangle obtained by performing the Delaunay triangulation The feature points of the three visual field images constituting the vertices of the Delaunay triangle in which the visual line position of the user in the visual field image is included are extracted as the visual line position neighboring feature points. 6. A method of detecting a gaze position according to 6.   Based on the coordinates on the image of the corresponding point extracted by the corresponding point extraction step, a parameter for projective transformation between the point of the visual field image and the point of the visual recognition target image is derived, and the derived parameter is used. Comparing the position on the image obtained by projective transformation of one of the corresponding points with the position on the other image of the corresponding point for each of the corresponding points extracted by the corresponding point extraction step The line-of-sight position detection method according to claim 6, further comprising a corresponding point removal step of removing the corresponding point based on the result. The corresponding point extracting step includes a feature amount of the feature point of the visual field image extracted by the visual field image feature point extraction step, and a feature amount of the feature point of the visual recognition target image extracted by the visual recognition target image feature point extraction step. A derivation process for deriving the error of
Among the feature points of the visual target image extracted by the visual recognition object image feature point extracting step, the error derived by the derivation step is the feature point of the visual field image extracted by the visual field image feature point extraction step. A search step for searching for a feature point of the viewing target image that is the smallest and a feature point of the viewing target image that has the second smallest error;
When the difference between the error of the feature point of the viewing target image with the smallest error and the error of the feature point of the viewing target image with the second smallest error is larger than a threshold value, The eye-gaze position detection according to any one of claims 6 to 8, further comprising: an extraction step of extracting the feature point of the visual target image with the smallest error as the corresponding point. Method. The visual recognition target image feature point extraction step extracts feature points of the visual recognition target image acquired in the visual recognition target image acquisition step by SIFT (Scale-Invariant Feature Transform),
The said visual field image feature point extraction process extracts the feature point of the visual field image acquired by the said visual field image acquisition process by SIFT (Scale-Invariant Feature Transform), The any one of Claims 6-9 characterized by the above-mentioned. The line-of-sight position detection method according to item.   A computer program for causing a computer to execute each step of the eye gaze position detecting method according to any one of claims 6 to 10.

Images