JP2011194105A - Gazing point measuring device, gazing point measuring method, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform appropriate measurement without a wearer of an eye tracking device unconsciously turning eyes to a marker, and to accurately and automatically measure the position of a gazing point in an original form image of a target object even if the target object is deformed.SOLUTION: In the gazing point measuring device, in a corresponding point retrieval process 31, a real space characteristic point 51a and an original form characteristic point 51b are extracted, and a set of a real space corresponding point 42a and an original form corresponding point 42b are retrieved. In a planar projective transformation matrix computing process 33, a planar projective transformation matrix 44 for planar-projective-transforming coordinates of a visual field image 4 to coordinates of the original form image 3 is output. In a curved face mesh generating process 35, a planar mesh 45 is generated by the planar projective transformation of an original form mesh 43 based on the planar projective transformation matrix 44, and a planar corresponding point 42c is computed by the planar projective transformation of the rear space corresponding point 42a, and a curved face mesh 46 is generated by deforming the planar mesh 45 based on the displacement 54 between the planar corresponding point 42c and the real space corresponding point 42a.

Description

  The present invention relates to a gazing point measurement device or the like that measures the position of a gazing point when a medium relating to a product or the like is analyzed using an eye tracking device.

Traditionally, in order to appropriately appeal to consumers about brands related to products or services, such as poster designs and signboards posted in various places such as product package designs, brochures, leaflets, and flyers, etc. Analysis of used POP (Point of purchase advertising), advertisements published in magazines, newspapers, etc., and product instruction manuals and service explanation documents (hereinafter referred to as “appeal medium”) is performed.
In the analysis of the appeal medium, the consumer's consumption behavior is analyzed from the product recognition to selection, purchase, use (or service recognition to selection, contract, use), psychology, etc., the brand value is the consumer It is important to know if it is properly communicated to.

One method for analyzing appeal media is to use an eye tracking system. The eye tracking system is a system for analyzing the behavior of the consumer including the behavior that the consumer is performing unconsciously by measuring and analyzing the movement of the eyeball of the consumer.
For example, consider a case where a consumer takes a product brochure and analyzes the behavior until the consumer recognizes the product. By using the eye tracking system to know the gaze point, trajectory, and gaze stop time of the consumer ’s line of sight with respect to the pamphlet, it is possible to determine whether the consumer is viewing the part that the pamphlet wants to convey. You can see, “Which are you watching?” And “Which part is the longest time in the brochure?”

  For example, in a head-mounted eye tracking device, the direction in which the face is facing can be photographed by a camera, and the gaze point of the wearer's line of sight can be mapped to the photographed image. In order to analyze the appeal medium, information indicating the position of the target object that the wearer is looking at (the position of the gazing point in the target object) is very important. Since the camera also moves, the position of the target object in the captured image changes. For this reason, the position of the gazing point in the target object is generally confirmed or specified by an analyst visually. However, the human load in the analysis is very large, which is a big problem in terms of time and cost.

In order to eliminate the above-described human load, several techniques for automatically specifying the position of a gazing point in a target object by a computer have been developed.
For example, in the technique described in Patent Document 1, markers are attached to the four corners of the target object. The coordinates of the marker are recognized from the first frame of the video imaged by the eye camera, and the parameters to be converted into the gazing point coordinates in the target object are calculated. In the second and subsequent frames, the conversion to the gazing point coordinates in the target object is shifted in accordance with the marker shift from the first frame.
Further, for example, in the technique described in Patent Document 2, the coordinates of the marker are recognized for each frame of the video captured by the eye camera, and the parameters to be converted into the gazing point coordinates in the target object are recalculated for each frame.

Sho 62-53630

Hei 3-77533

However, since the technique described in Patent Document 1 supports only the displacement caused by the parallel movement of the marker, it cannot cope with the rotation of the head or the target object. On the other hand, the technique described in Patent Document 2 can cope with the rotation of the head and the target object. However, both of the techniques described in Patent Document 1 and Patent Document 2 require markers at the four corners of the target object. Since the marker needs to be automatically recognized by image processing, it must have a conspicuous shape and color. Since the marker is more conspicuous than the appealing medium, the line of sight unconsciously turns to the marker, and appropriate measurement cannot be performed. In addition, measurement cannot be performed when a marker is removed from an image captured by an eye tracking device (hereinafter referred to as “field-of-view image”).
Furthermore, the techniques described in Patent Document 1 and Patent Document 2 can be applied only to a target object that is flat and does not deform.

As mentioned above, for appealing media such as pamphlets, it is normal for consumers to pick up and observe, and the edges of the appealing media are curved or the inside of the appealing media is uneven. Deformation different from rotation occurs.
On the other hand, when analyzing the appeal medium, the original image (hereinafter referred to as “original image”) of the target object (= appeal medium) such as the layout image of the appeal medium created by layout software or the like. And the position of the gazing point in the original image is specified in association with the position of the eye mark in the visual field image. If the deformation of the target object in the visual field image is ignored, the position of the gazing point in the original image of the target object cannot be specified accurately.
Therefore, in the analysis of appealing media of low-rigidity materials such as sheets, paper, and films, the analyst still has to visually compare the visual field image with the original image and specify the position of the gazing point in the original image. Don't be.

  The present invention has been made in view of the above-described problems, and the purpose of the present invention is to prevent the eye gaze of the wearer of the eye tracking device from unconsciously facing the marker, so that the marker deviates from the visual field image. Is to provide a gaze point measuring device or the like that can perform appropriate measurement. In addition, when the target object is a low-rigidity appeal medium, even if the target object is deformed, it is possible to automatically and accurately measure the position of the point of interest in the original image of the target object. It is to provide a device or the like.

In order to achieve the above-described object, the first invention relates to the original image of the target object included in the visual field image in association with the position of the eye mark in the visual field image that is an image captured by the eye tracking device. A gaze point measuring apparatus for measuring a position of a viewpoint, wherein a real space feature point that is a feature point of the visual field image and an original feature point that is a feature point of the original image are extracted, and the set of the real space feature points and Based on the set of original feature points, as a set of corresponding points for associating the visual field image with the original image, corresponding to the real space corresponding point of the visual field image and original correspondence corresponding point of the original image Corresponding point search means for searching for a set of points, and plane projection conversion of the coordinates of the visual field image to the coordinates of the original image based on a set of the set of the corresponding points of the real space and the original shape A transformation matrix calculation unit configured to calculate a plane projective transformation matrix, a gazing point measuring apparatus characterized by comprising a.
In the first invention, since a marker for specifying the position of the gazing point is unnecessary, the wearer of the eye tracking device does not unintentionally turn his / her line of sight to the marker, and appropriate measurement can be performed.

According to a first aspect of the present invention, there is provided an original mesh generating means for generating an original mesh that is a mesh of the original image, and planar projection transformation based on the planar projection transformation matrix with respect to the original mesh so that the planar mesh of the visual field image is obtained. A plane mesh generation means for generating a plane corresponding point by performing the plane projective transformation on the real space corresponding point, and based on a positional deviation between the plane corresponding point and the real space corresponding point It is desirable to further comprise curved surface mesh generating means for generating a curved surface mesh of the visual field image by deforming a planar mesh.
Accordingly, when the appeal medium made of a material having low rigidity is used as the target object, the position of the gazing point in the original image of the target object can be accurately and automatically measured even when the target object is deformed.

Further, the curved mesh generation means in the first invention repeatedly deforms the planar mesh based on a determination formula including a positional shift between the plane corresponding point and the real space corresponding point and a deformation complexity. It is desirable to generate the optimal curved mesh.
As a result, the displacement can be corrected by smooth deformation.

Further, the curved surface mesh generation means in the first invention does not employ, for example, a pair of corresponding points whose positional deviation between the plane corresponding points and the real space corresponding points is larger than a threshold value in the calculation processing of the determination formula. . Accordingly, it is possible to select a pair of corresponding points to be employed in the determination formula calculation process by excluding a pair of corresponding points that are highly likely to be associated by mistake.
In this case, it is preferable that the threshold value is decreased every time the deformation of the planar mesh is repeated. Thereby, in each case of the beginning of the repetition and the end of the repetition, an appropriate deformation can be performed using an appropriate set of corresponding points.

Further, the curved surface mesh generation means according to the first aspect of the invention includes, for example, a process of calculating the determination formula using a fixed number of corresponding points from the one having a larger positional deviation between the plane corresponding points and the real space corresponding points. Not adopted for. Accordingly, it is possible to select a pair of corresponding points to be employed in the determination formula calculation process by excluding a pair of corresponding points that are highly likely to be associated by mistake.
In this case, it is desirable to increase the certain number every time the deformation of the planar mesh is repeated. Thereby, in each case of the beginning of the repetition and the end of the repetition, an appropriate deformation can be performed using an appropriate set of corresponding points.

In the first invention, the position of the gazing point is specified on the assumption that the element of the original mesh corresponding to the element of the planar mesh or the curved mesh including the eye mark in the visual field image includes the gazing point. It is desirable to further include a gazing point position specifying means.
Thereby, the element of the original mesh including the gazing point can be specified as the position of the gazing point. The position of the gazing point can be specified in more detail by finely setting the size of the mesh.

According to a second aspect of the present invention, a gaze point that measures a position of a gaze point in an original image of a target object included in the field-of-view image in association with a position of an eye mark in the field-of-view image that is a video photographed by the eye tracking device. A measurement method that extracts a real space feature point that is a feature point of the visual field image and an original feature point that is a feature point of the original image, and is based on the set of the real space feature points and the set of the original feature points As a set of corresponding points for associating the visual field image with the original image, corresponding points for searching a set of corresponding points of the real image corresponding to the visual field image and a corresponding point of the original shape corresponding to the original image Based on the search step and a set of the real space corresponding points and the original corresponding points, a plane projective transformation matrix for plane projecting the coordinates of the visual field image to the coordinates of the original image is calculated. A transformation matrix calculation step, a gazing point measuring method, which comprises a.
In the second invention, since a marker for specifying the position of the gazing point is not necessary, the wearer of the eye tracking device does not unintentionally point the line of sight to the marker, and appropriate measurement can be performed.

According to a second aspect of the present invention, there is provided an original mesh generation step of generating an original mesh that is a mesh of the original image, and planar projection transformation based on the planar projection transformation matrix is performed on the original mesh so that the planar mesh of the visual field image A plane mesh generation step of generating a plane corresponding point by performing the plane projective transformation on the real space corresponding point, and based on a positional deviation between the plane corresponding point and the real space corresponding point Preferably, the method further includes a curved mesh generation step of generating a curved mesh of the visual field image by deforming the planar mesh.
Accordingly, when the appeal medium made of a material having low rigidity is used as the target object, the position of the gazing point in the original image of the target object can be accurately and automatically measured even when the target object is deformed.

The third invention is a program for causing a computer to function as the gazing point measuring device of the first invention.
By distributing the third invention through a network and installing it on a general-purpose computer, the gaze point measuring device of the first invention can be provided.

A fourth invention is a computer-readable storage medium storing a program for causing a computer to function as the gazing point measuring device of the first invention.
A gazing point measurement device according to the first invention is provided by mounting the fourth invention on a storage medium reader provided in a general-purpose computer and installing the program stored in the fourth invention in a general-purpose computer. be able to.

  According to the present invention, it is possible to provide a gaze point measuring device and the like capable of performing appropriate measurement even when the eye tracking device wearer does not unintentionally turn his / her line of sight to the marker and the marker is removed from the visual field image. it can. In addition, when the target object is a low-rigidity appeal medium, even if the target object is deformed, it is possible to automatically and accurately measure the position of the point of interest in the original image of the target object. A device or the like can be provided.

The figure which shows the outline | summary of the gaze point measuring system which concerns on this invention Schematic diagram of original image 3 and visual field image 4 Hardware configuration diagram of gaze point measuring device 1 Hardware configuration diagram of eye tracking device 2 The figure which shows the data concerning the gaze point measuring device 1, and the flow of a process The figure explaining the corresponding point search process 31 Schematic diagram of original mesh 43 The flowchart which shows the flow of the plane projection transformation matrix calculation process 33 The figure which shows an example of the plane projection transformation matrix 44 The figure which shows an example of the plane projection transformation matrix 44 The flowchart which shows the flow of the plane mesh generation process 34 and the curved surface mesh generation process 35 The figure which shows an example of the plane mesh 45 The figure explaining position shift 54 of real space corresponding point 42a and plane corresponding point 42c The figure which shows an example of the curved surface mesh 46 The figure explaining an example of a calculation process of the gaze point position 48 The flowchart which shows the 1st example of the selection process of the corresponding point group employ | adopted as calculation processing of a judgment formula The flowchart which shows the 2nd example of the selection process of the corresponding point group employ | adopted as calculation processing of a judgment formula

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a diagram showing an outline of a gazing point measurement system according to the present invention. FIG. 2 is a schematic diagram of the original image 3 and the visual field image 4.
As shown in FIG. 1, the gazing point measurement system according to the present invention includes a gazing point measurement device 1, an eye tracking device 2, and the like.
The gazing point measurement system according to the present invention is optimal when the target object 5 shown in FIG. 2 is an appeal medium made of a material having low rigidity. By performing the gaze point measurement of the appeal medium according to the present invention, the human burden in analyzing the appeal medium is drastically reduced. Here, a material having low rigidity means a material that can be easily deformed by human power. Examples of the material having low rigidity include sheets, paper, and films, but are not limited to these examples.

The gazing point measurement device 1 associates the position of the gazing point 7 in the original image 3 that is a still image of the target object 5 with the position of the eye mark 6 in the field-of-view image 4 that is an image taken by the eye tracking device 2. Measure.
The original image 3 is an original image of the target object 5, that is, an image that is not deformed by human power or the like. The original image 3 includes characters, figures, symbols, photographs, and the like. When the target object 5 is an appeal medium, the original image 3 is a layout image generated by a computer in which layout software is installed, for example. The original image 3 is stored in the gazing point measurement apparatus 1.

  The eye tracking device 2 is, for example, a head-mounted type, which captures a direction in which the wearer's face is facing by a camera, and maps the gaze point of the wearer's line of sight as an eye mark on the captured visual field image 4. . The captured visual field image 4 is transmitted to the gazing point measurement apparatus 1.

  As shown in FIG. 2, the original image 3 has a two-dimensional plane shape. FIG. 2 schematically shows the original image 3. The hatched portion at the top of the original image 3 indicates that the same pattern and color are given. A rectangle inside the original image 3 indicates a photograph. A plurality of horizontal lines inside the original image 3 indicate sentences.

  In addition, as shown in FIG. 2, the visual field image 4 includes a target object 5. In FIG. 2, the visual field image 4 is schematically shown. The target object 5 included in the field-of-view image 4 is irregularly deformed by human power or the like, and has a curved end or unevenness in the appeal medium. In order to make the drawing easy to understand, objects such as human hands and tables that are actually present in the real space are omitted, and all areas except the target object 5 are left blank.

  The eye tracking device 2 cannot distinguish the target object 5 from other objects and the background in the visual field image 4. The eye tracking device 2 simply maps, as an eye mark 6, where the wearer of the eye tracking device 2 is looking in the visual field image 4. Therefore, for example, if the wearer keeps an eye on the target object 5, the eye mark 6 is mapped to a region that is not the target object 5, but the eye tracking device 2 indicates that the wearer has taken the eye off the target object 5. Is not recognized.

Next, the hardware configuration of the gazing point measurement device 1 and the eye tracking device 2 will be described with reference to FIGS. 3 and 4.
FIG. 3 is a hardware configuration diagram of the gazing point measurement apparatus 1. Note that the hardware configuration of FIG. 3 is an example, and various configurations can be adopted depending on the application and purpose.

  The gazing point measurement apparatus 1 includes a control unit 11, a storage unit 12, a media input / output unit 13, a communication control unit 14, an input unit 15, a display unit 16, a peripheral device I / F unit 17, and the like connected via a bus 18. Is done.

  The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

The CPU calls and executes a program stored in the storage unit 12, ROM, recording medium, or the like to a work memory area on the RAM, drives and controls each device connected via the bus 18, and controls the gazing point measurement device 1. The process to be described later is realized.
The ROM is a non-volatile memory and permanently holds a computer boot program, a program such as BIOS, data, and the like.
The RAM is a volatile memory, and temporarily stores programs, data, and the like loaded from the storage unit 12, ROM, recording medium, and the like, and includes a work area used by the control unit 11 for performing various processes.

The storage unit 12 is an HDD (hard disk drive), and stores a program executed by the control unit 11, data necessary for program execution, an OS (operating system), and the like. With respect to the program, a control program corresponding to an OS (operating system) and an application program for causing a computer to execute processing described later are stored.
Each of these program codes is read by the control unit 11 as necessary, transferred to the RAM, read by the CPU, and executed as various means.
The original image 3 is stored in the storage unit 12.

The media input / output unit 13 (drive device) inputs / outputs data, for example, media such as a CD drive (-ROM, -R, -RW, etc.), DVD drive (-ROM, -R, -RW, etc.) Has input / output devices.
The communication control unit 14 includes a communication control device, a communication port, and the like, and is a communication interface that mediates communication between a computer and a network, and performs communication control between other computers via the network. The network may be wired or wireless.

The input unit 15 inputs data and includes, for example, a keyboard, a pointing device such as a mouse, and an input device such as a numeric keypad.
An operation instruction, an operation instruction, data input, and the like can be performed on the computer via the input unit 15.
The display unit 16 includes a display device such as a liquid crystal panel, and a logic circuit or the like (video adapter or the like) for realizing a video function of the computer in cooperation with the display device.

The peripheral device I / F (interface) unit 17 is a port for connecting a peripheral device to the computer, and the computer transmits and receives data to and from the peripheral device via the peripheral device I / F unit 17. An example of the peripheral device is the eye tracking device 2. The peripheral device I / F unit 17 is configured by USB, IEEE 1394, RS-232C, or the like, and usually includes a plurality of peripheral devices I / F. The connection form with the peripheral device may be wired or wireless.
The bus 18 is a path that mediates transmission / reception of control signals, data signals, and the like between the devices.

  FIG. 4 is a hardware configuration diagram of the eye tracking device 2. Note that the hardware configuration in FIG. 4 is merely an example, and various configurations can be adopted depending on the application and purpose.

  In the eye tracking device 2, a control unit 21, a storage unit 22, a visual field camera 23, an eye mark detection unit 24, a power supply unit 25, a switch 26, an external output unit 27, and the like are connected via a bus 28.

  The control unit 21 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

The CPU calls and executes a program stored in the storage unit 22, ROM, or the like in the work memory area on the RAM, and drives and controls each device of the eye tracking device 2 connected via the bus 28.
The ROM is a non-volatile memory and permanently holds programs, data, and the like.
The RAM is a volatile memory, and programs and data loaded from the storage unit 22 and ROM, field-of-view video 4 captured by the field-of-view camera 23, coordinate data indicating the position of the eye mark 6 detected by the eye mark detection unit 24, and the like. And a work area used by the control unit 21 to perform various processes.

  The storage unit 22 stores a program executed by the control unit 21, data necessary for program execution, and the like. Each program code is read by the control unit 21 as necessary, transferred to the RAM, and read and executed by the CPU.

The field-of-view camera 23 photographs the direction in which the wearer's face is facing.
The eye mark detection unit 24 shoots the eyeball of the wearer, analyzes the eyeball movement and direction (line of sight) of the wearer from the photographed eyeball image, stores the eyemark 6 on the visual field image 4 and stores it. Coordinate data indicating the position of the mark 6 is stored.

The power supply unit 25 supplies the electromotive force of the eye tracking device 2. The power supply unit 25 may be a built-in battery or an adapter that transmits power from the outside.
The switch 26 is an input device for turning on / off the power of the eye tracking device 2 and performing various calibrations.
The external output unit 27 outputs, for example, coordinate data indicating the position of the visual field image 4 and the eye mark 6 to the outside. The external output unit 27 may be wired or wireless.

Next, processing performed by the gazing point measurement apparatus 1 will be described with reference to FIGS.
FIG. 5 is a diagram illustrating a flow of data and processing related to the gazing point measurement apparatus 1. As shown in FIG. 5, the control unit 11 of the gazing point measurement device 1 includes a corresponding point search process 31, an original mesh generation process 32, a plane projection transformation matrix calculation process 33, a plane mesh generation process 34, a curved surface mesh generation process 35, A gazing point position specifying process 36, a display process 37, and the like are executed.

  In the corresponding point search process 31, the control unit 11 inputs the visual field image 4 received from the eye tracking device 2 and the original image 3 stored in the storage unit 12, and associates the visual field image 4 with the original image 3. A set of corresponding points 42 is output.

FIG. 6 is a diagram for explaining the corresponding point search processing 31.
In the corresponding point search processing 31, the control unit 11 extracts the real space feature point 51a that is the feature point of the visual field image 4 and the original feature point 51b that is the feature point of the original image 3, and the real space feature point 51a and the original feature Based on the set of points 51 b, as a set of corresponding points 42 for associating the visual field image 4 with the original image 3, the corresponding point 42 of the visual field image 4 is the real space corresponding point 42 a and the corresponding point 42 of the original image 3. A set of original corresponding points 42b is searched.
In FIG. 6, the real space feature points 51a and the original feature points 51b are indicated by black circles.

As a method for realizing the corresponding point search processing 31, for example, SIFT (Scale Invariant Feature Transform) is known. SIFT is a technique for extracting feature points from an image and describing feature quantities. Each feature point is defined with a scale and a 128-dimensional vector indicating various feature amounts. Then, using the Euclidean distance between the 128-dimensional vectors, a set of corresponding points 42 is searched from the set of real space feature points 51a and the set of original feature points 51b. The pair with the smallest distance (the pair of the real space feature point 51a and the original feature point 51b) is a candidate for the pair of corresponding points 42, but the difference from the pair with the second smallest distance is a certain value or more. Only a pair of corresponding points 42 is determined. That is, there are a real space feature point 51a and an original feature point 51b that are not determined as a set of corresponding points 42. This is to prevent erroneous association when there are a plurality of feature points having similar feature amounts.
As an input of the planar projective transformation matrix calculation process 33 described later, it is sufficient if there are four or more sets of corresponding points 42. In order to reduce the accuracy of the plane projection transformation matrix calculation process 33 when there is a pair of corresponding points 42 that are associated with each other in error rather than when there are fewer pairs of corresponding points 42, in the corresponding point search process 31, As much as possible, a set of corresponding points 42 that are erroneously associated is not output.

  SIFT has an effect that it is not easily affected by the rotation, size change, and illumination change of the target object 5. However, the corresponding point search processing 31 according to the present invention is not limited to SIFT, and a known technique can be applied as appropriate.

Returning to the description of FIG.
In the original mesh generation process 32, the control unit 11 inputs the original image 3 stored in the storage unit 12 and outputs an original mesh 43 that is a mesh of the original image 3. The mesh is obtained by finely dividing a calculation target region with an arbitrary polygon (for example, a triangle, a quadrangle, etc.) as a minimum unit. The mesh size (the size of one side in the case of regular triangles, regular squares, etc.) can be appropriately determined by the user according to the application.
In the original mesh generation process 32, for example, the original image 3 is equally divided into arbitrary polygons by a known technique.

  FIG. 7 is a schematic diagram of the original mesh 43. The original mesh 43 shown in FIG. 7 is obtained by dividing the original image 3 shown in FIG. The control unit 11 stores the coordinates of the vertices of each triangle in the storage unit 12 together with an ID for identifying each triangle.

Returning to the description of FIG.
In the plane projective transformation matrix calculation process 33, the control unit 11 inputs a set of corresponding points 42 output by the original mesh generation process 32, and based on a set of a set of real space corresponding points 42a and original corresponding points 42b, A plane projection transformation matrix 44 for planar projection transformation of the coordinates of the visual field image 4 to the coordinates of the original image 3 is output.

  In the corresponding point search process 31 described above, there is a possibility that a set of corresponding points 42 that are erroneously associated with each other is output. In the plane projective transformation matrix calculation process 33, if a pair of corresponding points 42 that are erroneously associated is employed for calculation, an accurate plane projective transformation matrix 44 cannot be obtained. In the planar projective transformation matrix calculation process 33 of the present invention, for example, a technique called RANSAC (RANdom Sample Sample Consensus) is used so as not to calculate by adopting a pair of corresponding points 42 that are incorrectly associated.

FIG. 8 is a flowchart showing the flow of the planar projective transformation matrix calculation process 33. 9 and 10 are diagrams illustrating an example of the planar projective transformation matrix 44.
The flowchart shown in FIG. 8 basically shows calculation processing by RANSAC. In FIG. 9 and FIG. 10, ruled lines drawn like a grid are attached so that it is easy to grasp the deformation of the plane and the position of each point.

The control unit 11 determines a point to be adopted for calculation of the plane projective transformation matrix 44 from among a set of all corresponding points 42 (S101).
9 and 10, the real space corresponding point 42a in the visual field image 4 is indicated by a numbered square. In the example shown in FIGS. 9 and 10, there are six real space corresponding points 42a. The control unit 11 determines, for example, four real space corresponding points 42a from among the six real space corresponding points 42a as random points to be employed in the calculation of the planar projective transformation matrix 44 by random numbers.
In the example of FIG. 9, the real space corresponding points 42a of “1”, “2”, “3”, and “4” are used for matrix calculation. In the example of FIG. 10, the real space corresponding points 42a of “1”, “3”, “4”, and “6” are used for matrix calculation.

Next, the control unit 11 converts the set of all corresponding points 42 of the visual field image 4 using the calculated plane projective transformation matrix 44, and sets the converted points 52 in the original image 3 (S102).
9 and 10, the conversion points 52 in the original image 3 are indicated by numbered squares. 9 and 10, the original corresponding point 42b in the original image 3 referred to in S103 described later is indicated by a numbered circle.

Next, the control unit 11 calculates the number of conversion points 52 whose positions coincide with the original corresponding points 42b (S103). Here, “the positions match” includes not only the case where the x coordinate and the y coordinate completely match, but also the case where the Euclidean distance between the original corresponding point 42b and the conversion point 52 is within a certain value. Further, the comparison target of whether the positions match is only the original corresponding point 42b associated with the real space corresponding point 42a that is the conversion source of the conversion point 52.
9 and 10, the positions of the original corresponding point 42 b and the conversion point 52 coincide with each other by overlapping circles and squares.
In the example of FIG. 9, the number of conversion points 52 whose positions coincide with the original corresponding point 42 b is only “2”. In the example of FIG. 10, the number of conversion points 52 whose positions coincide with the original corresponding point 42b is three, “1”, “3”, and “6”.

Next, the control part 11 confirms whether completion | finish conditions are satisfy | filled (S104).
The end condition is, for example, the number of repetition processes, that is, the processes from S101 to S103 are executed a predetermined number of times.
The end condition is, for example, that the number of conversion points 52 whose positions coincide with the original corresponding point 42b, that is, the calculation result of S103 exceeds a predetermined number.
When the termination condition is not satisfied (No in S104), the control unit 11 repeats from S101.
When the end condition is satisfied (Yes in S104), the control unit 11 proceeds to S105.

Next, the control unit 11 stores the planar projection transformation matrix 44 having the largest number of transformation points 52 whose positions coincide with the original corresponding point 42b as the optimal solution in the storage unit 12 or the RAM (S105).
Comparing the examples of FIG. 9 and FIG. 10, since the number of conversion points 52 matching the original shape corresponding point 42b is larger in FIG. 10, “1”, “3”, “4”, “6” The plane projection transformation matrix 44 calculated by adopting the real space corresponding point 42a is the optimal solution.
If the plane projection transformation matrix 44 is obtained if the target object 5 is a plane that is not curved or the like in FIG. 2, the above-described original mesh generation process 32, plane mesh generation process 34 described later, and curved mesh generation are performed. Even if the process 35 is not performed, the gazing point 7 of the original image 3 can be obtained from the position coordinates of the eye mark 6 in the visual field image 4.

Returning to the description of FIG.
In the plane mesh generation process 34, the control unit 11 performs a set of the plane projection conversion matrix 44 output by the plane projection conversion matrix calculation process 33, the set of corresponding points 42 output by the corresponding point search process 31, and the original mesh generation process 32. The original mesh 43 to be output is input, and the planar mesh 45 of the visual field image 4 is output.
In the curved surface mesh generation process 35, the control unit 11 inputs the planar mesh 45 output by the planar mesh generation process 3 and outputs the curved surface mesh 46 of the visual field image 4.

  FIG. 11 is a flowchart showing the flow of the plane mesh generation process 34 and the curved surface mesh generation process 35. FIG. 12 is a diagram illustrating an example of the planar mesh 45. FIG. 13 is a diagram for explaining the positional deviation 54 between the real space corresponding point 42a and the plane corresponding point 42c. FIG. 14 is a diagram illustrating an example of the curved mesh 46.

In the plane mesh generation process 34, the control unit 11 generates a plane mesh 45 of the visual field image 4 by performing plane projection conversion on the original mesh 43 based on the plane projection conversion matrix 44.
Further, in the curved surface mesh generation processing 35, the control unit 11 calculates a plane corresponding point 42c by performing plane projective transformation on the real space corresponding point 42a with respect to the plane mesh 45, and calculates the plane corresponding point 42c and the real corresponding point 42c. The curved mesh 46 of the visual field image 4 is generated by deforming the planar mesh 45 based on the positional deviation 54 with respect to the space corresponding point 42a.

As shown in FIG. 11, the control unit 11 performs the planar projective transformation of the original mesh 43 using the inverse matrix of the optimal solution of the planar projective transformation matrix 44 calculated in S105 of FIG. Generate (S201).
In FIG. 12, conversion from the original mesh 43 to the planar mesh 45 is shown. For convenience of explanation, the plane mesh 45 is shown superimposed on the target object 5 in the visual field image 4.
As shown in FIG. 2, when the end of the target object 5 is curved or has irregularities inside, the planar mesh 45 is subject to irregularly deformed visual field image 4 as shown in FIG. 12. Deviation from the object 5 occurs. On the other hand, by performing a curved mesh generation process 35, which will be described later, the deviation due to the end of the target object 5 being curved or uneven inside is corrected, and the position of the gazing point 7 is accurately specified. Can do.

Next, the control unit 11 aligns the plane mesh 45 with a set of corresponding points 42 (set of real space corresponding points 42a and plane corresponding points 42c) (S202).
Specifically, the control unit 11 deforms the planar mesh 45 based on a determination formula including the positional deviation 54 between the planar corresponding point 42c and the real space corresponding point 42a and the complexity of the deformation. The determination formula is the following formula (1).

S in Expression (1) is a set of vertices of each element of the planar mesh 45 and is a matrix. ε (S) is a judgment formula. λ _D is an adjustment parameter. ε _D (S) is the complexity of the deformation. ε _C (S) is a value based on the positional deviation 54 between the real space corresponding point 42a and the plane corresponding point 42c.
N in the formula (2) is a subscript indicating each element of the planar mesh 45. N is the number of elements of the planar mesh 45. ε _D (S (n)) is, for example, the square root of the sum of squares of the difference between the sides before and after deformation, as shown in Expression (2). Instead of the square root of the sum of squares of the differences, a sum of squares of the differences, variance, standard deviation, or the like may be employed. l _i (n) is the length of the side of the element n before deformation. l _i ′ (n) is the length of the side of the element n after deformation. When the element is a triangle, i = 1 to 3 because there are three sides. If the element is a quadrangle, i = 1 to 4 because there are four sides.
In Expression (3), k is a subscript indicating a set of corresponding points 42. K is the number of sets of corresponding points 42. ε _C (S (k)) is, for example, the Euclidean distance between the real space corresponding point 42a and the plane corresponding point 42c, as shown in Expression (3). x ₁ (k) and y ₁ (k) are the x coordinate and y coordinate of the real space corresponding point 42a. x ₂ (k) and y ₂ (k) are the x coordinate and y coordinate of the plane corresponding point 42c.

ε _D (S) increases in value if the deformation is complicated. Further, the value of ε _C (S) increases as the positional deviation 54 increases.
The controller 11 reduces the positional deviation 54 between the plane corresponding point 42c and the real space corresponding point 42a by simple deformation, that is, smooth deformation, and the position of the plane corresponding point 42c matches the position of the real space corresponding point 42a. In order to perform alignment, S that minimizes the determination equation ε (S) shown in Equation (1) is calculated. Therefore, the control unit 11 calculates S at which the partial differential of the determination formula ε (S) shown in Formula (1) becomes zero.

  Next, the control unit 11 deforms the planar mesh 45 based on the alignment result in S202 (S203). Here, the control part 11 should just move the vertex of each element of the plane mesh 45 based on the set S of the vertex of each element of the plane mesh 45 calculated in S202. In the modification of S203, the number of elements does not change, but the area and shape of the elements change.

Next, the control part 11 confirms whether completion | finish conditions are satisfy | filled (S204).
The end condition is, for example, the number of repetition processes, that is, the process from S202 to S203 is executed a predetermined number of times.
The termination condition is, for example, that the maximum value of the positional deviation 54, that is, the maximum value of the positional deviation 54 between the plane corresponding point 42c and the real space corresponding point 42a calculated in S202 is equal to or less than a certain value.
When the termination condition is not satisfied (No in S204), the control unit 11 repeats from S202.
When the end condition is satisfied (Yes in S204), the control unit 11 proceeds to S205.

In FIG. 13, the real space corresponding point 42a is indicated by a numbered square, and the plane corresponding point 42c is indicated by a numbered circle. In the example of FIG. 13, for the identification numbers “1”, “2”, “5”, and “6” of the pair of corresponding points 42, the positional deviation 54 between the plane corresponding point 42c and the real space corresponding point 42a is large. . On the other hand, for the identification numbers “3” and “4” of the corresponding points 42, the positional deviation 54 between the plane corresponding point 42c and the real space corresponding point 42a is small.
As the control unit 11 repeats the processing from S202 to S203, the positional deviation 54 shown in FIG. 13 gradually decreases.

Next, the control unit 11 stores the deformed planar mesh 45 at the current time in the storage unit 12 as the curved mesh 46 (S205). The control unit 11 repeatedly deforms the plane mesh 45 to generate an optimal curved surface mesh 46.
FIG. 14 shows a schematic diagram of the curved mesh 46. By performing the processing shown in FIG. 11, a curved mesh 46 in which the target object 5 of the visual field image 4 is accurately expressed is generated. As shown in FIG. 14, the area of the curved mesh 46 coincides with the area of the target object 5 in the visual field image 4. Further, each element of the curved mesh 46 is deformed in position, size, and shape in accordance with the deformation of the end of the target object 5 and the unevenness inside the object, as compared with the elements of the planar mesh 45. .

Returning to the description of FIG.
In the gazing point position specifying process 36, the control unit 11 detects the eye mark position 47 received from the eye tracking device 2, the curved surface mesh 46 output by the curved surface mesh generating process 35 (or the plane output by the planar mesh generating process 34). The mesh 45) is input and the gaze point position 48 is output.
As illustrated in FIG. 14, the element 56 including the eye mark of the curved mesh 46 (or the planar mesh 45) can be determined based on the eye mark position 47 in the visual field image 4. Since each element of the curved mesh 46 (or the planar mesh 45) and the original mesh 43 are associated with each other, the element of the original mesh 43 corresponding to the element 56 including the eye mark of the curved mesh 46 (or the planar mesh 45). Is identified as the element 57 including the gaze point. Then, the control unit 11 calculates and outputs a gaze point position 48 based on the element 56 including the eye mark, the element 57 including the gaze point, and the eye mark position 47.

FIG. 15 is a diagram for explaining an example of the calculation process of the gazing point position 48.
FIG. 15 shows an element 56 including an eye mark and an element 57 including a gazing point. Points A, B, and C are vertices of an element 56 (triangle of the curved mesh 46 or the planar mesh 45) including the eye mark, and the coordinates of the points A, B, and C are known. Similarly, the points A ′, B ′, and C ′ are the vertices of the element 57 (the triangle of the original mesh 43) that includes the gazing point, and the coordinates of the points A ′, B ′, and C ′ are known. The point P is the eye mark position 47, and the coordinates of the point P are known.
Here, if the area ratio of the triangles PBC, PCA, and PAB is (α, β, γ), the coordinates of the points A, B, C, and P are known, so (α, β, γ) can be calculated. It is.
Further, the point P ′ is set as the gazing point position 48. Further, the area ratio of the triangles P′B′C ′, P′C′A ′, and P′A′B ′ is (α ′, β ′, γ ′).
For example, the control unit 11 calculates the coordinates of the point P ′ where (α ′, β ′, γ ′) = (α, β, γ), and outputs the coordinates as the coordinates of the gazing point position 48.
However, the calculation process of the gazing point position 48 is not limited to this example, and a known technique can be applied as appropriate.

In the description of the alignment process described above (S202 in FIG. 11), ε _C (S) in Expression (3) is calculated using a set of all corresponding points 42 (k = 1,..., K). However, the present invention is not limited to this example.
As described above, in the corresponding point search process 31, there is a possibility that a set of corresponding points 42 associated with each other may be output. In the pair of corresponding points 42 that are erroneously associated with each other, the positional deviation 54 between the planar corresponding point 42c and the real space corresponding point 42a becomes large. On the other hand, in S202 of FIG. 11, alignment is performed so as to reduce the positional deviation 54 between the plane corresponding point 42c and the real space corresponding point 42a. Therefore, the control unit 11 may perform processing so as to match the positions of the pair of corresponding points 42 that are erroneously associated with each other, and may not be able to generate an accurate curved surface mesh 46.

  In the following, with reference to FIGS. 16 and 17, the selection of the pair of corresponding points 42 to be employed in the determination formula calculation process in order to exclude the pair of corresponding points 42 incorrectly associated from the calculation of the determination formula. Processing will be described.

FIG. 16 is a flowchart illustrating a first example of a process for selecting a pair of corresponding points employed in the determination formula calculation process.
The control unit 11 selects one set of corresponding points 42 (S301), and calculates a positional deviation 54 between the plane corresponding point 42c and the real space corresponding point 42a (S302).
Next, the control unit 11 checks whether or not the calculation result of S302 is within a threshold value (S303).
When the calculation result in S302 is within the threshold (Yes in S303), the control unit 11 employs the pair of corresponding points 42 selected in S301 for the determination formula calculation process (S304), and proceeds to S305.
When the calculation result in S302 is larger than the threshold (No in S303), the control unit 11 proceeds to S305 without adopting the set of corresponding points 42 selected in S301 in the determination formula calculation process.

Next, the control part 11 confirms whether the process was complete | finished with respect to the group of all the corresponding points 42 (S305).
If the process has not been completed for all the pairs of corresponding points 42 (No in S305), the control unit 11 repeats the process from S301.
When the process has been completed for all the pairs of corresponding points 42 (Yes in S305), the control unit 11 ends the process.

  By the process of FIG. 16, it is possible to select a pair of corresponding points 42 to be employed in the determination formula calculation process by excluding a pair of corresponding points 42 that are highly likely to be associated by mistake.

Note that it is desirable that the threshold value in S303 be reduced as the deformation of the planar mesh 45 is repeated.
By appropriately changing the threshold value, in the early iterations, a rough deformation is performed using a set of many corresponding points 42 (including a set of corresponding points 42 erroneously associated to some extent). Can do.
Further, in the final stage of the repetition, it is possible to accurately perform fine deformation using a set of a few corresponding points 42 (there are few sets of corresponding points 42 that are erroneously associated).

FIG. 17 is a flowchart illustrating a second example of the corresponding point pair selection process employed in the determination formula calculation process.
The control unit 11 selects one set of corresponding points 42 (S401), and calculates a positional deviation 54 between the plane corresponding point 42c and the real space corresponding point 42a (S402).
Next, the control unit 11 checks whether or not the processing has been completed for all the pairs of corresponding points 42 (S403).
When the process has not been completed for all the pairs of corresponding points 42 (No in S403), the control unit 11 repeats the process from S401.
When the processing has been completed for all pairs of corresponding points 42 (Yes in S403), the control unit 11 proceeds to S404.

  Next, the control unit 11 employs the pair of corresponding points 42 excluding the pair of corresponding points 42 having the largest positional deviation 54 in the determination formula calculation process (S404). Here, the set of corresponding points 42 to be excluded is not a set of corresponding points 42 with the largest positional deviation 54, but may be a set of a certain number of corresponding points 42 from the one with the largest positional deviation 54.

  With the processing in FIG. 17, it is possible to select a pair of corresponding points 42 to be employed in the determination formula calculation process by excluding a pair of corresponding points 42 that are highly likely to be associated by mistake.

In S404, when a certain number of pairs of corresponding points 42 are excluded from the larger position deviation 54, it is desirable to increase the certain number every time the deformation of the plane mesh 45 is repeated.
By appropriately changing a certain number, as in the first example, an appropriate transformation is performed using an appropriate set of corresponding points 42 in each case of the beginning of the iteration and the end of the iteration. Can do.

  The reason why appropriate modifications can be made in the first example and the second example is as follows. Corresponding points 42 that are erroneously associated with each other have different alignment corrections. On the other hand, since the corresponding points 42 correctly associated are corrected in the same direction, even if the corresponding points 42 incorrectly associated are included, the correction amount in the correct direction as a whole increases.

  As described above in detail, the gaze point measuring apparatus 1 according to the present invention does not require a marker for specifying the position of the gaze point, so that the wearer of the eye tracking device 2 unconsciously directs his / her line of sight to the marker. Therefore, appropriate measurement can be performed. Further, when the appeal medium made of a material having low rigidity is set as the target object 5, even if the target object 5 is deformed, the position of the gazing point in the original image 3 of the target object 5 can be automatically and accurately measured. it can.

  The preferred embodiments of the gaze point measuring apparatus and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the technical idea disclosed in the present application, and these naturally belong to the technical scope of the present invention. Understood.

1 ……… Gaze point measuring device 2 ……… Eye tracking device 3 ……… Original image 4 ……… Field of view image 5 ……… Target object 6 ……… Eye mark 7 ……… Gaze point 31 ……… Correspondence Point search process 32 ... Original mesh generation process 33 ... Planar projective transformation matrix calculation process 34 ... Plane mesh generation process 35 ... ... Curved mesh generation process 36 ... ... Gaze point specifying process 42 ... ... Corresponding point 42a ......... Real space corresponding point 42b ......... Original shape corresponding point 42c ......... Plane corresponding point 43 ......... Original mesh 44 ......... Plane projection transformation matrix 45 ......... Plane mesh 46 ......... Surface mesh 47 ......... Eye mark position 48 ......... Gaze point position 51a ......... Real space feature point 51b ......... Original feature point 52 ......... Transformation point 54 ......... Position shift 56 ......... Elements including eye marks 57 ……… Note Element containing viewpoint

Claims (12)

A gazing point measurement device that measures a position of a gazing point in an original image of a target object included in the field-of-view image in association with the position of an eye mark in the field-of-view image that is an image photographed by an eye tracking device,
A real space feature point that is a feature point of the visual field image and an original feature point that is a feature point of the original image are extracted, and based on the set of the real space feature point and the original feature point, the visual field image Corresponding point search means for searching for a set of corresponding points that correspond to the original image and a corresponding set of real space corresponding points of the visual field image and a corresponding point of original shapes that are corresponding points of the original image;
Transformation matrix calculation means for calculating a plane projective transformation matrix for plane projective transformation of the coordinates of the visual field image to the coordinates of the original image based on a set of sets of the real space corresponding points and the original corresponding points;
A gaze point measuring apparatus comprising: Original mesh generation means for generating an original mesh that is a mesh of the original image;
Plane mesh generation means for generating a plane mesh of the visual field image by performing plane projection conversion based on the plane projection conversion matrix for the original mesh;
A plane corresponding point is calculated by performing the plane projective transformation on the real space corresponding point, and the plane mesh is deformed based on a positional deviation between the plane corresponding point and the real space corresponding point, and the field of view is obtained. Curved surface mesh generating means for generating a curved mesh of an image;
The gazing point measurement apparatus according to claim 1, further comprising:   The curved mesh generation means repeatedly deforms the planar mesh based on a determination formula including a positional deviation between the plane corresponding point and the real space corresponding point and a complexity of deformation, thereby obtaining an optimal curved mesh. The gaze point measuring device according to claim 2, wherein the gaze point measuring device is generated.   The curved surface mesh generation means does not employ a pair of corresponding points in which a positional deviation between the plane corresponding point and the real space corresponding point is larger than a threshold value in the calculation process of the determination formula. The gaze point measuring device described.   The gaze point measuring apparatus according to claim 4, wherein the threshold value is decreased every time the deformation of the planar mesh is repeated.   The curved surface mesh generation means does not employ a certain number of pairs of corresponding points in the calculation process of the determination formula from the one with the larger positional deviation between the plane corresponding points and the real space corresponding points. Item 4. The gaze point measuring device according to item 3.   The gaze point measuring apparatus according to claim 6, wherein the certain number is increased each time the deformation of the planar mesh is repeated. Gazing point position specifying means for specifying the position of the gazing point as an element of the original mesh corresponding to the element of the planar mesh or the curved surface mesh including the eye mark in the visual field image includes the gazing point;
The gazing point measurement apparatus according to claim 1, further comprising: A gazing point measurement method for measuring a position of a gazing point in an original image of a target object included in the field-of-view image in association with the position of an eye mark in the field-of-view image that is an image photographed by an eye tracking device,
A real space feature point that is a feature point of the visual field image and an original feature point that is a feature point of the original image are extracted, and based on the set of the real space feature point and the original feature point, the visual field image Corresponding point search step for searching for a set of corresponding points that correspond to the original image and a set of corresponding points of the original image that correspond to real space corresponding points of the visual field image and a corresponding point of the original image;
A transformation matrix calculating step for calculating a plane projective transformation matrix for plane projective transformation of the coordinates of the visual field image to the coordinates of the original image based on a set of the real space corresponding point and the original shape corresponding point;
A gaze point measuring method characterized by including An original mesh generation step of generating an original mesh that is a mesh of the original image;
A plane mesh generation step of generating a plane mesh of the visual field image by performing plane projection conversion based on the plane projection conversion matrix with respect to the original mesh;
A plane corresponding point is calculated by performing the plane projective transformation on the real space corresponding point, and the plane mesh is deformed based on a positional deviation between the plane corresponding point and the real space corresponding point, and the field of view is obtained. A curved mesh generation step for generating a curved mesh of the video;
The gazing point measurement method according to claim 9, further comprising:   A program for causing a computer to function as the gazing point measurement device according to any one of claims 1 to 8.   A computer-readable storage medium storing a program for causing a computer to function as the gazing point measurement device according to any one of claims 1 to 8.

Images