JP2014509535A - Gaze point mapping method and apparatus - Google Patents

Abstract

An apparatus for mapping a target gazing point on a scene image to a gazing point in a reference image is provided.
A scene image and a reference image are taken by a camera from different positions, and the apparatus executes a feature detection algorithm on the reference image to identify a plurality of characteristics in the reference image and their positions; A module that executes a feature detection algorithm on a scene image and re-specifies the characteristics in the scene image and their positions, and the position of the scene image and the reference image based on the positions of the characteristics detected in the reference image and the scene image And a module for mapping a point of interest determined in the scene image using the point transfer mapping to a corresponding point in the reference image.
[Selection] Figure 1

Description

  The present invention relates to a gaze point mapping method and apparatus, and more particularly to a method and apparatus for mapping a target gaze direction to a gaze point in a scene.

  The problem to be solved is to find the point or object that the (possibly moving) person is gazing at, more specifically the part of the surface of the object. An existing solution to this problem is described below, which can be divided into separate parts.

  First, the person's gaze direction (or the pupil / CR combination, corneal center and pupil / edge, etc., display thereof) is found.

  This gaze direction is mapped to a scene image captured by a head-mounted scene camera or a scene camera at any fixed position. Since the head mounted scene camera is fixed with respect to the eyes, such mapping is possible once the corresponding calibration is executed.

  The next step is to change the target's movement in the reference image that corresponds to the object in the "real world" without moving the gaze point in the scene image captured by the head-mounted camera. It is to map to the point.

An eye tracker can be used to determine the gaze direction. The eye tracker observes eye features such as pupils, edges, blood vessels on the sclera, or light source reflection (corneal reflection) to calculate the gaze direction.
Any kind of eye tracker can be used as long as the gaze direction can be mapped to the image of the head mounted scene camera.

  If the subject's head does not move, once calibration is performed, the gaze point on the reference image is directly given by determining the gaze direction. When the head mounted camera does not move and is in a fixed position with respect to the reference image, the scene image and the reference image are the same because of the calibration in the special case that the head does not move. Gaze direction can be mapped to image points.

  However, when the head and eyes move, it is possible to determine the gazing point with the non-moving reference image based on the detection of the gaze direction for a certain scene image taken by the head mounted camera after the head moves. The image is more complex because it is no longer the same as the reference image used to calibrate the gaze direction with respect to the corresponding gazing point of the reference image.

  One possible approach to determining the point of gaze is to cross the direction of gaze on a virtual scene plane defined for the eye tracker. For this purpose, WO 2010/083853 A1 discloses the use of a plurality of active IR markers which are fixed in a certain position, for example a bookshelf. The positions of these markers are "reference" images obtained by a head mounted camera using two orthogonal IR line detectors that detect two orthogonal angles by detecting the maximum intensity of the two line sensors. A “test scene” is first detected. The detection angle of the IR source corresponds to its position in the reference image. Then, the angle of the marker is detected for a scene detected after taking a picture with a head-mounted camera from a different position, thereby detecting the position of the IR source in the subsequent scene image. When the head-mounted camera is at a different position, mapping is performed to convert the position of the IR source (scene image) detected in the image taken later to the position of the IR light source in the test image (or reference image). The “perspective projection” is determined. By this conversion, a gaze point determined later for the scene image can also be converted into a corresponding (actual) gaze point in the test image.

  Mapping a gaze point from an actual "scene image" to a stable reference image that is time-invariant is by defining a plane that maps the gaze point to a scene stability marker instead of an eye tracker (ET) It becomes possible. In this way, the reference image plane is stable over time, and other participants' gazes are mapped to it so that many participants over time, which were previously only possible with fixed-position eye trackers. It becomes possible to collect attention point information. For this reason, the prior art disclosed in WO2010 / 083853 A1 uses an IR source whose position can be detected by an orthogonal IR line detector as an artificial marker, and detects the maximum radiation angle.

  The method of using an IR source as a marker for determining the conversion from a gaze point scene image to a reference image is complicated and inconvenient. For this, an artificial IR light source must be attached and an additional IR detector consisting of two orthogonal line sensors must be provided. Therefore, it is desirable to provide an approach that allows gaze point mapping to be determined even if the head mounted scene camera moves without such external markers.

According to an embodiment, there is provided an apparatus for mapping a target gazing point on a scene image to a gazing point in a reference image, the scene image and the reference image taken with cameras from different positions. And the device is
A module for performing a feature detection algorithm on the reference image to identify a plurality of characteristics and their position in the reference image;
A module for executing the feature detection algorithm on the scene image and re-identifying the plurality of characteristics and their position in the scene image;
A module for determining a point transfer mapping for converting a point position between the scene image and the reference image based on the position of the reference image and the plurality of characteristics detected in the scene image;
A module for mapping the point of interest determined in the scene image to a corresponding point in the reference image using the point transfer mapping.

  This allows for the realization of gaze mapping that does not require any artificial IR source and IR detector. It can operate on normal, uncorrected images of natural scenes taken with a normal CCD camera operating in the visible frequency range.

  According to an embodiment, the point transfer mapping is estimated from features of a reference image and a scene image, and a planar homography that transfers point positions from the scene image to the reference image via a virtual transfer surface of a world scene. It is.

  This is a particularly preferred implementation of point transcription mapping.

According to an embodiment, the device is
It further comprises a module for determining one or more regions of interest in the reference image, executing the feature detection algorithm therein, and detecting and identifying the characteristic therein.
Thereby, a region suitable for the feature detection algorithm can be selected. This enhances the efficiency of the feature detection algorithm by allowing the user to select regions with particularly characteristic characteristics.

According to an embodiment, the feature detection algorithm for detecting the plurality of characteristics in the scene image, detecting and respecifying the characteristics in the scene image comprises any of the following:
Scale invariant feature transcription algorithm,
Fast and robust feature algorithm.

  These are particularly preferred implementations of feature detection and re-identification algorithms.

According to an embodiment, the device is
It further comprises a module for mapping gaze directions from different people and / or different times to corresponding gaze points in the reference image and recording the gaze points mapped over time, possibly also for different users.

  This allows gaze data to be accumulated and accumulated over time, even for different users.

According to an embodiment, the device is
The module further includes a module for displaying the gazing point mapped from the scene image to the reference image in the reference image, and visualizing the gazing point as the position expands as time passes.

  This makes it possible to visualize the gazing point mapped to the reference image even if it takes some time.

According to an embodiment, the device is
The gazing point mapped from the scene image to the reference image is further visualized with a visualization image different from the reference image, and the module includes:
A module for determining a point transfer mapping for transferring points from the reference image to the visualized image;
A module for mapping the point of interest from the reference image to its corresponding point in the visualized image using the point transfer mapping;
And a module for displaying the gazing point on the corresponding point in the visualized image.

  As a result, the gazing point can be mapped to an image different from the reference image and visualized here.

According to an embodiment, the device is
It further comprises a module that determines the point transfer mapping based on characteristics detected in the reference image by a feature detection algorithm and detected and re-specified in the visualized image by the feature detection algorithm.

  This is a preferred implementation of point transcription mapping for visualization.

According to an embodiment, the visualized image is one of the following:
Images selected from scene images taken from different positions, images generated by stitching two or more scene images taken from different positions,
A geometrically modified image of the scene,
An image of the scene with the distortion removed,
Images taken at different times from the actual measurement of the gazing point,
An image having a higher resolution than the reference image;
Cartoon or sketch,
Stylized figure,
Homemade illustration.

  These are preferred implementations of the visualized image different from the reference image.

According to an embodiment, the device is
Further comprising a module for visualizing the point of interest within a frame of a video sequence, the module comprising:
A module for determining a point transfer mapping from a point of the reference image to a corresponding point in a video frame of the video sequence;
A module that maps the point of interest from the reference image to the corresponding point into the corresponding point of the frame of the video sequence using the point transfer mapping.

  This makes it possible to visualize the point of interest in the video sequence.

According to an embodiment, the device is
From a module that maps a user's point of interest detected in a scene image taken by a first camera from a position different from the first camera to a corresponding position in a corresponding frame of a video taken by a second camera Become more.

  This enables visualization with a video sequence taken from a position different from the scene image.

According to an embodiment, the device is
Eye tracking module for tracking a subject's gaze and / or head mounted camera for taking a scene image and / or mapping a point of gaze to a corresponding point in a scene taken with the head mounted camera And further comprises a calibration module.

  This makes it possible to implement a complete system comprising an eye tracker, a calibration module, and a scene camera.

According to an embodiment, there is provided a method for mapping a target gazing point on a scene image to a gazing point in a reference image, wherein the scene image and the reference image are taken by a camera from different positions. The method is
A feature detection algorithm is executed on the reference image, a plurality of characteristics and positions thereof are specified in the reference image, and the feature detection algorithm is executed on the scene image to determine the characteristics in the scene image. Re-specify the position, and determine the point transfer mapping to convert the point position between the scene image and the reference image based on the positions of the plurality of characteristics detected in the reference image and the scene image Then, using the point transfer mapping, the gazing point determined in the scene image is mapped to a corresponding point in the reference image.

  Thus, the method according to the embodiment of the present invention can be implemented.

  The method can further include performing according to additional features of any of the other embodiments.

According to an embodiment, there is provided a computer program comprising computer program code that, when executed on a computer, causes the computer to execute a method according to any of the embodiments of the invention.

FIG. 1 schematically illustrates gaze point mapping according to an embodiment of the present invention. FIG. 2 schematically illustrates the mapping of the initial reference image to the final reference image according to an embodiment of the present invention.

According to certain embodiments, a method and apparatus are provided that can map a point of interest from a scene image to a reference image taken at a different location.
For this reason, a set of characteristics (reference features) is used in the reference image.

  According to one embodiment, these characteristics are identified by a feature detection algorithm that can search for identifiable features in the reference image. According to one embodiment, the feature detection algorithm is selected so that the detected features can be re-specified in images taken from different locations and later taken. Using the feature detection algorithm, even if the scene image is taken at a different position, characteristics such as a reference point, line, or region can be re-specified by the scene image taken later.

  Suitable features that can be detected by the feature detection algorithm are, for example, characteristic points or regions such as blobs, edges or corners in the reference image. These points or regions that can be used as characteristics can be identified by an image processing algorithm capable of detecting features in the images taken by the camera, and later taken by the camera from different locations using such feature detection algorithms. It can be re-specified in the image.

  Note that using a feature detection algorithm that can process the image and detect one or more characteristics eliminates the need to use artificial markers such as IR sources or additional detectors such as IR line detectors. It is. Instead, the natural scene image taken with the camera itself can be used to determine the mapping point without having to install any artificial components or light sources in the scene and without the need for additional detectors. can do. This is achieved by applying a feature detection algorithm that detects the position of the feature in the (unmodified or “natural”) scene image without the need for artificial markers provided by IR sources in the prior art. .

  According to certain embodiments, the user can manually select one or more regions (eg, with a mouse or other input means) that need to run a feature detection algorithm to detect a characteristic. Such points or areas are marked in the reference image by the user, for example by clicking on it with a mouse, or by limiting the area surrounding the rectangle or circle, for example, to execute a feature detection algorithm It can be selected manually by defining one or more “regions of interest”.

  The characteristics detected together with the corresponding positions in such a region of interest by the execution of the feature detection algorithm are described in detail below in the so-called “scene image” after being taken by the head-mounted scene camera from different positions. It is detected again by respecifying the characteristics in the scene image as described. Both the characteristic detected in the reference image and the characteristic re-specified in the scene image can be used to determine a point transfer mapping that converts the point position between the scene image and the reference image. One possible embodiment for such a mapping is a homography that can be estimated from feature positions in the reference image and the scene image, and that transfers the point position from the scene image to the reference image via the virtual transfer plane of the world scene. .

  The reference image taken before the actual measurement sequence consists of the projection of the real world scene onto the camera image plane. Scene images taken later from different positions of the camera at different positions then consist of different projections of the real world scene onto the image plane of the camera. However, if multiple characteristics (at least three or more) are first identified in the reference image and later re-specified in the scene image, it is possible to determine the mapping between points from the scene image to corresponding points in the reference image it can. Then, the gaze point mapping from the scene image to the corresponding point of the reference image can be performed by using the conversion as the mapping.

  However, first, a setup procedure for obtaining characteristic reference features and a corresponding calibration mechanism according to an embodiment will be described. First, a reference image is taken before the actual measurement sequence (a sequence in which “scene images” from different positions are taken and the direction of gaze is determined). This is used during eye tracker calibration to calibrate the correspondence between the gaze direction detected by the eye tracker and the corresponding point in the reference image taken by the scene camera. In this way, the gaze direction calibration can be performed on the corresponding points of the reference image and thus the corresponding points of the “real world scene” visible in the reference image. This corresponds to the conventional calibration that maps the gaze direction to the gaze point when the head-mounted scene camera does not move.

In later actual measurements, when the head-mounted scene camera and the head and eyes move, the image of the scene camera (“scene image”) changes. For this reason, the position of the characteristic detected in the reference image also changes accordingly.
In such a scenario, according to an embodiment, gaze point detection is performed as follows.

First, characteristic reference features are again detected or “re-identified” in scene images taken with a head-mounted scene camera from different positions.
For this reason, an image processing algorithm is used that can detect the characteristics and match the known reference features of the reference image determined during the setup procedure.

  The feature detection algorithm that is actually used to detect the characteristics in the reference image and re-specify them in the scene images taken from different positions can be selected according to the feature type most suitable for the application area.

  For example, if the corner is the characteristic that is detected in the reference image and re-detected and re-specified in the scene image, use a corner detection algorithm such as the Moravec angle detection algorithm, Harris & Stephens angle detection algorithm, Plessey angle detection algorithm, etc. be able to.

  Once the characteristics are detected or re-specified in the scene image taken from a position different from the reference image, they can be used to estimate the planar projection transformation from the point of the scene image to the point of the reference image.

  This will be described in more detail with reference to FIG.

  FIG. 1 shows a scene consisting of bookshelves. As an initial reference image, a scene image is taken by the scene camera SC at the first position (“reference position”), and is thereby projected onto the image plane of the scene camera SC. As shown in the figure, this scene comprises a bookshelf and is projected on the image plane of the scene camera SC when an image is taken.

  During the setup procedure, a set of characteristic reference features is defined by detecting characteristics in a reference image, or part thereof, such as within a region of interest such as a rectangle surrounding the bookshelf. This region may be selected as a “region of interest” by the user during a calibration procedure to define the region and execute the feature detection algorithm therein to detect the characteristic. The region of interest may be selected using, for example, a mouse, thereby selecting a rectangle surrounding the bookshelf in the reference image taken by the scene camera SC.

  Another possibility to define a set of characteristic reference features is to select, for example, four points, four front corners of the bookshelf. By selecting four corner points in this manner (for example, with a mouse), according to the present embodiment, it is possible to cope with selection of a small (predetermined) area around the selection point selected by mouse click. The feature detection algorithm is executed, and the feature and its position (within a predetermined area surrounding the point selected by the mouse click) are determined by the feature detection algorithm. In this way, the definition of the characteristic reference features can be made manually (to select a point of interest, such as a corner of a bookcase, with a mouse click) and to a feature (within a predetermined area around the point selected with the mouse click). It is executed in combination with automatic detection by execution of the detection algorithm.

  According to another embodiment, the entire reference image may form a “region of interest” in which characteristics are detected by a feature detection algorithm. Then, the “region of interest” like that shown by the rectangle surrounding the bookshelf in FIG. 1 is not necessary. However, in a preferred embodiment, the characteristic reference feature set is defined by selecting at least one shape, area or region of interest by constraining the feature detection algorithm by its content.

  With this reference image (or other image suitable for gaze calibration), gaze direction calibration can be performed in a conventional manner, which corresponds to the gaze direction determined by the eye tracker ET and the corresponding in the reference image. It means to perform mapping between gaze points. This can be done, for example, by having the object look at a definition point in the reference image (or other image suitable for gaze calibration) and determining the corresponding gaze direction measured by the eye tracker. Based on this data, it is possible to determine the correspondence between the gaze direction determined by the eye tracker and the corresponding gaze point in the reference image (and thus the actual scene image). This calibration is performed while the head mounted scene camera is in a fixed position and does not move, as in a conventional calibration procedure.

  By this calibration, as long as the head-mounted camera does not move, the gaze point for a certain period can be detected by the conventional method by detecting the gaze direction.

  However, when the head-mounted camera moves, as shown in FIG. 1, for example, a scene taken with the head-mounted scene camera as shown by the scene image on the image plane of the scene camera SC2 that is now at a different position. The image changes.

  According to one embodiment, previously defined characteristic reference features are detected in a scene image taken later (such as by a feature detection algorithm performed on an area defined by a rectangle surrounding the bookshelf). Therefore, an image processing algorithm is used that can at least partially re-specify previously defined characteristics (or objects) of a reference image in subsequent scene images taken from different positions of the head mounted scene camera.

  One suitable algorithm that can be used for this is the so-called scale invariant feature transformation (SIFT). With this algorithm, first, target features are extracted from one or more reference images (such as the “reference image” described above) and stored in a database. This feature is defined by a feature-descriptor-vector that captures information about the image location and the image area that surrounds that location in such a way that the feature can be compared by the descriptor-vector distance. Then, each feature of the new image is individually compared with this database, and a target candidate is recognized in the new image by finding a matching feature candidate based on the Euclidean distance of the feature vector. From the full set of matches, identify the target and the subset of features that match its location, the scale and orientation in the new image, and extract the correct match. Then, after more detailed model verification is performed on each cluster of four or more features that match the target and its pose, outliers are discarded. Finally, considering the number of false matches considered to be the accuracy of the match, the probability that a particular feature set indicates the presence of the object is calculated. An object or feature match that passes all these tests can be identified as reliable and correct.

  A more detailed description of the SIFT algorithm can be found, for example, in US Pat. No. 6,711,293.

  By using the SIFT algorithm, the characteristic reference feature of the image SC2 shown in FIG. 1 can be re-specified or recognized.

  Therefore, even if the head-mounted camera is moving, it is possible to determine a mapping that specifies which point in the reference image SC corresponds to which point in the scene image taken in SC2. For this reason, for example, by discarding outliers that are too far away from the implicitly represented virtual transfer surface using a robust estimator such as RANSAC that estimates homography from some or all of the matched features. Similar to mapping, a homography between the characteristics of the reference image and the characteristics re-specified in the scene image is determined. This plane is in most cases the average plane of scene target points corresponding to the characteristics shown in FIG.

  Using this mapping, it is possible to map the gaze direction to the gaze point in the reference image. Based on the determination of the gaze direction, the actual gaze point can first be determined in the scene image (based on initial calibration) and then the corresponding point in the reference image (based on homography) can be determined. This is also shown in FIG. 1 and will be described in more detail below.

  The detection of a certain gaze direction corresponds to a certain point in the scene image SC2 taken later, as indicated by a line intersecting the image plane of the image taken by the camera SC2. This correspondence was obtained during calibration, thereby providing a mapping of gaze directions to points in the scene image as described above in connection with the calibration procedure. This point in the scene image plane specified by the gaze direction corresponds to a point on a certain virtual transfer surface in FIG. 1 as can be seen from FIG. 1, and the virtual in FIG. 1 taken again by the camera SC2 at a different position. This point on the transfer surface also corresponds to a point in the reference image taken by the camera SC at the initial position during calibration. This mapping is determined by mapping a plurality of points of the scene image SC2 to the reference image SC and is obtained by homography determined based on feature recognition or re-identification and preferably robust estimation.

  As can be seen from the above description, according to one embodiment, a set of properties (points, corners, edges, etc.) that define a “virtual transfer surface” along with features identified in the reference image and re-specified in the scene image are used. . The camera image is a projection of the virtual transfer surface onto the image plane of the camera. When these images are taken from different camera positions, the positions of the characteristics in the different camera images are different. Even if the scene image changes due to the movement of the camera, this virtual transfer surface is a scene image taken later by re-specifying the characteristics in the scene image taken later from different positions depending on the head mounted camera. It may be regarded as a “scene stable surface” in the sense that it can be “respecified”. Instead of using an artificial IR light source that must be attached in a real-world scene after first defining a characteristic reference feature using image processing of a head-mounted scene camera image, a later scene by feature detection Find or re-specify the characteristics in the image. Instead, you can use “natural” uncensored images of real-world scenes taken with a regular camera, without artificial markers or additional IR detectors, and perform gaze point determination without using any external markers it can. Instead, even if the head-mounted camera moves, gaze point detection can be executed based only on unchanged scene images without artificial addition.

  The definition of a set of characteristic reference features in the above example is a “characteristic region” as shown in FIG. 1, or a “shelf as a region of interest” that defines the features identified by the feature detection algorithm. This was done by selecting the enclosing rectangle.

However, other regions of interest in the reference image that include characteristics suitable for re-specification in later scene images may be selected. For example, the four front corners of the bookshelf are selected with the mouse, the surrounding area of interest is limited, and in this case, for example, the feature detection algorithm such as the corner detection algorithm is used to select the feature point. Also good. Such corners are then re-identified in later scene images by suitable angle detection algorithms well known to those skilled in the art, such as the Moravec angle detection algorithm, Harris & Stephens angle detection algorithm, Plessey angle detection algorithm, etc. Good.
Further, the SIFT algorithm described in the above example may be used.

Another algorithm for feature identification and feature recognition that may be used in place of or in combination with the SIFT algorithm (based on the entire reference image or based on one or more regions of interest defined in the reference image) is: This is the so-called “speed-up robust feature” (SURF) algorithm. SURF uses the integrated image efficiently as a feature-descriptor based on the sum of approximate 2D Haar Wavelet responses around the feature location.
This finds feature locations using integer approximations to the determinants of the Hessian Blob detector, but can be calculated very quickly with integral images. The algorithm is described in, for example, Herbert Bay, Andreas Ess, Tinne Tuytelaars, Luc Van Gool SURF: “Speeded Up Robust Features”, Computer Vision and Image Understanding (CVIU), Vol. 110, No. 3, pp. 346- 359, 2008.

  As a feature detection algorithm, it is possible to detect the characteristics of points, blobs, regions, objects, etc. in the reference image, and even if the scene image is taken from a different position, the scene image taken later is also detected from the reference image Any image processing algorithm that can recognize or re-specify these characteristics may be used. Image processing algorithms must be able to detect such features based on “natural” images taken with conventional cameras. This avoids the need for artificial IR light sources and the need for additional detectors. Instead, select a feature detection algorithm that can detect unique properties in any “natural” scene image taken with a regular camera, as long as the scene image has some contrast and is not perfectly uniform and includes shapes and contours. To do. By using such a feature detection algorithm, the gazing point mapping approach of the embodiment of the present invention can be performed based on any scene image in principle, without an artificial scene enrichment or an IR light source scene image. There are a number of image processing algorithms that can achieve such functions, such as a somewhat simple one such as a corner detection algorithm, a slightly advanced one such as a SIFT algorithm, or a SURF algorithm. All of these algorithms can operate on natural images without the need for additional IR sources or additional IR detectors because they can detect and use image-specific properties that do not include artificial IR sources as markers. In common.

  According to one embodiment, such property determination is performed automatically by a feature detection algorithm, for example, a region of interest (eg, a mouse or any input or You may respond manually by selecting (by the selection device).

  However, according to an embodiment, characteristics such as characteristic points, blobs, regions, etc. can also be performed completely automatically based on the reference image. For example, the SIFT algorithm described above can identify feature points (or features), so-called “keypoints” that are later re-specified in later images. After selecting the rectangle shown in FIG. 1, the SIFT algorithm can therefore automatically identify "keypoints" within this rectangle. If instead of selecting a rectangle as described above, the entire reference image forms the basis for feature identification by the SIFT algorithm and a virtual transfer surface is defined, a fully automatic procedure can be acquired.

  To summarize the above description of the embodiment, mapping of the gaze direction of the eye up to the reference image of the scene can be executed as follows.

  First, the gaze from the eye tracker ET to the scene camera reference image SC is mapped using a standard procedure.

  Next, a set of characteristics in the reference image is defined using a feature detection algorithm.

  Next, the position of the characteristic reference feature in another scene camera image SC2 taken from a different position is found.

  Then, the transformation (homography) of the scene plane to the virtual transfer surface is acquired / calculated. Using this transformation, the gaze direction (and the corresponding point in the scene image as obtained from the calibration procedure) can be mapped to the corresponding gaze point in the reference image.

  It should be noted that the other scene images may be from other participants (that is, other people wearing head-mounted cameras and eye trackers) as well as from other positions and other times. The head mounted camera does not need to perform a new calibration (i.e., generation of a new reference image or its features) even if another subject wears in the next measurement sequence.

  In this way, even if the user's head moves or is a separate user, the gaze direction is mapped to the corresponding gaze point on the reference image for a certain period (and through different participants) Be able to record gaze points.

  Once you have obtained the gaze mapping for the reference image, for example, highlight the location where the user's gaze is directed, or display the gaze in the reference image by displaying a symbol such as a cross or other mark at the gaze location it can. When the measurement is performed during a certain period, the corresponding development of the gazing point can be displayed by moving the mark or symbol shown in the reference image. This is visualization of the gaze of the object in the reference image.

  However, the initial reference image may not be the best image for such visualization. For example, the initial reference image includes a perspective view instead of a front view of a bookshelf as shown in FIG. However, it is more preferable to use an image showing a gazing point in an image shown in a front view than in a perspective view of the bookshelf for visualization of a target gaze.

  Thus, according to some embodiments, a “final reference image” or “visualized image” may be generated that provides a “desired view” of the scene for determining and tracking the point of interest.

  In order to generate such a “final reference image”, for example, a suitable scene image showing a scene in a desired figure may be selected. For this scene image, a homography mapping between the initial reference image and the scene image has already been determined during the actual measurement phase.

  Using this conversion, all gaze points are mapped from the initial reference image to the “final reference image” or “visualized image” corresponding to the selected scene image.

  If there is no scene image showing the scene in the most desirable way, for example, from a desirable point of view at high resolution, an “external” or “additional” image of the scene can be taken as the final reference image by a different camera. For this final reference image, a feature detection algorithm is executed that detects or re-specifies the characteristic reference feature. Based on the recognized features, the corresponding transformation from the initial reference image to the final reference image (visualized image) is determined again, and using the determined conversion, the gaze point detected in the initial reference image is determined. Convert or map to a visualized image.

  According to one embodiment, the final reference image is generated to best represent the scene image so as to represent the point of interest when converted to the final reference image in the most desirable manner. This is, for example, a case where an object for determining a gazing point is shown as a front view instead of a perspective view, or a case where the final reference image is larger than the scene image or the resolution is higher.

  An example of geometrically modifying the final reference image relative to the initial reference image to provide a front view of the scene from the initial reference image is shown schematically in FIG. FIG. 2 shows the initial reference image of FIG. 1 and the final reference image providing a front view from the initial reference image. A projection diagram corresponding to the homography or transformation in which the initial reference image is projected onto the final reference image and mapping acquisition is determined is also shown schematically. Another example of geometric modification is changing the aspect ratio of the image.

  According to another embodiment, the final reference image may be obtained by removing distortion from the initial reference image, for example, by removing camera distortion, such as barrel distortion.

  According to an embodiment, the generation of the final reference image or visualization image may consist of an image stitch. It may also consist of modification of one or more scene images or their stylization, for example by marking an area or part or sketching a stylized figure. The visualized image may be a handmade drawing in some embodiments.

  Depending on the type of image used as the visualized image, various methods may be used to determine the point transfer mapping from the reference image to the visualized image. For example, if the visualized image is taken with the same camera as the reference image, the feature detection algorithm described above may be used for this purpose.

  However, if the visualized image is a stylized image, or even a sketch or a handmade drawing, the point transfer mapping may be defined manually, for example by a triangle or point-by-point mapping.

  According to one embodiment, the visualization of the mapped gazing point is not a single visualized image, but a frame of a video sequence that acts as a sequence of visualized images.

  Thus, it is possible to determine a point transfer mapping between the reference image and each frame of the video sequence and to use this mapping to map a point of interest of one or more users during a period to a frame of the video sequence. In this way, the gazing point captured and recorded by the head mounted video camera during the period can be mapped via the reference image to another video captured by the different camera from a different position or viewpoint. One or more user gazes along with the development over time can thus be mapped to the same video of the scene taken from different, more desirable perspectives.

  Once the final reference image is generated, a feature detection algorithm can be run on it to detect the characteristic reference features therein. Then, by calculating the homography between the features, it is possible to acquire the mapping of the gazing point to the final reference image. Other means for determining mapping that can convert points from the initial reference image to the corresponding position in the final reference image, such as defining hand homography (specifying 4-point matching, etc.), defining point-by-point conversion, etc. Can also be used.

  As described above with respect to other embodiments, for automatic image processing that detects characteristics, the image portion for which feature detection is performed is characterized by the area (region of interest) and the contents of the area, which allows the algorithm to determine the characteristics. It is possible to define a search area and an object that can be identified in the contents of the area of interest. The homography is calculated from the characteristic reference features in another scene image and the characteristic reference features in the reference image that define an implicit virtual scene plane that facilitates point transfer. Homography extends all points in the scene image to the reference image and to any reference image (corresponding to all points from the plane with parallax with respect to the distance from the virtual transfer plane point and the plane of the real world scene) This reduces the mapping of a moving person's point wearing a head-mounted eye tracker to that of a fixed eye tracker. Therefore, gaze tracking can be performed even when the head mounted camera moves.

  According to an embodiment, by defining an object (or one or more objects) in the reference image through its shape or contour (eg, through its border), the It can be compared whether it is in such a boundary. If it is within the boundary, it can be determined that the corresponding object is being watched, otherwise it is not determined.

  Whether or not an object is being watched is specified based on the gaze mapped on the reference image, not based on the gaze determined in the scene image. By replacing the entire scene image with one reference image and mapping the gaze (including from different participants and time) onto it, the accumulated gaze data can be visualized and statistics can be taken. Thus, analyzing the data of the head mounted eye tracker (at least for the gaze data of the virtual transfer surface) is the same as the analysis of the eye tracking data with the head fixed.

  Those skilled in the art will appreciate that the above-described embodiments may be implemented in hardware, software, or a combination of software and hardware. The modules and functions described in connection with embodiments of the present invention may be implemented in whole or in part by a microprocessor or computer suitably programmed to operate according to the methods described in connection with embodiments of the present invention. Good. This may include a suitable interface and / or measurement device and computer or microprocessor connection used in the field of eye tracking and image processing, as will be readily appreciated by those skilled in the art.

Claims (15)

An apparatus for mapping a target gazing point on a scene image to a gazing point in a reference image, wherein the scene image and the reference image are taken by a camera from different positions,
A module that performs a feature detection algorithm on the reference image and identifies a plurality of characteristics and positions in the reference image;
A module that executes the feature detection algorithm on the scene image and re-specifies the plurality of characteristics and their positions in the scene image;
A module for determining a point transfer mapping for converting a point position between the scene image and the reference image based on the positions of the plurality of characteristics detected in the reference image and the scene image;
An apparatus comprising: a module for mapping a gazing point determined in the scene image to a corresponding point in the reference image using the point transfer mapping.   The point transfer mapping is a homography that is estimated from characteristics of a reference image and a scene image, and that transfers a point position from the scene image to the reference image via a virtual transfer surface in a world scene. Item 2. The apparatus according to Item 1.   3. The apparatus of claim 1 or 2, further comprising a module that determines one or more regions of interest in the reference image and in which the feature detection algorithm is executed to detect and locate the characteristic therein. The feature detection algorithm for detecting the plurality of characteristics in the scene image, detecting the characteristics in the scene image, and respecifying,
A scale time invariant feature transformation algorithm,
The apparatus according to any one of the preceding claims, comprising one of the accelerated robust feature algorithms.   Any of the preceding claims further comprising a module for mapping gaze directions from different people and / or different times to corresponding gaze points in the reference image, and recording the gaze points mapped over time for different users The apparatus according to claim 1.   The said reference image WHEREIN: The said gazing point mapped from the said scene image to the said reference image is displayed on the said reference image, The said gazing point is further comprised from the module during the period of the position, The module which further comprises visualizes. Equipment. The module further comprises a module for visualizing the gazing point mapped from the scene image to the reference image in a visualized image different from the reference image,
A module for determining a point transfer mapping for transferring points from the reference image to the visualized image;
A module for mapping the point of interest from the reference image to its corresponding point in the visualized image using the point transfer mapping;
The apparatus according to claim 1, further comprising a module that displays the gazing point on the corresponding point on the visualized image.   8. The apparatus of claim 7, further comprising a module that determines the point transfer mapping based on characteristics that are detected by the feature detection algorithm in the reference image and detected by the feature detection algorithm in the visualized image and re-specified. The visualized image is
Images selected from scene images taken from different positions,
An image generated by stitching two or more scene images taken from different positions;
A geometrically modified image of the scene,
An image of the scene with the distortion removed,
External images taken with different cameras,
Images taken at different times from the actual measurement of the gaze point,
An image having a higher resolution than the reference image;
With cartoons or sketches,
A stylized figure and
9. Device according to claim 7 or 8, characterized in that it is one of the handmade figures. Further comprising a module for visualizing the point of interest in a frame of a video sequence, the module comprising:
A module for determining a point transfer mapping from a point of the reference image to a corresponding point in a video frame of the video sequence;
The module according to any one of the preceding claims, comprising a module that uses the point transfer mapping to map a gazing point from the reference image to a corresponding point to a corresponding point in a frame of a video sequence. Equipment.   A module for mapping a user's gaze point detected in a scene image taken by a first camera from a position different from the first camera to a corresponding position in a corresponding frame of a video taken by a second camera The apparatus of claim 10 further comprising: An eye tracking module that tracks the gaze of the object and / or a head mounted camera that takes a scene image and / or a calibration module that maps a point of interest to a corresponding point in the scene taken by the head mounted camera. An apparatus according to any one of the preceding claims. A method for mapping a target gazing point on a scene image to a gazing point in a reference image, wherein the scene image and the reference image are taken by a camera from different positions, and the method includes:
Performing a feature detection algorithm on the reference image to identify a plurality of characteristics and their positions in the reference image;
Performing the feature detection algorithm on the scene image to re-specify the plurality of characteristics and their positions in the scene image;
Determining a point transfer mapping for converting a point position between the scene image and the reference image based on the positions of the plurality of characteristics detected in the reference image and the scene image;
Mapping the point of interest determined in the scene image using the point transfer mapping to a corresponding point in the reference image.   14. A method according to claim 13, further comprising the step of performing according to the features defined in any one of claims 2-12.   A computer program comprising computer program code which, when executed on a computer, allows the computer to execute the method according to claim 13 or claim 14.

Images