JP2015046111A - Viewpoint detection device and viewpoint detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a viewpoint detection device and a viewpoint detection method which can enhance accuracy of viewpoint detection.SOLUTION: A viewpoint detection device comprises: a visual line detection unit; a viewpoint detection unit; an acquisition unit; and a correction unit. The visual line detection unit detects a direction of visual line of an operator. The viewpoint detection unit detects a viewpoint position indicating a position of viewpoint of the operator in a display area of a display unit on the basis of the direction of visual line. The acquisition unit acquires a specification position indicating a position specified by the operator in the display area. The correction unit corrects the viewpoint position on the basis of the specification position.

Description

  The present invention relates to a viewpoint detection device and a viewpoint detection method.

  2. Description of the Related Art Conventionally, viewpoint detection devices that detect a position where an operator or a subject is gazing on an observation surface such as a monitor screen have been proposed. For example, Patent Document 1 proposes a line-of-sight detection method using a stereo camera. In recent years, electronic devices have been developed that detect the line of sight of an operator looking at the screen from an image of an eye taken by a built-in camera or an external camera attached due to miniaturization of the camera. In such a viewpoint detection device, a device in which a touch panel or the like is provided on a display unit to improve operability is being studied.

  Patent Document 2 proposes a technique for detecting an operator's line of sight and generating event information corresponding to a touch operation when a touch point on the touch panel ahead of the line of sight is detected. This is a function for preventing operation due to a malfunction that is not intended by the operator.

  Patent Document 3 proposes a technique related to a touch panel correction device that improves operability in a customer operation type terminal using a touch panel. The touch panel correction apparatus includes a viewpoint position detection unit that detects a viewpoint position that determines a correlation between a button position and a corresponding detection position of the touch panel, and detects the touch panel detection position according to a correction amount corresponding to the deviation of the viewpoint position. to correct.

Japanese Patent No. 4517049 JP 2012-203671 A JP 09-044310 A

  Even with a viewpoint detection device that can indicate a position on a screen such as a touch panel, a more accurate viewpoint detection result is required. However, the accuracy is not sufficient due to individual differences in eye images, and it is necessary to improve the accuracy.

  The method of Patent Document 2 can check whether the touch on the touch panel is a malfunction by using the viewpoint detection result, but does not correct the viewpoint detection result. The method of Patent Document 3 corrects the detection position of the touch panel based on the viewpoint position, and does not correct the viewpoint position.

  The present invention has been made in view of the above, and an object of the present invention is to provide a viewpoint detection apparatus and a viewpoint detection method that can improve viewpoint detection accuracy.

  In order to solve the above-described problems and achieve the object, the present invention includes a gaze detection unit that detects the gaze direction of the operator, and the viewpoint of the operator in the display area of the display unit based on the gaze direction. A viewpoint detection unit that detects a viewpoint position that represents a position; an acquisition unit that acquires an indication position that indicates the position indicated by the operator in the display area; and a correction unit that corrects the viewpoint position based on the indication position And.

  The viewpoint detection device and viewpoint detection method according to the present invention have an effect of improving the accuracy of viewpoint detection.

FIG. 1 is a diagram illustrating a configuration of a tablet PC as an example of the present embodiment. FIG. 2 is a block diagram showing a functional configuration of a tablet PC as an example of the present embodiment. FIG. 3 is a flowchart illustrating an example of viewpoint detection processing according to the present embodiment. FIG. 4 is a diagram illustrating an example of a method for calculating a difference between the designated position and the viewpoint position. FIG. 5 is a diagram illustrating an example of a method for calculating the correction value from the difference between the designated position and the viewpoint position. FIG. 6 is a flowchart illustrating an example of a correction value calculation process according to the present embodiment. FIG. 7 is a diagram for explaining a viewpoint position correction method.

  Hereinafter, embodiments of a viewpoint detection device and a viewpoint detection method according to the present invention will be described in detail with reference to the drawings. In addition, this invention is not limited by this embodiment.

  An electronic device with a built-in camera or an externally connected camera can calculate a viewpoint position representing the position the operator is looking on the screen from the eye image of the operator captured by the camera. . However, the viewpoint position detected from the eye image varies greatly depending on the individual, and sufficient accuracy may not be obtained only by the result.

  In this embodiment, when the operator performs a touch operation on the touch panel, it is determined that the eye image when touched is looking around the touch position, and the touch position (instructed position) is determined as the viewpoint detection result (viewpoint position). Used for correction. This realizes more accurate viewpoint position detection.

  Note that applicable apparatuses are not limited to electronic devices including a touch panel. The present invention can be applied to any device as long as the device has a function of detecting the viewpoint of the operator on the display screen and a function of detecting a position (designated position) designated by the operator on the display screen. For example, instead of the touch panel, a device having a function of instructing a position on the screen with a mouse or a touch pad may be used. Further, a method of detecting the viewpoint of the operator using a device other than the camera may be applied.

  FIG. 1 is a diagram illustrating a configuration of a tablet PC 100 as an example of a viewpoint detection apparatus according to the present embodiment. FIG. 1 shows an example of the arrangement of a touch panel, a camera (stereo camera), and an infrared light source of the tablet PC 100. The tablet PC 100 includes a touch panel 102 having functions of a display unit and an operation unit. A pair of stereo cameras (the left camera 103 and the right camera 104) are arranged below the touch panel 102.

  Infrared LED light sources 105 and 106 are arranged in the circumferential direction immediately before the lens of the stereo camera. An infrared LED light source 105 is a light source for the left camera 103. An infrared LED light source 106 is a light source for the right camera 104. Infrared LED light sources 107 and 108 are disposed at positions away from the cameras. An infrared LED light source 107 is a light source for the left camera 103. An infrared LED light source 108 is a light source for the right camera 104. The infrared LED light sources 105 and 106 are lit to acquire an image obtained by reflecting the pupil with high luminance. The infrared LED light sources 107 and 108 are lit to acquire an image obtained by reflecting the pupil with low luminance.

  As a pupil detection method and a gaze detection method using a pupil or the like, for example, the method described in Patent Document 1 can be applied.

  FIG. 2 is a block diagram illustrating an example of a functional configuration of the tablet PC 100. The tablet PC 100 includes a touch panel 102, a control unit 120, a drive / IF unit 130, a left camera 103, a right camera 104, and infrared LED light sources 105 to 108.

  The control unit 120 controls the entire tablet PC 100. For example, the control unit 120 outputs the processing result to the touch panel 102 or the like. A touch panel 102 and a drive / IF unit 130 are connected to the control unit 120. The drive / IF unit 130 includes camera IFs 131R and 131L and an LED drive control unit 132.

  The left camera 103 and the right camera 104 are connected to the drive / IF unit 130 via the camera IFs 131R and 131L, respectively. The drive / IF unit 130 drives these cameras to image the operator and the like.

  A frame synchronization signal is output from the left camera 103. The frame synchronization signal is input to the right camera 104 and the LED drive control unit 132. As a result, the left and right wavelength 1 infrared light sources (infrared LED light sources 105 and 106) emit light at different timings in the first frame, and corresponding images are captured by the left and right cameras (right camera 104 and left camera 103). It is. In the second frame, the left and right infrared light sources (infrared LED light sources 107 and 108) emit light at different timings, and images from the left and right cameras are captured correspondingly.

  The infrared LED light sources 105 and 106 irradiate infrared rays having a wavelength of 1, for example. The infrared LED light sources 107 and 108 irradiate infrared rays having a wavelength of 2, for example.

  The wavelength 1 and the wavelength 2 are, for example, a wavelength of less than 900 nm and a wavelength of 900 nm or more, respectively. When the reflected light reflected by the pupil is irradiated with infrared light having a wavelength of less than 900 nm, a bright pupil image is obtained as compared with the case where reflected light reflected by the pupil is irradiated with infrared light having a wavelength of 900 nm or more. It is because it is obtained. In addition, about the wavelength of the infrared rays to irradiate, it is not restricted to the above, The result which imaged the reflected light reflected by the pupil by irradiating the infrared rays of wavelength 1, and the reflection reflected by the pupil after irradiating the infrared rays of wavelength 2 What is necessary is just to be able to make a difference with the result of imaging light.

  Further, the line-of-sight detection method is not limited to a method using such a stereo camera, and for example, a method using a monocular camera built in the tablet PC 100 may be used.

  The control unit 120 includes a line-of-sight detection unit 121, a viewpoint detection unit 122, an acquisition unit 123, a calculation unit 124, and a correction unit 125.

  The line-of-sight detection unit 121 detects the operator's line of sight (line-of-sight direction) from the captured image captured by the imaging unit (stereo camera). The process of detecting the line of sight includes the process of detecting the position of the operator's eyes. The viewpoint detection unit 122 detects the operator's viewpoint using the detected line-of-sight direction. The viewpoint detection unit 122 detects, for example, the position (viewpoint position) of the viewpoint (gaze point) that is the point on which the operator gazes on the display screen (display area) of the touch panel 102. As a gaze detection method by the gaze detection unit 121 and a viewpoint detection method by the viewpoint detection unit 122, any conventionally used method can be applied. Below, similarly to patent document 3, it demonstrates as an example the case where an operator's gaze direction and a gaze point are detected using a stereo camera.

  In this case, the line-of-sight detection unit 121 first detects the line-of-sight direction of the operator from the image captured by the stereo camera. For example, the line-of-sight detection unit 121 obtains a difference between an image captured by irradiating infrared light having a wavelength 1 and an image captured by irradiating infrared light having a wavelength 2, and generates an image in which a pupil image is clarified. The line-of-sight detection unit 121 uses the two images generated as described above from the images captured by the left and right cameras (the right camera 104 and the left camera 103), and uses the stereo vision technique to position the pupil of the operator. (Eye position) is calculated. In addition, the line-of-sight detection unit 121 calculates the position of the corneal reflection of the operator using images captured by the left and right cameras. Then, the line-of-sight detection unit 121 calculates a line-of-sight vector representing the line-of-sight direction of the operator from the position of the pupil of the operator and the corneal reflection position.

  Note that the method for detecting the eye position and line of sight of the operator is not limited to this. For example, the position and line of sight of the operator's eyes may be detected by analyzing an image captured using visible light instead of infrared light.

  The viewpoint detection unit 122 detects an intersection point between a line-of-sight vector expressed in a predetermined coordinate system and a plane corresponding to the display area of the touch panel 102 as an operator's gaze point. The coordinate system is, for example, with the center position of the display area of the touch panel 102 as the origin, the top and bottom are the Y coordinate (up is +), the side is the X coordinate (right is +), and the depth is the Z coordinate (front is +) Coordinate system can be used. When the gaze direction of both eyes is obtained, the gaze point may be measured by obtaining the intersection of the left and right gazes of the operator.

  The acquisition unit 123 acquires an indicated position that represents a position instructed by the operator within the display area of the display unit. When the touch panel 102 is used as the display unit, the acquisition unit 123 acquires an instruction position touched by an operator (such as a finger or a touch pen) on the touch panel 102. When an instruction function such as a mouse or a touch pad is used, the acquisition unit 123 acquires the position of the cursor on the display area indicated by the instruction function as the instruction position.

  The calculation unit 124 calculates a difference between the viewpoint position detected when the operator points in the display area and the designated position. For example, the calculation unit 124 calculates the difference between the designated position and the viewpoint position for each X coordinate and Y coordinate of the coordinate system. Further, the calculation unit 124 calculates a correction value for correcting the viewpoint position detected later from the difference. As will be described later, when the display area of the touch panel 102 is divided into a plurality of display areas, the calculation unit 124 calculates a difference and a correction value for each of the plurality of divided display areas (hereinafter also referred to as divided areas). Also good.

  The correction unit 125 corrects the viewpoint position detected by the viewpoint detection unit 122 based on the designated position acquired by the acquisition unit 123. For example, the correction unit 125 corrects the viewpoint position detected after the difference from the indicated position and the correction value are calculated using the correction value. When the display area of the touch panel 102 is divided into a plurality of divided areas, the correction unit 125 may correct the viewpoint position using a plurality of correction values calculated for each of the plurality of divided areas.

  Next, viewpoint detection processing by the tablet PC 100 according to the present embodiment configured as described above will be described with reference to FIG. FIG. 3 is a flowchart illustrating an example of viewpoint detection processing according to the present embodiment.

  First, the LED drive control unit 132 turns on the infrared LED light source 105 corresponding to the left camera 103 (step S101). The line-of-sight detection unit 121 acquires a bright pupil image captured by the left camera 103 when the infrared LED light source 105 is turned on (step S102). A bright pupil image is acquired similarly for the right camera 104. That is, the LED drive control unit 132 turns on the infrared LED light source 106 corresponding to the right camera 104 (step S103). The line-of-sight detection unit 121 acquires an image for a bright pupil captured by the right camera 104 when the infrared LED light source 106 is turned on (step S104).

  The infrared LED light sources 105 and 106 are bright pupil LEDs that reflect with high luminance at the pupil. The infrared LED light sources 107 and 108 are dark pupil LEDs that reflect with low luminance at the pupil.

  Next, in order to reflect the pupil with low luminance, the LED drive control unit 132 turns on the infrared LED light source 107 corresponding to the left camera 103 (step S105). The line-of-sight detection unit 121 acquires a dark pupil image captured by the left camera 103 when the infrared LED light source 107 is turned on (step S106). A dark pupil image is similarly acquired for the right camera 104. That is, the LED drive control unit 132 turns on the infrared LED light source 108 corresponding to the right camera 104 (step S107). The line-of-sight detection unit 121 acquires an image for the dark pupil imaged by the right camera 104 when the infrared LED light source 108 is turned on (step S108).

  Next, the line-of-sight detection unit 121 generates, for each of the bright pupil image and the dark pupil image, a difference image representing a difference (luminance difference) between the image captured by the left camera 103 and the image captured by the right camera 104. Generate (step S109). For example, the line-of-sight detection unit 121 generates a difference image by obtaining a difference in luminance at each position (each pixel) of an image captured by each of the left and right cameras. In the obtained difference image, the pupil remains bright, and portions other than the pupil are eliminated or reduced by taking the difference.

  Next, the line-of-sight detection unit 121 detects two regions as pupils from the high-brightness regions that are considered to be pupil candidates in the difference image (step S110). Further, the line-of-sight detection unit 121 calculates a region that is considered to be corneal reflection from an image near the region detected as a pupil (step S111). The line-of-sight detection unit 121 calculates the distance between the pupil and the camera from images acquired from the left and right cameras (step S112). The line-of-sight detection unit 121 calculates the line-of-sight direction of the operator from the calculated pupil position, corneal reflection position, and distance between the pupil and the camera. Then, the viewpoint detection unit 122 detects where the operator is looking on the display area of the touch panel 102 using the line-of-sight direction, that is, the position of the gazing point (viewpoint position) (step S113).

  Thereafter, in order to continue the operations of the gaze detection and the viewpoint detection, the process returns to step S101 and the same operation is repeatedly executed.

  Next, a method for calculating the difference between the designated position and the viewpoint position when the touch panel 102 is operated will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a method for calculating a difference between the designated position and the viewpoint position.

  The operator looks at the screen of the touch panel 102 when using the tablet PC 100. As described above, the images of the eyes of the operator at this time are the left and right with the bright pupil LEDs (infrared LED light sources 105 and 106) and the dark pupil LEDs (infrared LED light sources 107 and 108) turned on. Photographed by a stereo camera (left camera 103, right camera 104). Then, the viewpoint position is calculated from information about the position of the pupil and the position of corneal reflection calculated from the captured image.

  On the other hand, when the operator operates on the screen of the touch panel 102, it is considered that the operator gazes around the contact point 401 between the screen of the touch panel 102 and the finger. For this reason, the calculation unit 124 calculates the difference between the viewpoint positions at the contact point 401 from the coordinates of the viewpoint position obtained at this time and the coordinates of the position (instructed position) of the contact point 401, and a storage unit (not shown). ) Etc.

  FIG. 5 is a diagram illustrating an example of a method for calculating the correction value from the difference between the designated position and the viewpoint position. In the example of FIG. 5, areas obtained by dividing the entire display area of the touch panel 102 into four areas are defined as areas 1 to 4, respectively. The viewpoint position obtained from the camera image when the operator uses the touch panel 102 is a viewpoint position 511, and the position of the touch point on the touch panel 102 is an instruction position 512. Note that the viewpoint position 511 represents the viewpoint position after being corrected using a correction value calculated up to the previous time (the initial value (for example, 0) of the correction value may be used in the first processing). At this time, since the viewpoint position 511 exists in the area 2, the difference between the viewpoint position 511 and the designated position 512 is added to the correction value of the area 2. In this way, the correction value used in the subsequent correction is calculated by adding the difference calculated this time to the correction value calculated up to the previous time. In the example of FIG. 5, the difference is calculated from the viewpoint position 511 and the designated position 512. Then, using the correction value obtained from this difference, the corrected viewpoint position 522 is obtained from the viewpoint position 521 before correction.

  Next, the correction value calculation process of this embodiment will be described with reference to FIG. The correction value calculation process is, for example, a process executed when the touch panel 102 is operated, and correction for correcting the viewpoint position detected in the viewpoint detection process using the indication position detected by this operation. A value is calculated. FIG. 6 is a flowchart illustrating an example of a correction value calculation process according to the present embodiment.

  The acquisition unit 123 determines whether the touch panel 102 has been operated (step S201). If not operated (step S201: No), the process is repeated until it is operated. When operated (step S201: Yes), the acquisition unit 123 acquires a touch position (instructed position) on the display screen of the touch panel 102 (step S202). The acquisition unit 123 further acquires the viewpoint position calculated from the camera image at this time (step S203). The calculation unit 124 determines in which divided region the acquired viewpoint position is present among a plurality of divided regions obtained by dividing the screen (step S204). The calculation unit 124 calculates the difference between the touch position and the viewpoint position (step S205). The calculation unit 124 calculates the correction value of the corresponding divided region by adding the difference calculated this time to the correction value of the corresponding divided region (step S206).

  FIG. 7 is a diagram for explaining a method of correcting the viewpoint position when the touch panel 102 is not operated. A position 711 represents an example of the viewpoint position before correction in the area 1, and a position 721 represents an example of the viewpoint position after correction. A position 712 represents an example of the viewpoint position before correction in the area 2, and a position 722 represents an example of the viewpoint position after correction. A position 713 represents an example of the viewpoint position before correction in the area 3, and a position 723 represents an example of the viewpoint position after correction. A position 714 represents an example of the viewpoint position before correction in the area 4, and a position 724 represents an example of the viewpoint position after correction. An arrow connecting the viewpoint position before correction and the corrected viewpoint position corresponds to the correction value.

  For example, the correcting unit 125 corrects the detected viewpoint position using the correction value of the region including the detected viewpoint position. The correcting unit 125 may correct the viewpoint position using a part or all of a plurality of correction values obtained for each of the divided regions (regions 1 to 4). For example, when the viewpoint position 731 before correction is detected, the correction unit 125 determines the positional relationship between the viewpoint positions (position 711, position 712, position 713, position 714) before correction of each divided region and the viewpoint position 731. Accordingly, the correction values actually used may be calculated by weighting and adding the correction values of the respective divided regions. For example, the weight may be set to be larger as the correction value of the divided region including the viewpoint position that is closer to the viewpoint position 731. A position 732 in FIG. 7 represents an example of a corrected viewpoint position obtained by correcting the viewpoint position 731 in this way.

  As described above, according to the present embodiment, the viewpoint position of the operator when the touch panel 102 is not operated is obtained from the difference between the coordinates of the gazing point and the contact point calculated from the camera image when the touch panel 102 is operated. Correct with the correction value. As a result, it is possible to perform more accurate viewpoint position detection.

Preferred embodiments of the present invention are listed below.
(Embodiment 1) A line-of-sight detection unit that detects the line-of-sight direction of the operator;
A viewpoint detection unit that detects a viewpoint position that represents the position of the operator's viewpoint in the display area of the display unit based on the line-of-sight direction;
An acquisition unit for acquiring an indicated position representing a position indicated by the operator in the display area;
A correction unit that corrects the viewpoint position based on the indicated position;
A viewpoint detection apparatus comprising:
(Embodiment 2) For calculating a difference between the viewpoint position detected by the viewpoint detection unit when the operator indicates the inside of the display area and the specified position, and correcting the viewpoint position A calculation unit that calculates a correction value from the difference;
The correction unit corrects the viewpoint position detected after the difference is calculated by the correction value;
The viewpoint detection apparatus according to the first embodiment, characterized in that:
(Embodiment 3) The calculation unit calculates a new correction value by adding a difference between the viewpoint position corrected by the correction value and the designated position to the correction value,
The correction unit corrects the viewpoint position detected after a new correction value is calculated with the new correction value;
The viewpoint detection apparatus according to the second embodiment, characterized in that:
(Embodiment 4) The said calculation part calculates the said difference for every some display area which divided | segmented the said display part,
The correction unit corrects the viewpoint position using a plurality of the differences calculated for the plurality of display areas;
The viewpoint detection apparatus according to the second embodiment, characterized in that:
(Embodiment 5) A line-of-sight detection step of detecting an operator's line-of-sight direction;
A viewpoint detection step of detecting a viewpoint position representing the position of the operator's viewpoint in the display area of the display unit based on the line-of-sight direction;
An acquisition step of acquiring an indicated position representing a position indicated by the operator in the display area;
A correction step of correcting the viewpoint position based on the indicated position;
Viewpoint detection method.

As described above, according to the present embodiment, for example, the following effects can be obtained.
In addition to the conventional gaze detection method, it is possible to correct viewpoint detection by obtaining information indicating the position where the touch panel or the like is touched (instructed). Thereby, viewpoint detection with higher accuracy can be performed.

100 Tablet PC
DESCRIPTION OF SYMBOLS 102 Touch panel 103 Left camera 104 Right camera 105-108 Infrared LED light source 120 Control part 121 Eye-gaze detection part 122 View point detection part 123 Acquisition part 124 Calculation part 125 Correction part 130 Drive / IF part 131R, 131L Camera IF
132 LED drive control unit

Claims (3)

A line-of-sight detector that detects the line-of-sight direction of the operator;
A viewpoint detection unit that detects a viewpoint position that represents the position of the operator's viewpoint in the display area of the display unit based on the line-of-sight direction;
An acquisition unit for acquiring an indicated position representing a position indicated by the operator in the display area;
A correction unit that corrects the viewpoint position based on the indicated position;
A viewpoint detection apparatus comprising: A difference between the viewpoint position detected by the viewpoint detection unit when the operator points in the display area and the indicated position is calculated, and a correction value for correcting the viewpoint position is calculated. A calculation unit for calculating from the difference,
The correction unit corrects the viewpoint position detected after the difference is calculated by the correction value;
The viewpoint detection apparatus according to claim 1. A line-of-sight detection step for detecting the line-of-sight direction of the operator;
A viewpoint detection step of detecting a viewpoint position representing the position of the operator's viewpoint in the display area of the display unit based on the line-of-sight direction;
An acquisition step of acquiring an indicated position representing a position indicated by the operator in the display area;
A correction step of correcting the viewpoint position based on the indicated position;
Viewpoint detection method.

Images