JP5229928B1 - Gaze position specifying device and gaze position specifying program - Google Patents

Abstract

A position on a screen at which an operator is gazing is specified.
A gaze position specifying device 100 controls a screen display unit that displays a screen on a display device 104 and an imaging device 101 to capture an image including an operator's pupil, and an image of the operator's pupil. (Hereinafter, referred to as “pupil image”) and image processing of the pupil image acquired by the pupil image acquisition unit, and an image reflected in the operator's pupil is extracted as a target image. From the screens displayed by the target image extraction means and the screen display means, a position where an image having a high similarity with the target image extracted by the target image extraction means is specified as a gaze position by the operator. Gaze position specifying means.
[Selection] Figure 1

Description

  The present invention relates to a gaze position specifying device and a gaze position specifying program.

  The following gaze direction identification systems are known. This line-of-sight direction identifying system detects the corners of the eyes, both mouths, and the center of the black eye in the image captured by the imaging device, identifies the position and direction of the face from the corners of the eyes and the mouth, and determines the direction of the black eye from the center of the detected black eye. The line-of-sight direction is specified from the specified face position and direction and the black eye direction. Then, the keyboard is displayed on the display device, the direction of the user's line of sight is detected by detecting the user's line of sight direction on the displayed keyboard, or the user's line of sight direction is detected, and the display surface and line of sight of the display device are detected. When the directions cross each other, the cursor is displayed at the intersection to enable operation of the apparatus (for example, Patent Document 1).

JP 2006-141862 A

  However, in the conventional gaze direction specifying system, in order to specify which position of the display device the user is gazing at, the eye corners, the mouth corners, and the center of the black eye in the captured image are detected, and the eye corners are detected. It is necessary to specify the face position and direction from the mouth, the black eye direction from the detected black eye center, and the line-of-sight direction from the specified face position and direction and black eye direction. For this reason, since there are many elements to be detected and specified in order to specify the user's line-of-sight direction, there has been a problem that detection errors in the line-of-sight direction are likely to occur. In addition, there is a problem that the load of image processing for detecting the user's line-of-sight direction is high, and the processing takes time.

The gaze position specifying device according to the present invention controls a screen display unit that displays a screen on a display device and an imaging device to capture an image including an operator's pupil, and an image of the operator's pupil (hereinafter, “ A pupil image acquisition unit that acquires a pupil image), and a target image extraction unit that performs image processing on the pupil image acquired by the pupil image acquisition unit and extracts an image reflected in the operator's pupil as a target image And gaze position specifying means for specifying a position where an image having a high similarity with the target image extracted by the target image extraction means is displayed as a gaze position by the operator from the screen displayed by the screen display means It is characterized by providing.
The gaze position specifying program according to the present invention controls a screen display procedure for displaying a screen on a display device and an image pickup device to pick up an image including an operator's pupil, and an image of the operator's pupil (hereinafter, “ A pupil image acquisition procedure for acquiring a pupil image ”, a target image extraction procedure for performing image processing on the pupil image acquired in the pupil image acquisition procedure, and extracting an image shown in the operator's pupil as a target image; And a gaze position identifying procedure for identifying a position where an image having a high similarity to the target image extracted in the target image extraction procedure is displayed as a gaze position by the operator from the screens displayed in the screen display procedure. It is made to perform.

  According to the present invention, it is possible to specify the position on the screen where the operator is gazing just by specifying the object reflected in the operator's pupil, so that the accuracy of specifying the gazing position can be improved. it can. Further, the processing load can be reduced and the processing time can be shortened.

1 is a block diagram showing a configuration of an embodiment of a gaze position specifying device 100. FIG. It is the figure which showed the specific example of the difference calculation process typically. 5 is a diagram illustrating an example of an operation method of the gaze position specifying device 100. FIG. It is a flowchart figure which shows the flow of an operation reception process. It is a figure which shows typically the specific example of the identification method of the gaze position on the screen by the operator in a modification.

  FIG. 1 is a block diagram showing a configuration of an embodiment of a gaze position specifying device according to the present embodiment. The gaze position specifying device 100 uses various information terminals such as a personal computer, a smartphone, a mobile phone, and a tablet terminal, and includes an imaging device 101, a control device 102, a recording device 103, and a display device 104. Yes.

  The imaging device 101 includes a lens, an imaging element, and other peripheral circuits. The image sensor is composed of an image sensor such as a CCD or a CMOS, for example, and generates an analog image signal by photoelectrically converting a subject image by a light beam that has passed through a lens, and outputs the generated analog image signal to the control device 102 as image data. To do.

  In the present embodiment, the imaging device 101 is arranged so that the optical axis thereof is set in a direction in which the operator's pupil facing the display device 104 enters the angle of view. For example, the display device 104 may be attached as an external camera to the edge portion of the screen, or may be attached as a built-in camera embedded in the edge of the screen. Or you may make it install in the periphery of the display apparatus 104 so that the pupil of the operator who opposes the display apparatus 104 may enter into an angle of view.

  The control device 102 includes a CPU, a memory, and other peripheral circuits, and controls the gaze position specifying device 100 as a whole. In addition, the memory which comprises the control apparatus 102 is volatile memories, such as SDRAM, for example. This memory is used as a work memory for the CPU to expand the program when the program is executed and a buffer memory for temporarily recording data. For example, the image data acquired by the imaging device 101 is temporarily recorded in the buffer memory.

  The recording device 103 is a recording device for recording various data stored by the gaze position specifying device 100, data of a program to be executed by the control device 102, and the like, for example, HDD (Hard Disk Drive) or SSD (Solid). State Drive) or the like is used. The program data recorded in the recording device 103 is provided by being recorded on a recording medium such as a CD-ROM or DVD-ROM, or provided via a network, and the program data acquired by the user is recorded. By installing in the device 103, the control device 102 can execute the program.

  The display device 104 is a liquid crystal monitor, for example, and displays various display data output from the control device 102.

  In the gaze position specifying device 100 according to the present embodiment, the control device 102 controls the imaging device 101 to perform imaging, and obtains an image including the operator's eyes facing the display device 104. Then, the control device 102 extracts an image of one eye of the operator (hereinafter referred to as “pupil image”) from the acquired image including the pupil. As a process for this, for example, information related to the feature point of the person's face is recorded in the recording device 103 in advance, and the position of the operator's eye is determined from the image including the pupil based on the information related to the feature point. The pupil image may be extracted by specifying and extracting the image of the specified eye position. Alternatively, a template image imitating the shape of the pupil may be recorded in the recording device 103 in advance, and a template matching process using the template image may be performed to extract the pupil image. That is, the control device 102 calculates the similarity at each position while shifting the template image by a predetermined pixel, for example, one pixel within the image including the pupil, and extracts the image at the position with the highest similarity as the pupil image. You can do it. Note that it is only necessary to set in advance which eye image of the left or right eye of the operator is to be acquired.

  The control device 102 performs image processing on the thus-extracted pupil image, and extracts an image shown in the operator's pupil as a target image. Here, as described above, the imaging apparatus 101 is arranged so that the optical axis of the imaging apparatus 101 is set so that the operator's pupil facing the display apparatus 104 enters the angle of view. Shows an image of the screen displayed on the display device 104. Therefore, an image of the screen displayed on the display device 104 is extracted as the target image. When a plurality of images reflected on the operator's pupil are extracted from the pupil image, an image having the shortest distance from the center of the pupil, for example, the center of the black eye, may be extracted as the target image.

  As a method for extracting the target image from the pupil image, for example, the following method can be considered. First, an image of a pupil in a state where the operator is looking at a uniform plane is acquired in advance, and this is recorded in the recording device 103 as a reference image. As an image of a pupil in a state of looking at a uniform plane, for example, the control device 102 displays a uniform image such as a white color on the display device 104, controls the imaging device 101, and the operator at that time A pupil image is acquired. Note that the method for acquiring the pupil image used as the reference image is the same as the method for acquiring the pupil image described above.

  The control device 102 performs a difference calculation process (subtraction process) between the pupil image extracted as described above and the reference image recorded in advance, and calculates an absolute value thereof. As a result, as a difference between the two images, an image shown in the operator's pupil can be extracted as a target image. FIG. 2 is a diagram schematically showing a specific example of the difference calculation process. FIG. 2A is a reference image obtained by photographing the eyes of the operator, and FIG. 2B shows a pupil image in which a pentagonal image displayed on the monitor 104 is shown. The control device 102 performs a difference calculation process between the reference image shown in FIG. 2A and the pupil image shown in FIG. 2B, and calculates the absolute value thereof, so that FIG. The target image 2a can be extracted by obtaining the difference image shown.

  The control device 102 sets a template image based on the extracted image of the target image 2a, and sets a screen image displayed on the monitor 104 as the target image to perform template matching processing. Specifically, the control device 102 calculates the similarity between both images at each position while shifting the template image by a predetermined pixel, for example, one pixel on the target image, and the highest similarity is calculated. A position in the target image is specified. As the position in the target image, the center position of the template image when the similarity is calculated may be adopted.

  The position in the target image specified here corresponds to the center position of the target image on the screen and matches the center position of the target image shown in the operator's pupil. Therefore, this position is monitored by the operator. It can be regarded as the position being watched on the screen displayed on the screen 104. For this reason, the control apparatus 102 can specify the position where the operator is gazing on the screen displayed on the monitor 104 by the above processing.

  When the position where the operator is gazing on the screen is specified, the control device 102 acquires the coordinate value of the position on the screen and records this in the buffer memory as the coordinate value of the gazing position. The coordinate value on the screen may be obtained by setting the horizontal axis and the vertical axis with the lower left corner of the screen as the origin, or by setting the horizontal axis and the vertical axis with the center of the screen as the origin. May be.

  The control device 102 monitors the change of the gaze position of the operator on the screen by repeatedly executing the above processing at a predetermined time interval, for example, at an interval of 1 second. Specifically, the control device 102 reads the coordinate value of the gaze position in the previous frame recorded in the buffer memory, and calculates the difference from the coordinate value of the gaze position in the current frame, so that A time change in the coordinate value of the gaze position is detected. Thereby, the change of the operator's gaze position on the screen can be monitored. And the control apparatus 102 receives operation of the gaze position specific | specification apparatus 100 according to the time change of the detected gaze position. Thereby, the operator can operate the gaze position specifying device 100 by changing the gaze position on the screen.

  An example of an operation method of the gaze position specifying device 100 will be described with reference to FIG. The operator can perform an operation corresponding to the gaze position by continuing to gaze at one point on the screen for a predetermined time or more. For example, the operator can instruct an operation on the operation target user interface by watching the operation target user interface arranged on the screen for a predetermined time or more. For example, as shown in FIG. 3, it is possible to instruct a click operation of these buttons by gazing at an operation target user interface such as an “OK” button 3a or a “cancel” button 3b on a screen for a predetermined time or more. it can.

  That is, as a result of monitoring the change of the operator's gaze position on the screen by the above-described processing, the control device 102 does not change the gaze position of the operator for a predetermined time or more, for example, 5 seconds or more. In the case of corresponding to the display position of the user interface, processing is performed when the operation target user interface at that position is operated. For example, if the operation target user interface is the “OK” button 3a, a process to be executed when the “OK” button 3a is clicked is executed. If the operation target user interface is the “cancel” button 3b, a process to be executed when the “cancel” button 3b is clicked is executed.

  Further, the operator can instruct a preset operation by gazing at portions other than the operation target user interface on the screen for a predetermined time or more. For example, when the operator gazes at a part other than the user interface to be operated for a predetermined time or longer, and the operation is assigned to enlarge the screen around the gaze position, the operator By gazing at the point 3c for a predetermined time or more, it is possible to instruct an enlargement display operation of the screen around the point 3c. Alternatively, when the operator gazes at a part other than the user interface to be operated for a predetermined time or more, and the operation is assigned so as to reduce the screen around the gaze position, the operator By gazing at the point 3c for a predetermined time or more, it is possible to instruct a reduction display operation of the screen centering on the point 3c. Other operations may be assigned.

  The operator can instruct a scroll operation of the screen in the moving direction by moving the gaze position from one point on the screen to another point. For example, the operator can instruct screen scrolling in the direction indicated by the arrow 3f, that is, in the downward direction, by moving the gaze position from the point 3d on the screen to the point 3e on the screen. That is, as a result of monitoring the change in the operator's gaze position on the screen by the above-described processing, the control device 102 indicates that the operator's gaze position has moved in any direction on the screen during the predetermined time. If it is detected, the screen is scrolled in the moving direction.

  In this way, the control device 102 detects a change in the gaze position on the screen by the operator, and accepts an operation on the screen according to the detection result. It is possible to intuitively perform operations on the screen simply by changing the viewing position on the screen without using the existing operation member.

  FIG. 4 is a flowchart showing a flow of operation reception processing in the present embodiment. The process shown in FIG. 4 is executed by the control device 102 as a program that starts when the gaze position specifying device 100 is powered on. Note that the power of the gaze position specifying device 100 can be turned on and off by operating a power button (not shown).

  In step S <b> 10, the control device 102 controls the imaging device 101 to perform imaging, and acquires an image including the operator's pupil that faces the display device 104. Then, it progresses to step S20.

  In step S20, the control apparatus 102 extracts an operator's pupil image from the image containing the pupil acquired by step S10 by the process mentioned above. Then, it progresses to step S30.

  In step S30, the control device 102 determines whether or not a pupil image has been extracted in step S20. If a negative determination is made in step S30, the process proceeds to step S110 described later. On the other hand, if a positive determination is made in step S30, the process proceeds to step S40.

  In step S <b> 40, the control device 102 performs image processing on the pupil image as described above, and extracts an image shown in the operator's pupil as a target image. Thereafter, the process proceeds to step S50.

  In step S50, the control device 102 determines whether or not the target image has been extracted in step S40. If a negative determination is made in step S50, the process proceeds to step S110 described later. On the other hand, if a positive determination is made in step S50, the process proceeds to step S60.

  In step S60, as described above, the control device 102 specifies the position of the operator's gaze on the screen displayed on the monitor 104 by specifying the position of the target image extracted in step S40 on the screen. To do. Thereafter, the process proceeds to step S70.

  In step S70, the control apparatus 102 acquires the coordinate value of the gaze position of the operator on the screen specified in step S60, and records this in the buffer memory as the coordinate value of the gaze position. Thereafter, the process proceeds to step S80.

  In step S80, as described above, the control device 102 detects a temporal change in the operator's gaze position on the screen. Thereafter, the process proceeds to step S90.

  In step S90, as described above with reference to FIG. 3, the control device 102 receives an operation of the gaze position specifying device 100 according to the temporal change in the gaze position detected in step S80.

  Thereafter, the process proceeds to step S100, and as described above, processing according to the operation content received in step S90, for example, button click processing, screen enlargement / reduction processing, scroll processing, and the like are executed. Then, it progresses to step S110.

  In step S110, the control device 102 determines whether or not the power of the gaze position specifying device 100 is turned off based on an instruction from the operator. If a negative determination is made in step S110, the process returns to step S10 and is repeated. At this time, the control device 102 performs control so that detection of a change in the gaze position of the operator is repeated at a preset time interval, for example, one second interval, using a time measurement result by a timer (not shown). On the other hand, if an affirmative determination is made in step S110, the process ends.

According to the present embodiment described above, the following operational effects can be obtained.
(1) The control device 102 displays a screen on the display device 104, controls the imaging device 101 to capture an image of the operator's pupil, acquires a pupil image, performs image processing on the acquired pupil image, The image reflected in the operator's pupil is extracted as the target image. And the control apparatus 102 specified the position where the image with high similarity with a target image is displayed from the screen currently displayed on the display apparatus 104 as a gaze position by an operator. As a result, it is possible to specify the position on the screen where the operator is gazing just by specifying the target image reflected in the operator's pupil. The position where the person is watching

(2) The control device 102 detects a time change in the gaze position of the identified operator, identifies the content of the operation by the operator based on the detected time change in the gaze position, and performs processing based on the identified operation content Was made to run. Thus, the operator can intuitively operate the gaze position specifying device 100 by changing the gaze position on the screen.

(3) When the control device 102 monitors the change in the gaze position of the operator on the screen and detects that the gaze position of the operator has moved in any direction on the screen during a predetermined time Scrolls the screen in that direction. Thus, the operator can intuitively perform the scroll operation only by moving the gaze position in the direction in which the user wants to scroll on the screen.

(4) When the operator's gaze position on the screen does not change for a predetermined time or more and the position corresponds to the display position of the operation target user interface, the control device 102 corresponds to the operation target user interface at that position. It was specified that the operation was instructed, and the processing when the operation target user interface was operated was executed. Thus, the operator can intuitively operate the operation target user interface simply by gazing at the operation target user interface desired to be operated on the screen.

(5) When the operator's gaze position on the screen does not change for a predetermined time or more and the position corresponds to a display position other than the operation target user interface, the control device 102 displays the gaze position as a center. The screen enlargement / reduction processing was executed after specifying that the content enlargement / reduction was instructed. As a result, the operator can intuitively instruct the enlargement or reduction of the screen only by gazing at the part of the screen that is desired to be enlarged or reduced.

(6) The imaging device 101 is arranged so that its optical axis is set in a direction in which the pupil of the operator facing the display device 104 enters the angle of view. As a result, it is possible to capture the operator's pupil image that faces the display device 104.

(7) The control device 102 is reflected on the operator's pupil from the pupil image by subtracting a reference image obtained by photographing the operator's pupil in a state of viewing the uniform plane acquired in advance from the pupil image. The target image was extracted. As a result, it is possible to extract an image reflected on the operator's pupil with high accuracy by a simple process of subtracting the images and creating a difference image.

(8) The control device 102 calculates the similarity of both images at each position while shifting the template image by a predetermined pixel, for example, one pixel on the target image, and the target image for which the highest similarity is calculated The position inside is specified as the gaze position by the operator. Thereby, the gaze position on the screen by the operator can be specified with high accuracy.

-Modification-
Note that the gaze position specifying device 100 of the above-described embodiment can be modified as follows.
(1) In the above-described embodiment, the control device 102 sets the image of the target image 2a as a template image, and sets the image of the screen displayed on the monitor 104 as the target image, and performs template matching processing. An example has been described in which the position on the target image for which the highest similarity is calculated is specified as the gaze position on the screen by the operator. However, the method for specifying the gaze position on the screen by the operator is not limited to this. For example, in the case of a terminal equipped with a small monitor as the display device 104 such as a smartphone or a tablet terminal, there is a high possibility that an image of the entire monitor is reflected in the pupil. Therefore, the control device 102 extracts the image of the monitor frame by extracting the largest rectangular image from the pupil image. Then, the control device 102 extracts the operator's black eye from the pupil image and specifies the center position. As a result, as shown in FIG. 5, the frame image 5a corresponding to the outer periphery of the monitor and the center position 5b of the black eye are specified. The control device 102 determines the vertical ratio A between the distance S1 from the center position 5b of the black eye to the upper side of the frame image 5a and the distance S2 to the lower side, and the distance S3 to the right side and the distance to the left side. The horizontal ratio B to the distance S4 is calculated, and the operator pays attention to the position on the screen displayed on the display device 104 where the vertical ratio and the horizontal ratio match A and B described above. You may make it specify as a position which is doing.

(2) In the above-described embodiment, as an explanation of the operation method using FIG. 3, an example in which the operator can perform a button click operation, a screen enlargement / reduction operation, and a screen scroll operation has been described. . However, the present invention can also be applied when performing other operations.

  For example, there are other operation target user interfaces other than the operation buttons, and when the control device 102 detects that the operator has been gazing for a predetermined time or more, the operation target user interface is also operated. The same processing as when the target user interface is clicked may be performed. For example, when the operation target user interface is a radio button, the operator can instruct the selection operation of the radio button by watching the radio button on the screen for a predetermined time or more. When the operation target user interface is a check box, the operator can instruct a check operation on the check box by watching the check box on the screen for a predetermined time or more.

  In addition, when a text box for inputting characters is displayed on the screen, the control device 102 detects that the operator has watched the text box for a predetermined time or longer. While accepting input of characters into the box, a virtual keyboard image may be displayed on the screen. Then, the operator's gaze position on the displayed keyboard is specified, the input of the key arranged at the gaze position is accepted, and the characters are displayed in the text box.

(3) In the embodiment described above, the power of the gaze position specifying device 100 can be turned off by the operator operating a power button (not shown). However, the operator may be able to turn off the power of the gaze position specifying device 100 by changing the gaze position on the screen. For example, a button for instructing to turn off the power is arranged on the screen, and the controller 102 instructs the operator to turn off the power for a predetermined time, for example, 5 seconds or longer as a result of performing the above-described processing. When it is detected that the user is gazing at the button for doing so, the gaze position specifying device 100 may be turned off.

(4) In the above-described embodiment, an example is described in which the operator acquires an image of a pupil in a state of viewing a uniform plane in advance, and records this in the recording apparatus 103 as a reference image. Therefore, the example has been described in which the control device 102 displays a uniform image such as a white color on the display device 104, controls the imaging device 101, and acquires the pupil image of the operator at that time as a reference image. The reference image acquisition process may be performed by switching the mode of the gaze position specifying device 100. For example, the control device 102 switches the shooting mode of the imaging device 101 to a reference image shooting mode for shooting a reference image based on an instruction from the operator, and displays a uniform image on the display device 104. Then, the image pickup apparatus 101 may be controlled to acquire an image of the operator's pupil at that time.

(5) In the above-described embodiment, the control device 102 has described an example in which the time change of the gaze position of the operator on the screen is detected and the operation by the operator is received based on the detection result. However, an operation member such as a touch panel, a mouse, and a keyboard may be mounted on the gaze position specifying device 100, and reception of an operation via the operation member and reception of an operation based on a temporal change in the gaze position may be used in combination.

(6) In the above-described embodiment, when it is detected that the operator's gaze position has moved in any direction on the screen during a predetermined time, it is determined that the scroll operation has been performed, An example has been described in which it is determined that an operation related to the gaze position is instructed when the gaze position of the operator does not change for a predetermined time or more. However, the operator may perform the operation by combining the movement and fixation of the gaze position. For example, after an operator gazes and fixes a certain point on the screen for a predetermined time or longer and then moves the gaze position upward, the operator performs an enlargement operation of the screen around the gaze position that was initially fixed. After a certain point on the screen is fixed by gazing for a predetermined time, the gaze position is continuously moved downward so that the screen can be reduced around the initially fixed gaze position. It may be. Thereby, the operator can perform many operations by combining a plurality of gaze methods on the screen.

(7) In the above-described embodiment, an example has been described in which the control device 102 controls the imaging device 101 to perform imaging and acquires an image including the operator's pupil facing the display device 104. At this time, the control device 102 may perform control for acquiring a clearer pupil image in order to increase the accuracy of specifying the gaze position of the operator. For example, when taking a pupil image, the control device 102 uses the known subject recognition function to identify the eyes of the operator and controls the imaging device 101 to focus on the identified eyes. That's fine. Further, zooming may be performed so that the size of the eyes of the operator in the pupil image is a certain level or more. Alternatively, the control device 102 performs image processing such as enlargement processing, contour enhancement processing, and sharpening processing on an image including a pupil acquired by controlling the imaging device 101, and then extracts a pupil image. Also good.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired. Moreover, it is good also as a structure which combined the above-mentioned embodiment and a some modification.

100 gaze position specifying device 101 imaging device 102 control device 103 recording device 104 display device

Claims (22)

Screen display means for displaying a screen on a display device;
Pupil image acquisition means for controlling the imaging device to capture an image including the operator's pupil and acquiring an image of the operator's pupil (hereinafter referred to as “pupil image”);
Subject image extraction means for subjecting the pupil image acquired by the pupil image acquisition means to image processing, and extracting an image shown in the operator's pupil as a target image;
A gaze that identifies a position at which an image having a high similarity to the target image extracted by the target image extraction unit is displayed as the gaze position by the operator from the screen displayed by the screen display unit. A gaze position specifying device comprising: position specifying means. The gaze position specifying device according to claim 1,
The gaze position specifying device further comprising an operation accepting unit that accepts an operation by the operator on the screen based on a result of the gaze position specifying unit. The gaze position specifying device according to claim 2,
An operation content specifying means for detecting a time change of the gaze position specified by the gaze position specifying means and specifying an operation content by the operator based on the detected time change of the gaze position;
The gaze position identifying device, wherein the operation accepting unit accepts an operation based on the operation content specified by the operation content specifying unit. The gaze position specifying device according to claim 3,
The operation content specifying unit specifies that when the gaze position specified by the gaze position specifying unit changes within a predetermined time, a scroll operation toward the changed gaze position direction in the screen is specified. A gaze position specifying device characterized in that: In the gaze position specifying device according to claim 3 or 4,
The operation content specifying means specifies that an operation for the gaze position in the screen has been instructed when the gaze position specified by the gaze position specifying means does not change for a predetermined time or more. Gaze position identification device to do. The gaze position specifying device according to claim 5,
The operation content specifying means specifies that an operation for the operation target interface is instructed when the gaze position in the screen is a position where the operation target interface is displayed. Positioning device. The gaze position specifying device according to claim 5,
When the gaze position in the screen is a position where an interface other than the operation target interface is displayed, the operation content specifying unit specifies that an instruction to enlarge or reduce the screen around the gaze position is specified. A gaze position specifying device characterized in that: In the gaze position specifying device according to any one of claims 1 to 7,
The gaze position specifying device, wherein the optical axis is set in a direction in which the pupil of the operator facing the display device enters an angle of view. In the gaze position specifying device according to any one of claims 1 to 8,
The target image extracting means extracts the target image from the pupil image by subtracting a reference image obtained by photographing the operator's pupil in a state of viewing a uniform plane acquired in advance from the pupil image. A gaze position specifying device characterized by the above. The gaze position specifying device according to claim 9,
Mode switching means for switching a photographing mode by the imaging device to a reference image photographing mode for photographing the reference image;
Uniform image display means for displaying a uniform image on the display device when switched to the reference image capturing mode by the mode switching means;
Reference image acquisition means for controlling the imaging device to acquire an image of the eyes of the operator who is viewing the display device on which the uniform image is displayed as the reference image. Gaze position identification device to do. In the gaze position specifying device according to any one of claims 1 to 10,
The gaze position specifying unit sequentially matches the target image extracted by the target image extracting unit while shifting the predetermined image by a predetermined pixel within the image of the screen displayed by the screen display unit, and has the highest similarity. A gaze position identifying device that identifies a position as the gaze position. A screen display procedure for displaying a screen on the display device;
A pupil image acquisition procedure for controlling an imaging device to capture an image including the operator's pupil and acquiring an image of the operator's pupil (hereinafter referred to as “pupil image”);
A target image extraction procedure for performing image processing on the pupil image acquired in the pupil image acquisition procedure and extracting an image reflected in the pupil of the operator as a target image;
Gaze position identification that identifies, as the gaze position by the operator, a position at which an image having a high similarity to the target image extracted in the target image extraction procedure is displayed from the screen displayed in the screen display procedure A gaze position specifying program for causing a computer to execute the procedure. In the gaze position specifying program according to claim 12,
A gaze position specifying program further comprising an operation receiving procedure for receiving an operation by the operator on the screen based on a result of the gaze position specifying procedure. In the gaze position specifying program according to claim 13,
Further comprising an operation content specifying procedure for detecting a time change in the gaze position specified in the gaze position specifying procedure, and specifying an operation content by the operator based on the detected time change in the gaze position,
The gaze position specifying program characterized in that the operation receiving procedure receives an operation based on the operation content specified in the operation content specifying procedure. In the gaze position specifying program according to claim 14,
The operation content specifying procedure specifies that the scroll operation toward the changed gaze position direction in the screen is instructed when the gaze position specified in the gaze position specifying procedure changes within a predetermined time. A gaze position specifying program characterized by that. In the gaze position specifying program according to claim 14 or 15,
The operation content specifying procedure specifies that an operation for the gaze position in the screen has been instructed when the gaze position specified in the gaze position specifying procedure has not changed for a predetermined time or more. Gaze position identification program. In the gaze position specifying program according to claim 16,
The operation content specifying procedure specifies that an operation for the operation target user interface is instructed when the gaze position in the screen is a position where the operation target user interface is displayed. Gaze position identification program. In the gaze position specifying program according to claim 16,
In the operation content specifying procedure, when the gaze position in the screen is a position where an area other than the operation target user interface is displayed, enlargement or reduction of the screen centered on the gaze position is instructed. A gaze position specifying program characterized by specifying. In the gaze position specifying program according to any one of claims 12 to 18,
The gaze position specifying program characterized in that the imaging device sets an optical axis in a direction in which the pupil of the operator facing the display device enters an angle of view. In the gaze position specifying program according to any one of claims 12 to 19,
The target image extraction procedure extracts the target image from the pupil image by subtracting, from the pupil image, a reference image obtained by photographing the operator's pupil in a state of viewing a uniform plane acquired in advance. A gaze position specifying program characterized by that. In the gaze position specifying program according to claim 20,
A mode switching procedure for switching a shooting mode by the imaging device to a reference image shooting mode for shooting the reference image;
A uniform image display procedure for displaying a uniform image on the display device when the mode switching procedure is switched to the reference image capturing mode;
A reference image acquiring procedure for controlling the imaging device to acquire an image of the pupil of the operator who is viewing the display device on which the uniform image is displayed as the reference image. Gaze position identification program. In the gaze position specifying program according to any one of claims 12 to 21,
In the gaze position specifying procedure, in the image of the screen displayed by the screen display procedure, the target image extracted by the target image extraction procedure is sequentially matched while shifting by a predetermined pixel, and the position with the highest similarity is determined. A gaze position specifying program characterized by specifying as a gaze position.

Images