WO2022196093A1 - Information processing device, line-of-sight detection method, and program - Google Patents

Abstract

Provided is an information processing device comprising a line-of-sight detection computation unit that performs a line-of-sight detection process for detecting a line-of-sight direction on the basis of an eye image captured by an eye image capture unit and performs a correction process for correcting calibration data to be used in the line-of-sight detection process on the basis of roll angle information about a change in the relative angle between the eye and an image capture unit in the line-of-sight detection device unit.

Description

Information processing device, line-of-sight detection method, program

This technology relates to an information processing device, a line-of-sight detection method, and a program, and particularly to technology for line-of-sight detection processing.

For example, a technique for detecting a line of sight using a pupillary corneal reflection method is known. When performing such line-of-sight detection, it is necessary to perform calibration according to individual differences in eyes in advance.
Further, for example, when a viewfinder of an imaging device (camera) is equipped with a line-of-sight detection function, the user holds the camera vertically or horizontally, so the relative posture between the user and the camera is not constant. Therefore, the tilt of the eye detected by the line-of-sight detection device may be significantly different from that at the time of calibration. Therefore, for example, a plurality of calibration data corresponding to a plurality of camera orientations are required.

Patent Document 1 below discloses a method for detecting whether the camera is held vertically or horizontally by detecting the orientation of the camera with respect to gravity using an acceleration sensor or the like, and storing a plurality of calibration data according to the camera orientation. It is disclosed that

JP-A-7-255676

However, in order to save a plurality of pieces of calibration data according to camera orientations, calibration must be performed a plurality of times according to a plurality of camera orientations.
Also, even if the calibration data for vertical and horizontal holding are stored according to the camera posture, for example, when the user looks at the camera from an angle or the camera is placed at an angle, appropriate calibration may not be performed. option data does not exist.
As a result, accurate line-of-sight detection cannot be executed.
Although the camera has been described above as an example, the same situation exists in various information processing apparatuses that detect the line of sight.

Therefore, an object of the present disclosure is to be able to deal with various relative posture states using calibration data obtained in one calibration.

An information processing apparatus according to the present technology performs a line-of-sight detection process for detecting a line-of-sight direction based on an eye image captured by an eye-image capturing unit, and performs calibration data used for the line-of-sight detection process by: A line-of-sight detection calculation unit is provided for performing correction processing for correction based on roll angle information, which is information on a relative angle change between the eye and the eye image capturing unit.
In line-of-sight detection processing, calibration according to individual differences in the optical axis and the visual axis is required. Correction processing based on roll angle information is processing for further correcting the calibration data in accordance with the orientation of the user's eyes when the line of sight is detected.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit includes reference angle information used to calculate the roll angle information when performing calibration processing for acquiring calibration data used in the line-of-sight detection processing. and store the calibration data in association with the reference angle information.
For example, information that changes according to the relative posture relationship between the user's eyes and the information processing device is acquired during the calibration process, and stored as reference angle information in association with the calibration data.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit acquires current angle information, which is the same type of information as the reference angle information, during the line-of-sight detection processing. It is conceivable to calculate the roll angle information using the current angle information and perform the correction processing.
For example, as information that changes according to the relative posture relationship between the user's eyes and the information processing device, the same information as the reference angle information is acquired during the line-of-sight detection process and compared with the reference angle information to obtain the roll angle information.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit may calculate roll angle information from an iris code used in iris authentication processing.
There is iris authentication as a method of user personal authentication. The iris code used in this iris authentication is used to obtain roll angle information during line-of-sight detection processing.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit performs calibration processing for obtaining calibration data used in the line-of-sight detection processing at an opportunity to obtain an iris code, and converts the obtained iris code to the As the reference angle information used for calculating the roll angle information, it is conceivable to perform a process of storing the reference angle information in association with the calibration data.
The iris code used in the iris authentication process is information that changes according to the relative posture relationship between the user's eyes and the information processing device. This iris code is acquired when performing calibration, and stored as reference angle information in association with the calibration data.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit uses the iris code detected in the iris authentication process as current angle information, and uses the reference angle information and the current angle information to obtain the roll angle information. is calculated to perform the correction process.
When detecting the line of sight, the iris code obtained in the iris authentication process is used as the current angle information. This is compared with pre-stored reference angle information to obtain roll angle information, and the calibration data is corrected so that it can be used for line-of-sight detection processing.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit calculates the roll angle information based on the Hamming distance between the iris code as the reference angle information and the iris code as the current angle information. can be considered.
Since the iris code is information bit-shifted by a shift amount corresponding to the amount of change in the relative posture relationship, the change in angle can be obtained from the Hamming distance.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit detects the iris code detected in the iris authentication process and the iris code authenticated as belonging to the same person as the reference angle information. Alternatively, the iris code detected in the iris authentication process may be used as current angle information, and the correction process may be performed using the roll angle information calculated using the reference angle information and the current angle information.
Correction processing using the iris code is performed when it is determined that the person is a person whose iris code has been stored as reference angle information in the past.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit detects the iris code detected in the iris authentication process and the iris code authenticated as belonging to the same person as the reference angle information when the iris code is not stored as the reference angle information. Alternatively, the calibration process may be performed, and the iris code acquired in the iris authentication process may be stored as the reference angle information in association with the calibration data.
If the person is not a person whose iris code has been stored as reference angle information in the past, the roll angle information cannot be calculated correctly even if the reference angle information and the current angle information are compared, so correction processing is not performed.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit may calculate the roll angle information based on eyelid boundary information in the captured image of the eye.
By detecting features in the vicinity of the eyelids in the captured image of the eyes (eye image), roll angle information is obtained from changes in the feature points.
Note that the boundary information is information related to the boundary of the eyelids. For example, the boundary line of the eyelid, the feature points near the boundary line, the circumscribed triangle of the eyelid boundary, and the like.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit acquires the boundary information from the captured image of the eye when performing calibration processing for acquiring calibration data used in the line-of-sight detection processing. , the boundary information may be stored in association with the calibration data as reference angle information used for calculating the roll angle information.
Information about the shape of the eyelid boundary that appears in the captured image of the eye is information that changes according to the relative posture relationship between the user's eyes and the information processing device. This boundary information is obtained when performing calibration, and stored as reference angle information in association with the calibration data.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit acquires the boundary information from the captured image of the eye as current angle information, and uses the reference angle information and the current angle information to determine the roll angle. It is conceivable to calculate the angle information and perform the correction process.
In line-of-sight detection, boundary information is used as current angle information. This is compared with pre-stored reference angle information to obtain roll angle information, and the calibration data is corrected so that it can be used for line-of-sight detection processing.

In the information processing apparatus according to the present technology described above, it is conceivable that the line-of-sight detection calculation unit calculates the roll angle information based on the contact position information with respect to the touch panel.
For example, in the case where a touch panel is provided at the bottom of the viewfinder of the imaging device, roll angle information is obtained by detecting the contact position of the user's nose or the like on the touch panel when the user looks into the viewfinder.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit acquires the contact position information when performing calibration processing for acquiring calibration data used in the line-of-sight detection processing, and calculates the contact position information. It is conceivable to perform a process of storing the information in association with the calibration data as reference angle information used for calculating the roll angle information.
The contact position information on the touch panel is information that changes according to the relative posture relationship between the user's eyes and the information processing device. This contact position information is acquired when performing calibration, and stored as reference angle information in association with calibration data.

In the information processing apparatus according to the present technology described above, the line-of-sight detection calculation unit acquires the contact position information to use it as current angle information, and calculates the roll angle information using the reference angle information and the current angle information. It is conceivable that the correction process is performed as described above.
At the time of line-of-sight detection, contact position information is used as current angle information. This is compared with pre-stored reference angle information to obtain roll angle information, and the calibration data is corrected so that it can be used for line-of-sight detection processing.

The information processing device according to the present technology described above includes a detection unit that detects the posture of the device itself, and the line-of-sight detection calculation unit receives device posture information of the head-mounted device transmitted from the head-mounted device, It is conceivable to calculate the roll angle information based on the device orientation information detected by the detection unit.
For example, assuming a combination of an imaging device and a head-mounted device, the head-mounted device corresponds to the posture of the user. Therefore, it is possible to obtain roll angle information for correction processing by comparing apparatus attitude information.

It is conceivable that the information processing apparatus according to the present technology described above includes the eye image capturing unit.
That is, in an information processing apparatus integrally provided with an eye image capturing section, a line-of-sight detection process based on an eye image by the eye image capturing section and a correction process of calibration data are performed.

It is conceivable that the information processing apparatus according to the present technology described above includes an imaging unit that performs image capturing and the eye image capturing unit.
That is, it is an example configured as an imaging device that performs line-of-sight detection processing.

A line-of-sight detection method according to the present technology performs line-of-sight detection processing for detecting a line-of-sight direction based on an eye image captured by an eye image capturing unit, and performs calibration data used for the line-of-sight detection processing by: In the line-of-sight detection method, an information processing device executes a line-of-sight detection calculation for performing a correction process based on roll angle information, which is information about a relative angle change between an eye and the eye image capturing unit.
Thus, the information processing apparatus performs line-of-sight detection using the corrected calibration data.
A program according to the present technology is a program that causes an information processing apparatus to execute the line-of-sight detection calculation described above. This facilitates the realization of the above information processing apparatus.

1 is an explanatory diagram of an example of an information processing device according to an embodiment of the present technology; FIG. 1 is a perspective view of an imaging device according to an embodiment; FIG. 1 is a block diagram of an imaging device according to an embodiment; FIG. 1 is a block diagram of an information processing device according to an embodiment; FIG. FIG. 10 is an explanatory diagram of line-of-sight detection processing; FIG. 4 is an explanatory diagram of an optical axis and a visual axis; FIG. 10 is an explanatory diagram of calibration using a plurality of points; FIG. 4 is an explanatory diagram of generation and application of calibration data; 7 is a flow chart of an example of processing when recording calibration data according to the first embodiment; FIG. 10 is an explanatory diagram of comparing the line angles of the eyeballs of the same person using iris patterns; FIG. 10 is an explanatory diagram of Hamming distance comparison of iris codes according to the first embodiment; 7 is a flowchart of an example of correction processing according to the first embodiment; FIG. 11 is an explanatory diagram of detection of roll angle information from an eye image according to the second embodiment; FIG. 11 is a flow chart of an example of processing when recording calibration data according to the second embodiment; FIG. 10 is a flowchart of an example of correction processing according to the second embodiment; FIG. 11 is a flow chart of an example of processing when recording calibration data according to the third embodiment; FIG. FIG. 11 is a flow chart of an example of correction processing according to the third embodiment; FIG. It is explanatory drawing of the apparatus attitude|position of 4th Embodiment. FIG. 14 is a flow chart of an example of processing when recording calibration data according to the fourth embodiment; FIG. FIG. 16 is a flow chart of an example of correction processing according to the fourth embodiment; FIG. FIG. 5 is an explanatory diagram of an application example of roll angle information according to the embodiment;

Hereinafter, embodiments will be described in the following order.
<1. Applied device example>
<2. Configuration Example of Imaging Device>
<3. Configuration example of information processing device>
<4. Gaze detection and calibration>
<5. First Embodiment: Correction Processing Using Iris Authentication>
<6. Second Embodiment: Correction Processing Based on Eye Image>
<7. Third Embodiment: Correction Processing Using Touch Panel>
<8. Fourth Embodiment: Correction Processing Using Headset>
<9. Application of Roll Angle Information>
<10. Summary and Modifications>

<1. Applied device example>
First, a specific device example assumed as an information processing device of the present disclosure will be described.
An information processing apparatus according to the present disclosure is an apparatus capable of performing information processing, and specifically, an apparatus including a microprocessor or the like and capable of performing line-of-sight detection calculations.

A line-of-sight detection calculation is a calculation as a line-of-sight detection process for detecting a line-of-sight direction based on an eye image captured by an eye image pickup unit inside or outside the apparatus. Note that the “eye image” referred to in the present disclosure is an image of a person's eye.
In addition, in the case of the present embodiment, correction processing is also performed for correcting the calibration data used in the line-of-sight detection processing based on roll angle information, which is information on relative angle changes between the eyes and the eye image capturing unit. .
That is, a processor device such as a CPU or a DSP that performs at least line-of-sight detection processing and correction processing, or a device provided with such a processor device, is the information processing device according to the present disclosure.

FIG. 1 illustrates an imaging device 1, a terminal device 100, and a table top display 120 as specific device examples corresponding to information processing devices.

In the case of the imaging device 1, it is assumed that it has a function of detecting the line-of-sight direction of the user looking into the EVF (Electric Viewfinder) 5. FIG.
Alternatively, a configuration having a function of detecting the line-of-sight direction of the user viewing the rear monitor 4 in the imaging device 1 is also assumed.

As the terminal device 100, a tablet device, a smartphone, etc. are exemplified. For these terminal devices 100 as well, for example, a configuration example for detecting the line-of-sight direction of the user with respect to the screen is conceivable.

In the case of the table top display 120, a configuration example is assumed in which the line-of-sight direction of a person in the vicinity looking at the display on the top of the table is detected.

For example, as in these, it is assumed that various devices are equipped with a configuration for detecting the line-of-sight direction. conduct. In the case of the present embodiment, such a processor device further performs correction processing of the calibration data.

The equipment listed in FIG. 1 is only an example. The technology of the present disclosure can be applied to equipment that detects the line-of-sight direction. Devices corresponding to the information processing device of the present disclosure are extremely diverse, such as, for example, television receivers, game machines, personal computers, workstations, robots, monitoring devices, and sensor devices.

<2. Configuration Example of Imaging Device>
A configuration example of an imaging device 1 as an example of an information processing device according to the present disclosure will be described.
FIG. 2 is a perspective view of the imaging device 1 as seen from the rear side. For the sake of explanation, the subject side is the front side (front side), and the photographer side is the rear side (rear side).

The imaging device 1 includes a camera housing 2 and a lens barrel 3 that is detachable from the camera housing 2 and attached to the front surface portion 2a. It is an example that the lens barrel 3 is detachable as a so-called interchangeable lens, and the lens barrel may be a lens barrel that cannot be removed from the camera housing 2 .

A rear monitor 4 is arranged on the rear surface portion 2 b of the camera housing 2 . The rear monitor 4 displays live view images, reproduced images of recorded images, and the like.
The rear monitor 4 is configured by a display device such as a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display.

An EVF 5 is arranged on the upper surface portion 2 c of the camera housing 2 . The EVF 5 includes an EVF monitor 5a and a frame-shaped enclosing portion 5b that protrudes rearward so as to enclose an upper portion and left and right sides of the EVF monitor 5a.
The EVF monitor 5a is formed using an LCD, an organic EL display, or the like. An optical view finder (OVF) may be provided instead of the EVF monitor 5a.

Various operators 6 are provided on the rear surface portion 2b and the upper surface portion 2c. For example, it is a shutter button (release button), a reproduction menu activation button, an enter button, a cross key, a cancel button, a zoom key, a slide key, and the like. These manipulators 6 include various types of manipulators such as buttons, dials, pressable and rotatable composite manipulators. For example, shutter operation, menu operation, playback operation, mode selection/switching operation, focus operation, zoom operation, and parameter selection/setting such as shutter speed and F-number are enabled by operating elements 6 of various modes.

FIG. 3 shows the internal configuration of the imaging device 1. As shown in FIG.
The imaging apparatus 1 includes, for example, a lens system 11, an imaging element section 12, a camera signal processing section 13, a recording control section 14, a display section 15, a communication section 16, an operation section 17, a camera control section 18, a memory section 19, a driver section 22, and a , a sensor unit 23 and a line-of-sight detection device unit 41 .

The lens system 11 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, and the like. Light (incident light) from a subject is guided by the lens system 11 and condensed on the imaging element section 12 .

The imaging device unit 12 is configured by having an image sensor 12a (imaging device) such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type.
In the image sensor unit 12, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. are performed on an electrical signal obtained by photoelectrically converting light received by the image sensor 12a, and further A/D processing is performed. Performs (Analog/Digital) conversion processing. Then, the imaging signal as digital data is output to the camera signal processing section 13 and the camera control section 18 in the subsequent stage.

The camera signal processing unit 13 is configured as an image processing processor such as a DSP (Digital Signal Processor). The camera signal processing section 13 performs various signal processing on the digital signal (captured image signal) from the imaging element section 12 . For example, as a camera process, the camera signal processing unit 13 performs preprocessing, synchronization processing, YC generation processing, resolution conversion processing, and the like.

Then, the camera signal processing unit 13 performs, for example, compression encoding for recording and communication, formatting, generation and addition of metadata, etc. on the image data that has been subjected to the various types of processing described above, and converts the image data into image data for recording and communication. file is generated.
For example, an image file in a format such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), or GIF (Graphics Interchange Format) is generated as a still image file. It is also conceivable to generate an image file in the MP4 format, which is used for recording MPEG-4 compliant moving images and audio.
It is also conceivable to generate an image file as raw (RAW) image data.

The recording control unit 14 performs recording and reproduction on a recording medium such as a non-volatile memory. The recording control unit 14 performs a process of recording image files such as moving image data and still image data, and metadata including thumbnail images and the like on a recording medium, for example.
Various actual forms of the recording control unit 14 are conceivable. For example, the recording control unit 14 may be configured as a flash memory built in the imaging device 1 and its writing/reading circuit. Also, the recording control unit 14 may be configured by a card recording/reproducing unit that performs recording/reproducing access to a recording medium detachable from the imaging apparatus 1, such as a memory card (portable flash memory, etc.). Also, the recording control unit 14 may be implemented as an HDD (Hard Disk Drive) or the like as a form incorporated in the imaging device 1 .

The display unit 15 is a display unit that provides various displays to the photographer. For example, a liquid crystal panel (LCD: Liquid Crystal Display) or an organic EL ( display devices such as Electro-Luminescence) displays.
The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 18 .
For example, the display unit 15 displays a reproduced image of image data read from the recording medium by the recording control unit 14 .
The display unit 15 is supplied with the image data of the captured image whose resolution has been converted for display by the camera signal processing unit 13, and the display unit 15 responds to an instruction from the camera control unit 18 to display the image data of the captured image. may be displayed. As a result, a so-called through image (monitoring image of the subject), which is an image captured while confirming the composition or recording a moving image, is displayed.
Further, the display unit 15 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface) on the screen based on instructions from the camera control unit 18 .

The communication unit 16 performs wired or wireless data communication and network communication with external devices. For example, captured image data (still image files and moving image files) and metadata are transmitted and output to external information processing devices, display devices, recording devices, playback devices, and the like.
As a network communication unit, the communication unit 16 performs communication via various networks such as the Internet, a home network, and a LAN (Local Area Network), and can transmit and receive various data to and from servers, terminals, etc. on the network. can.
In addition, the communication unit 16 enables the imaging device 1 to communicate with, for example, a PC, a smartphone, a tablet terminal, headphones, earphones, a headset, or the like via Bluetooth (registered trademark), Wi-Fi communication, NFC, or the like. It may also be possible to perform mutual information communication by means of distance wireless communication or infrared communication.
Alternatively, the imaging device 1 and other equipment may be able to communicate with each other through wired connection communication.

The operation unit 17 collectively indicates an input device for a user to perform various operation inputs. Specifically, the operation unit 17 indicates various operators (keys, dials, touch panels, touch pads, etc.) provided on the housing of the imaging device 1 . For example, it is assumed that the touch panel is provided on the surface of the rear monitor 4 .
A user's operation is detected by the operation unit 17 , and a signal corresponding to the input operation is sent to the camera control unit 18 .

The line-of-sight detection device 41 is a device that captures an eye image for detecting the line of sight of the user. It is composed of an image capturing unit 51 (for example, an infrared camera) and the like.
For example, by arranging the line-of-sight detection device 41 having the eye image capturing unit 51 in the EVF 5 shown in FIG. Also, such a line-of-sight detection device unit 41 is arranged near the rear monitor 4 so that an eye image for detecting the line-of-sight direction of the user looking at the rear monitor 4 can be picked up. good too.

The camera control unit 18 is configured by a microcomputer (information processing device) having a CPU (Central Processing Unit).
The memory unit 19 stores information and the like that the camera control unit 18 uses for processing. As the illustrated memory unit 19, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like are comprehensively illustrated.
The memory section 19 may be a memory area built into a microcomputer chip as the camera control section 18, or may be configured by a separate memory chip.

The camera control unit 18 controls the entire imaging apparatus 1 by executing programs stored in the ROM of the memory unit 19, flash memory, or the like.
For example, the camera control unit 18 controls the shutter speed of the image sensor unit 12, instructs various signal processing in the camera signal processing unit 13, performs image capturing and recording operations in response to user operations, reproduces recorded image files, performs lens It controls the operations of necessary units for operations of the lens system 11 such as zoom, focus, and aperture adjustment in the lens barrel, user interface operations, and the like.

The camera control unit 18 has a function as a line-of-sight detection calculation unit 40 by an application program.
The line-of-sight detection calculation unit 40 analyzes the eye image obtained from the line-of-sight detection device unit 41 and performs line-of-sight detection processing for detecting the line-of-sight direction of the user. Also, in order to improve the accuracy of this line-of-sight detection, calibration processing is also performed. Furthermore, in the case of the present embodiment, the calibration data used for the line-of-sight detection process is corrected based on the roll angle information, which is the information on the relative angle change between the eye and the eye image capturing section in the line-of-sight detection device section 41. I am trying to process it.
Details of these processes will be described later.

The camera control unit 18 can also perform various controls based on the user's line-of-sight direction detected by the function of the line-of-sight detection calculation unit 40 . For example, depending on the line-of-sight direction, it is possible to set the focus area so that the subject in the line-of-sight direction is in just focus, or to perform aperture adjustment control according to the brightness of the subject in the line-of-sight direction.

The camera control unit 18 may perform so-called AI (artificial intelligence) processing for line-of-sight detection processing, correction processing, and other processing.

The RAM in the memory unit 19 is used as a work area for the CPU of the camera control unit 18 to perform various data processing, and is used for temporary storage of data, programs, and the like.
The ROM and flash memory (nonvolatile memory) in the memory unit 19 store an OS (Operating System) for the CPU to control each unit, content files such as image files, application programs for various operations, and firmware. , and used to store various setting information.
Various setting information includes communication setting information, exposure setting, shutter speed setting, mode setting as setting information related to imaging operation, white balance setting, color setting, image effect setting as setting information related to image processing, etc. There is

The memory unit 19 also stores programs for line-of-sight detection processing, calibration processing, and correction processing, which will be described later, and data used for these processing. For example, the memory unit 19 can also function as a database regarding calibration data for line-of-sight detection processing. For example, authentication data for iris authentication, which will be described later, code information obtained by encoding an iris pattern, pattern data used for object recognition by semantic segmentation, and the like are also stored.

The driver unit 22 includes, for example, a motor driver for the zoom lens drive motor, a motor driver for the focus lens drive motor, a motor driver for the motor of the aperture mechanism, and the like.
These motor drivers apply drive currents to the corresponding drivers in accordance with instructions from the camera control unit 18 to move the focus lens and zoom lens, open and close the diaphragm blades of the diaphragm mechanism, and the like.

The sensor unit 23 comprehensively indicates various sensors mounted on the imaging device.
As the sensor section 23, for example, an IMU (inertial measurement unit) may be mounted. The IMU can detect angular velocity with, for example, three-axis angular velocity (gyro) sensors of pitch, yaw, and roll, and acceleration with an acceleration sensor. This makes it possible to detect the orientation of the imaging device 1 with respect to the direction of gravity.
As the sensor unit 23, for example, a position information sensor, an illuminance sensor, a range sensor, etc. may be mounted.

For example, in such an imaging device 1, the line-of-sight detection device unit 41 is incorporated in the EVF 5 or near the rear monitor 4, and the line-of-sight direction of the user looking through the EVF 5 or looking at the rear monitor 4 can be detected. can. In this case, the camera control unit 18 (line-of-sight detection calculation unit 40) performs line-of-sight detection processing, calibration processing, and calibration data correction processing based on the eye image captured by the line-of-sight detection device unit 41. FIG.

<3. Configuration example of information processing device>
Next, FIG. 4 describes a configuration example of the terminal device 100 as a tablet device or a smartphone illustrated in FIG. 1 as an example of the information processing device of the present disclosure.

The CPU 71 of the terminal device 100 executes various programs according to a program stored in a ROM 72 or a non-volatile memory unit 74 such as an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or a program loaded from the storage unit 79 to the RAM 73. Execute the process. The RAM 73 also appropriately stores data necessary for the CPU 71 to execute various processes.
The CPU 71 , ROM 72 , RAM 73 and nonvolatile memory section 74 are interconnected via a bus 83 . An input/output interface 75 is also connected to this bus 83 .

Since the terminal device 100 is assumed to perform image processing and AI (artificial intelligence) processing, instead of the CPU 71 or together with the CPU 71, GPU (Graphics Processing Unit), GPGPU (General-purpose computing on graphics processing units) , an AI-dedicated processor, or the like may be provided.

The input/output interface 75 is connected to an input section 76 including operators and operating devices. For example, as the input unit 76, various operators and operation devices such as a keyboard, mouse, key, dial, touch panel, touch pad, remote controller, etc. are assumed.
A user's operation is detected by the input unit 76 , and a signal corresponding to the input operation is interpreted by the CPU 71 .
A microphone is also envisioned as input 76 . A voice uttered by the user can also be input as operation information.
Various sensing devices such as an image sensor (imaging unit), an acceleration sensor, an angular velocity sensor, a vibration sensor, an air pressure sensor, a temperature sensor, and an illuminance sensor are also assumed as the input unit.

The input/output interface 75 is connected integrally or separately with a display unit 77 such as an LCD or an organic EL panel, and an audio output unit 78 such as a speaker.
The display unit 77 is a display unit that performs various displays, and is configured by, for example, a display device provided in the housing of the terminal device 100, a separate display device connected to the terminal device 100, or the like.
The display unit 77 displays images for various types of image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71 . Further, the display unit 77 displays various operation menus, icons, messages, etc., ie, as a GUI (Graphical User Interface), based on instructions from the CPU 71 .

The input/output interface 75 may be connected to a storage unit 79 made up of a hard disk, a solid-state memory, etc., and a communication unit 80 made up of a modem or the like.
The storage unit 79 stores various programs, data files, and the like. A database may also be constructed.
The communication unit 80 performs communication processing via a transmission line such as the Internet, and communication by wired/wireless communication with various devices, bus communication, and the like.

A drive 81 is also connected to the input/output interface 75 as required, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately loaded.
Data files such as image files and various computer programs can be read from the removable recording medium 82 by the drive 81 . The read data file is stored in the storage unit 79 , and the image and sound contained in the data file are output by the display unit 77 and the sound output unit 78 . Computer programs and the like read from the removable recording medium 82 are installed in the storage unit 79 as required.

In this terminal device 100, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via the removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

The line-of-sight detection device section 41 is connected to the input/output interface 75 . 3, the line-of-sight detection device 41 is a device that captures an eye image for detecting the line of sight of the user. LED), an eye image capturing unit 51 (for example, an infrared camera) that captures an image of the user's eyes, and the like.

In this terminal device 100, the CPU 71 functions as the line-of-sight detection calculation unit 40 by means of an application program.
The line-of-sight detection calculation unit 40 analyzes the eye image obtained from the line-of-sight detection device unit 41 and performs line-of-sight detection processing for detecting the line-of-sight direction of the user, as described for the imaging device 1 in FIG. 3 . In addition, calibration processing and correction processing for correcting calibration data based on roll angle information, which is information on relative angle changes between the eyes and the eye image capturing unit in the sight line detection device unit 41, are performed.

The CPU 71 can also perform various controls based on the user's line-of-sight direction detected by the function of the line-of-sight detection calculation unit 40 . For example, it is possible to move a pointer on the display unit 77, switch images, output sound, and perform communication according to the line-of-sight direction.

The non-volatile memory unit 74, ROM 72, and storage unit 79 can also function as a database regarding calibration data for line-of-sight detection processing. For example, authentication data for iris authentication, which will be described later, code information obtained by encoding an iris pattern, pattern data used for object recognition by semantic segmentation, and the like are also stored.

Although the configuration example of the terminal device 100 has been described here, the table top display 120 in FIG. 1, game machines, personal computers, workstations, etc. have almost the same configuration. In addition to the configuration shown in FIG. 3, the television receiver has a tuner function, the monitoring device and the sensor device have an imaging unit and a detection unit necessary for monitoring and sensing, and the robot has the necessary functions for the operation of the robot. It is sufficient if the function is provided.

<4. Gaze detection and calibration >
The line-of-sight detection and calibration performed by the imaging device 1, the terminal device 100, and the like will be described.

In line-of-sight detection, a "reference point" and a "moving part (moving point)" of the eye are found, and the line of sight is detected from the position of the moving point with respect to the reference point.
Although the corneal reflection method and the image processing method are known as specific methods, the pupil corneal reflection method will be explained here.

In the corneal reflection method, the eye is photographed while the cornea is irradiated with a point light source, and the center of corneal curvature obtained from this corneal reflection image (Purkinje image) is used as a reference point and the moving point is measured as the pupil.
5A and 5B show a pupil 60, a cornea 61, an iris 62, an inner corner 63, and an outer corner 64 as structures of the eye.

FIG. 5C schematically shows an infrared irradiation section 50 and an eye image pickup section 51 as an example of the line-of-sight detection device section 41 shown in FIGS. The infrared irradiation unit 50 is composed of, for example, an infrared LED (light emitting diode). The eye image pickup unit 51 is configured by, for example, an infrared camera.
In this case, the infrared irradiation unit 50 irradiates the cornea 61 with a point light source. The eye image capturing unit 51 captures an image of the eye in a state where the cornea 61 is irradiated with the point light source, and outputs the captured eye image.
This eye image is supplied to the line-of-sight detection calculation unit 40, that is, the camera control unit 18 in FIG. 3 and the CPU 71 in FIG.
The line-of-sight detection calculation unit 40 detects the pupil 60 and the corneal reflection image 65 from the eye images shown in FIGS. 5D and 5E as line-of-sight detection processing. The line-of-sight direction is calculated from the positions of the moving point and the reference point. That is, the viewpoint (coordinates) is measured by applying the imaging information of the corneal reflection image 65 to the eyeball model.

In order to perform such line-of-sight detection processing, calibration corresponding to individual differences is required. FIG. 6 shows the optical axis AX1 and visual axis AX2 of the human eye.
The optical axis AX1 is the corneal normal passing through the center of the pupil 60 . This optical axis AX1 can be estimated by the eyeball 3D model.
A visual axis AX2 is an axis that connects the nodal point (central posterior surface of the lens 66) and the fovea 70. FIG. What a person actually sees is this visual axis AX2.

The optical axis AX1 and the visual axis AX2 are tilted by about 5 degrees. There are individual differences in this inclination, and it is generally about 4 to 8 degrees.
In the line-of-sight detection process, the direction of the optical axis AX1 can be calculated by observing the pupil 60, so by correcting it to the direction of the visual axis AX2, the line-of-sight direction of the person (actual viewing direction) can be accurately detected. become.
Therefore, calibration data for correcting the deviation between the optical axis AX1 and the visual axis AX2 for each individual is collected by performing calibration processing. As a result, correction can be performed using the calibration data in subsequent actual line-of-sight detection.

An example of calibration processing will be described.
First, the optical axis AX1 when viewing a certain "point" in the field of view is estimated.
Then, the difference between the vector from the center of curvature of the cornea 61 to that "point" and the vector of the optical axis AX1 is measured. Using the difference, the visual axis AX2 at that time is estimated from the optical axis AX1 when an arbitrary point is viewed.
A plurality of points (for example, 5 to 9 points) in the normal visual field are used because the calibration parameters differ depending on the orientation of the eyeball.

FIG. 7 shows an example of calibration processing using multiple points.
A gaze point marker 67 is displayed on the screen viewed by the user. The user is made to gaze at this marker 67 . Observe the line of sight at that time. The observable line-of-sight direction is the direction of the optical axis AX1. Also, the direction toward the center of the marker 67 is the direction of the visual axis AX2.
After obtaining the data at the point of the marker 67, ie, the vector of the optical axis AX1 and the visual axis AX2 at that time, the marker 67 is moved to the next point.
As shown in the figure, data is collected while moving the marker 67 to, for example, the center, upper left corner, upper right corner, lower left corner, and lower right corner of the screen.

By performing such measurements, vectors G and T shown in FIG. 8A are obtained for each point. Vector G is the detected viewing direction (that is, optical axis AX1), and vector T is a vector (that is, visual axis AX2) connecting the center of the user's eye and the calibration point (the point indicated by marker 67).

A rotational transformation Q from this vector G to vector T is obtained. The rotation transform Q is a quaternion. The rotational transformation Q for each vector G is stored in the database 52 as calibration data for the person who made the measurement. This rotational transformation Q becomes a correction amount for each individual.
The database 52 may be constructed, for example, in the memory section 19 of FIG. 3 or the non-volatile memory section 74 or storage section 79 of FIG.

When applying the calibration data obtained by the above calibration processing when actually performing the line-of-sight detection processing, for example, as shown in FIG. Extract P3. The three neighboring points P1, P2, and P3 are three points close to the current gaze point P0 among the points for which the user is caused to gaze and the rotational transformation Q is obtained during the calibration process.
Then, each correction amount is weighted by the reciprocal of the distance from the gaze point P0 to the neighboring three points P1, P2, and P3, and added. The correction amount is applied to the current gazing point P0 to obtain the post-correction gazing point P0A.
This makes it possible to accurately detect the direction in which the user is actually looking.

However, such correction is effective when the user views the image at the same relative angle as when the calibration process was performed.
For example, when the calibration process is performed while the user is looking into the EVF 5, the relative posture relationship (angular relationship) between the user's eyes and the eye image capturing unit 51 of the line-of-sight detection device unit 41 Calibration data are obtained.
If the user holds the imaging apparatus 1 horizontally and performs the calibration process in a normal standing posture with the EVF 5 removed, calibration data of the relative posture relationship can be obtained. After that, when the user holds the imaging device 1 vertically or looks into the EVF 5 from an oblique angle, the posture relationship differs from that at the time of the calibration process. not a direction.

Then, in order to correspond to various posture relationships, calibration processing is performed in various posture relationships, and calibration data for each posture relationship is obtained. Accordingly, calibration data must be selected and applied to the correction process.
This makes the calibration process troublesome for the user, and storing a plurality of pieces of calibration data is not efficient for the imaging device 1 and the terminal device 100 . Furthermore, it is not realistic to perform calibration processing so as to cover all posture-related states, and in fact, it is assumed that there may be cases where accurate line-of-sight detection cannot be performed.

Therefore, in the present embodiment, the difference between the attitude relationship when the calibration process is performed and the attitude relationship when the line-of-sight detection is actually performed can be represented by the amount of rotation in the roll direction. Correction processing is performed to correct the calibration data according to the information. The roll angle can be said to be information about the relative angle between the user's eyes and the eye image capturing unit 51, with reference to the time of the calibration process (angle change amount).

When detecting the line of sight, the corrected calibration data is used to correct from the optical axis AX1 to the visual axis AX2. To detect a line of sight with high accuracy only with obtained calibration data.

<5. First Embodiment: Correction Processing Using Iris Authentication>
As a first embodiment, calibration data correction processing using iris authentication will be described.
When performing the processing of the first embodiment, the camera control unit 18 or the CPU 71 also has a function of performing iris authentication processing.

Iris authentication is known as a technology that uses eyeballs. This technique encodes an iris pattern into an iris code, and compares the iris code to perform personal authentication. When the Hamming distance is obtained by shifting the iris code when performing this comparison, the shift amount that minimizes it is proportional to the roll angle of the eyeball. Therefore, correction is performed by rotating the calibration data by the difference (roll angle) between the angle at the time of calibration processing and the current angle.

The processing of the line-of-sight detection calculation unit 40 for performing such processing will be described. The processing of the line-of-sight detection calculation unit 40 described below is processing performed by an information processing device such as the camera control unit 18 in FIG. 3 and the CPU 71 in FIG.

FIG. 9 shows calibration processing of the line-of-sight detection calculation unit 40 .
In step S101, the line-of-sight detection calculation unit 40 acquires an image captured by the eye image capturing unit 51 of the line-of-sight detection device unit 41, that is, an eye image of the user.

In step S102, the camera control unit 18 or the CPU 71 equipped with the line-of-sight detection calculation unit 40 performs encoding for iris authentication by the iris authentication function also provided. That is, an iris code is generated by encoding the iris pattern observed from the user's eye image.

The iris code will be described with reference to FIGS. 10 and 11. FIG.
For example, assume that an eye image (image IM1) as shown in FIG. 10A is captured. The donut-shaped portion of the iris 62 is extracted by polar coordinate transformation, and as shown in FIGS. 11A and 11B, 256 divisions in the angular direction and 4 lines in the radial direction are obtained.
Then, it is converted into a 2048-bit code as shown in FIG. 11C by a Gapole wavelet filter. For example, one pixel has 2 bits of filter response real part and imaginary part, which is 2048 bits as 256 degrees×2 bits×4 lines.
This is the iris code obtained from the eye image as image IM1.

In step S103 of FIG. 9, the line-of-sight detection calculation unit 40 collects calibration data. For example, as described with reference to FIGS. 7 and 8, the vector G and the vector T are determined while the marker 67 is gazed at, and the rotational transformation Q is determined with the position of the marker 67 as the gaze point. While changing the position of the marker 67, this is performed for a plurality of fixation points.
As a result, calibration data is obtained in the posture relation in which the eye image as the image IM1 was obtained.

In step S104, the line-of-sight detection calculation unit 40 stores the calibration data and the iris code as a pair in the database 52 in association with each other.
This means that an iris code for a certain individual is stored for personal authentication, and that iris code is used as reference angle information of the roll angle for correction of calibration data. The reference angle information is, for example, information indicating a roll angle=0 degree state. Therefore, the calibration data stored in a pair with the iris code in this case is positioned as the calibration data when the roll angle is set to 0 degrees.
It should be noted that the iris code paired with the calibration data and stored in step S104 is used at least as information used for line-of-sight detection. Separately from this storage processing, the camera control unit 18 or the CPU 71 may perform registration processing separately as information for personal authentication for the generated iris code when the iris code is generated in step S102. Alternatively, the iris code stored for the line-of-sight detection process in step S104 may also be used as information for later personal authentication.
In the above description, the iris code registration for personal authentication is performed simultaneously with the calibration process shown in FIG. may be held on another occasion.

Next, an example of processing when performing line-of-sight detection processing for actually detecting the line-of-sight direction will be described with reference to FIG.
In step S201, the line-of-sight detection calculation unit 40 acquires an image captured by the eye image capturing unit 51 of the line-of-sight detection device unit 41, that is, an eye image of the user.
In this case, as in image IM1 in FIG. 10A, an eye image in the same posture relationship as in the calibration process may be obtained. In some cases, an eye image with a pose relationship can be obtained. In this case, the image IM2 is an image with a convolution angle θ compared to the image IM1.

In step S202, the camera control unit 18 or the CPU 71 having the line-of-sight detection calculation unit 40 performs iris authentication using the iris authentication function also provided.
In this case, the camera control unit 18 or CPU 71 first generates an iris code from the eye image in the same manner as described above. If an eye image exactly the same as the image IM1 with zero rotation angle is obtained for the same person who obtained the iris code in the past, the same iris code as the iris code in FIG. 11C is obtained. However, although there may be some bit differences depending on the imaging state of the eye image and the determination of the iris pattern, the codes are generally the same.
10B for the same person, the iris code is bit-shifted as compared to the iris code in FIG. 11C, as in FIG. 11D. you get the code.

In this case, while bit-shifting the obtained iris code, the Hamming distance (that is, the amount of difference in bits) from the iris code stored for iris authentication in the past is calculated for each bit-shifted state. Then, when the obtained iris code is in a certain shift state, the Hamming distance is minimized.
The iris code stored for iris authentication in the past is the iris code of the user who is registered as a person whose personal authentication is OK. The iris code may be stored during the calibration process of FIG. 9, or may be stored during a separate registration process.
When the minimum value of the Hamming distance is found as described above, the Hamming distance is compared with a predetermined threshold value for authentication.
If the minimum value of the Hamming distance is equal to or less than a predetermined value, it can be determined that the current iris code belongs to the same person as the iris code stored for iris authentication in the past. This is because the smaller the Hamming distance, the higher the similarity of the iris code.
On the other hand, if the minimum value of the Hamming distance is not equal to or less than the predetermined value, it can be determined that the user has not stored an iris code as a person who should be authenticated in the past.
The camera control unit 18 or the CPU 71 can thus perform personal authentication using the iris authentication method.

Subsequently, from step S203 onwards, the camera control unit 18 or the CPU 71 continues processing using the function of the line-of-sight detection calculation unit 40. In step S203, the line-of-sight detection calculation unit 40 determines whether or not the iris code acquired this time and the iris code of the same person are stored in the database 52 in a pair with the calibration data.

If the current iris code is found in the database 52 as having been paired with the calibration data in the process of step S104 of FIG. The roll angle is calculated based on the Hamming distance between the iris code detected this time and the iris code stored in the database 52 .
In this case, the iris code detected this time is positioned as the current angle information and compared with the iris code as the reference angle information stored in step S104 of FIG.
For example, if the iris code of the image IM2 is obtained this time, the iris code detected this time as current angle information is bit-shifted from the stored iris code by the amount corresponding to the rotation angle θ. . Therefore, when the Hamming distance is obtained while bit-shifting during iris authentication in step S202, the bit shift amount when the Hamming distance becomes the minimum corresponds to the rotation angle θ. That is, the roll angle can be calculated from the bit shift amount up to the minimum Hamming distance.

In step S206, the line-of-sight detection calculation unit 40 calculates a calibration vector at the current gaze point from the stored calibration data.
Then, in step S207, the calibration vector is rotated by the roll angle to obtain the corrected calibration vector.
This means that the calibration data has been corrected according to the current relative attitude relationship. Therefore, after that, using the corrected calibration data, the line-of-sight detection process can be performed as described with reference to FIG. 8B.

Note that if the iris code has not been stored as a pair with the calibration data in the past, the process of FIG. 12 proceeds from step S204 to step S210.
The line-of-sight detection calculation unit 40 collects calibration data in step S210. Then, in step S211, the line-of-sight detection calculation unit 40 stores the calibration data and the iris code in the database 52 in a state in which they are associated as a pair.
These steps S210 and S211 are the same processing as steps S103 and S104 in FIG. As a result, it becomes possible to correct the calibration data for the user who generated the iris code in this process.

It should be noted that such storage of the iris code may be executed independently of personal authentication by iris authentication.
In iris authentication, for example, the iris code of a user whose authentication is acceptable is registered, and if the iris code obtained during personal authentication is determined to be of the same person as the registered iris code, the authentication is accepted. be.
On the other hand, the iris code stored in pairs with the calibration data is stored at least in the sense of reference angle information for correcting the calibration data. Therefore, regardless of whether the user is authenticated OK or not, when performing the calibration processing in step S103 of FIG. 9 or step S210 of FIG. They may be stored in pairs.
However, it is conceivable to prevent the processing of steps S210 and S211 from being performed for users who have not been authenticated.

As shown in FIG. 12, there are various possible timings for iris authentication, roll angle detection, and calibration data correction. It may be executed irregularly by some trigger, or may be executed periodically. For example, in the case of a configuration in which the EVF 5 of the imaging device 1 detects the line of sight, it is conceivable to execute the processing in FIG.

In addition, the movement of the imaging device 1 or the terminal device 100 is acquired by an acceleration sensor, a gyro sensor, or the like, and if there is a large movement, the inclination may have changed. good too. In such a case, roll angle detection and calibration data correction may be performed intermittently only when movement such as rotation stops.

<6. Second Embodiment: Correction Processing Based on Eye Image>
As a second embodiment, an example of detecting the roll angle based on the eye image and correcting the calibration data will be described.
This process is particularly applicable to devices that do not perform iris authentication. Specifically, the eyelid area is detected by semantic segmentation of the eyeball image, and the roll angle relative to the eyelid area shape at the time of calibration is obtained.
When performing the processing of the second embodiment, the line-of-sight detection calculation unit 40 also has a function of performing object recognition processing from an image by semantic segmentation.

Assume that FIG. 13A is an eye image captured by the eye image capturing unit 51 . FIG. 13B shows the eyelid region 68 of this eye image with diagonal lines.

14 and 15 show processing examples of the line-of-sight detection calculation unit 40. FIG.
FIG. 14 is an example of calibration processing.
In step S<b>120 , the line-of-sight detection calculation unit 40 acquires an eye image from the eye image capturing unit 51 .
In step S121, the line-of-sight detection calculation unit 40 determines the eyelid region 68 in the eye image by semantic segmentation processing, and acquires the eyelid boundary information.

Boundary information is information related to the boundary of the eyelids. For example, information on the boundary line 69 can be used as boundary information. Alternatively, feature points near the boundary line 69 may be extracted and used as boundary information. Further, a circumscribing triangle 86 with respect to the boundary of the eyelid may be detected as indicated by the dashed line in FIG. 13B, and the information of the circumscribing triangle 86 may be used as the boundary information.

In step S122 in FIG. 14, the line-of-sight detection calculation unit 40 performs calibration processing and collects calibration data in the same manner as in step S103 in FIG. 9 described above.
In step S<b>123 , the line-of-sight detection calculation unit 40 pairs the calibration data and the boundary information and stores them in the database 52 .
That is, in this case, boundary information is stored as reference angle information for setting the calibration data to be stored to the roll angle=0 state.

FIG. 15 shows an example of calibration data correction processing.
In step S<b>220 , the line-of-sight detection calculation unit 40 acquires an eye image from the eye image capturing unit 51 .
In step S221, the line-of-sight detection calculation unit 40 determines the eyelid region 68 in the eye image by semantic segmentation processing, and acquires the eyelid boundary information.

In step S222, the line-of-sight detection calculation unit 40 compares the boundary information acquired in step S221 with the boundary information stored in the database 52 to obtain the roll angle.
The boundary information detected this time is used as the current angle information, and is compared with the boundary information as the reference angle information stored in step S123 of FIG.
By comparing the boundary information stored in this manner with the current boundary information, it is possible to detect the difference in the position state according to the rotation angle θ of the eye, and thus the roll angle can be calculated.

After calculating the roll angle, the line-of-sight detection calculation unit 40 calculates a calibration vector at the current gaze point from the stored calibration data in step S223 of FIG. Then, in step S224, the calibration vector is rotated by the roll angle to obtain the corrected calibration vector.
As a result, the calibration data is corrected in accordance with the current relative posture relationship, and thereafter the line-of-sight detection process can be performed using the corrected calibration data.

<7. Third Embodiment: Correction Processing Using Touch Panel>
As a third embodiment, an example of detecting a roll angle using a touch panel and correcting calibration data will be described.

Assume that the imaging device 1 has a touch panel provided on the surface of the rear monitor 4 . When the user looks into the EVF 5, the user's nose may touch or come close to the touch panel.
If there is a response on the right side of the touch panel, you know you are looking with your left eye.
If there is a response on the left side of the touch panel, you know that you are looking with your right eye.
Further, since the deviation of the detection position on the touch panel shows the change in the roll angle, the roll angle can be estimated therefrom.

16 and 17 show processing examples of the line-of-sight detection calculation unit 40. FIG.
FIG. 16 is an example of calibration processing.
In step S140, the line-of-sight detection calculation unit 40 performs calibration processing and collects calibration data in the same manner as in step S103 of FIG. 9 described above.

In step S<b>141 , the line-of-sight detection calculation unit 40 acquires contact position information from information on the touch panel of the operation unit 17 . For example, the response of the touch panel while the user is looking into the EVF 5 for the calibration process is accumulated to generate contact position information on the touch panel with the nose during the calibration process.

In step S<b>142 , the line-of-sight detection calculation unit 40 pairs the calibration data and the contact position information and stores them in the database 52 .
That is, in this case, the contact position information is stored as the reference angle information for setting the calibration data to be stored so that the roll angle is zero.

FIG. 17 shows an example of calibration data correction processing.
In step S240, the line-of-sight detection calculation unit 40 acquires contact position information on the touch panel.
The contact position information detected this time becomes the current angle information to be compared with the contact position information as the reference angle information stored in step S142 of FIG.
Therefore, in step S241, the line-of-sight detection calculation unit 40 compares the contact position information acquired in step S240 with the contact position information stored in the database 52 to obtain the roll angle. The difference in contact position on the touch panel corresponds to the roll angle centering on the position of the EVF 5, for example.

After calculating the roll angle, the line-of-sight detection calculation unit 40 calculates a calibration vector at the current gaze point from the stored calibration data in step S242. Then, in step S243, the calibration vector is rotated by the roll angle to obtain the corrected calibration vector.
As a result, the calibration data is corrected in accordance with the current relative posture relationship, and thereafter the line-of-sight detection process can be performed using the corrected calibration data.

<8. Fourth Embodiment: Correction Processing Using Headset>
As a fourth embodiment, an example using an imaging device 1 and a headset 130 as shown in FIG. 18 will be described. FIG. 18 shows the yaw axis Yax, the pitch axis Pax, and the roll axis Rax.

It is assumed that the user uses the imaging device 1 while wearing the headset 130 on the head. The sensor unit 23 of the imaging device 1 has a function of detecting the orientation of the device itself by using an acceleration sensor, an angular velocity sensor, or the like. For example, it detects postures in three axial directions of yaw, pitch, and roll. Similarly, the headset 130 can also detect the posture of the device itself in three axial directions of yaw, pitch, and roll using an acceleration sensor, an angular velocity sensor, and the like.
It is assumed that the imaging apparatus 1 receives the posture information of the headset 130 through the communication unit 16, and the camera control unit 18 can detect the posture state of the headset 130 in real time.

In this case, the pose of headset 130 can be assumed to be the pose of the user's head. Therefore, the line-of-sight detection calculation unit 40 of the image pickup device 1 detects the relative relationship between the orientation of the image pickup device 1 itself and the orientation of the headset 130 and the relative relationship between the user's eye and the eye image pickup unit 51 for the EVF 5 and the rear monitor 4 . posture relationship (tilt amount) can be detected. That is, the current roll angle can be determined.
For example, the roll angle of the eye-to-eye image pickup unit 51 is the difference in roll angle when rotation correction is applied so that the yaw angles in the depth direction of the imaging device 1 and the depth direction of the headset 130 match.

19 and 20 show processing examples of the line-of-sight detection calculation unit 40. FIG.
FIG. 19 is an example of calibration processing.
In step S150, the line-of-sight detection calculation unit 40 performs calibration processing and collects calibration data in the same manner as in step S103 of FIG. 9 described above.

In step S151, the line-of-sight detection calculation unit 40 acquires device orientation information of the imaging device 1 itself.
In step S<b>152 , the line-of-sight detection calculation unit 40 acquires device orientation information of the headset 130 .

In step S<b>153 , the line-of-sight detection calculation unit 40 associates the device orientation information of the imaging device 1 , the device orientation information of the headset 130 , and the calibration data, and stores them in the database 52 .
That is, in this case, the calibration data to be stored includes device orientation information of the imaging device 1 and device orientation information of the headset 130 as reference angle information for setting the roll angle to zero. Note that the relative angle obtained from the device orientation information of the imaging device 1 and the device orientation information of the headset 130 at this time may be used as the reference angle information.

FIG. 20 shows an example of calibration data correction processing.
In step S<b>250 , the line-of-sight detection calculation unit 40 acquires current device orientation information of the imaging device 1 .
In step S<b>251 , the line-of-sight detection calculation unit 40 acquires current device orientation information of the headset 130 .
The device orientation information of the imaging device 1 and the device orientation information of the headset 130 acquired this time become current angle information that is compared with both device orientation information as the reference angle information stored in step S157 of FIG.

In step S252, the line-of-sight detection calculation unit 40 calculates the roll angle based on the device posture information. For example, if the database 52 stores device posture information of the imaging device 1 and device posture information of the headset 130, a relative angle is obtained as the relative posture difference. Also, the relative angle is obtained for each device orientation information acquired in steps S250 and S251.
The difference between the stored relative angle of both apparatus attitude information and the current relative angle of both attitude information is the current roll angle.

After calculating the roll angle, the line-of-sight detection calculation unit 40 calculates a calibration vector at the current gaze point from the stored calibration data in step S253. Then, in step S254, the calibration vector is rotated by the roll angle to obtain the corrected calibration vector.
As a result, the calibration data is corrected in accordance with the current relative posture relationship, and thereafter the line-of-sight detection process can be performed using the corrected calibration data.

<9. Application of Roll Angle Information>
Next, an application example of roll angle information will be described.
As in each of the above embodiments, when the roll angle is obtained for correcting the calibration data, it is conceivable to perform processing using the roll angle.

Here, an example of application to screen display as a UI (user interface) will be described.
First, it is conceivable to rotate the UI screen according to the roll angle.
Automatic tilt correction can be performed for monocular devices such as laser rangefinders.

Also, in devices such as head-mounted displays in AR (Augmented Reality) / VR (Virtual Reality), when the tilt of the device with respect to the eyes is detected as the roll angle, the image can be corrected based on the roll angle.

When the user wears the head-mounted display, it is difficult to wear it so that it is strictly horizontal. There is also mounting misalignment, and it may be slightly tilted depending on the user's head shape. However, even if there is a slight horizontal shift, there is a function in the human brain to correct the shift of the images projected on both eyes. For this reason, users usually do not mind a slight deviation. However, there is also a research report that fatigue accumulates by continuing such brain correction.
Therefore, when the roll angle is detected, the inclination of the image is corrected based on the roll angle even if the deviation is slight. As a result, it is possible to provide a device that does not make the user feel tired easily.

Alternatively, even if the image is not corrected, if there is a roll angle with respect to the horizontal, processing may be performed to notify the user that it is recommended to correct the inclination of the wearing state.

As an example of application to the UI, use in arranging icons on the screen is also conceivable.
Correction for eliminating the difference between the optical axis AX1 and the visual axis AX2 in the line-of-sight detection process uses three neighboring points P1, P2, and P3 with respect to the current gaze point P0, as described with reference to FIG. 8B.
Therefore, it is possible to improve the accuracy of the points gazed at during the calibration process performed in advance, that is, the areas included in the plurality of points on which the markers 67 are displayed as shown in FIG.

FIG. 21A assumes, for example, the screen of the EVF 5 or the rear monitor 4, and shows a hatched area 90 in the screen as an area with a high calibration effect.
For example, the four corners of this area 90 correspond to the four corners showing the markers 67 as shown in FIG. Within this area 90, three points P1, P2, and P3 near the current gaze point P0 can be obtained as shown in FIG. 8B, so that the calibration effect is enhanced. However, when the current gaze point P0 is out of the area 90, the accuracy is lowered because three neighboring points cannot be obtained.

Based on such circumstances, the rotation of the screen is considered.
If the screen is rotated vertically and horizontally when the area 90 with high calibration accuracy is not square, the area outside the area 90 becomes relatively wide in the vertical direction, as shown in FIG. 21B.
Therefore, for example, as shown in FIG. 21C, for a UI that uses the entire screen based on when the image capturing apparatus 1 is held horizontally, when the screen is rotated when held vertically, the icons used for the UI, etc., are changed as shown in FIG. 21D. , so as to be within the area 90 . The number of icons to be displayed and the contents may be changed without changing the scale.

By doing so, it is possible to always arrange the icons and the like in areas where the calibration effect is high regardless of the rotation of the screen.
For example, in the case of enabling operation by line of sight, it is extremely preferable to arrange controls such as icons in a state in which line-of-sight direction detection accuracy is high due to the calibration effect.

Note that FIG. 21 shows an example in which the screen of the EVF 5 or the rear monitor 4 is rotated from a horizontally elongated state to a vertically elongated state by holding the imaging device 1 vertically. In this case, if it is rotated horizontally, the area outside the area 90 can be widened to the left and right. In such a case, placement of icons and the like, size conversion, etc. are similarly applicable.
Also, if the face is slanted, it is possible to maintain the previous state.

Also, as an application to a UI screen, it is conceivable to change the size of UI parts such as icons according to the detection accuracy of the roll angle. Since the detection accuracy differs depending on the method of obtaining the roll angle, if the accuracy is low, the UI parts are made larger to provide a screen that allows deviation.

<10. Summary and Modifications>
According to the above embodiment, the following effects can be obtained.
The information processing apparatus (imaging device 1, terminal device 100, etc.) of the embodiment includes a line-of-sight detection calculation unit 40. FIG. The line-of-sight detection calculation unit 40 performs line-of-sight detection processing for detecting the line-of-sight direction based on the eye image captured by the eye image capturing unit 51 in the line-of-sight detection device unit 41, and also calculates the calibration data used for the line-of-sight detection processing from the eye. Correction processing is performed based on the roll angle information, which is the information on the relative angle change between the eye image pickup unit 51 and the eye image pickup unit 51 .
Accordingly, once the calibration process is executed, even if the user's way of holding the imaging device 1 or the posture changes, it is possible to obtain the calibration data corresponding to it and perform the line-of-sight detection process. Therefore, there is no need to perform calibration processing again, and there is no need to store a plurality of pieces of calibration data according to the user's posture and holding method.
Moreover, since it is based on relative angular changes, it is possible to appropriately correct the calibration data even when the user obliquely removes the imaging device 1 placed horizontally, for example.
Therefore, it is possible to improve usability of the imaging device 1, the terminal device 100, etc., improve processing efficiency, and improve the efficiency of the amount of calibration data to be stored, while improving the accuracy of line-of-sight detection.

In the embodiment, the line-of-sight detection calculation unit 40 acquires reference angle information used for calculating roll angle information when performing calibration processing for acquiring calibration data used for line-of-sight detection processing, and performs calibration data and the reference angle information. An example of performing processing for storing angle information in association with each other has been given.
The reference angle information is angle information with zero roll angle. For example, the iris code, boundary information, contact position information, device attitude information, etc. acquired when performing calibration. By storing such reference angle information in pairs with calibration data, the relative posture relationship for the acquired calibration data, that is, the relative posture relationship for the calibration data when the roll angle information is set to zero. It is possible to grasp the relationship between physical postures. This can be used as a reference for calculating the roll angle during the subsequent line-of-sight detection processing.
Note that the reference angle information is not limited to the iris code, boundary information, contact position information, and device posture information, and various other types of information are conceivable. Information that changes according to the relative posture relationship between the user's eyes and the eye image capturing unit 51 of the imaging device 1 or the terminal device 100 may be used as the reference angle information.

In the embodiment, the line-of-sight detection calculation unit 40 acquires current angle information, which is the same type of information as the reference angle information, during line-of-sight detection processing, and calculates roll angle information using the reference angle information and the current angle information. An example of performing correction processing by
The current angle information is angle information at the time of line-of-sight detection processing. For example, the iris code, boundary information, contact position information, device orientation information, etc., acquired during the line-of-sight detection process. Roll angle information can be easily calculated by comparing the current angle information and the reference angle information.
The current angle information may be the same type of information as the reference angle information, and is not limited to the iris code, boundary information, contact position information, and apparatus orientation information. Various types of current angle information are conceivable according to the type of information employed as the reference angle information.

In the first embodiment, the line-of-sight detection calculation unit 40 calculates the roll angle information from the iris code used in the iris authentication process.
This makes it possible to obtain roll angle information using information obtained by iris authentication processing, and eliminates the need to separately perform detection processing only for roll angle information. Therefore, processing can be made more efficient.

In the first embodiment, the line-of-sight detection calculation unit 40 performs calibration processing to acquire calibration data used for line-of-sight detection processing at an opportunity to acquire an iris code, and uses the acquired iris code to calculate roll angle information. This is an example in which the reference angle information is stored in association with the calibration data (see FIG. 9).
By storing the iris code as reference angle information in pairs with the calibration data, it becomes possible to grasp the relative attitude relationship when the roll angle information for the stored calibration data is set to zero.

In the first embodiment, the line-of-sight detection calculation unit 40 uses the iris code detected in the iris authentication process as the current angle information, calculates the roll angle information using the reference angle information and the current angle information, and performs the correction process. An example was given (see FIG. 12).
The roll angle information can be easily calculated by comparing the reference angle information with the iris code acquired during the line-of-sight detection process as the current angle information. Therefore, when iris authentication is performed, calibration data correction processing can be executed without special detection.

In the first embodiment, the line-of-sight detection calculation unit 40 calculates the roll angle information based on the Hamming distance between the iris code as the reference angle information and the iris code as the current angle information.
Both the current angle information and the iris code as the reference angle information have a bit-shifted relationship according to the relative change in the eye posture of the imaging device 1 and the user's eye. Therefore, the shift amount when the Hamming distance is the minimum represents the angle change amount, that is, the roll angle information. Accordingly, roll angle information can be calculated, and calibration data correction processing can be executed.

In the first embodiment, if the iris code detected in the iris authentication process and the iris code authenticated as belonging to the same person are stored as the reference angle information, the line-of-sight detection calculation unit 40 detects the iris code in the iris authentication process. An example has been described in which the calculated iris code is used as current angle information, and correction processing is performed using roll angle information calculated using the reference angle information and current angle information.
When it can be determined that the person is the same person as when the iris code was paired with the calibration data as reference angle information and stored in the past, correction processing using the iris code is performed in steps S205 to S207 of FIG. do.
As a result, calculation of the roll angle information using the iris code becomes appropriate, and correction processing of the calibration data appropriate for the current person is ensured.

Further, in the first embodiment, if the iris code detected in the iris authentication process and the iris code authenticated as belonging to the same person are not stored as the reference angle information, the line-of-sight detection calculation unit 40 performs the calibration process. and stores the iris code acquired in the iris authentication process as reference angle information in association with the calibration data. That is, it is the processing of steps S210 and S211 in FIG.
In this case, even if the reference angle information and the current angle information are compared, the roll angle information cannot be calculated correctly, so correction processing is not performed. Accordingly, correction processing based on inaccurate roll angle information can be prevented from being performed.
Also, when detecting a new person, by storing the current iris code as reference angle information in association with the calibration data, the calibration data for that person can be appropriately corrected thereafter. .

In the second embodiment, the line-of-sight detection calculation unit 40 calculates the roll angle information based on the eyelid boundary information in the captured image of the eye. Boundary information is information representing features related to the eyelids.
By acquiring the roll angle information using the information obtained from the eye image, it is possible to easily obtain the roll angle information and correct the calibration data. This is particularly useful for devices that do not require personal authentication or personal devices.

In the second embodiment, the line-of-sight detection calculation unit 40 acquires boundary information from the captured image of the eye when performing calibration processing for obtaining calibration data used in the line-of-sight detection processing, and rolls the boundary information. As the reference angle information used for calculating the angle information, an example is given in which processing is performed to store the reference angle information in association with the calibration data (see FIG. 14).
By storing the boundary information recognized from the eye image as reference angle information in pairs with the calibration data, it is possible to grasp the relative attitude relationship when the roll angle information for the stored calibration data is set to zero. become.

In the second embodiment, the line-of-sight detection calculation unit 40 acquires boundary information from the captured image of the eye and uses it as current angle information, calculates roll angle information using the reference angle information and the current angle information, and performs correction processing. (see FIG. 15).
During the line-of-sight detection process, the roll angle information can be easily calculated by acquiring the boundary information, which is the information on the boundary shape of the eyelids, from the eye image and using it as the current angle information, and comparing it with the reference angle information.

In the third embodiment, the line-of-sight detection calculation unit 40 calculates the roll angle information based on the contact position information on the touch panel.
By using the contact position on the touch panel, it is possible to easily obtain the roll angle information and perform correction processing of the calibration data. In particular, the device does not perform personal authentication processing or object recognition from images, and is suitable for use in devices in which the user looks into the display unit 15, such as the imaging device 1 and the telescope.

In the third embodiment, the line-of-sight detection calculation unit 40 acquires contact position information when performing calibration processing for acquiring calibration data used in line-of-sight detection processing, and converts the contact position information into roll angle information. As the reference angle information used for the calculation, an example is given in which the reference angle information is stored in association with the calibration data (see FIG. 16).
By storing the contact position information as reference angle information in pairs with the calibration data, it is possible to grasp the relative attitude relationship when the roll angle information for the stored calibration data is set to zero.

In the third embodiment, the line-of-sight detection calculation unit 40 acquires the contact position information and uses it as current angle information, calculates the roll angle information using the reference angle information and the current angle information, and performs the correction process. (see FIG. 17).
During the line-of-sight detection process, the roll angle information can be easily calculated by acquiring the boundary information, which is the information on the boundary shape of the eyelids, from the eye image and using it as the current angle information, and comparing it with the reference angle information.

In the fourth embodiment, the information processing device (for example, the imaging device 1) includes a detection unit (for example, the sensor unit 23) that detects its own posture, and the line-of-sight detection calculation unit 40 is connected to an external head-mounted device (for example, An example of calculating the roll angle information based on the device orientation information transmitted from the headset 130) and the device orientation information detected by the detection unit has been given (see FIGS. 18, 19, and 20).
For example, by using the device orientation information of the imaging device 1 and the headset 130, the relative relationship between the orientation of the user's head and the line of sight of the imaging device 1 can be known. Therefore, the roll angle can be calculated, and correction processing of the calibration data can be performed.

An example in which the terminal device 100 and the imaging device 1 as the information processing device of the embodiment include the line-of-sight detection device section 41 having the eye image imaging section 51 is shown. That is, in the information processing apparatus integrally including the eye image capturing unit 51, the line-of-sight detection processing and calibration data correction processing are performed by the line-of-sight detection calculation unit 40 based on the detection result of the line-of-sight detection device unit 41. .
As a result, the terminal device 100 and the imaging device 1 can implement a device that detects the line of sight after correcting the calibration data. Therefore, highly accurate line-of-sight detection processing can be performed in the terminal device 100 or the like, and highly accurate line-of-sight direction detection results can be applied to various types of processing.
In particular, in the terminal device 100 exemplified as a smartphone or tablet device, the angle at which the user removes the screen is diverse. Therefore, the calibration data correction process of the present technology is extremely useful.
Also, in the case of a device such as the tabletop display 120, the relative posture relationship varies depending on the user's standing position with respect to the table, and a user standing in a corner may appear oblique, especially to the eye imaging device. be. In such a situation, the correction processing of the present technology is useful.

Note that the device including the line-of-sight detection device unit 41 and the device including the line-of-sight detection calculation unit 40 are configured separately. A configuration in which the information is transmitted to an information processing apparatus including

In the embodiments, the imaging device 1 is given as an example of an information processing device that includes an imaging unit that captures an image, the line-of-sight detection device unit 41 that has the eye image capturing unit 51 , and the line-of-sight detection calculation unit 40 . The imaging section is an imaging system including the lens system 11 , the imaging element section 12 and the camera signal processing section 13 .
As a result, the imaging apparatus 1 is realized as an apparatus that detects the line of sight after correcting the calibration data. Therefore, highly accurate line-of-sight detection processing becomes possible, and the result of line-of-sight detection can be effectively used for imaging operations. For example, focus control according to the line-of-sight direction can be realized with high precision.

In any of the above embodiments, when storing calibration data for each of the right eye and left eye, reference angle information (iris code, boundary information, contact position information, device posture information) is stored for each of the right and left eyes. etc.) are assumed to be stored in association with the calibration data.

Also, when the entire face is captured from a relatively distant place, as in the case of the terminal device 100 or the tabletop display 120, the tilt of the face may be obtained using skeleton estimation or face detection.

A program according to an embodiment is a program that causes a CPU, DSP, GPU, GPGPU, AI processor, etc., or a device including these to execute the calibration data correction process described above.
That is, the program according to the embodiment performs line-of-sight detection processing for detecting the line-of-sight direction based on the eye image captured by the eye image capturing unit 51, and also performs the calibration data used for the line-of-sight detection processing in relation to the eyes and the eye image capturing unit. It is a program that causes an information processing device such as the camera control unit 18 or the CPU 71 to execute a line-of-sight detection calculation that performs a correction process based on roll angle information, which is information about a change in angle.
With such a program, the calibration data correction process referred to in the present disclosure can be realized by various information processing apparatuses.

These programs can be recorded in advance in a HDD as a recording medium built in equipment such as a computer device, or in a ROM or the like in a microcomputer having a CPU.
Alternatively, a flexible disc, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a Blu-ray disc (Blu-ray Disc (registered trademark)), a magnetic disc, a semiconductor memory, It can be temporarily or permanently stored (recorded) in a removable recording medium such as a memory card. Such removable recording media can be provided as so-called package software.
In addition to installing such a program from a removable recording medium to a personal computer or the like, it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

Also, such a program is suitable for widely providing the information processing apparatus of the present disclosure. For example, by downloading a program to a terminal device 100 such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), etc., these devices function as the information processing device of the present disclosure. be able to.

It should be noted that the effects described in this specification are merely examples and are not limited, and other effects may also occur.

Note that the present technology can also adopt the following configuration.
(1)
A line-of-sight detection process for detecting a line-of-sight direction is performed based on an eye image captured by an eye image capturing unit, and calibration data used for the line-of-sight detection process is obtained by detecting a relative angle change between the eye and the eye image capturing unit. An information processing apparatus comprising a line-of-sight detection calculation unit that performs correction processing for correcting based on roll angle information, which is information of.
(2)
The line-of-sight detection calculation unit is
When performing the calibration process for acquiring the calibration data used for the line-of-sight detection process, the reference angle information used for calculating the roll angle information is acquired, and the calibration data and the reference angle information are stored in association with each other. The information processing apparatus according to (1) above.
(3)
The line-of-sight detection calculation unit is
During the line-of-sight detection process, current angle information that is the same type of information as the reference angle information is acquired, and the roll angle information is calculated using the reference angle information and the current angle information, and the correction process is performed. The information processing apparatus according to (2) above.
(4)
The line-of-sight detection calculation unit is
The information processing apparatus according to any one of (1) to (3) above, wherein roll angle information is calculated from an iris code used in iris authentication processing.
(5)
The line-of-sight detection calculation unit is
performing calibration processing for acquiring calibration data used in the line-of-sight detection processing at an opportunity to acquire the iris code,
The information processing apparatus according to (4) above, wherein the acquired iris code is stored as reference angle information used for calculating the roll angle information in association with calibration data.
(6)
The line-of-sight detection calculation unit is
The information processing apparatus according to (5) above, wherein the iris code detected in the iris authentication process is used as current angle information, and the roll angle information is calculated using the reference angle information and the current angle information to perform the correction process. .
(7)
The line-of-sight detection calculation unit is
The information processing apparatus according to (6), wherein the roll angle information is calculated based on a Hamming distance between the iris code as the reference angle information and the iris code as the current angle information.
(8)
The line-of-sight detection calculation unit is
When the iris code detected in the iris authentication process and the iris code authenticated as belonging to the same person are stored as the reference angle information, the iris code detected in the iris authentication process is used as the current angle information, and the reference angle The information processing apparatus according to any one of (6) to (7) above, wherein the correction process is performed using the roll angle information calculated using the information and the current angle information.
(9)
The line-of-sight detection calculation unit is
If the iris code detected in the iris authentication process and the iris code authenticated as belonging to the same person are not stored as the reference angle information, the calibration process is performed, and the iris code acquired in the iris authentication process is used as the iris code. The information processing apparatus according to any one of (6) to (8) above, wherein the reference angle information is stored in association with the calibration data.
(10)
The line-of-sight detection calculation unit is
The information processing apparatus according to any one of (1) to (3) above, wherein roll angle information is calculated based on eyelid boundary information in a captured image of an eye.
(11)
The line-of-sight detection calculation unit is
obtaining the boundary information from the captured image of the eye when performing calibration processing for obtaining calibration data used in the line-of-sight detection processing;
The information processing apparatus according to (10), wherein the boundary information is stored in association with calibration data as reference angle information used for calculating the roll angle information.
(12)
The line-of-sight detection calculation unit is
The information according to (11) above, wherein the boundary information is obtained from the captured image of the eye and used as current angle information, and the roll angle information is calculated using the reference angle information and the current angle information to perform the correction process. processing equipment.
(13)
The line-of-sight detection calculation unit is
The information processing apparatus according to any one of (1) to (3) above, wherein roll angle information is calculated based on contact position information with respect to the touch panel.
(14)
The line-of-sight detection calculation unit is
Acquiring the contact position information when performing a calibration process for acquiring calibration data used in the line-of-sight detection process,
The information processing apparatus according to (13), wherein the contact position information is stored in association with calibration data as reference angle information used for calculating the roll angle information.
(15)
The line-of-sight detection calculation unit is
The information processing apparatus according to (14), wherein the contact position information is obtained as current angle information, the roll angle information is calculated using the reference angle information and the current angle information, and the correction process is performed.
(16)
Equipped with a detection unit that detects its own posture,
The line-of-sight detection calculation unit is
calculating the roll angle information based on the device posture information of the head-mounted device transmitted from the head-mounted device and the device posture information detected by the detection unit; any of the above (1) to (3) 1. The information processing device according to claim 1.
(17)
The information processing apparatus according to any one of (1) to (16) above, including the eye image capturing section.
(18)
an imaging unit that captures an image;
the eye image capturing unit;
The information processing apparatus according to any one of (1) to (17) above.
(19)
A line-of-sight detection process for detecting a line-of-sight direction is performed based on an eye image captured by an eye image capturing unit, and calibration data used for the line-of-sight detection process is obtained by detecting a relative angle change between the eye and the eye image capturing unit. The line-of-sight detection calculation that performs the correction process based on the roll angle information, which is the information of
A line-of-sight detection method executed by an information processing device.
(20)
A line-of-sight detection process for detecting a line-of-sight direction is performed based on an eye image captured by an eye image capturing unit, and calibration data used for the line-of-sight detection process is obtained by detecting a relative angle change between the eye and the eye image capturing unit. The line-of-sight detection calculation that performs the correction process based on the roll angle information, which is the information of
A program to be executed by an information processing device.

1 imaging device 4 rear monitor 5 EVF
15 display unit 16 communication unit 18 camera control unit 19 memory unit 23 sensor unit 24 line-of-sight detection unit 40 line-of-sight detection calculation unit 41 line-of-sight detection device unit 50 infrared irradiation unit 51 eye image capturing unit 52 database 71 CPU
100 terminal device 120 table top display 130 headset

Claims (20)

1. A line-of-sight detection process for detecting a line-of-sight direction is performed based on an eye image captured by an eye image capturing unit, and calibration data used for the line-of-sight detection process is obtained by detecting a relative angle change between the eye and the eye image capturing unit. An information processing apparatus comprising a line-of-sight detection calculation unit that performs correction processing for correcting based on roll angle information, which is information of.
2. The line-of-sight detection calculation unit is
   When performing the calibration process for acquiring the calibration data used for the line-of-sight detection process, the reference angle information used for calculating the roll angle information is acquired, and the calibration data and the reference angle information are stored in association with each other. The information processing device according to claim 1 .
3. The line-of-sight detection calculation unit is
   During the line-of-sight detection process, current angle information that is the same type of information as the reference angle information is acquired, and the roll angle information is calculated using the reference angle information and the current angle information, and the correction process is performed. The information processing apparatus according to claim 2.
4. The line-of-sight detection calculation unit is
   The information processing apparatus according to claim 1, wherein roll angle information is calculated from an iris code used in iris authentication processing.
5. The line-of-sight detection calculation unit is
   performing calibration processing for acquiring calibration data used in the line-of-sight detection processing at an opportunity to acquire the iris code,
   5. The information processing apparatus according to claim 4, wherein the obtained iris code is stored in association with calibration data as reference angle information used for calculating the roll angle information.
6. The line-of-sight detection calculation unit is
   6. The information processing apparatus according to claim 5, wherein the iris code detected in the iris authentication process is used as current angle information, and the roll angle information is calculated using the reference angle information and the current angle information to perform the correction process.
7. The line-of-sight detection calculation unit is
   7. The information processing apparatus according to claim 6, wherein the roll angle information is calculated based on a Hamming distance between the iris code as the reference angle information and the iris code as the current angle information.
8. The line-of-sight detection calculation unit is
   When the iris code detected in the iris authentication process and the iris code authenticated as belonging to the same person are stored as the reference angle information, the iris code detected in the iris authentication process is used as the current angle information, and the reference angle The information processing apparatus according to claim 5, wherein the correction processing is performed using the roll angle information calculated using the roll angle information and the current angle information.
9. The line-of-sight detection calculation unit is
   If the iris code detected in the iris authentication process and the iris code authenticated as belonging to the same person are not stored as the reference angle information, the calibration process is performed, and the iris code acquired in the iris authentication process is used as the iris code. The information processing apparatus according to claim 5, wherein the reference angle information is stored in association with the calibration data.
10. The line-of-sight detection calculation unit is
    The information processing apparatus according to claim 1, wherein roll angle information is calculated based on eyelid boundary information in a captured image of an eye.
11. The line-of-sight detection calculation unit is
    obtaining the boundary information from the captured image of the eye when performing calibration processing for obtaining calibration data used in the line-of-sight detection processing;
    11. The information processing apparatus according to claim 10, wherein the boundary information is stored in association with calibration data as reference angle information used for calculating the roll angle information.
12. The line-of-sight detection calculation unit is
    12. The information processing according to claim 11, wherein the boundary information is acquired from the captured image of the eye and used as current angle information, and the roll angle information is calculated using the reference angle information and the current angle information to perform the correction process. Device.
13. The line-of-sight detection calculation unit is
    The information processing apparatus according to claim 1, wherein roll angle information is calculated based on contact position information with respect to the touch panel.
14. The line-of-sight detection calculation unit is
    Acquiring the contact position information when performing a calibration process for acquiring calibration data used in the line-of-sight detection process,
    The information processing apparatus according to claim 13, wherein the contact position information is stored in association with calibration data as reference angle information used for calculating the roll angle information.
15. The line-of-sight detection calculation unit is
    15. The information processing apparatus according to claim 14, wherein the contact position information is obtained as current angle information, and the roll angle information is calculated using the reference angle information and the current angle information to perform the correction process.
16. Equipped with a detection unit that detects its own posture,
    The line-of-sight detection calculation unit is
    The information processing apparatus according to claim 1, wherein the roll angle information is calculated based on device posture information of the head mounted device transmitted from the head mounted device and device posture information detected by the detection unit.
17. The information processing apparatus according to claim 1, comprising the eye image capturing section.
18. an imaging unit that captures an image;
    the eye image capturing unit;
    The information processing apparatus according to claim 1, comprising:
19. A line-of-sight detection process for detecting a line-of-sight direction is performed based on an eye image captured by an eye image capturing unit, and calibration data used for the line-of-sight detection process is obtained by detecting a relative angle change between the eye and the eye image capturing unit. The line-of-sight detection calculation that performs the correction process based on the roll angle information, which is the information of
    A line-of-sight detection method executed by an information processing device.
20. A line-of-sight detection process for detecting a line-of-sight direction is performed based on an eye image captured by an eye image capturing unit, and calibration data used for the line-of-sight detection process is obtained by detecting a relative angle change between the eye and the eye image capturing unit. The line-of-sight detection calculation that performs the correction process based on the roll angle information, which is the information of
    A program to be executed by an information processing device.

Images