JP2014064634A - Image processing apparatus, line-of-sight specifying apparatus, and pupil image detecting program and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a pupil image even under an influence of external light.SOLUTION: An image processing apparatus (10) comprises: an infrared emission unit (40); an infrared camera (42) for imaging a user's pupil with infrared light emitted from the infrared emission unit (40); and a CPU (24). While controlling on/off of the infrared emission unit (S5, S11), the CPU (24) detects the pupil image (S15) on the basis of comparison between a first image (72) imaged by the infrared camera while the infrared emission unit is turned on and a second image (74) imaged by the infrared camera while the infrared emission unit is turned off.

Description

  The present invention relates to an image processing device, a line-of-sight identification device, and a pupil image detection program and method, and in particular, an image having an infrared light emitting unit and an infrared camera that images a user's pupil with infrared rays emitted from the infrared light emitting unit. The present invention relates to a processing apparatus, and a pupil image detection program and method for such an image processing apparatus.

Conventional devices of this type are disclosed in Patent Documents 1 to 3. In the background art of Patent Document 1, a pupil illuminated by a light source is photographed by a camera, and the eye vector to the camera is measured by detecting the pupil from the photographed image and performing elliptic approximation. In the background art of Patent Document 2, in order to prevent the accuracy of line-of-sight detection from being deteriorated due to the incidence of light from outside the light source (external light: for example, infrared rays contained in sunlight), A film for limiting incidence is provided. In the background art of Patent Document 3, in order to improve the accuracy of line-of-sight detection, an infrared cut filter that cuts the wavelength range of infrared rays from the infrared light emitting diode is provided in front of the LCD.
Japanese Patent Laying-Open No. 2005-13752 [A61B 3/113] JP-A-6-294928 [G02B 7/28, A61B 3/113, G03B 13/36] JP-A-10-179521 [A61B 3/113, H04N 5/225]

  However, the background art of Patent Document 1 has a problem that it becomes difficult to detect the pupil and thus the gaze vector due to the influence of external light such as sunlight. Since a film or a filter is used to limit or cut outside light, there is a problem that there are many components.

  Therefore, a main object of the present invention is to provide a novel image processing apparatus and line-of-sight identification apparatus, and a pupil image detection program and method.

  Another object of the present invention is to provide an image processing device, a line-of-sight identification device, and a pupil image detection program and method capable of detecting a pupil image even under the influence of external light.

  The present invention employs the following configuration in order to solve the above problems. Note that reference numerals in parentheses, supplementary explanations, and the like indicate correspondence with embodiments to be described later in order to help understanding of the present invention, and do not limit the present invention.

  The first aspect is an infrared light emitting unit, an infrared camera that images a user's pupil with infrared rays emitted from the infrared light emitting unit, a control unit that controls on / off of the infrared light emitting unit, and an infrared ray when the infrared light emitting unit is in an on state. An image processing apparatus including a first detection unit that detects a pupil image based on a comparison between a first image captured by a camera and a second image captured by an infrared camera when the infrared light emitting unit is in an off state. .

  In the first aspect, in the image processing apparatus (10) including the infrared light emitting unit (40) and the infrared camera (42) that images the user's pupil with infrared rays emitted from the infrared light emitting unit, for example, the CPU (24) By executing the detection program (54), the control unit and the first detection unit are realized. The control unit controls on / off of the infrared light emitting unit (S5, S11), and the first detection unit and the first image (72) captured by the infrared camera when the infrared light emitting unit is in the on state and the infrared light emission. A pupil image is detected based on the comparison with the second image (74) captured by the infrared camera when the unit is in the off state (S15).

  Therefore, even if the first image includes the pupil image by the infrared light from the infrared light emitting unit and the reflected image by the external light, the second image does not include the pupil image by the infrared light from the infrared light emitting unit, Since only the reflected image by the external light is included, the pupil image by the infrared rays from the infrared light emitting unit can be detected without using a filter or the like by comparing the two images with each other.

  According to the first aspect, a pupil image can be detected even under the influence of external light.

  The second mode is that, in the first mode, the control unit turns on / off the infrared light emitting unit in the first cycle, and the first detection unit is in the on state of the infrared light emitting unit in one cycle of the first cycle. The first image captured during the ON period is compared with the second image captured during the OFF period when the infrared light emitting unit is in the OFF state, and the pupil image is detected.

  According to the second aspect, the first image and the second image captured in the on period and the off period constituting one cycle are compared, that is, the first image and the second image that are close in time are compared. Even if the movement of the line of sight is fast, the pupil image can be detected.

  In another aspect, the first image and the second image that are separated in time may be compared. If the movement of the line of sight is sufficiently slow with respect to the temporal distance between the first image and the second image, the pupil image can be detected.

  In a second aspect, the third aspect further includes an imaging control unit that causes the infrared camera to perform imaging in a second period that is half the length of the first period.

  In the third aspect, an imaging control unit is further realized, and the imaging control unit causes the infrared camera to perform imaging in a second period that is half the length of the first period (S7, S13).

  According to the third aspect, by causing the infrared camera to perform imaging in the first period, that is, the second period that is half the on / off period of the infrared light emitting unit, there is no useless on period, and Since the imaging is performed once in each of the on period and the off period, the imaging for detecting the line of sight can be performed efficiently.

  In another aspect, the infrared camera is caused to pick up an image in the second period shorter than half the length of the first period, and the image pickup result of the infrared camera is sampled once in the on period and in the off period. May be.

  According to a fourth aspect, in any one of the first to third aspects, the first detection unit subtracts the value of the image data of the second image from the value of the image data of the first image, and obtains the image data after the subtraction. A pupil image is detected based on the third image (76) with the value of.

  According to the fourth aspect, the reflected image of the external light is reduced by the subtraction, and only the pupil image is included in the third image. Therefore, the pupil image can be easily detected.

  A fifth aspect is a line-of-sight identifying apparatus that identifies a user's line of sight based on an image of a pupil detected by the image processing apparatus according to any one of the first to fourth aspects, and is detected by the first detection unit. A second detection unit that detects a pupil image and an infrared light emitting unit image reflected on the pupil from the pupil image, and a positional relationship between the pupil image and the infrared light emitting unit image detected by the second detection unit A specifying unit for specifying the user's line of sight.

  In the fifth aspect, in the line-of-sight identifying device (10) that identifies the line of sight of the user based on the pupil image detected by the image processing apparatus according to any one of the first to fourth aspects, for example, the CPU (24) By executing the line-of-sight detection program (54), the second detection unit and the identification unit are realized. Note that the image processing device and the line-of-sight specifying device may be independent from each other, or one may be included in the other.

  The second detection unit detects the image of the pupil and the image of the infrared light emitting unit reflected in the pupil from the pupil image detected by the first detection unit (S17). The specifying unit specifies the line of sight of the user based on the positional relationship between the pupil image and the infrared light emitting unit image detected by the second detection unit (S19).

  In general, when the line of sight moves, the positional relationship between the image of the pupil and the image of the infrared light emitting part reflected on the pupil, that is, the light spot image changes. According to the fifth aspect, the line of sight can be accurately identified based on the positional relationship between the two. As the line of sight moves, the flatness of the pupil image and the light spot image also changes. Therefore, when specifying the line of sight, the flatness of the pupil image and the light spot image may be taken into consideration as necessary.

  In the sixth aspect, the CPU (24) of the image processing apparatus (10) having an infrared light emitting unit (40) and an infrared camera (42) that captures an image of the user's pupil with infrared rays emitted from the infrared light emitting unit emits infrared light. The control unit (S5, S11) for controlling the on / off of the unit, and the first image (72) captured by the infrared camera when the infrared light emitting unit is on and the infrared camera when the infrared light emitting unit is off A pupil image detection program (52) that functions as a first detection unit (S15) that detects a pupil image based on a comparison with the second image (74).

  The seventh aspect is a pupil detection method performed by an image processing apparatus (10) having an infrared light emitting unit (40) and an infrared camera (42) that images the user's pupil with infrared rays emitted from the infrared light emitting unit. The control step (S5, S11) for controlling on / off of the infrared light emitting unit, and the first image (72) captured by the infrared camera when the infrared light emitting unit is on and the infrared light emitting unit are off A first detection step (S15) for detecting a pupil image based on a comparison with the second image (74) captured by the infrared camera is included.

  According to each of the sixth and seventh aspects, similarly to the first aspect, the pupil image can be detected with a simple configuration even under the influence of external light.

  According to the present invention, an image processing device, an eye gaze identification device, and a pupil image detection program and method that can detect a pupil image even under the influence of external light are realized.

It is a block diagram which shows the structure of the portable terminal which is one Example of this invention. It is an illustration figure which shows the display, infrared LED, and infrared camera which were provided in the one main surface (front surface) of the portable terminal. It is an illustration figure which shows the structure of the pupil imaged with an infrared camera, and the image (light spot) of the infrared LED reflected on the pupil. It is an illustration figure which shows on / off control of infrared LED, and the imaging timing of an infrared camera. It is an illustration figure for demonstrating the method to detect a pupil image and a light spot image based on the result which subtracted the image imaged in the off period from the image imaged in the on period of infrared LED, (A) is an on period. The captured image is the image captured when (B) is off, and (C) shows the result of subtracting the image of (B) from the image of (A) (image after subtraction). It is a memory map figure which shows the content of the main memory of a portable terminal. It is a flowchart which shows the gaze input process by CPU of a portable terminal.

  FIG. 1 shows a hardware configuration of the mobile terminal 10. Referring to FIG. 1, mobile terminal 10 according to an embodiment of the present invention includes a CPU 24. The CPU 24 is connected to a key input device 26, a touch panel 32, a main memory 34, a flash memory 36, an infrared camera 42 and an image processing circuit 44, and the antenna 12 is connected to the A / D converter 16 via the wireless communication circuit 14. Are connected to the microphone 18, the D / A converter 20 to the speaker 22, the driver 28 to the display 30, and the LED driver 38 to the infrared LED 40.

  The antenna 12 captures (receives) a radio signal from a base station (not shown) and emits (transmits) a radio signal from the radio communication circuit 14. The radio communication circuit 14 demodulates and decodes a radio signal received by the antenna 12, and encodes and modulates a signal from the CPU 24. The microphone 18 converts the sound wave into an analog audio signal, and the A / D converter 16 converts the audio signal from the microphone 18 into digital audio data. The D / A converter 20 converts the audio data from the CPU 24 into an analog audio signal, and the speaker 22 converts the audio signal from the D / A converter 20 into a sound wave.

  The key input device 26 includes various keys and buttons (not shown) operated by the user, and inputs a signal (command) corresponding to the operation to the CPU 24. The driver 28 displays an image corresponding to the signal from the CPU 24 on the display 30. The touch panel 32 is provided on the display surface of the display 30 and inputs a signal indicating the position of the touch point to the CPU 24.

  The LED driver 38 turns on (turns on) / off (turns off) the infrared LED 40 in accordance with a command from the CPU 24. The infrared camera 42 images the user's pupil with infrared rays emitted from the infrared LED 40. The image processing circuit 44 performs image processing such as binarization on an image (captured image) captured by the infrared camera 42.

  The main memory 34 is composed of, for example, an SDRAM or the like, and stores programs, data, and the like (see FIG. 6) for causing the CPU 24 to execute various processes, and provides a necessary work area for the CPU 24. The flash memory 36 is composed of, for example, a NAND flash memory and is used as a storage area for programs and data.

  Based on the program (52) and data (62, 72-76) stored in the main memory 34, the CPU 24 executes various processes while using other hardware (12-22, 26-44). . Current time information and timing signals necessary for execution of processing are supplied from an RTC (Real Time Clock) 24a.

  With the mobile terminal 10 configured as described above, a call mode for performing a call, a data communication mode for performing data communication, an information processing mode for performing information processing, and the like can be selected through a menu screen (not shown).

  When the call mode is selected, the mobile terminal 10 functions as a call device. Specifically, when a call operation is performed by the key input device 26 or the touch panel 32, the CPU 24 controls the wireless communication circuit 14 to output a call signal. The output call signal is output via the antenna 12 and transmitted to the other telephone through a mobile communication network (not shown). The telephone starts ringing with a ring tone or the like. When the other party performs an incoming call operation, the CPU 24 starts a call process. On the other hand, when the call signal from the other party is captured by the antenna 12, the wireless communication circuit 14 notifies the CPU 24 of the incoming call, and the CPU 24 starts calling by a ringing tone from the speaker 22 or vibration of a vibrator (not shown). . When an incoming call operation is performed by the key input device 26 or the touch panel 32, the CPU 24 starts a call process.

  Call processing is performed as follows, for example. The received voice signal sent from the other party is captured by the antenna 12, demodulated and decoded by the wireless communication circuit 14, and then given to the speaker 22 via the D / A converter 20. As a result, the received voice is output from the speaker 22. On the other hand, the transmitted voice signal captured by the microphone 18 is sent to the wireless communication circuit 14 via the A / D converter 16, encoded and modulated by the wireless communication circuit 14, and then transmitted to the other party through the antenna 12. Is done. The other party's telephone also demodulates and decodes the transmitted voice signal and outputs the transmitted voice.

  When the data communication mode is selected, the mobile terminal 10 functions as a data communication device. Specifically, the CPU 24 performs data communication with a mail server or web server (not shown) on the Internet via the wireless communication circuit 14 to acquire mail data and hypertext data, and based on this, e-mail and HTML The document is displayed on the display 30 via the driver 28.

  When the information processing mode is selected, the mobile terminal 10 functions as an information processing device. Specifically, the CPU 24 executes various types of information processing according to operations of the key input device 26 and the touch panel 32, and displays a screen based on the processing result on the display 30 via the driver 28.

  Further, when the user calls the menu screen by a key operation or the like with the call mode, the data communication mode, the information processing mode, or the like selected as described above, and turns on the line-of-sight input function, the icon is displayed on various screens thereafter. The operation of selecting or pressing a soft key can be performed by line-of-sight input using the infrared LED 40 and the infrared camera 42.

  Specifically, when the line-of-sight input function is turned on, under the control of the CPU 24, the LED driver 38 starts to drive the infrared LED 40 (on / off control), and at the same time or substantially simultaneously, the infrared camera 42 is activated, Start imaging. An image (captured image) captured by the infrared camera 42 is input to an image processing circuit 44 where image processing such as binarization is performed. The CPU 24 detects the pupil image from the captured image after the image processing is performed, identifies the user's line of sight, and determines that the icon or the like on the screen has been selected. Perform the associated process.

  The detection of the pupil image and the identification of the line of sight are performed as follows, for example. As shown in FIG. 2, the infrared LED 40 and the infrared camera 42 are respectively arranged on the upper right and lower left of the display 30. The user's pupil (black eye) receives infrared rays from the infrared LED 40 in the eyeball as shown in FIG. 3. And the iris surrounding the pupil.

  The arrangement shown in FIG. 2 is an example, and if the display 30, the infrared LED 40, and the infrared camera 42 are provided on the same surface (usually the front surface) (the optical axis of the infrared LED 40 and the optical axis of the infrared camera 42 are mutually aligned). If the optical axes are parallel or substantially parallel and these optical axes are perpendicular to the screen of the display 30 or an inclination angle equal to or less than a certain angle), the line of sight can be input.

  When the infrared LED 40 is turned on while the user's line of sight is facing the screen of the display 30, the infrared light emitted from the infrared LED 40 is reflected by the user's pupil (particularly the pupil), and this reflected light is captured by the infrared camera 42. . For this reason, in the image captured by the infrared camera 42 with the infrared LED 40 turned on, the image of the pupil in the pupil (pupil image) and the image of the infrared LED 40 reflected in the pupil (light spot image) As a result, the result is brighter (conspicuous) (see FIG. 5A: described later). Therefore, if image processing such as binarization is performed on the captured image, a pupil image and a light spot image can be detected from the captured image. In this embodiment, the pupil image and the light spot image thus detected from the captured image may be collectively referred to as “pupil image”.

  Next, the positional relationship between the detected pupil image and the light spot image (for example, a vector from the center A of the light spot image to the center B of the pupil image) and the shape (for example, the pupil image and the light spot image with respect to the pupil and the light spot). From the flatness, etc., the user's line of sight (for example, the position of the intersection of the normal of the pupil and the screen of the display 30) is specified. The specific identification method is not an essential content of this embodiment, and thus the description thereof is omitted.

  However, if an attempt is made to input a line of sight outdoors in a sunny day or in a room where an infrared source such as an incandescent lamp or an electric heater is placed, a pupil image and a light spot image by the infrared LED 40 are displayed as shown in FIG. In addition, since the reflected image of light (external light) from the sun and other infrared sources appears bright, the reflected image of external light becomes noise, which may make it difficult to detect a pupil image.

  Therefore, in this embodiment, imaging by the infrared camera 42 is performed while the infrared LED 40 is repeatedly turned on / off (intermittent light emission), and an image captured when the infrared LED 40 is in an on state (“image when LED is on”). Or the image data of the image captured in the off state (sometimes called “image when LED is off” or “second image”) from the value of the image data of the image data (may be called “first image”) A value is subtracted, and an image of a pupil is detected based on an image (sometimes referred to as a “third image”) based on the value of the image data after the subtraction.

  Specifically, when the line-of-sight input function is turned on, the CPU 24 activates the infrared camera 42 and starts ON / OFF control of the infrared LED 40 as shown in FIG. 4 simultaneously or substantially simultaneously. Under the control of the CPU 24, the infrared camera 42 repeats an operation of performing imaging in an on period and an off period constituting one on / off cycle.

  For example, the infrared LED 40 is turned on / off at a cycle of 200 μs (on period 100 μs + off period 100 μs), and imaging by the infrared camera 42 is performed, for example, at a cycle of 100 μs (that is, a half value of the on / off cycle). . In this case, in one on / off cycle, imaging is performed twice in total, once when the infrared LED 40 is turned on and once when the infrared LED 40 is turned off. Thus, an image ( The first image (see FIG. 5A) and the image captured when the LED is off (second image: see FIG. 5B) are given to the CPU 24.

  In the example of FIG. 4, the on period and the off period constituting one on / off cycle are the same length (each 100 μsec), but are different from each other (for example, the on period 80 μsec + off period 120 μm). Seconds).

  In the modification, the infrared camera 42 captures an image with a cycle (for example, 20 μsec cycle) shorter than the on / off cycle (for example, 200 μsec) of the infrared LED 40, and the image processing circuit 44 performs an on / off cycle for the captured image. Sampling may be performed at half the value (for example, a cycle of 100 μs).

  Also, instead of performing imaging (sampling) once in each of the on period and the off period constituting one on / off cycle, imaging (sampling) is performed a plurality of times in the on period and the same number of times in the off period, The plurality of first images and the plurality of second images captured in each of the on period and the off period may be averaged, and the averaged first image and the averaged second image may be provided to the CPU 24.

  The CPU 24 takes a second image captured with the infrared LED 40 off as shown in FIG. 5B from a first image captured with the infrared LED 40 turned on as shown in FIG. Subtract images. The first image includes a pupil image and a light spot image based on infrared rays from the infrared LED 40, and a reflection image of external light such as the sun, while the second image includes only a reflection image of external light. Since the pupil image and the light spot image are not included, in the third image after subtracting the second image from the first image, as shown in FIG. (Noise) is reduced (cancelled), resulting in only the pupil image and light spot image remaining (prominent). Therefore, even under the influence of external light, the eyes can be identified by detecting the pupil image (pupil image and light spot image), so that the eyes can be input without any problem.

  The visual line input including the detection of the pupil image and the identification of the visual line as described above is performed by the CPU 24 based on, for example, the program (52) and data (62, 72 to 76) shown in FIG. This is realized by executing a line-of-sight input process according to the flow shown in FIG.

  Specifically, referring to FIG. 6, first, the main memory 34 includes a program area 50, a non-transition data area 60, and a transition data area 70. The line-of-sight input program 52 is in the program area 50, and the non-transition data area 60 is on. / Off period information 62, and in the transition data area 70, an LED 72 image 72 (also referred to as “first image”), an LED OFF image 74 (also referred to as “second image”), and an image after subtraction. 76 (also referred to as “third image”) are stored.

  The line-of-sight input program 52 detects the pupil image by subtracting the image captured in the off period from the image captured in the off period from the image captured in the on period while the infrared LED 40 is repeatedly turned on / off. A program for specifying the user's line of sight based on the detected pupil image and inputting a command corresponding to the specified line of sight, and corresponds to the flowchart of FIG.

  Although not shown, the program area 50 accepts user operations via various control programs for realizing the above-described call mode, data communication mode, information processing mode, and the like, the key input device 26, and the like. In addition, an input / output control program for displaying various screens on the display 30 via the driver 28 is also stored.

  The on / off cycle information 62 is information indicating the cycle of turning on / off the infrared LED 40. For example, “200 μs (on period = 100 μs, off period = 100 μs)” indicating the on / off cycle of FIG. Numerical values (or waveform data) are described.

  The image 72 when the LED is on is an image (first image) captured in a state where the infrared LED 40 is on. For example, as shown in FIG. 5A, a pupil image and a light spot image by infrared rays from the infrared LED 40. And a reflection image of outside light. The image 74 when the LED is off is an image (second image) captured in a state in which the infrared LED 40 is off. For example, as shown in FIG. The infrared pupil image and light spot image are not included. The subtracted image 76 is an image (third image) after the second image is subtracted from the first image. For example, as shown in FIG. 5C, the reflected image of the outside light is not included, and the pupil image And light spot images only.

  Referring to FIG. 7, when the line-of-sight input function is turned on, CPU 24 first turns on infrared camera 42 (issues an activation command) in step S1, and then in step S3, on / off cycle information. 62 is determined based on the ON / OFF cycle described in 62 and the current time information from the RTC 24a, and it is determined whether the timing for turning on the infrared LED 40 has arrived. If NO, the same determination is made after a predetermined waiting time. repeat.

  If “YES” in the step S3, the infrared LED 40 is turned on (a voltage is applied to the infrared LED 40 via the LED driver 38) in a step S5, and an imaging command is issued to the infrared camera 42 in a step S7. The infrared camera 42 performs imaging in accordance with an imaging command, and the captured image is subjected to image processing such as binarization by the image processing circuit 44, and is then used as an image 72 (first image) when the LED is on. It is written in the transition data area 70.

  Next, in step S9, based on the on / off cycle described in the on / off cycle information 62 and the current time information from the RTC 24a, it is determined whether or not the timing for turning off the infrared LED 40 has arrived. If so, the same determination is repeated after a predetermined waiting time.

  If YES in step S9, the infrared LED 40 is turned off in step S11 (voltage application to the infrared LED 40 is stopped), and an imaging command is issued to the infrared camera 42 in step S13. The infrared camera 42 performs imaging in accordance with an imaging command, and the captured image is subjected to image processing such as binarization by the image processing circuit 44, and is then used as an image 74 (second image) when the LED is off. It is written in the transition data area 70.

  When the LED-on image 72 and the LED-off image 74 are stored in the transition data area 70 in this way, the process proceeds to step S15, and the LED-on image 72 is changed (for each pixel) from the LED-on image 72. Subtract. The subtraction result is written in the transition data area 70 as an image 76 (third image) after subtraction.

  Next, in step S17, the pupil and the light spot (see FIG. 3) are detected based on the subtracted image 76, and in step S19, the line of sight is specified from the positional relationship between the pupil and the light spot. Since the image 76 after subtraction does not include the reflected image of the external light, but includes only the pupil image and the light spot image (the pupil image and the light spot image can be distinguished because they have different sizes). Detection and gaze identification without problems.

  Specifically, for example, a vector from the center A of the light spot image to the center B of the pupil image is obtained, and if necessary, the normal of the pupil is calculated in consideration of the flatness of the pupil image and the light spot image. To do. If the direction of the normal is regarded as the direction of the user's line of sight, the user's line of sight (such as an icon on the screen where the line of sight is directed) is determined by obtaining the intersection of the normal and the screen of the display 30. Can do.

  After accepting an input of a command corresponding to the line of sight thus identified in step S21, the process returns to step S3 to repeat the same processing as described above. The process corresponding to the input command is executed under the control of another program (not shown).

  Note that the present invention is not limited to the embodiment of the present invention. For example, as a modification, immediately after the infrared camera 42 is turned on in step S1, a cycle shorter than the half value of the on / off cycle of the infrared LED 40 (for example, a cycle of 20 μs) In step S7 and S13, a sampling execution command for the image processing circuit 44 may be issued instead of the imaging command.

  As is clear from the above, the portable terminal 10 of this embodiment includes the infrared LED 40 and the infrared camera 42 that images the user's pupil with infrared rays emitted from the infrared LED 40. The CPU 24 of the portable terminal 10 turns on / off the infrared LED 40 repeatedly (S5, S11), and the first image captured by the infrared camera 42 when the infrared LED 40 is on (image 72 when the LED is on). And the second image captured by the infrared camera 42 when the infrared LED 40 is in the off state (image 74 when the LED is off), more specifically, the pupil image (more specifically, the pupil image and the pupil). The image of the infrared LED 40) is detected (S15, S17).

  Therefore, even if the first image includes the pupil image by the infrared light from the infrared LED 40 and the reflected image by the external light, the second image does not include the pupil image by the infrared light from the infrared LED 40 and the external light. Therefore, by comparing the two images with each other, the pupil image by the infrared LED 40 can be detected without using a filter or the like. As a result, the pupil image can be detected with a simple configuration even under the influence of external light.

  Preferably, the CPU 24 turns on / off the infrared LED 40 in a first period (for example, 200 μsec period: on period = 100 μsec, off period = 100 μsec), and the infrared LED 40 of one first period is in an on state. A first image captured during an on period and a second image captured during an off period when the infrared LED 40 is in an off state are compared with each other to detect a pupil image. Thus, by comparing the first image and the second image captured in the on period and the off period constituting one cycle, that is, the first image and the second image that are close in time, the movement of the line of sight is faster. However, the pupil image can be detected. If the movement of the line of sight is slow, the pupil image can be detected even by comparing the first image and the second image that are separated in time.

  More preferably, the CPU 24 causes the infrared camera 42 to perform imaging in a first period, that is, a second period (for example, 100 μsec: see FIG. 4) that is a half value (half length) of the on / off period of the infrared LED 40. (S7, S13). As a result, the useless on-period is eliminated, and imaging is performed once in each of the on-period and the off-period, so that it is possible to efficiently perform imaging for eye-gaze detection.

  In the above, although the portable terminal 10 was demonstrated, this invention has an infrared rays light emission part (LED, organic EL, etc.) and an infrared camera which images a user's pupil with the infrared rays irradiated from an infrared light emission part. It can be applied to image processing devices (smartphones, tablet PCs, mobile phone terminals, portable information terminals, information appliances, etc.).

10 ... mobile terminal 24 ... CPU
30 ... Display 34 ... Main memory 40 ... Infrared LED
42: Infrared camera 44: Image processing circuit

Claims (7)

Infrared light emitting section,
An infrared camera that images the user's pupil with infrared rays emitted from the infrared light emitting unit;
A control unit that controls on / off of the infrared light emitting unit; and a first image captured by the infrared camera when the infrared light emitting unit is on and an image captured by the infrared camera when the infrared light emitting unit is off. An image processing apparatus comprising: a first detection unit that detects an image of the pupil based on a comparison with the second image. The control unit turns on / off the infrared light emitting unit in a first cycle,
The first detection unit includes the first image captured in the on period in which the infrared light emitting unit is in an on state and the off period in which the infrared light emitting unit is in an off state in one cycle of the first period. The image processing apparatus according to claim 1, wherein the image of the pupil is detected by comparing the captured second image.   The image processing apparatus according to claim 2, further comprising an imaging control unit that causes the infrared camera to perform imaging in a second period that is half the length of the first period.   The first detection unit subtracts the value of the image data of the second image from the value of the image data of the first image, and detects the image of the pupil based on the third image based on the value of the image data after the subtraction. The image processing apparatus according to any one of claims 1 to 3. A line-of-sight specifying device that specifies the line of sight of the user based on the image of the pupil detected by the image processing device according to claim 1,
A second detector for detecting an image of a pupil and an image of the infrared light emitting part reflected in the pupil from the image of the pupil detected by the first detector; and the pupil detected by the second detector A line-of-sight specifying device comprising: a specifying unit that specifies the line of sight of the user based on a positional relationship between the image of the image and the image of the infrared light emitting unit. A CPU of an image processing apparatus having an infrared light emitting unit and an infrared camera that images a user's pupil with infrared rays emitted from the infrared light emitting unit;
A control unit that controls on / off of the infrared light emitting unit; and a first image captured by the infrared camera when the infrared light emitting unit is on and an image captured by the infrared camera when the infrared light emitting unit is off. A pupil image detection program that functions as a first detection unit that detects an image of the pupil based on comparison with the second image. A pupil detection method performed by an image processing apparatus having an infrared light emitting unit and an infrared camera that images a user's pupil with infrared rays emitted from the infrared light emitting unit,
A control step for controlling on / off of the infrared light emitting unit; and a first image captured by the infrared camera when the infrared light emitting unit is on and an image captured by the infrared camera when the infrared light emitting unit is off. A pupil image detection method including a first detection step of detecting an image of the pupil based on a comparison with the second image.

Images