JP2017131446A - Pupil detection apparatus and pupil detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a pupil detection apparatus and the like for detecting a pupil by simple processing.SOLUTION: The pupil detection apparatus includes: an imaging part for capturing an image including a pupil of a person; and a control part for extracting the pupil of the person from the image captured by the imaging part. The control part includes: a pattern matching part that matches a pattern corresponding to luminance changes in the pupil and a periphery of the pupil, with the image, and extracts candidate positions of the pupil; and a pupil detection part for analyzing the candidate positions of the pupil extracted by the pattern matching part and detecting whether or not the pupil is present thereon.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a pupil detection device and a pupil detection method.

  With the advancement of high accuracy, high processing speed, and miniaturization of cameras, a line-of-sight detection device that detects the position where the subject is gazing on an observation surface such as a monitor screen from the face image captured by the camera has been proposed. Yes. Initially, there were many techniques for fixing the subject's head and attaching a detection device to the head. Recently, in order to reduce the burden on the subject, a non-contact type that does not require a device or the like to be attached to the head has been developed, and a highly accurate gaze detection device is required. In the non-contact type gaze detection apparatus, gaze detection is performed based on the positional relationship between the pupil on the camera image and the corneal reflection. For this reason, it is important to accurately obtain the pupil center coordinates and the corneal reflection center coordinates on the camera image. Patent Literature 1 describes a line-of-sight detection device having a position detection unit that identifies a pupil region and a corneal reflection region from an image obtained by capturing an eye.

JP2015-126850A

  Here, when detecting a pupil on an image, a pupil estimated to have a pupil is detected. The pupil detection device, for example, performs face recognition processing on an image and detects a pupil region on the screen, but the amount of processing may increase.

  The present invention has been made in view of the above, and an object of the present invention is to provide a pupil detection device and a pupil detection method capable of detecting a pupil with simpler processing.

  In order to solve the above-described problems and achieve the object, a pupil detection device of the present invention extracts an image capturing unit that captures an image including a human pupil, and extracts the human pupil from the image acquired by the image capturing unit. A control unit, and the control unit performs matching between the pupil and a pattern corresponding to a luminance change around the pupil and the image, and extracts a candidate position of the pupil, and the pattern A pupil detection unit that analyzes the candidate position of the pupil extracted by the matching unit and detects whether there is a pupil.

  The present invention produces an effect that a pupil can be detected with a small amount of processing by detecting a pupil from the candidate position after extracting the candidate position of the pupil by pattern matching.

FIG. 1 is a diagram showing a state of a subject's eyes when one light source is used. FIG. 2 is a diagram showing the eye of the subject when two light sources are used. FIG. 3 is a diagram illustrating an example of the arrangement of the display unit, the stereo camera, the infrared light source, and the subject according to the present embodiment. FIG. 4 is a diagram illustrating an example of the arrangement of the display unit, the stereo camera, the infrared light source, and the subject according to the present embodiment. FIG. 5 is a diagram illustrating an outline of functions of the diagnosis support apparatus. FIG. 6 is a block diagram illustrating an example of detailed functions of the respective units illustrated in FIG. FIG. 7 is a diagram for explaining an outline of processing when it is assumed that one light source is used. FIG. 8 is a diagram for explaining an overview of processing executed by the diagnosis support apparatus of this embodiment. FIG. 9 is a diagram for explaining calculation processing for calculating the distance between the pupil center position and the corneal curvature center position. FIG. 10 is a flowchart illustrating an example of calculation processing according to the present embodiment. FIG. 11 is a diagram showing a method of calculating the position of the corneal curvature center using the distance obtained in advance. FIG. 12 is a flowchart illustrating an example of a line-of-sight detection process according to the present embodiment. FIG. 13 is a diagram for explaining a calculation process of the modification. FIG. 14 is a flowchart illustrating an example of a modification calculation process. FIG. 15 is a diagram illustrating an outline of functions of the position detection unit. FIG. 16 is a diagram illustrating an example of a classifier stored in a database. FIG. 17 is a flowchart illustrating an example of processing of the position detection unit. FIG. 18 is a diagram for explaining the processing of the position detection unit. FIG. 19 is a diagram for explaining the pattern matching processing of the position detection unit. FIG. 20 is a diagram for explaining the pattern matching processing of the position detection unit. FIG. 21 is a diagram for explaining the pattern matching processing of the position detection unit.

  Embodiments of a pupil detection device and a pupil detection method according to the present invention will be described below in detail with reference to the drawings. In addition, this invention is not limited by this embodiment. In the following, an example will be described in which a pupil detection device is used as a diagnosis support device that supports the diagnosis of developmental disabilities using the line-of-sight detection result. Applicable devices are not limited to diagnosis support devices.

  The diagnosis support apparatus according to the present embodiment detects a line of sight using illumination units installed at two locations. In addition, the diagnosis support apparatus according to the present embodiment calculates the corneal curvature center position with high accuracy by using a result obtained by gazing at one point on the subject before the line-of-sight detection.

  In addition, an illumination part is an element which can irradiate light to a test subject's eyeball including a light source. The light source is an element that generates light, such as an LED (Light Emitting Diode). A light source may be comprised from one LED, and may be comprised by combining several LED and arrange | positioning in one place. Hereinafter, the “light source” may be used as a term representing the illumination unit in this way.

  In order to accurately detect the viewpoint, it is important that the pupil position can be detected correctly. It has been found that when the near-infrared light source is turned on and the image is taken with a camera, the pupil becomes darker than other parts if the distance between the camera and the light source is more than a certain distance. The pupil position is detected using this feature.

  In the present embodiment, two light sources are arranged outside each camera for two cameras. Then, these two light sources are turned on at different timings, and an image is taken with a camera having a longer (far) distance from the lit light source. This makes it possible to photograph the pupil more darkly and distinguish the pupil from other parts with higher accuracy.

  In this case, since the light sources to be turned on are different, it is not possible to simply apply the normal stereo method for three-dimensional measurement. That is, the straight line connecting the light source and the corneal reflection when obtaining the viewpoint cannot be calculated in world coordinates. Therefore, in this embodiment, the positional relationship between the cameras used for imaging and the positional relationship between the light sources to be lit at two timings are symmetric with respect to the position of the virtual light source (virtual light source position). . Then, the two coordinate values obtained when each of the two light sources is turned on are converted into world coordinates as a coordinate value by the left camera and a coordinate value by the right camera. Accordingly, it is possible to calculate a straight line connecting the virtual light source and the corneal reflection in the world coordinates using the corneal reflection position obtained when each of the two light sources is turned on, and to calculate the viewpoint based on the straight line. .

  FIG. 1 is a diagram showing a state of the eye 11 of a subject when one light source is used. As shown in FIG. 1, the difference in darkness between the iris 12 and the pupil 13 is not sufficient, making it difficult to distinguish. FIG. 2 is a diagram showing a state of the eye 21 of the subject when two light sources are used. As shown in FIG. 2, the difference in darkness between the iris 22 and the pupil 23 is larger than that in FIG.

  3 and 4 are diagrams illustrating an example of the arrangement of the display unit, stereo camera (imaging unit), infrared light source, and subject of the present embodiment.

  As shown in FIG. 3, the diagnosis support apparatus according to the present embodiment includes a display unit 101, a right camera 102a and a left camera 102b that constitute a stereo camera, and LED light sources 103a and 103b. The right camera 102a and the left camera 102b are disposed below the display unit 101. The LED light sources 103a and 103b are arranged at positions outside the right camera 102a and the left camera 102b, respectively. The LED light sources 103a and 103b are light sources that irradiate near infrared rays having a wavelength of 850 nm, for example. FIG. 3 shows an example in which LED light sources 103a and 103b (illuminating units) are configured by nine LEDs. The right camera 102a and the left camera 102b use lenses that can transmit near infrared light having a wavelength of 850 nm. The LED light sources 103a and 103b may be disposed at positions inside the right camera 102a and the left camera 102b by reversing the positions of the LED light sources 103a and 103b and the right camera 102a and the left camera 102b. .

  As shown in FIG. 4, the LED light sources 103 a and 103 b irradiate near-infrared light toward the eyeball 111 of the subject. When the LED light source 103a is irradiated, shooting is performed with the left camera 102b, and when the LED light source 103b is irradiated, shooting is performed with the right camera 102a. By appropriately setting the positional relationship between the right camera 102a and the left camera 102b and the LED light sources 103a and 103b, the pupil 112 is reflected with low brightness and darkens in the photographed image, and becomes a virtual image in the eyeball 111. The resulting corneal reflection 113 is reflected with high brightness and becomes brighter. Accordingly, the positions of the pupil 112 and the corneal reflection 113 on the image can be acquired by each of the two cameras (the right camera 102a and the left camera 102b).

  Further, the three-dimensional world coordinate values of the positions of the pupil 112 and the corneal reflection 113 are calculated from the positions of the pupil 112 and the corneal reflection 113 obtained by the two cameras. In the present embodiment, as the three-dimensional world coordinates, the center position of the screen of the display unit 101 is set as the origin, the top and bottom are Y coordinates (up is +), the side is X coordinates (right is +), and the depth is Z coordinates ( The front is +).

  FIG. 5 is a diagram illustrating an outline of functions of the diagnosis support apparatus 100. FIG. 5 shows a part of the configuration shown in FIGS. 3 and 4 and a configuration used for driving the configuration. As shown in FIG. 5, the diagnosis support apparatus 100 includes a right camera 102a, a left camera 102b, an LED light source 103a for the left camera 102b, an LED light source 103b for the right camera 102a, a speaker 205, and a drive / IF. (Interface) section 313, control section 300, storage section 150, and display section 101 are included. In FIG. 5, the display screen 201 shows the positional relationship between the right camera 102 a and the left camera 102 b in an easy-to-understand manner, but the display screen 201 is a screen displayed on the display unit 101. The drive unit and the IF unit may be integrated or separate.

  The speaker 205 functions as an audio output unit that outputs an audio or the like for alerting the subject during calibration.

  The drive / IF unit 313 drives each unit included in the stereo camera. The drive / IF unit 313 serves as an interface between each unit included in the stereo camera and the control unit 300.

  The control unit 300 includes, for example, a control device such as a CPU (Central Processing Unit), a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), a communication IF connected to a network for communication, It can be realized by a computer having a bus for connecting each part.

  The storage unit 150 stores various information such as a control program, a measurement result, and a diagnosis support result. The storage unit 150 stores, for example, an image to be displayed on the display unit 101. The display unit 101 displays various information such as a target image for diagnosis.

  FIG. 6 is a block diagram illustrating an example of detailed functions of the respective units illustrated in FIG. As shown in FIG. 6, the display unit 101 and the drive / IF unit 313 are connected to the control unit 300. The drive / IF unit 313 includes camera IFs 314 and 315, an LED drive control unit 316, and a speaker drive unit 322.

  The right camera 102a and the left camera 102b are connected to the drive / IF unit 313 via the camera IFs 314 and 315, respectively. The driving / IF unit 313 drives these cameras to image the subject. A frame synchronization signal is output from the right camera 102a. The frame synchronization signal is input to the left camera 102b and the LED drive control unit 316. As a result, the LED light sources 103a and 103b are caused to emit light, and corresponding images are captured by the left and right cameras.

  The speaker driving unit 322 drives the speaker 205. The diagnosis support apparatus 100 may include an interface (printer IF) for connecting to a printer as a printing unit. In addition, the printer may be provided inside the diagnosis support apparatus 100.

  The control unit 300 controls the entire diagnosis support apparatus 100. The control unit 300 includes a lighting control unit 351, a position detection unit 352, a curvature center calculation unit 353, a line-of-sight detection unit 354, a viewpoint detection unit 355, an output control unit 356, and an evaluation unit 357. Yes.

  Each element (lighting control unit 351, position detection unit 352, curvature center calculation unit 353, line-of-sight detection unit 354, viewpoint detection unit 355, output control unit 356, and evaluation unit 357) included in the control unit 300 is executed by a CPU or the like. May be realized by software (program) to be executed, may be realized by a hardware circuit, or may be realized by using software and a hardware circuit in combination.

  When implemented by a program, the program is a file in an installable or executable format, such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), a DVD ( Digital Versatile Disk) is recorded on a computer-readable recording medium and provided as a computer program product. The program may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program may be provided or distributed via a network such as the Internet. The program may be provided by being incorporated in advance in a ROM or the like. The program executed by the diagnosis support apparatus 100 according to the present embodiment has a module configuration including the above-described units. As actual hardware, a CPU (processor) reads the program from the storage medium and executes it. Each of the above parts is loaded on the main storage device, and each part is generated on the main storage device.

  The lighting control unit 351 uses the LED drive control unit 316 to control lighting of the LED light sources 103a and 103b. For example, the lighting control unit 351 controls the LED light sources 103a and 103b to light at different timings. The timing difference (time) may be set to a time that is predetermined as a time that does not affect the visual line detection result due to movement of the visual line of the subject, for example.

  The position detection unit 352 detects the pupil region from the eyeball image captured by the stereo camera, detects the pupil from the pupil region, detects the pupil contour, and determines the pupil center indicating the pupil center from the pupil contour. Calculate the position. The position detection unit 352 calculates the position of the corneal reflection center indicating the center of corneal reflection from the captured image of the eyeball.

  The curvature center calculation unit 353 calculates the corneal curvature center from a straight line connecting the virtual light source position and the corneal reflection center. For example, the curvature center calculation unit 353 calculates a position on the straight line where the distance from the corneal reflection center is a predetermined value as the corneal curvature center. As the predetermined value, a value determined in advance from a general radius of curvature of the cornea or the like can be used.

  Since there may be individual differences in the radius of curvature of the cornea, there is a possibility that an error will increase if the center of corneal curvature is calculated using a predetermined value. Therefore, the curvature center calculation unit 353 may calculate the corneal curvature center in consideration of individual differences. In this case, the curvature center calculation unit 353 first uses the pupil center and the corneal reflection center calculated when the target position is watched by the subject, the straight line connecting the pupil center and the target position, the corneal reflection center, and the virtual center. The intersection of the straight line connecting the light source positions is calculated. Then, the curvature center calculation unit 353 calculates the distance between the pupil center and the calculated intersection, and stores it in the storage unit 150, for example.

  The target position may be a position that is determined in advance and can calculate a three-dimensional world coordinate value. For example, the center position of the display screen 201 (the origin of the three-dimensional world coordinates) can be set as the target position. In this case, for example, the output control unit 356 displays an image (target image) or the like that causes the subject to gaze at the target position (center) on the display screen 201. Thereby, a test subject can be made to gaze at a target position.

  The target image may be any image as long as it can attract the attention of the subject. For example, an image in which a display mode such as luminance or color changes, an image in which the display mode is different from other regions, or the like can be used as the target image.

  The target position is not limited to the center of the display screen 201, and may be an arbitrary position. If the center of the display screen 201 is set as the target position, the distance from an arbitrary end of the display screen 201 is minimized. For this reason, it becomes possible to make the measurement error at the time of gaze detection smaller, for example.

  The processing up to the calculation of the distance is executed in advance, for example, before the actual gaze detection is started. At the time of actual line-of-sight detection, the curvature center calculation unit 353 calculates, on the straight line connecting the virtual light source position and the corneal reflection center, a position where the distance from the pupil center is a previously calculated distance as the corneal curvature center. . The curvature center calculation unit 353 corresponds to a calculation unit that calculates the corneal curvature center from the virtual light source position, a predetermined position indicating the target image on the display unit, the pupil center position, and the corneal reflection center position. .

  The gaze detection unit 354 detects the gaze of the subject from the pupil center and the corneal curvature center. For example, the gaze detection unit 354 detects the direction from the corneal curvature center to the pupil center as the gaze direction of the subject.

  The viewpoint detection unit 355 detects the viewpoint of the subject using the detected gaze direction. The viewpoint detection unit 355 detects, for example, a viewpoint (gaze point) that is a point on the display screen 201 where the subject gazes. The viewpoint detection unit 355 detects, for example, the intersection of the line-of-sight vector represented in the three-dimensional world coordinate system as shown in FIG. 2 and the XY plane as the gaze point of the subject.

  The output control unit 356 controls the output of various information to the display unit 101, the speaker 205, and the like. For example, the output control unit 356 causes the target image to be output to the target position on the display unit 101. Further, the output control unit 356 controls the output to the display unit 101 such as a diagnostic image and an evaluation result by the evaluation unit 357.

  The diagnostic image may be an image corresponding to the evaluation process based on the line-of-sight (viewpoint) detection result. For example, in the case of diagnosing a developmental disorder, a diagnostic image including an image (such as a geometric pattern image) preferred by a subject with a developmental disorder and other images (such as a person image) may be used. Moreover, as a diagnostic image used in autism spectrum disease (ASD), it is an image including at least a human face.

  The evaluation unit 357 performs an evaluation process based on the diagnostic image and the gazing point detected by the viewpoint detection unit 355. For example, in the case of diagnosing a developmental disability, the evaluation unit 357 analyzes the diagnostic image and the gazing point, and evaluates whether or not the image portion preferred by the subject with the developmental disability has been gazed.

  FIG. 7 is a diagram for explaining an outline of processing when it is assumed that one light source is used. The elements described in FIGS. 3 to 6 are denoted by the same reference numerals and description thereof is omitted. In the example of FIG. 7, one LED light source 203 is used instead of the two LED light sources 103a and 103b.

  The pupil center 407 and the corneal reflection center 408 respectively represent the center of the pupil and the center of the corneal reflection point detected when one LED light source 203 is turned on. The corneal curvature radius 409 represents the distance from the corneal surface to the corneal curvature center 410. Here, the LED light source 203 is a single LED, but it may be a combination of several small LEDs arranged in one place.

  FIG. 8 is a diagram illustrating an outline of processing executed by the diagnosis support apparatus 100 according to the present embodiment. The elements described in FIGS. 3 to 6 are denoted by the same reference numerals and description thereof is omitted.

  A corneal reflection point 621 represents a corneal reflection point on the image taken by the left camera 102b. A corneal reflection point 622 represents a corneal reflection point on an image taken by the right camera 102a. In the present embodiment, the right camera 102a and the right camera LED light source 103b, and the left camera 102b and the left camera LED light source 103a are, for example, left and right with respect to a straight line passing through an intermediate position between the right camera 102a and the left camera 102b. Symmetric positional relationship. For this reason, it can be considered that the virtual light source 303 exists at an intermediate position (virtual light source position) between the right camera 102a and the left camera 102b. A corneal reflection point 624 represents a corneal reflection point corresponding to the virtual light source 303. The world coordinate value of the cornea reflection point 624 is calculated by converting the coordinate value of the cornea reflection point 621 and the coordinate value of the cornea reflection point 622 using a conversion parameter for converting the coordinate value of the left and right cameras into the three-dimensional world coordinates. it can. A center of curvature 505 of the cornea exists on a straight line 523 connecting the virtual light source 303 and the cornea reflection point 624. Therefore, the viewpoint can be detected by a method equivalent to the method of detecting the line of sight shown in FIG.

  The positional relationship between the right camera 102a and the left camera 102b and the positional relationship between the LED light source 103a and the LED light source 103b are not limited to the above-described positional relationship. For example, the positional relationship may be symmetrical with respect to the same straight line, and the right camera 102a, the left camera 102b, the LED light source 103a, and the LED light source 103b may not be on the same straight line. Good.

  FIG. 9 is a diagram for explaining calculation processing for calculating the corneal curvature center position and the distance between the pupil center position and the corneal curvature center position before performing viewpoint detection (line-of-sight detection). The elements described in FIGS. 3 to 6 are denoted by the same reference numerals and description thereof is omitted.

  The target position 605 is a position for displaying a target image or the like at one point on the display unit 101 and causing the subject to stare. In this embodiment, the center position of the screen of the display unit 101 is set. A straight line 613 is a straight line connecting the virtual light source 303 and the corneal reflection center 612. A straight line 614 is a straight line connecting the target position 605 (gaze point) that the subject looks at and the pupil center 611. A corneal curvature center 615 is an intersection of the straight line 613 and the straight line 614. The curvature center calculation unit 353 calculates and stores the distance 616 between the pupil center 611 and the corneal curvature center 615.

  FIG. 10 is a flowchart illustrating an example of calculation processing according to the present embodiment.

  First, the output control unit 356 reproduces the target image at one point on the screen of the display unit 101 (step S101), and causes the subject to gaze at the one point. Next, the lighting control unit 351 uses the LED drive control unit 316 to light one of the LED light sources 103a and 103b toward the eyes of the subject (step S102). The control unit 300 images the eyes of the subject with the camera having the longer distance from the lit LED light source among the left and right cameras (the right camera 102a and the left camera 102b) (step S103). Next, the lighting control unit 351 lights the other of the LED light sources 103a and 103b toward the eyes of the subject (step S104). The control unit 300 images the eyes of the subject with the camera having the longer distance from the lit LED light source among the left and right cameras (step S105).

  Note that it is not necessary to stop imaging by a camera other than the camera having a long distance from the lit LED light source. That is, it is only necessary that the subject's eyes are imaged with a camera having a longer distance from the lit LED light source and the captured image can be used for coordinate calculation or the like.

  By irradiation with the LED light source 103a or the LED light source 103b, the pupil part (pupil region) is detected as a dark part (dark pupil). Further, a corneal reflection virtual image is generated as a reflection of LED irradiation, and a corneal reflection point (corneal reflection center) is detected as a bright portion. That is, the position detection unit 352 detects the pupil portion from the captured image, and calculates coordinates indicating the position of the pupil center. The position detection unit 352 detects, for example, a region below a predetermined brightness including the darkest part in a certain region including the eyes as a pupil part, and a region above the predetermined brightness including the brightest part as corneal reflection. To detect. In addition, the position detection unit 352 detects a corneal reflection portion (corneal reflection region) from the captured image, and calculates coordinates indicating the position of the corneal reflection center. Note that the position detection unit 352 calculates each coordinate value for each of the two images acquired by the left and right cameras (step S106).

  Note that the left and right cameras are pre-calibrated with a stereo calibration method to obtain three-dimensional world coordinates, and conversion parameters are calculated. As the stereo calibration method, any conventionally used method such as a method using Tsai's camera calibration theory can be applied.

  Using this conversion parameter, the position detection unit 352 converts the coordinates of the left and right cameras into the three-dimensional world coordinates of the pupil center and the corneal reflection center (step S107). For example, the position detection unit 352 uses the coordinates obtained from the image captured by the left camera 102b when the LED light source 103a is turned on as the coordinates of the left camera, and images by the right camera 102a when the LED light source 103b is turned on. Using coordinates obtained from the image as coordinates of the right camera, conversion to three-dimensional world coordinates is performed using conversion parameters. The world coordinate value obtained as a result corresponds to the world coordinate value obtained from the image captured by the left and right cameras when it is assumed that light is emitted from the virtual light source 303. The curvature center calculation unit 353 obtains a straight line connecting the obtained world coordinates of the corneal reflection center and the world coordinates of the center position of the virtual light source 303 (step S108). Next, the curvature center calculation unit 353 calculates a straight line connecting the world coordinates of the center of the target image displayed at one point on the screen of the display unit 101 and the world coordinates of the pupil center (step S109). The curvature center calculation unit 353 obtains an intersection between the straight line calculated in step S108 and the straight line calculated in step S109, and sets the intersection as the corneal curvature center (step S110). The curvature center calculation unit 353 calculates the distance between the pupil center and the corneal curvature center at this time, and stores it in the storage unit 150 or the like (step S111). The stored distance is used to calculate the corneal curvature center at the time of subsequent detection of the viewpoint (line of sight).

  The distance between the pupil center and the corneal curvature center when looking at one point on the display unit 101 in the calculation process is kept constant within a range in which the viewpoint in the display unit 101 is detected. The distance between the center of the pupil and the center of corneal curvature may be obtained from the average of all the values calculated during playback of the target image, or from the average of several values of the values calculated during playback. You may ask for it.

  FIG. 11 is a diagram illustrating a method of calculating the corrected position of the corneal curvature center using the distance between the pupil center and the corneal curvature center that is obtained in advance when performing viewpoint detection. A gazing point 805 represents a gazing point obtained from a corneal curvature center calculated using a general curvature radius value. A gazing point 806 represents a gazing point obtained from a corneal curvature center calculated using a distance obtained in advance.

  The pupil center 811 and the corneal reflection center 812 indicate the position of the pupil center and the position of the corneal reflection center calculated at the time of viewpoint detection, respectively. A straight line 813 is a straight line connecting the virtual light source 303 and the corneal reflection center 812. The corneal curvature center 814 is the position of the corneal curvature center calculated from a general curvature radius value. The distance 815 is the distance between the pupil center and the corneal curvature center calculated by the prior calculation process. The corneal curvature center 816 is the position of the corneal curvature center calculated using the distance obtained in advance. The corneal curvature center 816 is obtained from the fact that the corneal curvature center exists on the straight line 813 and the distance between the pupil center and the corneal curvature center is the distance 815. Thus, the line of sight 817 calculated when a general radius of curvature value is used is corrected to the line of sight 818. Also, the gazing point on the screen of the display unit 101 is corrected from the gazing point 805 to the gazing point 806.

  FIG. 12 is a flowchart illustrating an example of a line-of-sight detection process according to the present embodiment. For example, the line-of-sight detection process of FIG. 12 can be executed as a process of detecting the line of sight in the diagnostic process using the diagnostic image. In the diagnosis process, in addition to the steps in FIG. 12, a process of displaying a diagnostic image, an evaluation process by the evaluation unit 357 using the detection result of the gazing point, and the like are executed.

  Steps S201 to S207 are the same as steps S102 to S108 in FIG.

  The curvature center calculation unit 353 calculates, as the corneal curvature center, a position that is on the straight line calculated in step S207 and whose distance from the pupil center is equal to the distance obtained by the previous calculation process (step S208).

  The line-of-sight detection unit 354 determines a vector (line-of-sight vector) connecting the pupil center and the corneal curvature center (step S209). This vector indicates the line-of-sight direction viewed by the subject. The viewpoint detection unit 355 calculates the three-dimensional world coordinate value of the intersection between the line-of-sight direction and the screen of the display unit 101 (step S210). This value is a coordinate value representing one point on the display unit 101 that the subject gazes in world coordinates. The viewpoint detection unit 355 converts the obtained three-dimensional world coordinate value into a coordinate value (x, y) represented in the two-dimensional coordinate system of the display unit 101 (step S211). Thereby, the viewpoint (gaze point) on the display part 101 which a test subject looks at can be calculated.

(Modification)
The calculation process for calculating the distance between the pupil center position and the corneal curvature center position is not limited to the method described with reference to FIGS. Hereinafter, another example of the calculation process will be described with reference to FIGS. 13 and 14.

  FIG. 13 is a diagram for explaining the calculation process of the present modification. The elements described in FIGS. 3 to 6 and FIG. 9 are denoted by the same reference numerals and description thereof is omitted.

  A line segment 1101 is a line segment connecting the target position 605 and the virtual light source position. A line segment 1102 is a line segment that is parallel to the line segment 1101 and connects the pupil center 611 and the straight line 613. In this modification, the distance 616 between the pupil center 611 and the corneal curvature center 615 is calculated and stored using the line segment 1101 and the line segment 1102 as follows.

  FIG. 14 is a flowchart illustrating an example of calculation processing according to the present modification.

  Steps S301 to S309 are the same as steps S101 to S109 in FIG.

  The curvature center calculation unit 353 calculates a line segment (the line segment 1101 in FIG. 13) that connects the center of the target image displayed at one point on the screen of the display unit 101 and the virtual light source position, and the calculated line. The length of minutes (referred to as L1101) is calculated (step S310).

  The curvature center calculation unit 353 calculates a line segment (line segment 1102 in FIG. 13) that passes through the pupil center 611 and is parallel to the line segment calculated in step S310, and calculates the length of the calculated line segment (L1102). Is calculated (step S311).

  The curvature center calculation unit 353 includes a triangle having the corneal curvature center 615 as a vertex and the line segment calculated in step S310 as a lower side, and a triangle having the corneal curvature center 615 as a vertex and the line segment calculated in step S311 as a lower side. Is a similar relationship, the distance 616 between the pupil center 611 and the corneal curvature center 615 is calculated (step S312). For example, the curvature center calculation unit 353 makes the ratio of the length of the line segment 1102 to the length of the line segment 1101 equal to the ratio of the distance 616 to the distance between the target position 605 and the corneal curvature center 615. The distance 616 is calculated.

The distance 616 can be calculated by the following equation (1). L614 is the distance from the target position 605 to the pupil center 611.
Distance 616 = (L614 × L1102) / (L1101−L1102) (1)

  The curvature center calculation unit 353 stores the calculated distance 616 in the storage unit 150 or the like (step S313). The stored distance is used to calculate the corneal curvature center at the time of subsequent detection of the viewpoint (line of sight).

  Next, with reference to FIGS. 15 to 21, processing for specifying a pupil region, which is a target region for which the position detection unit 352 detects a pupil, will be described. As described above, the position detection unit 352 detects the center of the pupil from the detected pupil region, and detects the position of corneal reflection. FIG. 15 is a diagram illustrating an outline of functions of the position detection unit. FIG. 16 is a diagram illustrating an example of a classifier stored in a database. FIG. 17 is a flowchart illustrating an example of processing of the position detection unit. FIG. 18 is a diagram for explaining the processing of the position detection unit. 19 to 21 are diagrams for explaining the pattern matching processing of the position detection unit.

  The position detection unit 352 includes a pattern matching unit 370, a pupil detection unit 372, and a database 374. Here, the image to be processed by the position detection unit 352 is preferably a monochrome image. The pattern matching unit 370 performs matching between the pupil and the pattern corresponding to the luminance change around the pupil and the image, and extracts candidate positions of the pupil. The pattern matching unit 370 includes a pattern control unit 380 and a comparison unit 382. The pattern control unit 380 selects a pattern (identifier) to be used and moves the pattern on the image. The comparison unit 382 performs extraction processing using a plurality of patterns, and compares the candidates extracted by the extraction processing. The pattern matching unit 370 identifies one pattern as the extraction process. The pattern matching unit 370 compares the identified pattern with each region of the image, and extracts a region that matches the luminance distribution of the pattern. The pattern matching unit 370 determines the priority order of the extracted region when it matches the pattern based on the result of the extraction process with a plurality of patterns, and sets the candidate with the higher priority order as the candidate position of the pupil.

  The pupil detection unit 372 analyzes the candidate position of the pupil extracted by the pattern matching unit 370 and detects whether there is a pupil. The pupil detection unit 372 detects a position where the presence of the pupil boundary and corneal reflection can be confirmed from the image of the candidate position of the pupil as a pupil.

  The database 374 stores patterns (identifiers) used for pattern matching by the pattern matching unit 370. As shown in FIG. 16, the database 374 of the present embodiment includes a first discriminator 220, a second discriminator 222, a third discriminator 224, a fourth discriminator 226, a fifth discriminator 228, a sixth discriminator 230, Eight classifiers of the seventh classifier 232 and the eighth classifier 234 are stored. The discriminator has a first area and a second area. In the discriminator, the first area is associated with the first luminance, and the second area is associated with the second luminance. The discriminator is set so that the first luminance is brighter than the second luminance, and the distribution of the first luminance and the second luminance of the pattern (discriminator) matches the luminance distribution of each region of the image. Extracted as a candidate position of the pupil. The first luminance and the second luminance are average luminances of the respective areas. For example, when the first luminance that is the average luminance of the image that is superimposed on the first region of the pattern is higher than the second luminance that is the average luminance of the image that is superimposed on the second region of the pattern, the candidate position of the pupil Extract as

  The eight discriminators shown in FIG. 16 are superimposed on the image in a preset direction. The first discriminator 220 has a first area 250a and a second area 250b. In the first discriminator 220, the first area 250a and the second area 250b are separated in the vertical direction of the image. In the first discriminator 220, the first area 250a is arranged above the second area 250b in the image vertical direction. The first discriminator 220 is a square pattern with one side of a pixel. The second discriminator 222 has a first area 252a and a second area 252b. In the second discriminator 222, the first region 252a and the second region 252b are separated in the left-right direction of the image. In the second discriminator 222, the first area 252a is disposed on the right side of the second area 252b in the horizontal direction of the image.

  The third discriminator 224 has first regions 254a, 254b and a second region 250c. In the third discriminator 224, the first areas 254a, 254b and the second area 250c are separated in the vertical direction of the image. In the third discriminator 224, the first region 254a is disposed above the second region 254c in the image vertical direction, and the first region 254b is disposed below the second region 254c in the image vertical direction. In the third discriminator 224, the length of the first area 254a, 254b in the vertical direction of the image is shorter than that of the second area 254c.

  The fourth discriminator 226 has first regions 256a, 256b and a second region 256c. In the fourth discriminator 226, the first areas 256a, 256b and the second area 256c are separated in the vertical direction of the image. In the fourth discriminator 226, the first region 256a is disposed above the second region 256c in the image vertical direction, and the first region 256b is disposed below the second region 256c in the image vertical direction. In the fourth discriminator 226, the lengths of the first regions 256a and 256b in the vertical direction of the image are longer than those of the second region 256c.

  The fifth discriminator 228 has first regions 258a, 258b and a second region 258c. In the fifth discriminator 228, the first regions 258a, 258b and the second region 258c are separated in the left-right direction of the image. In the fifth discriminator 228, the first area 258a is arranged on the left side of the second area 258c in the horizontal direction of the image, and the first area 258b is arranged on the right side of the second area 258c in the horizontal direction of the image. In the fifth discriminator 228, the length of the first region 258a, 258b in the left-right direction of the image is shorter than that of the second region 258c.

  The sixth discriminator 230 has first regions 260a and 260b and a second region 260c. In the sixth discriminator 230, the first areas 260a and 260b and the second area 260c are separated in the horizontal direction of the image. In the sixth discriminator 230, the first area 260a is arranged on the left side of the second area 260c in the horizontal direction of the image, and the first area 260b is arranged on the right side of the second area 260c in the horizontal direction of the image. In the sixth discriminator 230, the length of the first regions 260a and 260b in the left-right direction of the image is longer than that of the second region 260c.

  The seventh discriminator 232 has a first area 262a and a second area 262b. The seventh discriminator 232 is a quadrangle, and the first region 262a and the second region 262b are separated with a diagonal line of the quadrangle as a boundary. In the seventh discriminator 232, the first area 262a is arranged on the upper left side of the image of the second area 262b.

  The eighth discriminator 234 has a first area 264a and a second area 264b. The eighth discriminator 234 is a quadrangle, and the first region 264a and the second region 264b are separated from each other with a quadrilateral diagonal as a boundary. In the eighth discriminator 234, the first area 264a is arranged on the upper left side of the image of the second area 264b.

  Next, an example of processing executed on the image by the position detection unit 352 will be described with reference to FIGS. 17 and 18. FIG. 17 is a flowchart showing the flow of processing. FIG. 18 is a diagram schematically illustrating processing for a screen. The position detection unit 352 determines a pattern (identifier) used for pattern matching (step S401). Next, the position detection unit 352 performs pattern matching with the determined pattern (step S402). Specifically, as shown in FIGS. 18 and 19, the target area 291 of the image 290 is specified, the second classifier 222 is overlaid on the specified target area 291, and the second classifier 222 and the image 290 overlap. Pattern matching is performed with the position. Here, the image 290 is an image having b pixels in the vertical direction and c pixels in the horizontal direction. For example, if a of the a pixel of the first discriminator 220 is 64 pixels, the b pixel of the image 290 is 480 pixels and the c pixel is 752 pixels.

  The position detection unit 352 determines whether all the patterns have been processed (step S403). If the position detection unit 352 determines that all the patterns have not been processed (No in step S403), the position detection unit 352 returns to step S401, and performs the processes from step S401 to step S402 on the pattern that has not been subjected to pattern matching. To do. The position detection unit 352 sequentially performs pattern matching with eight patterns on the target region.

  If the position detection unit 352 determines that all the patterns have been processed (Yes in step S403), in this embodiment, if it is determined that pattern matching has been performed with eight discriminators, the entire area of the image is detected. Is determined (step S404).

  If it is determined that the entire area of the image has not been detected (No in step S404), the position detection unit 352 moves the target area for pattern matching (step S405), returns to step S401, and moves to the moved target area. On the other hand, the processing from step S401 to step S404 is performed.

  The position detection unit 352 moves the target region 290 to be subjected to pattern matching with a pattern in the left-right direction of the image. When the position detection unit 352 performs matching from one end to the other end in the left-right direction of the image 290 as indicated by the symmetry region 291a, the position detection unit 352 moves the image 290 downward as in the target region 291b. Thereby, the position detection unit 352 overlaps each position of the image 290 with the first discriminator 220. Note that the amount of movement of the target region may be one pixel, but is preferably several pixels, for example, eight pixels. By moving the target area by a plurality of pixels, the processing can be reduced. That is, as shown in FIG. 18, when the target area is determined, the processing from step S401 to step S405, that is, the pattern matching is sequentially performed by the eight discriminators and the target area is moved by several pixels. Next, the processing from step S401 to step S405 is also performed on the moved target region, that is, the pattern matching is performed in order by the eight discriminators, and the target region is moved by several pixels. Pattern matching is performed on the entire image while moving the target area 291 on the screen.

  If it is determined that the entire area of the image has been detected (Yes in step S404), the position detection unit 352 determines the candidate position of the analysis target pupil from the candidate position of the pupil extracted in each pattern (step S406). Specifically, the position detection unit 352 determines a position that has become a candidate position for a pupil with more patterns as a candidate position for a pupil having a higher priority based on the pattern matching results of a plurality of discriminators. As shown in FIG. 20, the position detection unit 352 extracts the region 292 as a pupil candidate position by pattern matching of the fourth discriminator 226 (step S501), and the region 292 of the pupil by pattern matching of the first discriminator 220. When extracted as a candidate position (step S502) and the region 292 is not extracted as a candidate position of the pupil by pattern matching of the seventh discriminator 232 (step S503), the candidate position of the pupil is not set as an analysis target. As shown in FIG. 21, the position detector 352 extracts the region 294 as a candidate pupil position by pattern matching of the fourth discriminator 226 (step S601), and the region 294 of the pupil by pattern matching of the first discriminator 220. When extracted as a candidate position (step S602) and the region 294 is extracted as a candidate position of the pupil by pattern matching of the seventh discriminator 232 (step S603), it is set as an analysis target as a candidate position of the pupil. Thereby, as shown in an image 290a in FIG. 18, a plurality of candidate positions 295 of the pupil are extracted from the image. The candidate pupil position 295 may be included in a region other than the pupil.

  When the candidate position of the pupil to be analyzed is determined, the position detection unit 352 performs masking on a region other than the candidate position of the pupil (Step S407). Specifically, as shown in the screen 290b of FIG. 18, a process of covering the area other than the candidate pupil position 295 with a mask, specifically, the area other than the candidate pupil position 295 is made the same color pixel. After performing the masking process, the position detection unit 352 analyzes the candidate position of the pupil with the pupil detection unit 372 for the masking process, and detects the pupil (step S408). Specifically, a masking process is performed as shown in FIG. 18, and a process for detecting a pupil is performed on an image 290c in which only the image at the candidate position 295 of the pupil remains. As an example, the darkest position in the image 290c is detected, and it is determined whether the dark position satisfies the pupil condition. By performing the process of detecting the pupil, the position 296 of the pupil is detected as shown in the image 290d. In the image 290d, since there is one person in the image 290, both eyes of one person are detected as the pupil position 296.

  As described above, the position detection unit 352 uses the pattern matching unit 370 to identify candidate positions of the pupil from the luminance distribution of the region, and the pupil detection unit detects the pupil only at the identified candidate position, thereby detecting the pupil. It is possible to reduce the processing amount required for the process. In the present embodiment, since pattern matching is performed only by comparing the brightness of each region of the pattern, an increase in the processing amount in pattern matching can be suppressed. In addition, pattern matching is performed using a plurality of patterns, and based on the result, priority is determined, and a position having a high priority is set as a candidate position of the pupil, so that pupil detection omission can be suppressed. . In the present embodiment, the pattern of eight discriminators is described, but the number and pattern are not limited to the above. Further, the pattern matching process is not limited to the above-described process, and it is determined whether the image superimposed on the first area of the pattern has the first luminance, and the image superimposed on the second area of the pattern is the second. It is possible to determine whether or not the brightness is the same, and if both match, the candidate positions may be extracted.

  Further, when the pattern matching is executed, the position detection unit 352 may mask, for example, a black and white area where there is no luminance difference between the two areas and exclude it from the pattern matching processing target. Thereby, the processing amount can be further reduced. Further, the position detection unit 352 may perform processing from patterns that are easily detected by pattern matching, and after performing a predetermined number of patterns, may perform pattern matching only on candidate positions of the pupil.

  The position detection unit 352 can simplify processing by masking positions other than the candidate positions of the pupils extracted by the pattern matching unit 370. Note that the masking process may not be performed.

DESCRIPTION OF SYMBOLS 100 Diagnosis support apparatus 101 Display part 102a Right camera 102b Left camera 103a, 103b LED light source 150 Storage part 201 Display screen 205 Speaker 300 Control part 313 Drive / IF part 316 LED drive control part 322 Speaker drive part 351 Lighting control part 352 Position detection Unit 353 center of curvature calculation unit 354 gaze detection unit 355 viewpoint detection unit 356 output control unit 357 evaluation unit 370 pattern matching unit 372 pupil detection unit 374 database 380 pattern control unit 382 comparison unit

Claims (6)

An imaging unit that captures an image including a human pupil;
A control unit that extracts the pupil of the person from the image acquired by the imaging unit;
Have
The control unit performs matching between the image corresponding to a luminance change around the pupil and the pupil and the image, and a pattern matching unit that extracts the candidate position of the pupil;
A pupil detection unit that analyzes a candidate position of the pupil extracted by the pattern matching unit and detects whether there is a pupil; The pattern matching unit further includes a comparison unit that performs extraction processing using a plurality of patterns and compares candidates extracted using a plurality of patterns,
The pupil detection device according to claim 1, wherein the pattern includes a first region having a first luminance and a second region having a second luminance lower than the first luminance.   The extraction process extracts one pattern, reads a luminance distribution area of the extracted pattern, performs matching with the image, determines a priority order based on a result of matching a plurality of patterns, The pupil detection apparatus according to claim 1, wherein a candidate having a high priority is set as a candidate position of the pupil. The control unit further includes a masking unit that masks positions other than the candidate positions of the pupil extracted by the pattern matching unit,
The pupil detection device according to claim 1, wherein the pupil detection unit processes an image masked by the masking unit.   The said pupil detection part detects the position which confirmed the presence of the boundary of a pupil and corneal reflection from the image of the said candidate position of a pupil as said pupil. Pupil detection device. A pattern matching step of performing matching between an image including a human pupil and a pattern corresponding to a change in luminance around the pupil and the pupil, and extracting candidate positions of the pupil;
A pupil detection method, comprising: analyzing a candidate position of the pupil extracted in the pattern matching step and detecting whether or not there is a pupil.