JP6217446B2 - Control device, diagnosis support device, control method, and control program - Google Patents

Description

  The present invention relates to a technique for supporting diagnosis.

  Recently, people with developmental disabilities are said to be on the rise. It is known that developmental disabilities are found early and start treatment to reduce symptoms and increase the effect of adapting to society. In Japan, we are aiming for early detection through interviews at the age of 1 and a half. However, there are problems such as shortage of psychiatrists and time-consuming interviews, and the effect is not sufficient. Therefore, there is a need for an objective and efficient diagnosis support device for developmental disabilities.

  For early detection of developmental disabilities, it is ideal to be able to diagnose at the age of 1 and a half years. One of the characteristics of children with developmental disabilities is the tendency to see geometric patterns more than people. There has been proposed a method for diagnosing developmental disability by applying a method of detecting a gazing point by photographing a human face with a camera and calculating a corneal reflection and a pupil position.

JP 2005-185431 A JP 2008-125619 A JP 2005-198743 A JP 2011-206542 A

Karen Pierce, “Preference for Geometric Patterns Early in Life as a Risk Factor for Autism”, Arch Gen Psychiatry 2011; 68 (1): 101-109.

  However, in the conventional method for detecting a gazing point, there are cases where diagnosis cannot be appropriately supported depending on the diagnostic image presented to the subject, and a more accurate detection method has been demanded.

In order to solve the above problems, a control device according to an aspect of the present invention displays a diagnostic image obtained by combining a natural image and a pattern image on a display unit, and displays the diagnostic image on the first diagnostic image display unit. After the first diagnostic image is displayed, an output control unit is provided that displays on the display unit a second diagnostic image in which a natural image portion and a pattern image are similar to the first diagnostic image.
Another aspect of the present invention is a diagnosis support apparatus, the output control unit, an imaging unit that images the subject, a gaze detection unit that detects a gaze direction of the subject from a captured image captured by the imaging unit, and a gaze And a viewpoint detection unit that detects the viewpoint of the subject in the display area of the display unit based on the direction.
Another aspect of the present invention is a control method, in a control method for controlling the display of a diagnostic image combining a natural image and a pattern image, after displaying the first diagnostic image, compared to the first diagnostic image. A step of displaying a second diagnostic image in which the natural image portion and the pattern image are similar to each other on the display unit;
It should be noted that any combination of the above-described constituent elements and a representation of the present invention converted between a method, an apparatus, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, it is possible to improve the accuracy of diagnosis.

It is a figure which shows an example of arrangement | positioning of the display part used in 1st Embodiment, a stereo camera, and a light source. It is a figure which shows the outline | summary of the function of the diagnosis assistance apparatus of 1st Embodiment. It is a block diagram which shows an example of the detailed function of each part shown in FIG. It is a figure which shows an example of the relationship between the light emission timing of an infrared light source, and the imaging timing of a left-right camera. It is a figure which shows an example of the detection of the eyes and distance at the time of using two cameras. It is a figure which shows an example of the diagnostic image in 1st Embodiment. It is a figure which shows an example of the diagnostic image in 1st Embodiment. It is a figure which shows an example of the diagnostic image in 1st Embodiment. It is a figure which shows an example of the diagnostic image in 1st Embodiment. It is a figure which shows an example of the diagnostic image in 1st Embodiment. It is a figure which shows an example of the time chart in the case of displaying the diagnostic image | video in 1st Embodiment. It is a figure which shows the ratio of the gaze point position of the test subject of a fixed form development in 1st Embodiment. It is a figure which shows the ratio of the gaze point position of the test subject of a developmental disorder in 1st Embodiment. It is an example of the diagnostic image in 1st Embodiment. It is an example of the diagnostic image in 1st Embodiment. It is a flowchart which shows an example of the diagnostic assistance process in 1st Embodiment. It is a flowchart which shows an example of the diagnostic assistance process in 1st Embodiment. It is a flowchart which shows an example of the diagnostic assistance process in 1st Embodiment. FIG. 19 is a diagram illustrating an example of the arrangement of the display unit, the stereo camera, the infrared light source, and the subject according to the second embodiment. FIG. 20 is a diagram illustrating an example of the arrangement of the display unit, the stereo camera, the infrared light source, and the subject according to the second embodiment. FIG. 21 is a diagram illustrating an outline of functions of the diagnosis support apparatus. FIG. 22 is a block diagram illustrating an example of detailed functions of the respective units illustrated in FIG. FIG. 23 is a diagram illustrating an overview of processing executed by the diagnosis support apparatus according to the second embodiment. FIG. 24 is an explanatory diagram showing the difference between the method using two light sources and the second embodiment using one light source. FIG. 25 is a diagram for explaining calculation processing for calculating the distance between the pupil center position and the corneal curvature center position. FIG. 26 is a flowchart illustrating an example of calculation processing according to the second embodiment. FIG. 27 is a diagram illustrating a method of calculating the position of the corneal curvature center using the distance obtained in advance. FIG. 28 is a flowchart illustrating an example of a line-of-sight detection process according to the second embodiment.

  Hereinafter, a control device, a diagnosis support device, a control method, and a control program of the present invention will be described with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a diagram illustrating an example of an arrangement of a display unit, a stereo camera, and a light source used in the first embodiment. As shown in FIG. 1, in this embodiment, a set of stereo cameras 102 is arranged below the display screen 101. The stereo camera 102 is an imaging unit that can perform stereo shooting with infrared rays, and includes a right camera 202 and a left camera 204.

  Infrared LED (Light Emitting Diode) light sources 203 and 205 are arranged in the circumferential direction immediately before the lenses of the right camera 202 and the left camera 204, respectively. The infrared LED light sources 203 and 205 include an inner peripheral LED and an outer peripheral LED having different wavelengths for emitting light. The pupils of the subject are detected by the infrared LED light sources 203 and 205. As a pupil detection method, for example, the method described in Patent Document 2 can be applied.

  When detecting the line of sight, the space is expressed by coordinates to identify the position. In this embodiment, the center position of the display screen 101 is the origin, the top and bottom are the Y coordinate (up is +), the side is the X coordinate (right is +), and the depth is the Z coordinate (front is +). .

  FIG. 2 is a diagram illustrating an outline of functions of the diagnosis support apparatus 100. FIG. 2 shows a part of the configuration shown in FIG. 1 and a configuration used for driving the configuration. As illustrated in FIG. 2, the diagnosis support apparatus 100 includes a right camera 202, a left camera 204, infrared LED light sources 203 and 205, a speaker 105, a drive / IF (InterFace) unit 208, and a control unit 300. A storage unit 150 and a display unit 210. In FIG. 2, the display screen 101 shows the positional relationship between the right camera 202 and the left camera 204 in an easy-to-understand manner, but the display screen 101 is a screen displayed on the display unit 210. The drive unit and the IF unit may be integrated or separate.

  The speaker 105 functions as an audio output unit that outputs an audio or the like for alerting the subject during calibration.

  The drive / IF unit 208 drives each unit included in the stereo camera 102. The drive / IF unit 208 serves as an interface between each unit included in the stereo camera 102 and the control unit 300.

  The storage unit 150 stores various information such as a control program, a measurement result, and a diagnosis support result. The storage unit 150 stores, for example, an image to be displayed on the display unit 210. The display unit 210 displays various information such as a target image for diagnosis.

  FIG. 3 is a block diagram illustrating an example of detailed functions of the respective units illustrated in FIG. As shown in FIG. 3, a display unit 210 and a drive / IF unit 208 are connected to the control unit 300 (control device). The drive / IF unit 208 includes camera IFs 314 and 315, an LED drive control unit 316, and a speaker drive unit 322.

  A right camera 202 and a left camera 204 are connected to the drive / IF unit 208 via camera IFs 314 and 315, respectively. The driving / IF unit 208 drives these cameras to image the subject.

  FIG. 4 is a diagram illustrating an example of the relationship between the light emission timing of the infrared light source and the shooting timing of the left and right cameras. A frame synchronization signal FS is output from the right camera 202. The frame synchronization signal FS is input to the left camera 204 and the LED drive control unit 316. This causes the left and right wavelength 1 infrared light sources (wavelength 1-LED 303, wavelength 1-LED 305) to emit light at different timings in the first frame, and correspondingly by the left and right cameras (right camera 202, left camera 204). In the second frame, the left and right wavelength 2 infrared light sources (wavelength 2-LED 304, wavelength 2-LED 306) are caused to emit light in the second frame, and the images from the left and right cameras are captured correspondingly. Thereafter, the processing of the first frame and the second frame is repeated according to the frame synchronization signal FS.

  Returning to FIG. 3, the infrared LED light source 203 includes a wavelength 1 -LED 303 and a wavelength 2 -LED 304. The infrared LED light source 205 includes a wavelength 1-LED 305 and a wavelength 2-LED 306.

  Wavelength 1-LEDs 303 and 305 emit infrared light having wavelength 1. Wavelength 2-LEDs 304 and 306 irradiate wavelength 2 infrared rays.

  The wavelength 1 and the wavelength 2 are, for example, a wavelength of less than 900 nm and a wavelength of 900 nm or more, respectively. When the reflected light reflected by the pupil is irradiated with infrared light having a wavelength of less than 900 nm, a bright pupil image is obtained as compared with the case where reflected light reflected by the pupil is irradiated with infrared light having a wavelength of 900 nm or more. It is because it is obtained. In addition, about the wavelength of the infrared rays to irradiate, it is not restricted to the above, The result which imaged the reflected light reflected by the pupil by irradiating the infrared rays of wavelength 1, and the reflection reflected by the pupil after irradiating the infrared rays of wavelength 2 What is necessary is just to be able to make a difference with the result of imaging light.

  The speaker driving unit 322 drives the speaker 105. The diagnosis support apparatus 100 may include an interface (printer IF) for connecting to a printer as a printing unit. In addition, the printer may be provided inside the diagnosis support apparatus 100.

  The control unit 300 controls the entire diagnosis support apparatus 100. The control unit 300 includes a line-of-sight detection unit 351, a viewpoint detection unit 352, an output control unit 353, and an evaluation unit 354. The control unit 300 may be a control device such as a PC (Personal Computer).

  The gaze detection unit 351 detects the gaze (gaze direction) of the subject from the captured image captured by the imaging unit (stereo camera 102). The process for detecting the line of sight includes a process for detecting the eye position of the subject. The viewpoint detection unit 352 detects the viewpoint of the subject using the detected gaze direction. For example, the viewpoint detection unit 352 detects a viewpoint (gaze point) that is a point that the subject gazes out of the target image displayed on the display screen 101. As a gaze detection method by the gaze detection unit 351 and a viewpoint detection method by the viewpoint detection unit 352, any conventionally used method can be applied. Below, similarly to patent document 3, it demonstrates as an example the case where a test subject's gaze direction and a gaze point are detected using a stereo camera.

  In this case, the gaze detection unit 351 first detects the gaze direction of the subject from the image captured by the stereo camera 102. The gaze detection unit 351 detects the gaze direction of the subject using, for example, the methods described in Patent Documents 1 and 2. Specifically, the line-of-sight detection unit 351 obtains the difference between the image captured by irradiating the infrared light having the wavelength 1 and the image captured by irradiating the infrared light having the wavelength 2, and the image in which the pupil image is clarified. Generate. The line-of-sight detection unit 351 uses the two images generated as described above from the images captured by the left and right cameras (the right camera 202 and the left camera 204) and uses the stereo vision technique to determine the position of the subject's pupil ( Eye position) is calculated. The line-of-sight detection unit 351 calculates the position of the subject's corneal reflection using images taken by the left and right cameras. Then, the gaze detection unit 351 calculates a gaze vector representing the gaze direction of the subject from the position of the pupil of the subject and the corneal reflection position.

  In addition, the detection method of the eye position and the line of sight of the subject is not limited to this. For example, the eye position and line of sight of the subject may be detected by analyzing an image captured using visible light instead of infrared light.

  The viewpoint detection unit 352 detects, for example, the intersection of the line-of-sight vector represented in the coordinate system as shown in FIG. 1 and the XY plane as the gaze point of the subject. When the gaze direction of both eyes is obtained, the gaze point may be measured by obtaining the intersection of the left and right gazes of the subject.

  FIG. 5 is a diagram illustrating an example of eye and distance detection when two cameras (the right camera 202 and the left camera 204) are used. For the two cameras, a camera calibration theory based on a stereo calibration method is applied in advance to obtain camera parameters. As the stereo calibration method, any conventionally used method such as a method using Tsai's camera calibration theory can be applied. Using the eye position detected from the image captured by the right camera 202, the eye position detected from the image captured by the left camera 204, and the camera parameters, the three-dimensional coordinates of the eye in the world coordinate system are obtained. It is done. Thereby, the distance between the eyes and the stereo camera 102 and the pupil coordinates can be estimated. The pupil coordinate is a coordinate value representing the position of the subject's eye (pupil) on the XY plane. The pupil coordinates can be, for example, coordinate values obtained by projecting the eye position expressed in the world coordinate system onto the XY plane. Usually, the pupil coordinates of the left and right eyes are obtained. A diagnostic image 401 is displayed on the display screen 101. As will be described later, the diagnostic image 401 includes a natural image displayed in the display area of the display screen 101 and a pattern image including a pattern similar to the natural image displayed in a partial area included in the display area.

  The pattern image is an image (geometric image) including one or more geometric patterns, for example. The natural image may be a natural object or an image reminiscent of a natural object other than a geometric image. For example, an image (still image or moving image) obtained by capturing a person, an animal, a plant, a natural landscape, or the like with a camera may be used as the natural image. Moreover, you may use the image (still image, moving image) of the character imitating a person, an animal, etc. as a natural image.

  Returning to FIG. 3, the output control unit 353 controls the output of various information to the display unit 210, the speaker 105, and the like. For example, the output control unit 353 controls the output to the display unit 210 such as the diagnostic image 401 and the evaluation result by the evaluation unit 354. The output control unit 353 displays a plurality of diagnostic images. For example, the output control unit 353 displays the first diagnostic image on the display unit 210 and then displays the second diagnostic image in which the position, brightness, hue, saturation, and the like of the pattern image is different from the first diagnostic image. May be displayed. In this case, the viewpoint detection unit 352 detects the viewpoint of the subject when the first diagnostic image and the second diagnostic image are displayed. By using a plurality of diagnostic images, it is possible to detect a viewpoint with higher accuracy and support diagnosis.

  The evaluation unit 354 calculates an evaluation value as an index related to the degree of developmental disability based on the diagnostic image and the gazing point detected by the viewpoint detection unit 352. The evaluation unit 354, for example, based on the position of the gaze point of the subject when displaying a diagnostic image such as FIGS. Is calculated as an evaluation value, and an evaluation value indicating that the lower the evaluation value is, the higher the possibility of developmental disability is calculated. The evaluation unit 354 may calculate the evaluation value based on the diagnostic image and the gazing point, and the calculation method is not limited to the embodiment.

  6-10 is explanatory drawing which shows an example of the diagnostic image 401 (401a-401e) in 1st Embodiment. Hereinafter, an example in which a moving image is used as a diagnostic image will be described. The diagnostic image is not limited to a moving image, and for example, a plurality of still images may be used as the diagnostic image. In the present embodiment, a video in which a video of a person (person video) occupies most of the display area is used as a diagnostic image. Such images tend to be viewed by subjects with routine development. An image of a geometric pattern (geometric pattern image) is displayed in a partial region F (partial region) of this image. Geometric patterns tend to be favored by subjects with developmental disabilities. As a result of experiments, it has been confirmed that subjects with developmental disabilities particularly like viewing fractal images such as Julia sets and Mandelbrot sets. Subjects with developmental disabilities tend to prefer geometric pattern images, but may not be aware of a geometric pattern image if the person image and the geometric image are similar. Therefore, in the present embodiment, the geometric pattern image is conspicuous in the diagnostic image during the diagnosis start period so that a subject with developmental disabilities notices the geometric pattern image. Note that the similarity between the natural image (including the person image) portion and the geometric pattern image is, for example, that the color, brightness, saturation, and the like described later are similar.

  What is important here is that the image in the region F is conspicuous in the period immediately after the diagnosis is started (hereinafter also referred to as the diagnosis period A) with respect to surrounding human images. Since subjects with steady development have a strong tendency to view surrounding human images, a diagnosis period (diagnosis period A has ended) after a diagnosis has started even if attention is first drawn to the geometric pattern image In a later period (hereinafter sometimes referred to as diagnosis period B), there is a tendency to view surrounding human images. In this diagnosis period B, compared with the period immediately after the diagnosis is started, the geometrically-patterned image is made inconspicuous so that the subject with steady development tends to see the surrounding person image. On the other hand, subjects with developmental disabilities have a strong tendency to prefer geometric pattern images. Therefore, in order to notify that there is a geometric pattern image, it is made to stand out in the period immediately after the start of diagnosis. Even if the geometric pattern image is less noticeable than the surrounding human image by increasing the degree of similarity between the natural image and the geometric pattern image in the diagnosis period B by noticing the presence of the geometric pattern. Subjects with developmental disabilities have a strong tendency to continue to see geometric pattern images. Therefore, in the diagnosis period B, the difference in the ratio of viewing the geometric pattern image between the subject with steady development and the subject with developmental disability increases.

6 to 10 are obtained by extracting continuously changing diagnostic images in the time chart of FIG. FIG. 6 shows images at the start of display (0 seconds), FIG. 7 shows images after 2 seconds, FIG. 8 after 3 seconds, FIG. 9 after 6 seconds, and FIG. 10 after 8 seconds. Note that these times are examples. In the time chart of FIG. 11, the entire video is 8 seconds, and the difference between the natural image and the geometric pattern image in the diagnostic image is 3 seconds (diagnosis period A, FIGS. 6, 7, and 8) immediately after the diagnosis is started. The period is large (the degree of similarity is low), and the difference between the natural image and the geometric pattern image in the diagnostic image is small (similarity) in the latter 3 seconds to 8 seconds (diagnosis period B, FIGS. 8, 9, and 10). Period). The difference between the natural image and the geometric pattern image in the diagnostic image is large (the degree of similarity is low). For example, there may be a large difference in color, brightness, and saturation between the natural image and the geometric pattern image. . Regarding the difference between the natural image and the geometric pattern image, a part of the natural image may be compared with the geometric pattern image. Further, the change amount of the plurality of geometric pattern images included in the diagnosis period A may be made larger than the change amount of the plurality of geometric pattern images included in the diagnosis period B. As an example of this, if the geometric pattern image is compared with the previous geometric pattern image, the color, brightness, and saturation change greatly over time, or if the geometric pattern image is provided as a video, The amount of change (for example, rotation or the like) may be increased. As described above, in the display period (diagnosis period A) immediately after the diagnosis is started, the difference in color, brightness, and saturation between the natural image and the geometric pattern image is determined so that the subject can notice the geometric pattern image. Make the geometric pattern image stand out by enlarging it or increasing the amount of change in the geometric pattern image. On the other hand, in the diagnosis period B, which is a period after the diagnosis period A, the difference in color, brightness, and saturation between the natural image and the geometric pattern is made smaller (to increase the degree of similarity) compared to the diagnosis period A, or the diagnosis is performed. Compared with the period A, the amount of change is made small so that the geometric pattern image in the diagnostic image is not so noticeable. In the diagnosis period B, the difference between the natural image and the geometric pattern image is preferably within the following range. In the Munsell color system, if it is a hue, in the case of 20 colors of Munsell hue ring, it is within 3 colors adjacent to the color. For example, a similar range of red is reddish purple, reddish purple, purpledish red, yellowed red, yellowed red, reddish yellow. In the Munsell color system, the brightness difference (lightness) is within 3 brightness differences, and the saturation is within 6 saturation differences.
In the time chart of FIG. 11, the change amount of the diagnostic image is largely changed at 3 seconds, but may be gradually changed as time elapses.

  FIG. 12 is a diagram illustrating an example of the ratio of the gazing point position when a diagnostic image is viewed by the subject with a normal development. The upper part of FIG. 12 shows the ratio of the gazing point position of the subject in the diagnosis period A until 3 seconds elapse from the start of diagnosis. The lower part of FIG. 12 shows the ratio of the gaze point position of the subject in the diagnosis period B from 3 seconds to 8 seconds. In the diagnosis period A, the proportion of subjects (G1) who look at the geometric pattern is larger than the rate (N1) in which the subjects (natural images) are gazed at the diagnosis period A. N2) is larger than the ratio (G2) of viewing the geometric pattern.

  FIG. 13 is an example showing the ratio of the gazing point position when a diagnostic image is viewed by a subject with developmental disabilities. The upper part of FIG. 13 shows the ratio of the gazing point position of the subject in the diagnosis period A until 3 seconds elapse from the start of diagnosis. The lower part of FIG. 13 shows the ratio of the gazing point position of the subject from the elapse of 3 seconds to the elapse of 8 seconds in the diagnosis period B. In the diagnosis period A, subjects with developmental disabilities have a higher rate of watching the geometric pattern (G3), and in the diagnosis period B, the rate of watching the geometric pattern image (G4) is higher. In addition, the ratio (N3) of gazing at the person (natural image) in the diagnosis period A and the ratio (N4) of gazing at the person (natural image) in the diagnosis period B are both small.

  When comparing a subject with a typical development and a subject with developmental disabilities, the difference in the ratio of gaze at the geometric pattern during the diagnosis period B (the difference between G2 in FIG. 12 and G4 in FIG. 13) is large. Therefore, by examining the ratio of gazing at the geometric pattern in the diagnosis period B, it becomes one guideline for discriminating between subjects with a fixed development and those with developmental disabilities.

  FIG. 14 is an explanatory diagram regarding the coordinates of an image displayed on the display unit 210. For the sake of explanation, the upper left corner of the coordinates on the display unit 210 is the origin, the top and bottom are the Y coordinate (bottom is +), and the horizontal is the X coordinate (right is +). For convenience of explanation, it is different from the above-described world coordinates (spatial coordinates) for detecting a gazing point.

  In the example of FIG. 14, the number of pixels of the display unit 210 is Xmax × Ymax. The X coordinate of the left side of the region F is X1, the X coordinate of the right side is X2, the Y coordinate of the upper side is Y1, and the Y coordinate of the lower side is Y2. Details of the diagnostic processing using such images and coordinates will be described with reference to flowcharts of FIGS. 16 to 18 (described later).

  FIG. 15 is a diagram illustrating an example of an image (diagnostic image) different from the images (FIGS. 6 to 11 and FIG. 14) used in the above description. In the diagnostic video of FIG. 15, the content of the video and the image area of the geometric pattern are different from the images of FIGS. 15 and FIGS. 6 to 10 and 14 only need to be different in at least the position of the area of the geometric pattern image.

  In FIG. 15, the X coordinate of the left side of the region F is X3, the X coordinate of the right side is X4, the Y coordinate of the upper side is Y3, and the Y coordinate of the lower side is Y4. Details of the diagnostic processing using such images and coordinates will be described with reference to flowcharts of FIGS. 16 to 18 (described later).

  In FIG. 14 and FIG. 15, the position of the region F of each diagnostic image is different. Specifically, it is point-symmetric with the center of the display area of the display screen 101 as the symmetry point. By displaying in this way, it is possible to diagnose the viewing tendency of the subject such as from which direction the screen is viewed, and it is possible to perform the diagnosis by excluding the viewer's tendency Become.

  Next, diagnosis support processing by the diagnosis support apparatus 100 according to the first embodiment configured as described above will be described with reference to FIGS. 16 to 18. 16 to 18 are flowcharts illustrating an example of the diagnosis support process in the present embodiment.

  First, referring to FIG. 16, a flowchart in the case of providing diagnosis support with one video will be described. One video is a continuously changing diagnostic image displayed in, for example, the time chart of FIG. For example, it is composed of a plurality of diagnostic images as shown in FIGS. 6 is an image at the start of display, FIG. 7 is an image after 2 seconds, FIG. 8 is an image after 3 seconds, FIG. 9 is an image after 6 seconds, and FIG. 10 is an image after 8 seconds. Here, FIGS. 6 and 7 are images (first diagnosis images) displayed in a period immediately after the diagnosis is started (diagnosis period A), and FIGS. 9 and 10 are a predetermined period after the diagnosis is started. It is the image | video (2nd diagnostic image) displayed in the set period (diagnosis period B). The diagnostic image (second diagnostic image) displayed in the diagnostic period B is similar in natural image part and geometric pattern image to the diagnostic image (first diagnostic image) displayed in the diagnostic period A. There is a feature that. In the following, it is assumed that a natural image (including a person image) and a geometric pattern image are synthesized in advance.

  First, the control unit 300 starts playing a video (step S1001). Next, the control unit 300 resets a timer that measures a time slightly shorter than the video playback time (step S1002). Next, the control unit 300 resets the counter 1 that counts up when the inside of the area F is watched and the counter 2 that counts up when the outside of the area F is watched (step S1003). Next, it is confirmed whether a predetermined time (3 seconds) has elapsed (step S1004). When a predetermined time has elapsed, the process proceeds to the next step. The period from the start of video playback to the elapse of a predetermined time corresponds to the diagnosis period A described above.

  Note that the gaze point measurement described below is performed for each frame of the stereo camera 102 that captures images synchronously, for example. That is, the gazing point is measured at predetermined time intervals. Therefore, the count values of the counter 1 and the counter 2 correspond to the gaze times in the area F and the area F, respectively.

  Next, the line-of-sight detection unit 351 and the viewpoint detection unit 352 perform gaze point detection (step S1005). Next, the control unit 300 determines whether or not gazing point detection has failed (step S1006). For example, gaze point detection fails when an image of the pupil and corneal reflection cannot be obtained due to blinking or the like. Further, when the gazing point does not exist in the screen of the display unit 210 (when the screen other than the screen of the display unit 210 is viewed), the viewpoint detection fails.

  If gazing point detection has failed (step S1006: Yes), the process from step S1007 to step S1012 is skipped and the process proceeds to step S1013 in order not to affect counter 1 and counter 2.

  When the gaze point detection is successful (step S1006: No), the control unit 300 checks whether the X coordinate “x” of the gaze point on the display unit 210 is larger than X1 (step S1007). If larger (step S1007: Yes), the controller 300 checks whether “x” is smaller than X2 (step S1008). If it is smaller (step S1008: Yes), the control unit 300 checks whether the Y coordinate “y” of the gazing point on the display unit 210 is greater than Y1 (step S1009). If larger (step S1009: Yes), the controller 300 checks whether “y” is smaller than Y2 (step S1010). When “y” is smaller than Y2 (step S1010: Yes), the gazing point exists in the region F. Accordingly, the control unit 300 counts up the counter 1 (step S1012). If “y” is not smaller than Y2 (step S1010: No), the subject is watching a person image or the like outside the area F. Therefore, the control unit 300 counts up the counter 2 (step S1011). Similarly, when “x” is not larger than X1 (step S1007: No), “x” is not smaller than X2 (step S1008: No), and “y” is not larger than Y1 (step S1009: No), the controller 300 counts up the counter 2 (step S1011).

  Next, in order to confirm the end of the video, the control unit 300 checks the completion of the timer (step S1012). For example, the control unit 300 determines that the timer has been completed when the value of the timer reaches a predetermined value corresponding to the video end time. If the timer has not been completed (step S1013: No), the process returns to step S1005 and is repeated.

  When the timer is completed (step S1013: Yes), the control unit 300 stops the video reproduction (step S1014). The period until the timer is completed in step S1013 after the predetermined time has elapsed in step S1004 corresponds to the diagnosis period B. Next, the control unit 300 outputs the data (value) of the counter 1 (step S1015). The data of the counter 1 corresponds to the gaze time in the area F. Next, the control unit 300 outputs the data (value) of the counter 2 (step S1016). The data of the counter 2 corresponds to the gaze time outside the area F. Next, the evaluation unit 354 calculates the ratio between the counter 1 and the counter 2 (step S1017). The evaluation unit 354 outputs the calculated evaluation value (step S1018). For example, the evaluation unit 354 calculates an evaluation value that represents the ratio of the value of the counter 1 to the value of the counter 2. Such an evaluation value is a guideline for the possibility of developmental disabilities. Note that the evaluation value calculation method is not limited to this. Any evaluation value may be used as long as it can be determined which of the natural image and the pattern image is being watched. The higher the proportion of the area F that is watched, the higher the possibility that the subject has a developmental disability.

FIG. 17 and FIG. 18 show a flowchart in the case where diagnosis is supported by two images. The two images displayed in FIGS. 17 and 18 are diagnostic images displayed in the time chart of FIG. 11, for example. The first video out of the two videos is called video 1. Here, video 1 is a continuously changing diagnostic image displayed in the time chart of FIG. In the time chart of FIG. 11, FIG. 6 is an image at the start of display, FIG. 7 is an image after 2 seconds, FIG. 8 is after 3 seconds, FIG. 9 is after 6 seconds, and FIG. Here, FIGS. 6 and 7 are images (first diagnosis images) displayed in a period immediately after the diagnosis is started (diagnosis period A), and FIGS. 9 and 10 are a predetermined period after the diagnosis is started. It is the image | video (2nd diagnostic image) displayed in the set period (diagnosis period B). The diagnostic image (second diagnostic image) displayed in the diagnostic period B is similar in natural image part and geometric pattern image to the diagnostic image (first diagnostic image) displayed in the diagnostic period A. There is a feature that.
Next, the second video out of the two videos is called video 2. Here, video 2 is a continuously changing diagnostic image displayed in the time chart of FIG. As shown in FIGS. 14 and 15, the image 2 is different in at least the position of the geometric pattern region. For example, at the start of display, a diagnostic image in which the position of the geometric pattern area is changed with respect to FIG. 6, two seconds later, a diagnostic image in which the position of the geometric pattern area is changed with respect to FIG. Is a diagnostic image in which the position of the geometric pattern area is different from that in FIG. 8, 6 seconds later is a diagnostic image in which the position of the geometric pattern area is different from that in FIG. 9, and 9 seconds later is in FIG. On the other hand, it is a diagnostic image in which the position of the area of the geometric pattern is changed. As an example, these diagnostic images are the same as those in FIGS. 6 to 10 except for the position of the area of the geometric pattern image. The diagnostic image provided in the period corresponding to the diagnostic period A is the third diagnostic image, and the diagnostic image provided in the period corresponding to the diagnostic period B is the fourth diagnostic image. The diagnostic image (fourth diagnostic image) displayed in the diagnostic period B is similar in natural image part and geometric pattern image to the diagnostic image (third diagnostic image) displayed in the diagnostic period A. There is a feature that. As in FIG. 16, it is assumed that a natural image (including a person image) and a geometric pattern image are synthesized in advance. Video 1 (first diagnostic image, second diagnostic image) and video 2 (third diagnostic image, fourth diagnostic image) may be connected to form one continuous video. Video 1 and video 2 are configured with predetermined times corresponding to diagnostic period A and diagnostic period B, respectively.

  FIG. 17 corresponds to a process for displaying the first video (video 1) of the two videos. Steps S1101 to S1113 are the same as steps S1001 to S1013 in FIG. In FIG. 17, since the video reproduction is not stopped yet, the processing corresponding to step S1014 in FIG. 16 is deleted. Steps S1114 to S1117 are the same as steps S1015 to S1018 in FIG. In this example, the roles of the counter 1 and the counter 2 are different from those in FIG.

  In FIGS. 17 and 18, the counter 1 and the counter 2 are counters that are counted through the time during which the two images, the video 1 (first diagnostic video) and the video 2 (second diagnostic video) are reproduced. In other words, counting is performed for the entire two images. However, at the end of the video 1, the counting result as the video 1 and its ratio are output (steps S1114 to S1117). Thereafter, the image 2 is continuously counted and the total value is output (steps S1133 to S1136).

  FIG. 18 will be described. FIG. 18 corresponds to processing when displaying the second video (video 2) of the two videos. Immediately before step S1118, playback of video 2 has started.

  The control unit 300 resets a timer that measures a time slightly shorter than the playback time of the video 2 (step S1118). Next, the control unit 300 resets the counter 3 that counts up when the inside of the region F is watched in the video 2 and the counter 4 that counts up when the outside of the region F is watched (step S1119). Next, it is confirmed whether a predetermined time (3 seconds) has elapsed (step S1120). When a predetermined time has elapsed, the process proceeds to the next step.

  Next, the line-of-sight detection unit 351 and the viewpoint detection unit 352 perform gaze point detection (step S1121). Next, the control unit 300 determines whether or not gazing point detection has failed (step S1122).

  If the gazing point detection has failed (step S1122: Yes), the process from step S1123 to step S1130 is skipped and the process proceeds to step S1131 so that the counters 3 and 4 are not affected.

  When the gaze point detection is successful (step S1122: No), the control unit 300 checks whether the X coordinate “x” of the gaze point on the display unit 210 is larger than X3 (step S1123). If larger (step S1123: Yes), the controller 300 checks whether “x” is smaller than X4 (step S1124). If small (step S1124: Yes), the control unit 300 checks whether the Y coordinate “y” of the gazing point on the display unit 210 is larger than Y3 (step S1125). If larger (step S1125: Yes), the controller 300 checks whether “y” is smaller than Y4 (step S1126). When “y” is smaller than Y4 (step S1126: Yes), the gazing point exists in the region F. Therefore, the control unit 300 counts up the counter 1 (step S1129) and counts up the counter 3 (step S1130). If “y” is not smaller than Y4 (step S1126: No), the subject is watching a person image or the like outside the area F. Therefore, the control unit 300 counts up the counter 2 (step S1127) and counts up the counter 4 (step S1128).

  Next, in order to confirm the end of the video, the control unit 300 checks completion of the timer (step S1131). If the timer has not been completed (step S1131: No), the process returns to step S1121 and is repeated.

  When the timer is completed (step S1131: Yes), the control unit 300 stops the reproduction of the video (step S1132). Next, the controller 300 outputs the data of the counter 1 (step S1133). The data of the counter 1 corresponds to the gaze time in the area F when the video 1 and the video 2 are reproduced. Next, the control unit 300 outputs the data of the counter 2 (step S1134). The data of the counter 2 corresponds to the gaze time outside the area F when the video 1 and the video 2 are reproduced. Next, the evaluation unit 354 calculates an evaluation value representing the ratio between the counter 1 and the counter 2 (step S1135). The evaluation unit 354 outputs this evaluation value (step S1136).

  Further, the evaluation unit 354 outputs the data of the counter 3 (step S1137). The data of the counter 3 corresponds to the gaze time in the area F in the video 2. Next, the evaluation unit 354 outputs the data of the counter 4 (step S1138). The data of the counter 4 corresponds to the gaze time outside the area F in the video 2. Next, the evaluation unit 354 calculates an evaluation value representing the ratio between the counter 3 and the counter 4 (step S1139). The evaluation unit 354 outputs this evaluation value (step S1140).

By comparing the counting result as video 1 and its ratio (steps S1114 to S1117) and the counting result as video 2 and its ratio (steps S1137 to S1140), the tendency of the subject can be known. For example, when there is a tendency to see from the right side of the screen, the count value in the region F tends to increase in the video 1 and the count value in the region F tends to decrease in the video 2. If both are balanced, it is thought that they started watching from the center and focused on their own preference.

  In 1st Embodiment, it demonstrated using about the tendency which looks at a test subject's screen about evaluation using the image | video 1 and the image | video 2. FIG. As for the evaluation value indicating the rate at which the area F is watched, a larger value or a smaller value may be adopted using the video 1 and the video 2 and may be used as a final evaluation value. Further, the two evaluation values may be averaged to obtain a final evaluation value.

As described above, according to the first embodiment, for example, the following effects can be obtained.
(1) A region having a geometric pattern is arranged in a person image, and the portion is configured in combination with a portion having a display mode such as luminance, hue, and saturation close to the image portion outside the region. The shape was changed. This makes it difficult for regular subjects to notice the area, while subjects with developmental disabilities can find the area accurately. Therefore, the gaze point difference becomes larger than before, and the detection accuracy is improved.
(2) Measure a plurality of times while changing the geometric pattern area centered on the center of the screen to be point-symmetric. Thereby, the tendency can be canceled for a subject who tends to look in a specific direction.

(Second Embodiment)
In the second embodiment, a line-of-sight detection device and a line-of-sight detection method capable of further simplifying the apparatus configuration are realized as compared with the first embodiment.

  Hereinafter, the line-of-sight detection device and the line-of-sight detection method of the second embodiment will be described in detail with reference to the drawings. In addition, this invention is not limited by this embodiment. In the following, an example in which the line-of-sight detection apparatus is used as a diagnosis support apparatus that supports diagnosis of developmental disabilities using the line-of-sight detection result will be described. Applicable devices are not limited to diagnosis support devices.

  The line-of-sight detection apparatus (diagnosis support apparatus) of the present embodiment detects the line of sight using an illumination unit installed at one place. In addition, the line-of-sight detection device (diagnosis support device) of the present embodiment calculates the corneal curvature center position with high accuracy by using a result obtained by gazing at one point on the subject before the line-of-sight detection.

  In addition, an illumination part is an element which can irradiate light to a test subject's eyeball including a light source. The light source is an element that generates light, such as an LED (Light Emitting Diode). A light source may be comprised from one LED, and may be comprised by combining several LED and arrange | positioning in one place. Hereinafter, the “light source” may be used as a term representing the illumination unit in this way.

  19 and 20 are diagrams illustrating an example of the arrangement of the display unit, the stereo camera, the infrared light source, and the subject according to the second embodiment. In addition, about the structure similar to 1st Embodiment, the same code | symbol may be attached | subjected and description may be abbreviate | omitted.

  As illustrated in FIG. 19, the diagnosis support apparatus according to the second embodiment includes a display unit 210, a stereo camera 2102, and an LED light source 2103. The stereo camera 2102 is disposed below the display unit 210. The LED light source 2103 is arranged at the center position of two cameras included in the stereo camera 2102. The LED light source 2103 is a light source that irradiates near infrared rays having a wavelength of 850 nm, for example. FIG. 19 shows an example in which an LED light source 2103 (illumination unit) is configured by nine LEDs. Note that the stereo camera 2102 uses a lens that can transmit near-infrared light having a wavelength of 850 nm.

  As shown in FIG. 20, the stereo camera 2102 includes a right camera 2202 and a left camera 2203. The LED light source 2103 irradiates near-infrared light toward the eyeball 111 of the subject. In the image acquired by the stereo camera 2102, the pupil 112 is reflected and darkened with low brightness, and the corneal reflection 113 generated as a virtual image in the eyeball 111 is reflected and brightened with high brightness. Accordingly, the positions of the pupil 112 and the corneal reflection 113 on the image can be acquired by each of the two cameras (the right camera 2202 and the left camera 2203).

  Further, the three-dimensional world coordinate values of the positions of the pupil 112 and the corneal reflection 113 are calculated from the positions of the pupil 112 and the corneal reflection 113 obtained by the two cameras. In the present embodiment, as the three-dimensional world coordinates, with the center position of the display screen 101 as the origin, the top and bottom are the Y coordinate (up is +), the side is the X coordinate (right is +), and the depth is the Z coordinate (front is +).

  FIG. 21 is a diagram illustrating an outline of functions of the diagnosis support apparatus 2100 according to the second embodiment. FIG. 21 shows a part of the configuration shown in FIGS. 19 and 20 and a configuration used for driving the configuration. As illustrated in FIG. 21, the diagnosis support apparatus 2100 includes a right camera 2202, a left camera 2203, an LED light source 2103, a speaker 105, a drive / IF (interface) unit 208, a control unit 2300, and a storage unit 150. And a display unit 210. In FIG. 21, the display screen 101 shows the positional relationship between the right camera 2202 and the left camera 2203 in an easy-to-understand manner, but the display screen 101 is a screen displayed on the display unit 210. The drive unit and the IF unit may be integrated or separate.

  The speaker 105 functions as an audio output unit that outputs an audio or the like for alerting the subject during calibration.

  The drive / IF unit 208 drives each unit included in the stereo camera 2102. The drive / IF unit 208 serves as an interface between each unit included in the stereo camera 2102 and the control unit 2300.

  The control unit 2300 is a communication I / F that communicates with a control device such as a CPU (Central Processing Unit) and a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory) by connecting to a network. And a computer equipped with a bus for connecting each unit.

  The storage unit 150 stores various information such as a control program, a measurement result, and a diagnosis support result. The storage unit 150 stores, for example, an image to be displayed on the display unit 210. The display unit 210 displays various information such as a target image for diagnosis.

  FIG. 22 is a block diagram illustrating an example of detailed functions of the respective units illustrated in FIG. As shown in FIG. 22, a display unit 210 and a drive / IF unit 208 are connected to the control unit 2300. The drive / IF unit 208 includes camera IFs 314 and 315, an LED drive control unit 316, and a speaker drive unit 322.

  A right camera 2202 and a left camera 2203 are connected to the drive / IF unit 208 via camera IFs 314 and 315, respectively. The driving / IF unit 208 drives these cameras to image the subject.

  The speaker driving unit 322 drives the speaker 105. The diagnosis support apparatus 2100 may include an interface (printer IF) for connecting to a printer as a printing unit. Further, the printer may be provided inside the diagnosis support apparatus 2100.

  The control unit 2300 controls the entire diagnosis support apparatus 2100. The control unit 2300 includes a first calculation unit 2351, a second calculation unit 2352, a third calculation unit 2353, a gaze detection unit 2354, a viewpoint detection unit 2355, an output control unit 2356, and an evaluation unit 2357. I have. Note that the line-of-sight detection device may include at least the first calculation unit 2351, the second calculation unit 2352, the third calculation unit 2353, and the line-of-sight detection unit 2354.

  Each element included in the control unit 2300 (the first calculation unit 2351, the second calculation unit 2352, the third calculation unit 2353, the line-of-sight detection unit 2354, the viewpoint detection unit 2355, the output control unit 2356, and the evaluation unit 2357) It may be realized by software (program), may be realized by a hardware circuit, or may be realized by using software and a hardware circuit in combination.

  When implemented by a program, the program is a file in an installable or executable format, such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), a DVD ( Digital Versatile Disk) is recorded on a computer-readable recording medium and provided as a computer program product. The program may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program may be provided or distributed via a network such as the Internet. The program may be provided by being incorporated in advance in a ROM or the like.

  The first calculation unit 2351 calculates the position of the pupil center (first position) indicating the center of the pupil from the eyeball image captured by the stereo camera 2102. The second calculator 2352 calculates the position of the corneal reflection center (second position) indicating the center of corneal reflection from the captured image of the eyeball.

  The third calculator 2353 calculates the corneal curvature center (third position) from the straight line connecting the LED light source 2103 and the corneal reflection center. For example, the third calculation unit 2353 calculates a position on the straight line where the distance from the corneal reflection center is a predetermined value as the corneal curvature center. As the predetermined value, a value determined in advance from a general radius of curvature of the cornea or the like can be used.

  Since there may be individual differences in the radius of curvature of the cornea, there is a possibility that an error will increase if the center of corneal curvature is calculated using a predetermined value. Therefore, the third calculation unit 2353 may calculate the corneal curvature center in consideration of individual differences. In this case, the third calculation unit 2353 uses the pupil center and the corneal reflection center calculated when the target position is first watched by the subject, the straight line connecting the pupil center and the target position, the corneal reflection center, and the LED. The intersection (fourth position) of the straight line connecting the light source 2103 is calculated. Then, the third calculation unit 2353 calculates the distance (first distance) between the pupil center and the calculated intersection, and stores the calculated distance in the storage unit 150, for example.

  The target position may be a position that is determined in advance and can calculate a three-dimensional world coordinate value. For example, the center position of the display screen 101 (the origin of the three-dimensional world coordinates) can be set as the target position. In this case, for example, the output control unit 2356 displays an image (target image) or the like that causes the subject to gaze at the target position (center) on the display screen 101. Thereby, a test subject can be made to gaze at a target position.

  The target image may be any image as long as it can attract the attention of the subject. For example, an image in which a display mode such as luminance or color changes, an image in which the display mode is different from other regions, or the like can be used as the target image.

  The target position is not limited to the center of the display screen 101, and may be an arbitrary position. If the center of the display screen 101 is set as the target position, the distance from an arbitrary end of the display screen 101 is minimized. For this reason, it becomes possible to make the measurement error at the time of gaze detection smaller, for example.

  The processing up to the calculation of the distance is executed in advance, for example, before the actual gaze detection is started. At the time of actual line-of-sight detection, the third calculation unit 2353 calculates, on the straight line connecting the LED light source 2103 and the corneal reflection center, a position where the distance from the pupil center is a previously calculated distance as the corneal curvature center. .

  The line-of-sight detection unit 2354 detects the line of sight of the subject from the pupil center and the corneal curvature center. For example, the gaze detection unit 2354 detects the direction from the corneal curvature center to the pupil center as the gaze direction of the subject.

  The viewpoint detection unit 2355 detects the viewpoint of the subject using the detected gaze direction. The viewpoint detection unit 2355 detects, for example, a viewpoint (gaze point) that is a point on the display screen 101 where the subject gazes. The viewpoint detection unit 2355 detects, for example, the intersection of the line-of-sight vector represented in the three-dimensional world coordinate system as shown in FIG. 20 and the XY plane as the gaze point of the subject.

  The output control unit 2356 controls output of various information to the display unit 210, the speaker 105, and the like. For example, the output control unit 2356 outputs the target image at the target position on the display unit 210. Further, the output control unit 2356 controls the output to the display unit 210 such as the diagnostic image and the evaluation result by the evaluation unit 2357.

  The diagnostic image may be an image corresponding to the evaluation process based on the line-of-sight (viewpoint) detection result. For example, in the case of diagnosing a developmental disorder, a diagnostic image including an image (such as a geometric pattern image) preferred by a subject with a developmental disorder and other images (such as a person image) may be used.

  The evaluation unit 2357 performs an evaluation process based on the diagnostic image and the gazing point detected by the viewpoint detection unit 2355. For example, in the case of diagnosing a developmental disorder, the evaluation unit 2357 analyzes the diagnostic image and the gazing point, and evaluates whether or not the image preferred by the subject with the developmental disorder has been gazed.

  The output control unit 2356 may display the same diagnostic image as that of the first embodiment, and the evaluation unit 2357 may perform the same evaluation process as that of the evaluation unit 354 of the first embodiment. In other words, the gaze detection process (gaze detection unit 351) of the first embodiment is replaced with the gaze detection process (first calculation unit 2351, second calculation unit 2352, third calculation unit 2353, gaze detection of the second embodiment. Part 2354). Thereby, in addition to the effect of 1st Embodiment, the effect (simplification of an apparatus structure etc.) of 2nd Embodiment can be achieved.

  FIG. 23 is a diagram illustrating an outline of processing executed by the diagnosis support apparatus 2100 of this embodiment. The elements described in FIGS. 19 to 22 are denoted by the same reference numerals and description thereof is omitted.

  The pupil center 407 and the corneal reflection center 408 represent the center of the pupil and the center of the corneal reflection point detected when the LED light source 2103 is turned on, respectively. The corneal curvature radius 409 represents the distance from the corneal surface to the corneal curvature center 410.

  FIG. 24 is an explanatory diagram showing a difference between a method using two light sources (illumination units) (hereinafter referred to as method A) and the present embodiment using one light source (illumination unit). The elements described in FIGS. 19 to 22 are denoted by the same reference numerals and description thereof is omitted.

  Method A uses two LED light sources 511 and 512 instead of the LED light source 2103. In the method A, a straight line 515 connecting the cornea reflection center 513 and the LED light source 511 when the LED light source 511 is irradiated, and a straight line 516 connecting the cornea reflection center 514 and the LED light source 512 when the LED light source 512 is irradiated. An intersection is calculated. This intersection is the corneal curvature center 505.

  In contrast, in the present embodiment, a straight line 523 connecting the corneal reflection center 522 and the LED light source 2103 when the LED light source 2103 is irradiated is considered. A straight line 523 passes through the corneal curvature center 505. It is also known that the radius of curvature of the cornea is almost constant with little influence from individual differences. Thus, the corneal curvature center when the LED light source 2103 is irradiated exists on the straight line 523 and can be calculated by using a general curvature radius value.

  However, if the viewpoint is calculated using the position of the center of corneal curvature calculated using a general radius of curvature, the viewpoint position may deviate from the original position due to individual differences in the eyeballs, and accurate viewpoint position detection cannot be performed. There is.

  FIG. 25 is a diagram for explaining calculation processing for calculating the corneal curvature center position and the distance between the pupil center position and the corneal curvature center position before performing viewpoint detection (line-of-sight detection). The elements described in FIGS. 19 to 22 are denoted by the same reference numerals and description thereof is omitted.

  The target position 605 is a position for displaying a target image or the like at one point on the display unit 210 and causing the subject to stare. In this embodiment, the center position of the display screen 101 is set. A straight line 613 is a straight line connecting the LED light source 2103 and the corneal reflection center 612. A straight line 614 is a straight line connecting the target position 605 (gaze point) that the subject looks at and the pupil center 611. A corneal curvature center 615 is an intersection of the straight line 613 and the straight line 614. The third calculation unit 2353 calculates and stores the distance 616 between the pupil center 611 and the corneal curvature center 615.

  FIG. 26 is a flowchart illustrating an example of calculation processing according to the present embodiment.

  First, the output control unit 2356 reproduces the target image at one point on the display screen 101 (step S101), and causes the subject to gaze at the one point. Next, the control unit 2300 turns on the LED light source 2103 toward the eyes of the subject using the LED drive control unit 316 (step S102). The controller 2300 images the eyes of the subject with the left and right cameras (the right camera 2202 and the left camera 2203) (step S103).

  The pupil part is detected as a dark part (dark pupil) by irradiation of the LED light source 2103. Further, a corneal reflection virtual image is generated as a reflection of LED irradiation, and a corneal reflection point (corneal reflection center) is detected as a bright portion. That is, the first calculation unit 2351 detects a pupil portion from the captured image, and calculates coordinates indicating the position of the pupil center. In addition, the second calculation unit 2352 detects a corneal reflection portion from the captured image, and calculates coordinates indicating the position of the corneal reflection center. The first calculation unit 2351 and the second calculation unit 2352 calculate coordinate values for each of the two images acquired by the left and right cameras (step S104).

  Note that the left and right cameras are pre-calibrated with a stereo calibration method to obtain three-dimensional world coordinates, and conversion parameters are calculated. As the stereo calibration method, any conventionally used method such as a method using Tsai's camera calibration theory can be applied.

  The first calculation unit 2351 and the second calculation unit 2352 use the conversion parameters to convert the coordinates of the left and right cameras into the three-dimensional world coordinates of the pupil center and the corneal reflection center (step S105). The 3rd calculation part 2353 calculates | requires the straight line which connects the world coordinate of the calculated | required corneal reflection center, and the world coordinate of the center position of the LED light source 2103 (step S106). Next, the third calculation unit 2353 calculates a straight line connecting the world coordinates of the center of the target image displayed at one point on the display screen 101 and the world coordinates of the pupil center (step S107). The 3rd calculation part 2353 calculates | requires the intersection of the straight line calculated by step S106, and the straight line calculated by step S107, and makes this intersection a cornea curvature center (step S108). The third calculation unit 2353 calculates the distance between the pupil center and the corneal curvature center at this time, and stores it in the storage unit 150 or the like (step S109). The stored distance is used to calculate the corneal curvature center at the time of subsequent detection of the viewpoint (line of sight).

  The distance between the pupil center and the corneal curvature center when looking at one point on the display unit 210 in the calculation process is kept constant within a range in which the viewpoint in the display unit 210 is detected. The distance between the center of the pupil and the center of corneal curvature may be obtained from the average of all the values calculated during playback of the target image, or from the average of several values of the values calculated during playback. You may ask for it.

  FIG. 27 is a diagram illustrating a method of calculating the corrected position of the corneal curvature center using the distance between the pupil center and the corneal curvature center that is obtained in advance when performing viewpoint detection. A gazing point 805 represents a gazing point obtained from a corneal curvature center calculated using a general curvature radius value. A gazing point 806 represents a gazing point obtained from a corneal curvature center calculated using a distance obtained in advance.

  The pupil center 811 and the corneal reflection center 812 indicate the position of the pupil center and the position of the corneal reflection center calculated at the time of viewpoint detection, respectively. A straight line 813 is a straight line connecting the LED light source 2103 and the corneal reflection center 812. The corneal curvature center 814 is the position of the corneal curvature center calculated from a general curvature radius value. The distance 815 is the distance between the pupil center and the corneal curvature center calculated by the prior calculation process. The corneal curvature center 816 is the position of the corneal curvature center calculated using the distance obtained in advance. The corneal curvature center 816 is obtained from the fact that the corneal curvature center exists on the straight line 813 and the distance between the pupil center and the corneal curvature center is the distance 815. Thus, the line of sight 817 calculated when a general radius of curvature value is used is corrected to the line of sight 818. Also, the gazing point on the display screen 101 is corrected from the gazing point 805 to the gazing point 806.

  FIG. 28 is a flowchart illustrating an example of a line-of-sight detection process according to the present embodiment. For example, the line-of-sight detection process of FIG. 28 can be executed as the process of detecting the line of sight in the diagnostic process using the diagnostic image. In the diagnostic process, in addition to the steps in FIG. 28, a process for displaying a diagnostic image, an evaluation process by the evaluation unit 2357 using the detection result of the gazing point, and the like are executed.

  Steps S201 to S205 are the same as steps S102 to S106 in FIG.

  The third calculation unit 2353 calculates, as the corneal curvature center, a position on the straight line calculated in step S205 and whose distance from the pupil center is equal to the distance obtained by the previous calculation process (step S206).

  The line-of-sight detection unit 2354 obtains a vector (line-of-sight vector) connecting the pupil center and the corneal curvature center (step S207). This vector indicates the line-of-sight direction viewed by the subject. The viewpoint detection unit 2355 calculates the three-dimensional world coordinate value of the intersection between the line-of-sight direction and the display screen 101 (step S208). This value is a coordinate value representing one point on the display unit 210 that the subject gazes in world coordinates. The viewpoint detection unit 2355 converts the obtained three-dimensional world coordinate value into a coordinate value (x, y) represented in the two-dimensional coordinate system of the display unit 210 (step S209). Thereby, the viewpoint (gaze point) on the display part 210 which a test subject looks at can be calculated.

As described above, according to the present embodiment, for example, the following effects can be obtained.
(1) It is not necessary to arrange light sources (illuminating units) at two places, and it becomes possible to perform line-of-sight detection with the light sources arranged at one place.
(2) Since the number of light sources is one, the apparatus can be made compact and the cost can be reduced.

  DESCRIPTION OF SYMBOLS 100 Diagnosis support apparatus, 101 Display screen, 102 Stereo camera, 105 Speaker, 150 Storage part, 202 Right camera, 203 Infrared LED light source, 204 Left camera, 205 Infrared LED light source, 208 Drive / IF part, 210 Display part, 300 control unit (control device), 351 gaze detection unit, 352 viewpoint detection unit, 353 output control unit, 354 evaluation unit

Claims (9)

In a control device that displays on a display unit a diagnostic image obtained by combining a natural image including a person, animal, or plant and a pattern image including a geometric pattern ,
After displaying the first diagnostic image on the display unit and displaying the first diagnostic image, a second diagnostic image in which the natural image portion and the pattern image are similar to the first diagnostic image is displayed. A control apparatus provided with the output control part displayed on a display part. The output control unit
After displaying the second diagnostic image, a third diagnostic image in which the position of the first diagnostic image and the pattern image are at least different and the natural image portion and the pattern image are not similar to the second diagnostic image. After the third diagnostic image is displayed, the position of the second diagnostic image and the pattern image is at least different and the natural image and the pattern image are similar to each other compared to the third diagnostic image. The control apparatus according to claim 1, wherein the fourth diagnostic image is displayed on the display unit.   2. The second diagnostic image is characterized in that at least one of hue, luminance, and saturation of the natural image portion and the pattern image is similar to that of the first diagnostic image. 3. The control device according to any one of 2. The output control unit according to any one of claims 1 to 3,
An imaging unit for imaging a subject;
From a captured image captured by the imaging unit, a gaze detection unit that detects the gaze direction of the subject,
A viewpoint detection unit that detects the viewpoint of the subject in the display area of the display unit based on the line-of-sight direction;
A diagnostic support apparatus comprising:   The diagnosis support apparatus according to claim 4, wherein the viewpoint detection unit detects a viewpoint when the second diagnostic image is displayed.   The diagnosis support apparatus according to claim 4, further comprising an evaluation unit that determines whether the natural image or the pattern image is being watched. An illumination unit including a light source that emits light;
A first calculation unit that calculates a first position indicating the center of the pupil from an image of the eyeball of the subject that is illuminated by the illumination unit and imaged by the imaging unit;
A second calculation unit that calculates a second position indicating the center of corneal reflection from the captured image of the eyeball;
A third calculator that calculates a third position indicating a corneal curvature center based on a straight line connecting the light source and the second position;
The line-of-sight detection unit detects the line of sight of the subject based on the first position and the third position;
The diagnosis support apparatus according to any one of claims 4 to 6. In a control method for controlling display of a diagnostic image combining a natural image including a person, an animal, or a plant and a pattern image including a geometric pattern ,
After displaying the first diagnostic image, the display unit has a step of displaying a second diagnostic image in which the natural image portion and the pattern image are similar to the first diagnostic image,
Control method. To a computer of a control device that controls display of a diagnostic image combining a natural image including a person, an animal, or a plant and a pattern image including a geometric pattern ,
After displaying the first diagnostic image, causing the display unit to display a second diagnostic image in which the natural image portion and the pattern image are similar to the first diagnostic image;
Control program.