JP2001340300A - Pupil detecting device and method, fixation point detecting device and method, and three-dimensional image display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a pupil detecting device and a method capable of highly accurately detecting a pupil position for a short time even when an observer wears spectacles. SOLUTION: An infrared image and a visible image of the observer are acquired (S102), and gravity center coordinates of a high luminance area in the infrared image is determined (S105). A density value of visible image picture elements corresponding to the gravity center coordinates is determined, and the high luminance area including the gravity center coordinates corresponding to the visible image picture elements having the lowest density value is determined as a retina reflected image of the observer (S107), and the gravity center coordinates are decided as the pupillary position.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

TECHNICAL FIELD The present invention relates to a subject (observer)
The present invention relates to a pupil position detection device and a pupil position detection method for detecting a pupil position of a pupil, and more particularly to a pupil position detection device and a pupil detection method with improved detection accuracy. The present invention further relates to a viewpoint position detecting device and method using the pupil position detecting device and method, and a stereoscopic image display system.

[0002]

2. Description of the Related Art Conventionally, there is a so-called direct-view type stereoscopic image display device capable of stereoscopic viewing with the naked eye. There are various types of stereoscopic image display devices.
No. 4 discloses a rear cross lenticular type. FIG. 6 is a perspective view of an essential part showing an example of a rear cross lenticular type stereoscopic image display device. In the figure, 6
Is a display device for displaying images, and is composed of, for example, a liquid crystal element (LCD). In the figure, a polarizing plate, a color filter, an electrode, a black matrix, an antireflection film, and the like are omitted.

[0003] Reference numeral 10 denotes a backlight (surface light source) serving as an illumination light source. Display device 6 and backlight 1
Between 0, a mask substrate (mask) 7 on which a mask pattern having a checkered opening 8 is formed is arranged. The mask pattern is manufactured by patterning a metal deposited film such as chromium or a light absorbing material on a mask substrate 7 made of glass or resin. The backlight 10, the mask substrate 7, and the like constitute one element of the light source.

A first lenticular lens 3 and a second lenticular lens 4 made of transparent resin or glass are arranged between the mask substrate 7 and the display device 6. The first lenticular lens 3 is a vertical cylindrical lens array formed by arranging vertically long vertical cylindrical lenses in the left-right direction, and the second lenticular lens 4 is formed by vertically arranging horizontal cylindrical lenses long in the horizontal direction. It is a cylindrical lens array.

As shown in the figure, the image displayed on the display device 6 divides the left and right parallax images R and L into a large number of horizontal stripe pixels R and L in the vertical direction, and divides them into, for example, LRLLRLR from the screen. ...
And a horizontal stripe image alternately arranged to form one image.

The light from the backlight 10 is applied to the mask substrate 7
Then, the display device 6 is illuminated through each of the openings 8, and the left and right stripe pixels R and L are separately observed by both eyes of the observer.

That is, the mask substrate 7 is
0 illuminates, and light exits from the aperture 8. The first lenticular lens 3 is arranged on the observer side of the mask substrate 7, and the lens curvature is designed so that the mask substrate 7 is almost at the focal position of each cylindrical lens. In this cross section, the second lenticular lens 4 has no optical effect, so that the light beam emitted from one point on the opening 8 is converted into a substantially parallel light beam in this cross section.

The pair of openings and light-shielding portions of the mask pattern are set to substantially correspond to one pitch of the first lenticular lens 3.

Further, based on the relationship between the optical distance from a predetermined position of the observer to the first lenticular lens 3 and the optical distance from the first lenticular lens 3 to the mask pattern, the first lenticular lens 3 is formed. By determining the pitch of the mask pattern and the pitch of the pair of openings and the light-shielding portion of the mask pattern, light from the openings 8 can be uniformly collected to the left eye or the right eye over the entire width of the screen.
In this way, the left and right stripe pixels on the display device 6 are observed separately in the horizontal direction into the left eye and right eye regions.

The second lenticular lens 4 condenses all light beams emitted from each point on the opening 8 of the mask 7 on the right-eye or left-eye stripe pixel of the display device 6, illuminates and transmits the same. It diverges only in the vertical direction according to the NA at the time of condensing, and provides an observation area in which the left and right stripe pixels are uniformly separated from the predetermined eye height of the observer over the entire vertical width of the screen. .

However, the viewing angle of such a stereoscopic image display device is narrow, and if the observer's viewpoint deviates from the viewing angle, the stereoscopic display cannot be recognized. Therefore, by detecting the viewpoint position of the observer and controlling the image display in response to the movement of the viewpoint position, by dynamically moving the viewing angle range in which stereoscopic viewing is possible, a substantial viewing angle is obtained. A technique for expanding the size of the device has been proposed. For example, Japanese Patent Application Laid-Open No. 10-2323
No. 67 discloses a technique for expanding a stereoscopically visible region (viewing angle) by moving a mask pattern or a lenticular lens in parallel to a display surface.

FIG. 7 is a diagram of a stereoscopic image display device disclosed in Japanese Patent Application Laid-Open No. 10-232367. 7, the same reference numerals are given to the same components as those in FIG. 6, and the description will be omitted. Since the three-dimensional image display device in FIG. 7 has a single lenticular lens, it does not include the second lenticular lens 4 in FIG.

In the three-dimensional image display device having such a configuration, control according to the movement of the observer 54 is performed as follows. First, the position sensor 51 detects a horizontal displacement of the observer 54 from a preset reference position, and sends the information to the control unit 52. The control unit 52 responds to the displacement information by using the display drive circuit. When the image control signal is output to the display 50, the display drive circuit 50 displays the first or second horizontal stripe image on the display 6. At the same time, the control unit 52 generates an actuator drive signal based on the displacement information and drives the actuator 53 that moves the mask pattern 7 in the horizontal direction, so that the mask pattern 7 can be separated from the left and right stripe images by the observer 54 in the best position. Move to As a result, even if the viewpoint position of the observer 54 changes, the range that can be stereoscopically viewed is expanded.

In addition to the method of physically moving the mask pattern, for example, as described in Japanese Patent Application Laid-Open No. H10-78563, the mask pattern is formed by a transmissive liquid crystal display element to form a pattern (opening). The viewing angle can also be increased by controlling the size of.

FIG. 8 is a diagram of a stereoscopic image display device disclosed in Japanese Patent Application Laid-Open No. 10-78563. 8, the same reference numerals are given to the same components as those in FIG. 6 or FIG. 7, and the description will be omitted.

The stereoscopic image display device shown in FIG. 8 is a cross-lenticular type direct-view type stereoscopic image display device, in which the mask pattern 7 can dynamically change the size of a pattern (opening) of a transmission type liquid crystal element or the like. The display unit has the same configuration as that of the display device shown in FIG. On the other hand, as a control part of the display, a position sensor 51 for detecting the position of the observer 54 in the same manner as the display device shown in FIG. 7, and the opening area and shape of the mask pattern are changed according to the position information detected by the position sensor. It has a control unit 55.

Specifically, the control unit 55 changes the position and width of the opening (the transparent portion of the pattern) and the light-shielding portion (the same black portion) according to the position of the observer, so that the right and left of the observer can be changed. And realizes good stereoscopic image display without crosstalk and reverse stereoscopic vision following the movement of. The principle and specific method for expanding the viewing angle by moving the mask pattern and changing the pattern are not directly related to the present invention, and therefore, detailed descriptions thereof are omitted from the descriptions of the above publications and will not be repeated here.

As described above, when the display device is controlled in accordance with the observer's viewpoint position, an image display suitable for the observer's viewpoint can be performed even if the detection accuracy is low or the processing time for detection is long. I can't. Therefore, it is very important how to accurately detect the viewpoint position of the observer in a short time in the performance of this type of display device.

As a method of detecting the viewpoint position of the observer (measured person), a method of irradiating the observer with infrared light and detecting the reflection of the retina (for example, see the design of the pupil photographing optical system for detecting the pupil of the pupil) Law, IEICE Transactions D-II, Vol. J74
-D-II, No.6, pp.736-747, described in June 1991) to detect the pupil position and use this pupil position as the viewpoint position or from the visible image using the detected pupil position A method of detecting a viewpoint position is known.

These methods make use of the fact that a human pupil has the property of retroreflecting near-infrared light (returning light in the same direction as the incident direction). Pupil reflected light is obtained as a sharp reflection peak, and usually shows a higher reflectance than the face, etc., so that only the pupil part is captured by imaging the observer using an infrared imaging device whose optical axis is coaxial with the light source. Bright images can be taken. If this photographed image is binarized with an appropriate threshold, an accurate viewpoint position can be detected from the extracted pupil position.

[0021]

As described above, pupil position detection using an infrared image utilizes the near-infrared reflection characteristics of the retina. Normally, there is no part of human face that reflects infrared rays strongly (like the retina), but infrared light is reflected by sebum such as nose and cheeks, reflection by objects around the observer, lighting, etc. May be detected in the infrared image.

In particular, when the observer wears glasses, if infrared light is reflected by the lens or the frame, the reflected image is detected near the pupil position, and the correct pupil position cannot be detected. Although detection is possible, problems such as a large error occur. FIG. 9 shows an infrared image (FIG. 9A) and a visible image (FIG. 9) when the observer wears glasses.
FIG. 10 shows an example of (b)) and FIG. 10 shows an example of an infrared image (FIG. 10 (a)) and an example of a visible image (FIG. 10 (b)) when there is illumination in the photographing range of the observer. As is clear from FIGS. 9 and 10, the infrared image includes various reflection images other than the retinal reflection image.

As a method for solving such a problem, for example, in the invention described in Japanese Patent Application Laid-Open No. Hei 7-218989, a reflected image is acquired for about several tens of seconds by utilizing the fact that a person blinks, and the reflected image is obtained. A method in which a reflected image that blinks in the image (exists or does not exist depending on time) is determined as a retinal reflected image, or an infrared image is acquired by irradiating infrared light from a different direction, and an image in which the luminance has changed And the like are disclosed.

Japanese Patent Application Laid-Open No. 8-185503 discloses that infrared images are binarized using two types of thresholds, taking advantage of the fact that reflection by eyeglasses is stronger than reflection by retinas. A method of discriminating a reflection image by eyeglasses from a binarized image is disclosed.

However, in the method described in Japanese Patent Application Laid-Open No. Hei 7-218989, since the determination is performed depending on the difference between the images acquired at different times, it is difficult to detect when the observer moves quickly. . In addition, when detection is performed using a difference between infrared images obtained by irradiating infrared light from different directions, a reflection image specific to the irradiation direction of infrared light may be included, and such a reflection image may be included. Cannot be erased even if the difference is taken, so that there is a possibility of erroneous detection. Further, in the method described in Japanese Patent Application Laid-Open No. 8-185503, it is necessary to increase the resolution of a captured image of an observer, and to increase the resolution, the observer must zoom in to take an image. Therefore, it is not realistic to assume that the observer moves.

The present invention has been made in view of such problems of the prior art, and can detect the position of the pupil in a short time and with high accuracy even when the observer wears glasses. Pupil detection device and method. It is another object of the present invention to provide a viewpoint position detecting device and method using pupil position information obtained by using the pupil position detecting device or method of the present invention, and a stereoscopic image display system.

[0027]

That is, the gist of the present invention is a pupil position detecting device for detecting a pupil position of a subject from an infrared image and a visible image of the subject and outputting pupil position information. A high-brightness area extraction unit that detects a high-brightness area having a brightness equal to or higher than a predetermined value in the infrared image; and a high-brightness area in which the brightness of a corresponding area in the visible image is lower than the predetermined value. A pupil position calculating device for calculating and outputting a pupil position of the subject based on predetermined coordinates in the luminance area.

Another aspect of the present invention is a pupil position detecting device for detecting a pupil position of a subject from an infrared image and a visible image of the subject and outputting pupil position information. From the outside image, a high-brightness area having a brightness equal to or more than a predetermined value is extracted, and for each of the extracted high-brightness areas, an area extraction unit that obtains at least one predetermined coordinate;
The image processing apparatus includes a gray-scale value obtaining unit that obtains a gray-scale value of a pixel corresponding to the predetermined coordinate, and a predetermined coordinate corresponding to a representative coordinate pixel having a gray-scale value smaller than the predetermined value among the gray-scale values obtained by the gray-scale value obtaining unit. A pupil position calculating device for detecting a high-luminance area, calculating and outputting a pupil position of the subject based on the representative coordinates.

Further, another gist of the present invention is that the pupil position detecting device of the present invention, a template creating means for creating a template for pattern matching from a visible image using a pupil position, and a template creating means are provided. A matching unit that detects a viewpoint position of the subject by performing pattern matching with a visible image of the subject continuously acquired at least every predetermined time using a template, and outputs a result as viewpoint position information. The present invention resides in a viewpoint position detecting device characterized in that:

Further, another gist of the present invention is that the pupil position detecting apparatus of the present invention, a template creating means for creating a template for pattern matching from a visible image using a pupil position, and a template creating means are provided. Using a template, a matching means for detecting a viewpoint position of the subject by performing pattern matching with a visible image of the subject continuously acquired at least every predetermined time, and outputting the detection result as viewpoint position information, There is provided a viewpoint position detecting device characterized by having a pupil position detecting device and a control means for controlling to generate a template again using the template generating means when a predetermined condition is satisfied.

Another aspect of the present invention is a stereoscopic image display system including the viewpoint position detecting device of the present invention and stereoscopic image display means, wherein the stereoscopic image display device controls a mask pattern for controlling an emission optical path. A stereoscopic image display system having a stereoscopic image display system characterized by changing a position or a pattern of a mask pattern using viewpoint position information received from a viewpoint position detecting device.

Another aspect of the present invention is a pupil position detecting method for detecting a pupil position of a subject from an infrared image and a visible image of the subject and outputting pupil position information. In the outer image, a high-brightness region extraction step of detecting a high-brightness region having a brightness equal to or more than a predetermined value, and among the high-brightness regions, a high-brightness region in which the brightness of a corresponding region in the visible image is less than a predetermined value A pupil position calculating step of calculating and outputting a pupil position of the subject based on predetermined coordinates.

Another aspect of the present invention is a pupil position detection method for detecting a pupil position of a subject from an infrared image and a visible image of the subject and outputting pupil position information. Extracting, from the outer image, a high-brightness region having a brightness equal to or more than a predetermined value, and obtaining an at least one predetermined coordinate for each extracted high-brightness region; Value obtaining step for obtaining a gray value of a pixel to be performed, and detecting, from among the gray values obtained by the gray value obtaining step, a high brightness area including predetermined coordinates corresponding to a representative coordinate pixel having a gray value smaller than a predetermined value A pupil position calculating step of calculating and outputting a pupil position of the subject based on the representative coordinates.

Further, another gist of the present invention is that, in addition to the pupil position detecting method of the present invention, a template creating step of creating a template for pattern matching from a visible image using a pupil position, and creating a template creating step A matching step of performing pattern matching with a visible image of the subject continuously acquired at least every predetermined time to detect the viewpoint position of the subject, and outputting the result as viewpoint position information using the template. The present invention resides in a method of detecting a viewpoint position.

Another gist of the present invention is that, in addition to the pupil position detecting method of the present invention, a template creating step for creating a template for pattern matching from a visible image using a pupil position, and a template creating step A matching step of performing pattern matching with the visible image of the subject obtained at least every predetermined time continuously to detect the viewpoint position of the subject, and outputting the detection result as viewpoint position information using the template. And a control step of performing control so as to generate a template again by using the pupil position detection method and the template generation step when a predetermined condition is satisfied.

Another aspect of the present invention resides in a computer-readable storage medium storing a pupil position detecting method of the present invention as a computer-executable program.

Further, another gist of the present invention resides in a computer-readable storage medium storing a viewpoint position detecting method of the present invention as a computer-executable program.

[0038]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the drawings. In the following description, a stereoscopic image capable of expanding a stereoscopically viewable range by controlling a stereoscopic image display device based on an observer's viewpoint position detected by a viewpoint position detection device having a pupil position detection device of the present invention. Although the display system will be described, it goes without saying that the pupil position detection device of the present invention can be used in any field and application.

In the following description, the viewpoint position means the coordinates of a certain point indicating the position of the observer's eyes.
The viewpoint position information output from the viewpoint position detecting device does not necessarily have to be the coordinate value of one point, but may be information indicating a certain area. Depending on the use, the position of the entire eye may be roughly known, and may be appropriately selected depending on the use.

FIG. 1 is a block diagram showing a configuration example of a stereoscopic image display system according to an embodiment of the present invention. The stereoscopic image display system includes a photographing unit 1, a viewpoint position detecting unit 2, and a stereoscopic image display unit 3. A photographing unit 1 that is built in the base of the stereoscopic image display unit 3 and captures a visible image for pupil position detection, viewpoint position detection, and tracking;
Infrared image capturing unit 1 that captures an infrared image for pupil position detection
And 2. Each of the visible image capturing unit 11 and the infrared image capturing unit 12 is provided with a position where the present system is installed so as to capture the front (face) of the observer's head of the stereoscopic image displayed on the stereoscopic image display unit 3. The angle of view and the imaging direction are determined according to the predicted position of the observer.

Visible image photographing section 11 and infrared image photographing section 1
2 can be constituted by a video camera, and the infrared image photographing unit 12 is configured so that only infrared light is incident on an internal light receiving element by a filter or the like. Also,
The infrared light source serving as the infrared light emitting unit 13 for infrared image capturing using an LED or the like preferably illuminates the observer with an optical axis equal to the lens center of the infrared image capturing unit 12. It is incorporated in the unit 12 and is configured to emit infrared light coaxially with the lens center of the infrared image capturing unit 12 using an optical path switching unit such as a half mirror. Further, the optical axes of the visible image capturing unit 11 and the infrared image capturing unit 12 may be coaxially arranged by an optical path switching unit such as a half mirror.

The viewpoint position detecting section 2 includes an infrared image photographing section 12
The pupil position of the observer is detected from the infrared image and the visible image captured by the visible image capturing unit 11, and the observer moves using the obtained pupil position and the visible image captured by the visible image capturing unit 11. While detecting the viewpoint position.

The viewpoint position information and the like obtained by the viewpoint position detector 2 are supplied to a display controller (not shown). The display control unit generates stereoscopic image data to be displayed on the stereoscopic image display unit 3.

The viewpoint position detecting section 2 can be composed of a general-purpose computer device capable of storing an image signal output from the photographing section 1, for example. The display control unit can also be constituted by a general-purpose computer device, but it is preferable to have a circuit such as a graphic accelerator that can execute image processing at high speed. Viewpoint position detector 2,
It is also possible to configure the display control unit with the same computer device.

The viewpoint position detector 2 includes a visible image storage 21
A pattern matching determination unit 22, a template creation unit 23, a pupil position detection processing unit 24, an infrared image storage unit 25, an infrared light emission control unit 26, and a coordinate / shade information storage unit 27. Have been.

The visible image storage unit 21 and the infrared image storage unit 2
5 is used as a means for storing image data taken by the corresponding photographing units 11 and 12. Even if it is constituted by a semiconductor memory such as a RAM, a magnetic / optical storage device such as a magnetic disk or an optical disk is used. You may comprise.

The pattern matching determination unit 22 uses the template supplied from the template creation unit 23 to display the position information of the area having the highest correlation with the template among the images stored in the visible image storage unit 21 as a three-dimensional image. Output to section 3. In addition, when the pattern matching fails, the infrared light emission control unit 26 outputs the infrared light emission unit 13 to emit light.

The template creating unit 23 uses the position information supplied from the pupil position detection processing unit 24 to convert the image data stored in the visible image storage unit 21 into a pattern matching template used by the pattern matching determination unit 22. Create

The pupil position detection processing unit 24 receives a signal indicating that the infrared light emission control unit 26 has emitted light from the infrared light emission unit 13 and stores an infrared image based on the signal information. The pupil position is detected from the infrared image stored in the unit 25, and the position information is supplied to the template creating unit 23. Further, the position information may be supplied to the stereoscopic image display unit 3.

The infrared light emission control unit 26 controls lighting of the infrared light emission unit 13 in accordance with the control of a control unit (not shown), a pupil position detection processing unit 24, a pattern matching determination unit 22, and the like. Each component of the viewpoint position detection unit 2 is operated by a system control unit 28 that controls the entire system and is realized by a microcomputer or the like.

The coordinate / shade information storage unit 27 stores, for each of the reflection images extracted from the infrared image, the coordinates of the center of gravity or the coordinates closest to the center of gravity, and the coordinates of the center of gravity or the coordinates of the visible image corresponding to the coordinates closest to the center of gravity. The gradation values (gradation values) are stored in association with each other. Hereinafter, the coordinates of the center of gravity or the coordinates closest to the center of gravity may be collectively referred to as the center of gravity coordinates.

The three-dimensional image display unit 3 displays the display image data supplied from the display control unit in accordance with a control signal also supplied from the display control unit. Further, in the present embodiment, the stereoscopic image display unit 3 is a direct-view type stereoscopic display device having a configuration in which the viewing angle can be expanded by following the viewpoint position of the observer. A mask pattern moving type disclosed in Japanese Patent Application Laid-Open No. H10-232367, a mask pattern changing type disclosed in Japanese Patent Application Laid-Open No. H10-78563, or a display device of another type may be used.

The control unit (not shown) moves the mask pattern or changes the mask pattern based on the viewpoint position information of the observer supplied from the pattern matching determination unit 22 and / or the pupil position detection unit 24. By doing so, the viewing angle is moved following the movement of the observer, and the viewing angle that can be viewed stereoscopically is expanded.

In the above configuration, the pupil position detecting device according to the present invention comprises the photographing unit 1, the visible image storage unit 21,
The pupil position detection unit 24, the infrared image storage unit 25, the infrared light emission control unit 26, and the coordinate / density information storage unit 27, and the viewpoint position detection device combines the imaging unit 1 and the viewpoint position detection unit 2. Configuration. However, neither the pupil position detecting device nor the viewpoint position detecting device necessarily includes the components included in the imaging unit 1, and only the pupil position detecting unit and the pupil position detecting unit that detect the pupil position and the viewpoint position from the acquired image are used.
It may be a viewpoint position detecting device.

(Pupil Position Detection Processing) Next, the pupil position detection processing operation in the stereoscopic image display system according to the present embodiment will be described with reference to the flowchart shown in FIG. The detection of the pupil position is performed every time the observer changes, and when the pupil position cannot be detected from the acquired infrared image, or when the viewpoint position detection process described later cannot follow the observer's movement, This is performed when it is necessary to perform the viewpoint position detection again, for example, in a case where the setting is made so as to obtain the target position.

First, an infrared image and a visible image are obtained (step S102). The control unit (not shown) instructs the infrared light emission control unit 26 to emit infrared light, and the infrared image photographing unit 12 and the visible image photographing unit 11 to acquire an image. In response to this instruction, the infrared light emission control unit 2
6 causes the infrared light emitting unit 13 to emit light, and the infrared image photographing unit 12
Then, an infrared image is acquired. Once the infrared image has been acquired,
In response to the notification from the infrared image capturing unit 12, the emission of the infrared light emitting unit 13 is terminated via the infrared light emission control unit 26, which is not shown. Alternatively, the end of the acquisition may be directly notified from the infrared image capturing unit 12 to the infrared light emission control unit 26, and the emission of the infrared light may be terminated in response to the notification.

On the other hand, it is desirable that the visible image photographing section 11 be constructed so as to acquire a visible image at the same time (at the same frame timing) as the acquisition of an infrared image. With this configuration, accurate template creation and pattern matching can be performed.

Then, the visible image and the infrared image acquired by the visible image capturing section 11 and the infrared image capturing section 12 are stored in the visible image storing section 21 and the infrared image storing section 25, respectively. Next, binarization of the acquired multi-tone infrared image is performed (step S103). The threshold value used for binarization is such that if a relatively strong reflection image such as an infrared reflection image by the retina is extracted and other unnecessary reflection images can be removed as noise, what value is set. Optional. Specifically, for example, an appropriate value may be obtained using an infrared image acquired in advance at the installation location of the stereoscopic image display system.

When a binarized infrared image is obtained, a labeling process is performed (step S104). The labeling process is
This is a process of detecting a region that satisfies a predetermined condition among the high-luminance regions (white pixel regions) in the binarized image, and assigning an identifier (a number or the like) to each region for use in subsequent processing.
The predetermined condition is, for example, the size of the region, and the labeling process is not performed on the region determined to be clearly smaller or larger than the reflected image of the pupil.

When the labeling process is completed, the barycentric coordinates of each of the labeled regions are calculated (step S105). At this time, it is preferable to calculate the degree of circularity representing the roundness of each area. When calculating the circularity, a region having a circularity equal to or more than a predetermined value, that is, a region having a shape not close to a circle (for example, a frame of glasses) can be configured to be excluded from a retinal reflection image determination target described later. .

Next, the grayscale values (gradation values) of the visible image pixels corresponding to the barycenter coordinates of each labeled area are stored in the coordinate / shade information storage unit 27 in association with each other (step S106). At this time, if the visible image photographing unit 11 has acquired a multi-tone monochrome image (gray scale image), the gray scale value of the pixel corresponding to the barycentric coordinates may be extracted as it is. On the other hand, when the visible image capturing unit 11 has acquired a color image, it converts the image into a grayscale image and then extracts the grayscale value.

Next, a retinal reflection image is determined using the extracted gray value (step S107). In particular,
Among the gray values of the visible image obtained in step S106,
A region including the barycentric coordinates corresponding to the two in order from the lowest value is determined as a retinal reflection image. This is because the human pupil is usually black and has a low gray value even in a visible image. On the other hand, the reflected image portion and the illumination of the spectacle lens are photographed brightly even in the visible image, so that the gray value is high. Therefore, it can be determined that the white area having the barycentric coordinates corresponding to the lowest and second lowest gray-scale values of the visible image pixels is the retinal reflection image. Then, in step S108, the coordinates of the center of gravity of the image determined to be a retinal reflection image are output as pupil positions. Note that the processes of steps S103 to S108 described above are performed by the pupil position detection unit 24 and the coordinate / density information storage unit 27.

The processing of steps S104 to S107 will be further described with reference to FIG. FIG. 3 shows step S1.
It is a figure which shows the example of the infrared image (FIG. 3 (a)) binarized by 03, and the corresponding visible image (FIG. 3 (b)).

In the labeling process in step S104, of the high brightness regions a to g, the region g has a size exceeding a predetermined size and is therefore excluded from the labeling. Then, labeling is performed for each of the remaining regions a to f.

Then, in step S105, barycentric coordinates (Xn, Yn) (where n = 1 to 6) are calculated for each of the labeled regions a to f. Also, the circularity of each area is calculated. FIG. 3 (a)
, Since the regions a to f all have a shape with a high degree of circularity, the coordinates of the center of gravity of the regions a to f and the corresponding grayscale values of the visible image pixels are stored in the coordinate / shade information storage unit 27 in step S106. Is done.

Next, in step S107, the coordinates
The grayscale values of the visible image pixels stored in the grayscale information storage unit 27 are compared. High-luminance areas a to of the infrared image shown in FIG.
In g, regions a and d are retinal reflection images, regions b and e are reflection images by spectacle lenses, and regions c, f, and g are illumination images. Regions a and d, ie, coordinates (X1, Y1) and (X4
The gray value of the visible image pixel corresponding to Y4) is low, and the gray value of the visible image pixel corresponding to the other coordinates is high.

Therefore, in step S107, the areas a and d
Is determined to be a retinal reflection image, and the barycentric coordinates (X1,
Y1) and (X4, Y4) are set as pupil positions in step S
It is output at 108. However, in the infrared image, the reflected image that appears to the left is the right eye of the observer, and the reflected image that appears to the right is the left eye of the observer.

Next, a specific operation of the entire system shown in FIG. 1 will be described with reference to a flowchart shown in FIG.

First, the respective units are started such as turning on the power of the photographing unit 1, the viewpoint position detecting unit 2, and the stereoscopic image display unit 3 (step S201). Next, it is checked whether a new observer has been detected.

The detection of a new observer is performed, for example, by processing a photographed image of the visible image acquiring unit 11 of the photographing unit 1 for detecting that a switch or the like provided on a chair on which the observer is sitting is turned on from off. The system can automatically detect when the observer has arrived at the observation position, or provide a switch that can be operated by the observer at the observation position and turn on the switch by the observer. It is also possible to make the detection by the active operation of.

Next, the pupil position detection processing (steps S102 to S108) described with reference to FIG. 2 is performed (step S202). If the pupil position detection has been normally completed in step S202, the template creation process is started (step S203). Although any pattern matching template can be used in the system of the present embodiment, a case where two child templates and one parent template are used will be described here as an example. Figure 5 for each type of template
This will be further described with reference to FIG.

FIGS. 5A and 5B are diagrams for explaining a child template and a parent template used in the present embodiment, respectively. As shown in the figure, the two child templates each have a viewpoint based on the viewpoint positions of the right eye and the left eye (indicated by X in the figure), and the parent template includes the viewpoint positions of the right eye and the left eye. One template is based on a point. Here, the viewpoint position in the template is a coordinate indicating one point of the coordinates in the image.

In the present embodiment, template creation processing is performed from creation of a child template. The template creating unit 23 uses the pupil positions (coordinate values on the image) of the right eye and the left eye detected by the pupil position detection processing unit 24 from the infrared image to extract the visible image stored in the visible image storage unit 21. , A child template 1 based on the right eye pupil position and a child template 2 based on the left eye pupil position. The size of the child template is determined from the following equation according to the distance between the pupil positions of the right eye and the left eye. Average distance between the pupil positions of the right and left eyes of the human: measured distance between the pupil positions of the right and left eyes = large enough to include the average human eye and eyebrows: the size of the child template The average value of the distance between pupil positions and the size of the eye and eyebrows can be, for example, a statistically obtained value.

When the child template has been created, the template creating section 23 creates a parent template. As described above, the parent template is a template including two pupil positions starting from the middle point of the two pupil positions of the right eye and the left eye. The size of the parent template is determined from the following equation based on the distance between the pupil positions of the right and left eyes. Average distance between pupil positions of right and left eyes of human: measured distance between pupil positions of right and left eyes = size enough to include average human face: size of parent template When creating child template alike,
A statistically determined value can be used as the average value. The template created by the template creation unit 23 is supplied to the pattern matching determination unit 22.

When the templates have been created, pattern matching using these templates is performed, and the viewpoint position is detected following the change in the viewpoint position of the observer (step S204). Specifically, pattern matching is performed using the visible image captured in the visible image storage unit 21 and the template created by the template creating unit 23.

The pattern matching determination section 22 first performs pattern matching between the parent template and the visible image.
Pattern matching can be performed using, for example, a normalized correlation function. For the pattern matching using the normalized correlation function, see, for example, “Matrox Imaging Library Version 5.1 User Guide (Matr
ox Imaging Library Version 5.1 User Guide) ", pages 154 to 155. The value obtained by the normalized correlation function is expressed as 0 to 100 (%), and 100% means a perfect match. Also, pattern matching is sometimes called template matching.

In the present embodiment, if a degree of correlation exceeding, for example, 85% is obtained, it is determined that the pattern matching has succeeded. In the pattern matching immediately after the template is created, since the image on which the template is based and the image data to be subjected to the pattern matching are the same, basically a correlation of almost 100% should be obtained.

When the result of pattern matching between the parent template and the visible image satisfies a predetermined degree of correlation, it is determined that pattern matching using the parent template has succeeded, and the pattern matching determination unit 22 sets the search area for the viewpoint position. I do. That is, the left half in the parent template is set as the search area for the right eye viewpoint position, and the right half in the parent template is set as the search area for the left eye viewpoint position. Then, pattern matching between the child template and the visible image is performed based on the set search area. In this way, by performing pattern matching hierarchically, narrowing down the search range in stages, and restricting the right eye left eye viewpoint position, detection of erroneous viewpoint position (not failure) is prevented, and accurate tracking is performed. Can be. Alternatively, pattern matching using the child template may be performed while the pattern matching is successful, and pattern matching may be performed hierarchically using the parent template and the child template while the pattern matching fails.

If the maximum correlation value satisfies the predetermined correlation value as a result of the pattern matching, it is determined that the pattern matching is successful in step S205, and the process proceeds to step S2.
Shift to 06. On the other hand, if the maximum correlation value is less than the predetermined correlation value, it is determined that the pattern matching has failed, and the process returns to step S202 and starts again from the pupil position detection process. In FIG. 4, when the pattern matching fails once
Although the process returns to S202, the process may return to S202 when two or more failures have occurred.

If it is determined in step S205 that the pattern matching is successful, the pattern matching determination unit 22
The finally obtained viewpoint position information (viewpoint position coordinates on the image) is output to the display control unit and the stereoscopic image display unit 3 (not shown) for each of the right eye and the left eye.

In step S206, the display control unit (not shown) generates display image data corresponding to the viewpoint position of the observer based on the viewpoint position information, and outputs the display image data to the stereoscopic image display unit 3. In the stereoscopic image display unit 3, a control unit (not shown) moves the mask pattern in the horizontal direction or changes the mask pattern based on the viewpoint position information, thereby changing the viewing angle at which the observer can view stereoscopically. Follow the viewpoint position of

In step S207, it is determined whether or not to end the system. If the end of the operation of the system is not instructed, a visible image is fetched from the visible image photographing unit 11, stored in the visible image storage unit 21, and then stored in step S20.
Return to 4.

Thereafter, the pattern matching for the visible image is continued, and if the pattern matching fails, the process automatically returns to the pupil position detecting process to perform the template re-creating process. When the end processing is instructed in step S207, predetermined end processing is performed (step S20).
8) End a series of processing.

[0084]

[Other Embodiments] In the above-described embodiment, a configuration has been described in which pattern matching with high accuracy and good tracking is performed by redoing from acquisition of an infrared image when the pattern matching result has failed. In an environment where the subject does not move, a similar effect can be expected by periodically recreating the template. In particular, when it is not desirable to repeatedly obtain an infrared image for the same observer, it is preferable to perform such processing.

Further, the observer can operate a switch or the like of a remote controller or the like and use it as a start signal of the pupil position detection processing. With such a configuration, the range in which the pupil position of the observer exists in the acquired infrared image can be narrowed, and more accurate detection can be performed.

Similarly, by operating this switch when it becomes difficult for the user to perform stereoscopic viewing, the pupil position detecting process starting from the infrared image acquiring operation can be performed again. With such a configuration, the template is updated at an accurate timing, and the viewpoint position detection can be performed with higher accuracy. As a result, a stereoscopic image display system with a wide stereoscopic view range can be realized. .

When emitting infrared light, the amount of emitted light is changed according to the brightness of the image display unit 3 or the distance to the observer, or the pupil position detection from the infrared image fails and is repeated. At this time, the light emission amount can be changed (increased or weakened) from the last time. Such control of the light emission amount increases the probability of successfully detecting the pupil position from the infrared image, and as a result, leads to obtaining a highly accurate viewpoint position detection result.

In the above-described embodiment, the case where the viewpoint position is detected using the detection result of the pupil position detecting device according to the present invention has been described as an example. However, the pupil position detecting device according to the present invention can be used for any purpose. Can be used.

In the above-described embodiment, the retinal reflection image is determined based on the gray value of the visible image pixel corresponding to the barycentric coordinate of the white area extracted from the binarized infrared image.
It is also possible to determine two or more coordinates for each white area and determine the retinal reflection image using the average value of the grayscale values of the visible image pixels corresponding to each coordinate. However, in this case, the coordinates output as the final pupil position need to be separately obtained (the coordinates of the center of gravity may be used).

Alternatively, the retinal reflection image can be determined using the average grayscale value of all pixels included in the visible image region corresponding to the white region extracted from the binarized infrared image. .

In the above-described embodiment, the retinal reflection image is determined based on the gray value of the visible image pixel corresponding to the barycentric coordinate of the white area extracted from the binarized infrared image.
A region having a predetermined brightness or higher (having a lighter / darker value than a predetermined value) may be extracted from the grayscale infrared image. In this case, the extracted area may be treated similarly to the white area in the embodiment.

When determining the retinal reflection image, two of the regions having the lowest gray value of the corresponding visible image pixel are not mechanically determined as the pupil reflection image. A final determination may be made from the region using the positional relationship between the left and right pupils, such as the average distance between the pupil positions of the right and left eyes of the human used when creating the template. With such a configuration, more accurate detection can be performed.

Of course, first, two regions from the lowest gray level value of the corresponding visible image pixel are mechanically provisionally determined as pupil reflection images, and if the above-mentioned positional relationship is not satisfied, the third or higher third from the bottom is further determined. You may make it determine sequentially about the area | region corresponding to the gray value of a visible image pixel.

Since the essence of the present invention is to determine a bright region in an infrared image and a dark region in a visible image as a retinal reflection image, any conditions other than the above conditions can be used.

Further, the specific methods described in the above embodiments, such as the pattern matching method and the template creation method, are not limited to those described in the embodiments, but may be equally applied. It is needless to say that may be used.

In the above-described embodiment, the viewpoint position, which is the pinpoint coordinate, is configured to be output. However, for example, as in the above-described embodiment, the finally obtained viewpoint position is displayed on the stereoscopic image display unit. If used for control,
Since minimum control is possible if the center positions of the viewpoint positions of the right eye and the left eye are known, the center position of the template may be output to the stereoscopic image display unit 3.

In the above embodiment, only the case where a liquid crystal display device using a backlight is used as the three-dimensional image display unit 3 has been described. It is also possible to realize a function equivalent to the combination of.

The present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), but can be applied to a single device (for example, a copier, a facsimile). Device).

Further, an object of the present invention is to supply a storage medium (or a recording medium) recording software program codes for realizing the functions of the above-described embodiments to a system or an apparatus, and to provide a computer (a computer) of the system or the apparatus. It is needless to say that the present invention can also be achieved by a CPU or an MPU) reading and executing the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. By executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written into the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, the program code is read based on the instruction of the program code. Needless to say, the CPU included in the function expansion card or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the above-described flowcharts (shown in FIG. 2 and / or FIG. 4).

[0103]

As described above, according to the present invention,
In a pupil position detection device that detects a pupil position of an observer using a retinal reflection image using infrared light, highly accurate pupil position detection can be performed in a short time even when the observer is wearing glasses. Can be realized.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a stereoscopic image display system according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating an infrared image acquisition operation according to the embodiment of the present invention.

FIG. 3 is a diagram illustrating a process of determining a retinal reflection image from an infrared image and a visible image.

FIG. 4 is a flowchart illustrating an operation of the stereoscopic image display system according to the embodiment of the present invention.

FIG. 5 is a diagram illustrating a configuration of a template used in the embodiment of the present invention.

FIG. 6 is a perspective view showing a configuration of a rear cross lenticular type stereoscopic image display device.

FIG. 7 is a perspective view illustrating an example of a stereoscopic image display device that performs display control based on a viewpoint position.

FIG. 8 is a perspective view illustrating an example of a stereoscopic image display device that performs display control based on a viewpoint position.

FIG. 9 is a diagram illustrating an example of an infrared image and a visible image obtained when an observer wears glasses.

FIG. 10 is a diagram showing an example of an infrared image and a visible image obtained when there is illumination on the background of the observer.

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] February 8, 2001 (2001.2.8)

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Claim 13

[Correction method] Change

[Correction contents]

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G06T 1/00 340 H04N 13/00 5C061 H04N 13/00 G02F 1/13 505 // G02F 1/13 505 H04N 7/18 B H04N 7/18 A61B 3/10 A F term (reference) 2F065 AA03 AA17 BB22 BB27 CC16 DD00 FF04 FF26 FF28 GG07 GG22 LL46 QQ24 QQ38 RR05 UU05 UU07 2H059 AA33 AB04 AB11 2H088BA20 BA02 CA16 CB13 CB16 CC03 CE12 DA07 DA08 DA16 DB02 DB05 DB08 DB09 DC06 DC14 DC22 DC34 DC39 5C054 CA04 CA05 CC03 FC01 FC04 FC05 FC12 FC15 FD02 GB12 HA05 HA12 5C061 AA07 AB17 AB24

Claims (28)

[Claims] 1. An infrared image and a visible image of a subject,
A pupil position detection device that detects a pupil position of the subject and outputs pupil position information, wherein a high-brightness region extracting unit that detects a high-brightness region having a brightness of a predetermined value or more in the infrared image. And a pupil that calculates and outputs a pupil position of the subject based on predetermined coordinates in the high-brightness area in which the brightness of a corresponding area in the visible image is less than a predetermined value in the high-brightness area. A pupil position detection device, comprising: a position calculation unit. 2. From an infrared image and a visible image of a subject,
A pupil position detection device that detects a pupil position of the subject and outputs pupil position information, wherein a high-brightness region having a brightness equal to or greater than a predetermined value is extracted from the infrared image. An area extracting unit that obtains at least one predetermined coordinate for each luminance area; a gray value obtaining unit that obtains a gray value of a pixel corresponding to the predetermined coordinate in the visible image; Among the gray values, the high-luminance area including the predetermined coordinates corresponding to the representative coordinate pixels having a gray value less than the predetermined value is detected, and the pupil position of the subject is calculated based on the representative coordinates. A pupil position calculating device for outputting the pupil position. 3. The pupil position calculating means according to claim 1, wherein said pupil position calculating means includes: a high-luminance area having a lowest luminance of a corresponding area in said visible image; 3. The pupil position detecting device according to claim 1, wherein the predetermined coordinates are output as the pupil position of the subject. 4. The pupil position detecting device according to claim 1, wherein the predetermined coordinates include a barycentric coordinate of the high-brightness area or a coordinate closest to the barycenter. . 5. The high-brightness region extracting unit according to claim 1, wherein the high-brightness region extracting unit extracts a region satisfying a predetermined area and / or circularity as the high-brightness region. The pupil position detecting device according to any one of the preceding claims. 6. A high-luminance area extracting means for generating a high-luminance area extracting means for generating a high-luminance area from the high-luminance area, comprising: The pupil position detecting device according to any one of claims 1 to 5, wherein the pupil position detecting device detects the pupil position. 7. The pupil position detecting device according to claim 1, wherein the visible image and the infrared image are obtained at the same time. 8. A pupil position detection device according to claim 1, wherein: a template creation unit that creates a template for pattern matching from the visible image using the pupil position. Using the template created by the template creating means, pattern matching is performed with the visible image of the subject, which is continuously acquired at least every predetermined time, to detect the viewpoint position of the subject, and the result is referred to as the viewpoint position. A viewpoint position detecting device, comprising: a matching unit that outputs the information as information. 9. A pupil position detection device according to claim 1, wherein: a template creation unit that creates a template for pattern matching from the visible image using the pupil position. Using the template created by the template creation unit, pattern matching is performed with the visible image of the person to be measured continuously acquired at least every predetermined time to detect the viewpoint position of the person to be measured, A viewpoint which comprises: matching means for outputting as position information; and control means for controlling the pupil position detecting device and the template creating means to create the template again when a predetermined condition is satisfied. Position detection device. 10. The control unit evaluates a detection result of the matching unit, and when it is determined that the detection has failed a predetermined number of times, the control unit re-executes the template using the pupil position detection device and the template creation unit. The viewpoint position detecting device according to claim 9, wherein control is performed to create the viewpoint position. 11. The viewpoint position according to claim 9, wherein the control means controls the pupil position detection device and the template creation means to create the template again at predetermined time intervals. Detection device. 12. The control unit evaluates the detection result of the matching unit, and when it is determined that the detection has failed a predetermined number of times, or every predetermined time, the control unit controls the pupil position detection device and the template creation unit. 10. The viewpoint position detecting apparatus according to claim 9, wherein the control is performed so that the template is created again using the template. 13. A three-dimensional image display system comprising the viewpoint position detecting device according to claim 8, and a three-dimensional image display means, wherein the three-dimensional image display device has a mask pattern for controlling an emission optical path. A stereoscopic image display system, wherein the position or pattern of the mask pattern is changed using the viewpoint position information received from the viewpoint position detection device. 14. The three-dimensional image display system according to claim 13, further comprising image generation means for generating an image to be displayed on said three-dimensional image display means according to the viewpoint position information of said person to be measured. 15. A pupil position detecting method for detecting a pupil position of the subject from an infrared image and a visible image of the subject, and outputting pupil position information, wherein the infrared image has a predetermined value. A high-brightness area extraction step of detecting a high-brightness area having the above brightness; and, among the high-brightness areas, predetermined coordinates in the high-brightness area where the brightness of a corresponding area in the visible image is less than a predetermined value A pupil position calculating step of calculating and outputting a pupil position of the subject based on the pupil position. 16. A pupil position detecting method for detecting a pupil position of the subject from an infrared image and a visible image of the subject, and outputting pupil position information, wherein a predetermined value is obtained from the infrared image. Extracting a high-brightness area having the above brightness, and obtaining at least one predetermined coordinate for each extracted high-brightness area; and shading of a pixel corresponding to the predetermined coordinate in the visible image. A gray value obtaining step of obtaining a value, and among the gray values obtained by the gray value obtaining step, detecting the high brightness area including the predetermined coordinates corresponding to a representative coordinate pixel having a gray value smaller than a predetermined value. A pupil position calculating step of calculating and outputting a pupil position of the subject based on the representative coordinates. 17. The pupil position calculating step, wherein, of the high-brightness areas, a high-brightness area having the lowest brightness of a corresponding area in the visible image and a next-highest brightness area
17. The pupil position detection method according to claim 15, wherein predetermined coordinates in the two high-luminance regions are output as pupil positions of the subject. 18. The pupil position detecting method according to claim 15, wherein the predetermined coordinates include a barycentric coordinate of the high-brightness area or a coordinate closest to the barycenter. . 19. The method according to claim 15, wherein the step of extracting a high-luminance region extracts a region satisfying a predetermined area and / or circularity as the high-luminance region. The pupil position detection method described in the above. 20. A binarized image generating step of generating a binarized image based on a predetermined threshold value from the infrared image, wherein the high-luminance area extracting step includes the step of extracting the high-luminance area from the binarized image. 16. The detection of
A pupil position detection method according to any one of claims 19 to 19. 21. The pupil position detecting method according to claim 15, wherein the visible image and the infrared image are obtained at the same time. 22. A pupil position detection method according to claim 15, further comprising: a template creation step of creating a template for pattern matching from the visible image using the pupil position. Using the template created in the template creation step, performing pattern matching with the visible image of the subject obtained continuously at least every predetermined time to detect the viewpoint position of the subject, A matching step of outputting as position information. 23. In addition to the pupil position detection method according to any one of claims 15 to 21, a template creation step of creating a template for pattern matching from the visible image using the pupil position. Using the template created in the template creation step, performing pattern matching with the visible image of the subject obtained continuously at least every predetermined time, detects the viewpoint position of the subject, and detects the detection result. A matching step of outputting as viewpoint position information; and a control step of controlling to create the template again by using the pupil position detection method and the template creation step when a predetermined condition is satisfied. Viewpoint position detection method. 24. The control step evaluates the detection result of the matching step, and when it is determined that the detection has failed a predetermined number of times, re-executes the template using the pupil position detection method and the template creation step. The viewpoint position detecting method according to claim 23, wherein the viewpoint position is controlled to be created. 25. The viewpoint position according to claim 23, wherein the control step performs control such that the template is created again using the pupil position detection method and the template creation step at predetermined time intervals. Detection method. 26. The control step evaluates the detection result of the matching step and executes the pupil position detection method and the template creation step when it is determined that the detection has failed a predetermined number of times or every predetermined time. The viewpoint position detecting method according to claim 23, wherein the control is performed so that the template is created again by using the same. 27. A computer-readable storage medium storing the pupil position detection method according to claim 15 as a computer-executable program. 28. A computer-readable storage medium storing the viewpoint position detection method according to claim 22 as a computer-executable program.