JP2000278716A - Device and method for detecting view point position and stereoscopic picture display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a device and a method for detecting a view point position with high accuracy and excellent tracking performance in a short time while suppressing an apprehension of the effect onto a human body with a simple configuration and to obtain a stereoscopic picture display system using this method. SOLUTION: A visual point position detector consisting of a photographing section 1 and a view point position detection section 2 has a visual picture photographing section 11 and an infrared ray picture photographing section 12, first a pupil position detection processing section 24 detects a pupil position from an infrared ray picture and a template generating section 23 generates a template for a visual picture by using the result of detection of a pupil position obtained from the infrared ray picture. Then a pattern matching discrimination section 22 applies pattern matching processing to a visual picture. The infrared ray emission time to acquire the infrared ray picture is minimized and using the pupil position from the infrared ray picture through the generated template relieves the processing load. Moreover, a picture display section 3 that is a stereoscopic picture display device is connected to the view point position detector to receive visual point position information so as to configure the stereoscopic picture display system where an available range of a stereoscopic vision by a viewer is wide.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a viewpoint detection apparatus and a viewpoint detection method for detecting a viewpoint position of a person to be measured, and more particularly to a viewpoint detection apparatus and a viewpoint detection method that achieve both high-speed processing and high detection accuracy. .

[0002]

2. Description of the Related Art Conventionally, there is a so-called stereoscopic image display device capable of stereoscopic viewing with the naked eye. There are various types of stereoscopic image display devices. For example, JP-A-09-31294 discloses a rear cross lenticular type. FIG. 11 is a perspective view of an essential part showing an example of a rear cross lenticular type stereoscopic image display device. In the figure, reference numeral 6 denotes a display device for displaying images, which is constituted by, for example, a liquid crystal element (LCD). In the figure, a polarizing plate, a color filter, an electrode, a black matrix, an antireflection film, and the like are omitted.

[0003] Reference numeral 10 denotes a backlight (surface light source) serving as an illumination light source. Display device 6 and backlight 1
Between 0, a mask substrate (mask) 7 on which a mask pattern having a checkered opening 8 is formed is arranged. The mask pattern is manufactured by patterning a metal deposited film such as chromium or a light absorbing material on a mask substrate 7 made of glass or resin. The backlight 10, the mask substrate 7, and the like constitute one element of the light source.

A first lenticular lens 3 and a second lenticular lens 4 made of transparent resin or glass are arranged between the mask substrate 7 and the display device 6. The first lenticular lens 3 is a vertical cylindrical lens array formed by arranging vertically long vertical cylindrical lenses in the left-right direction, and the second lenticular lens 4 is formed by arranging vertically long horizontal cylindrical lenses in the horizontal direction. It is a cylindrical lens array.

As shown in the figure, the image displayed on the display device 6 divides the left and right parallax images R and L into a large number of horizontal stripe pixels R and L in the vertical direction, and divides them into, for example, LRLLRLR from the screen. ...
And a horizontal stripe image alternately arranged to form one image.

The light from the backlight 10 is applied to the mask substrate 7
Then, the display device 6 is illuminated through each of the openings 8, and the left and right stripe pixels R and L are separately observed by both eyes of the observer.

That is, the mask substrate 7 is
0 illuminates, and light exits from the aperture 8. The first lenticular lens 3 is arranged on the observer side of the mask substrate 7, and the lens curvature is designed so that the mask substrate 7 is almost at the focal position of each cylindrical lens. In this cross section, the second lenticular lens 4 has no optical effect, so that the light beam emitted from one point on the opening 8 is converted into a substantially parallel light beam in this cross section.

The pair of openings and light-shielding portions of the mask pattern are set to substantially correspond to one pitch of the first lenticular lens 3.

Further, based on the relationship between the optical distance from a predetermined position of the observer to the first lenticular lens 3 and the optical distance from the first lenticular lens 3 to the mask pattern, the first lenticular lens 3 is formed. By determining the pitch of the mask pattern and the pitch of the pair of openings and the light-shielding portion of the mask pattern, light from the openings 8 can be uniformly collected to the left eye or the right eye over the entire width of the screen.
In this way, the left and right stripe pixels on the display device 6 are observed separately in the horizontal direction into the left eye and right eye regions.

The second lenticular lens 4 condenses all light beams emitted from each point on the opening 8 of the mask 7 on the right-eye or left-eye stripe pixel of the display device 6, illuminates and transmits the same. It diverges only in the vertical direction according to the NA at the time of condensing, and provides an observation area in which the left and right stripe pixels are uniformly separated from the predetermined eye height of the observer over the entire vertical width of the screen. .

However, the viewing angle of such a stereoscopic image display device is narrow, and if the observer's viewpoint deviates from the viewing angle, the stereoscopic display cannot be recognized. Therefore, there has been proposed a technique of detecting a viewpoint position of an observer and controlling image display in response to the movement of the viewpoint position, thereby enlarging a region where stereoscopic viewing is possible. For example, Japanese Patent Application Laid-Open No. H10-2323
No. 67 discloses a technique for expanding a stereoscopically visible area by moving a mask pattern or a lenticular lens in parallel to a display surface.

FIG. 12 is a diagram of a stereoscopic image display device disclosed in Japanese Patent Application Laid-Open No. 10-232367. 12, the same reference numerals are given to the same components as those in FIG. 11, and the description will be omitted. Since the three-dimensional image display device in FIG. 12 has a single lenticular lens, it does not include the second lenticular lens 4 in FIG.

In the three-dimensional image display device having such a configuration, control according to the movement of the observer 54 is performed as follows. First, the position sensor 51 detects a horizontal displacement of the observer 54 from a preset reference position,
The information is sent to the control unit 52, and when the control unit 52 outputs an image control signal to the display drive circuit 50 according to the shift information, the display drive circuit 50 displays the first or second horizontal stripe image on the display 6. I do. At the same time, the control unit 52 generates an actuator drive signal based on the displacement information and drives the actuator 53 that moves the mask pattern 7 in the horizontal direction, so that the mask pattern 7 can be separated from the left and right stripe images by the observer 54 in the best position. Move to
As a result, even if the viewpoint position of the observer 54 changes, the range that can be stereoscopically viewed is expanded.

As described above, when the display is controlled in accordance with the viewpoint position of the observer, an image suitable for the observer's viewpoint can be displayed even if the detection accuracy is low or the processing time for detection is long. Absent. Therefore, it is very important to detect the viewpoint position of the observer with high accuracy and in a short time in the performance of the display device.

As the method of detecting the viewpoint position of the observer (measured person), 1) a method of irradiating the observer with infrared light and detecting the reflection of the retina (for example, see Banno "Pupil optics for viewpoint detection" System Design Method, IEICE Transactions D-II, Vol.J74-D-
II, No.6, pp.736-747, described in June 1991) 2) A method of processing visible images to detect the eyes of the observer (for example, Sakaguchi et al. Real-Time Face Expression Recognition Using "IEICE Transactions
D-II, Vol.J80-D-II, No.6, pp.1547-1554 1997/6
3) A method of detecting an observer's eyes by image processing using an infrared image and a visible image (for example, see Japanese Patent Application Laid-Open No. 8-28721).
No. 6).

The method 1) utilizes the fact that the human pupil has the property of retroreflecting near-infrared light (returning light in the same direction as the incident direction). Pupil reflected light is obtained as a sharp reflection peak, and usually shows a higher reflectance than the face, etc., so that only the pupil part is captured by imaging the observer using an infrared imaging device whose optical axis is coaxial with the light source. Bright images can be taken. If this photographed image is binarized with an appropriate threshold, an accurate viewpoint position can be detected from the extracted pupil position.

In the method 2), the position of the observer in the photographing range is limited in advance, the observer blinks in that state, and the eye region is extracted by the inter-frame image of the visible image. are doing.

In the method 3), an infrared image and a visible color image are simultaneously photographed, a face region is extracted from these images, and then a characteristic region such as an eye is detected by using template matching. The infrared image is used to extract a person candidate area,
It is used to determine a temperature threshold used when extracting a skin color region from a color image.

[0019]

However, in the method (1), since it is necessary to continuously irradiate the observer with relatively strong infrared light, the influence of the infrared light on the observer is small. In addition to the problem of concern, there is also a problem that detection cannot be performed when the observer blinks because reflection by the retina is used. In addition, depending on the pupil enlargement or reduction due to the illuminance of the environment,
There is also a problem that it is difficult to follow the pupil reflection image.

In the method 2), the observer is required to perform operations such as adjustment of the observation position and blinking, which is troublesome for the observer. It requires time for operations such as adjustment of the observation position and blinking, which is not practical.

In the method 3), the irradiation intensity of infrared light may be lower than that in the method 1). However, after the intermediate processing result of the infrared image is obtained, the visible image is obtained by using the processing result. Is processed, the face area is detected using the processing result of the infrared image and the processing result of the visible image, and further pattern matching is performed. Therefore, the processing is very complicated. There is also a problem that it is not easy to create a template used for pattern matching. In addition, since the position of the face required to create a template for pattern matching is detected only from the visible image, there is a problem that the position accuracy is not very good.

Accordingly, it is an object of the present invention to provide a viewpoint position detecting apparatus and method which have high accuracy and followability in a short time, with a simple configuration and while suppressing the influence on the human body. It is another object of the present invention to provide a stereoscopic image display system having a stereoscopic image display device that controls display using viewpoint position information obtained by using the viewpoint position detecting device or method of the present invention. .

[0023]

That is, the gist of the present invention is a viewpoint position detecting device for detecting a viewpoint position of a person to be measured and outputting viewpoint position information, and acquiring an infrared image of the person to be measured. Infrared image acquiring means for acquiring, a visible image acquiring means for acquiring a visible image of the subject, detecting means for detecting a pupil position of the subject from the infrared image acquired by the infrared image acquiring means, A template creation unit that creates a template for pattern matching using the pupil position from the visible image acquired by the image acquisition unit, and a visible image acquired by the visible image acquisition unit using the template created by the template creation unit and template matching And a matching means for detecting the viewpoint position of the person to be measured and outputting the result as viewpoint position information. That.

Another aspect of the present invention is a viewpoint position detecting apparatus for detecting a viewpoint position of a person to be measured and outputting viewpoint position information. Acquisition means, visible image acquisition means for acquiring a visible image of the subject, from the infrared image acquired by the infrared image acquisition means,
Detecting means for detecting the pupil position of the subject, template creating means for creating a template for pattern matching using the pupil position from the visible image acquired by the visible image acquiring means, and a template created by the template creating means. Performing template matching with the visible image acquired by the visible image acquisition means using the same, detecting the viewpoint position of the person to be measured, a matching means for outputting the detection result as viewpoint position information, and a detection means when a predetermined condition is satisfied. And a control unit for controlling to create a template again by using the template creating unit.

Another aspect of the present invention is a stereoscopic image display system including the above-described viewpoint position detecting device according to the present invention and a stereoscopic image display device connected to the viewpoint position detecting device. There is provided a stereoscopic image display system that controls a stereoscopic image display device using viewpoint position information received from a detection device.

Another aspect of the present invention is a viewpoint position detecting method for detecting a viewpoint position of a person to be measured and outputting viewpoint position information, the method comprising detecting an infrared image of the person to be measured. An acquiring step, a visible image acquiring step of acquiring a visible image of the subject, a detecting step of detecting a pupil position of the subject from the infrared image acquired in the infrared image acquiring step, and a visible image acquiring step. From the acquired visible image,
A template creation step of creating a template for pattern matching using the pupil position, and using the template created in the template creation step, performing a template matching with the visible image acquired in the visible image acquisition step to determine the viewpoint position of the subject. And a matching step of detecting and outputting a result as viewpoint position information.

Another subject of the present invention is a viewpoint position detecting method for detecting a viewpoint position of a person to be measured and outputting viewpoint position information. An acquiring step, a visible image acquiring step of acquiring a visible image of the subject, a detecting step of detecting a pupil position of the subject from the infrared image acquired in the infrared image acquiring step, and a visible image acquiring step. From the acquired visible image,
A template creation step of creating a template for pattern matching using the pupil position, and performing template matching with the visible image acquired in the visible image acquisition step using the template created in the template creation step, and determining the viewpoint position of the subject. A matching step of detecting and outputting the detection result as viewpoint position information; and, if a predetermined condition is satisfied, creating a template again using the detection step and the template creation step; otherwise, a visible image acquisition step. The present invention resides in a viewpoint position detection method characterized by repeatedly performing a matching step.

Another aspect of the present invention resides in a computer-readable storage medium storing a viewpoint position detecting method according to the present invention as a computer-executable program.

[0029]

Embodiments of the present invention will be described below in detail with reference to the drawings. In the following description, a stereoscopic image display system in which the viewpoint position detecting device of the present invention is connected to a stereoscopic image display device will be described, but the use of the viewpoint position detecting device of the present invention is limited to the stereoscopic image display device system. is not.

Also, in the present invention, the viewpoint position means the coordinates of a certain point indicating the position of the eyes of the observer. The viewpoint position information output by the viewpoint position detecting device of the present invention is not necessarily the coordinate value of one point. It does not have to be, and may be information indicating a certain area. Depending on the use, the position of the entire eye may be roughly known, and may be appropriately selected depending on the use.

(First Embodiment) FIG. 1 is a block diagram showing a configuration of a viewpoint position detecting device according to the present invention. The viewpoint position detection device according to the present embodiment includes an imaging unit 1 and a viewpoint position detection unit 2, and the image display unit 3 is the above-described stereoscopic image display device. Hereinafter, the viewpoint position detecting device and the image display unit 3 may be collectively referred to as a stereoscopic image display system.

The photographing section 1 comprises a visible image photographing section 11, an infrared image photographing section 12, and an infrared light emitting section 13, and photographs a visible image and an infrared image of an observer. Each of the visible image photographing section 11 and the infrared image photographing section 12 can be constituted by a video camera, and the infrared image photographing section 12 causes only infrared light to enter the internal light receiving element by a filter or the like. The infrared light emitting unit 13 can be composed of an infrared light emitting element such as an LED, for example.
The light emission amount necessary for obtaining retinal reflection can be obtained at a distance within a range which can be normally taken with the above.

The viewpoint position detecting section 2 includes a visible image storing section 21
, A pattern matching determination unit 22, a template creation unit 23, a viewpoint position detection processing unit 24, an infrared image storage unit 25, and an infrared light emission control unit 26. The viewpoint position detecting section 2 can be constituted by a general-purpose personal computer capable of storing an image signal output from the photographing section 1, for example.

The visible image storage unit 21 and the infrared image storage unit 2
5 is used as a means for storing image data photographed by the corresponding photographing units 11 and 12, and may be constituted by a semiconductor memory such as a RAM or by using a storage device such as a magnetic disk or an optical disk. Is also good.

The pattern matching determination unit 22 uses the template supplied from the template creation unit 23 to display position information of an area having the highest correlation with the template in the image stored in the visible image storage unit 21 on the image display unit. Output to 3. In addition, when the pattern matching fails, the infrared light emission control unit 26 outputs the infrared light emission unit 13 to emit light.

The template creation unit 23 uses the position information supplied from the viewpoint position detection processing unit 24 to convert the image data stored in the visible image storage unit 21 into a pattern matching template used by the pattern matching determination unit 22. Create

The viewpoint position detection processing section 24 receives a signal indicating that the infrared light emission control section 26 has emitted light from the infrared light emission section 13 as an input, and stores an infrared image based on the signal information. The viewpoint position is detected from the infrared image stored in the unit 25, and the position information is supplied to the template creating unit 23. Further, the position information may be supplied to the image display unit 3.

The infrared light emission control unit 26 controls the lighting of the infrared light emission unit 13 according to the control of a control unit (not shown), a viewpoint position detection processing unit 24, a pattern matching processing unit, and the like. These components of the viewpoint position detection unit 2 operate under the control of a control unit (not shown).

Next, a specific operation of the system shown in FIG. 1 will be described with reference to FIGS. FIG.
5 is a flowchart illustrating an operation of the viewpoint position detecting device in the present embodiment.

First, the respective units are started, such as turning on the power of the photographing unit 1, the viewpoint position detecting unit 2, and the image display unit 3, and initializing (step S10). Next, the control unit (not shown) instructs the infrared light emission control unit 26 to emit infrared light. In response to this, the infrared light emission control unit 26 turns on the infrared light emission unit 13 (step S11) and notifies the viewpoint position detection processing unit 24 that the infrared light has been emitted. Then, the visible image captured by the visible image capturing unit 11 and the infrared image captured by the infrared image capturing unit 12 are loaded into the visible image storage unit 21 and the infrared image storage unit 25, respectively.
The infrared light emitting unit 13 is turned off. At this time, in order to create an accurate template and perform pattern matching,
It is preferable that the images taken into the respective image storage units have substantially the same timing (frame).

The emission of the infrared light is started, for example, by providing a sensor for detecting the presence or absence of an observer, automatically after the existence of the observer is confirmed, or by pressing the button through the observer. Any method can be used.

Next, the viewpoint position detection processing section 24 detects the viewpoint position from the infrared image fetched into the infrared image storage section 25 (step S12). The detection of the viewpoint position is basically performed by binarizing the image. However, in the infrared image, the reflected image that appears to the left is the right eye of the observer, and the reflected image that appears to the right is the left eye of the observer. FIG. 3 is an example of an infrared image, in which the pupil portion becomes much brighter than other portions due to retinal reflection of near-infrared light.

Since the reflection image due to retinal reflection is a partial area in the image, the position of the reflection image detected in step S12 is not a pinpoint coordinate value indicating one point in the image. The process of obtaining the viewpoint position, which is the pinpoint coordinate value in the image, from the reflection image may be appropriately performed until the template creation stage of pattern matching.

In step S13, if two points can be detected, the process proceeds to step S14. If the image cannot be detected normally due to the blinking of the observer, the process returns to step S11 to acquire an image again. Do.

If the viewpoint position detection has been completed normally in step S13, template creation processing is started.
The templates for pattern matching used in the present embodiment are two child templates and one parent template. Each type of template will be further described with reference to FIGS. 4 (a) and 4 (b).

FIGS. 4A and 4B are diagrams for explaining a child template and a parent template used in the present embodiment, respectively. As shown in the figure, the two child templates each have a viewpoint based on the viewpoint positions of the right eye and the left eye (indicated by X in the figure), and the parent template includes the viewpoint positions of the right eye and the left eye. One template is based on a point. Here, the viewpoint position in the template is a coordinate indicating one point of the coordinates in the image.

In the present embodiment, template creation processing is performed from the creation of a child template. The template creation unit 23 uses the right-eye and left-eye viewpoint positions (coordinate values on the image) detected from the infrared image by the viewpoint position detection processing unit 24 to convert the visible image stored in the visible image storage unit 21. Then, a child template 1 based on the right eye viewpoint position and a child template 2 based on the left eye viewpoint position are created (step S14). The size of the child template is determined from the following equation based on the distance between the right-eye and left-eye viewpoint positions. Average distance between the right and left eye viewpoint positions of the human: Measured distance between the right and left eye viewpoint positions = large enough to include the average human eye and eyebrows: the size of the child template As the average value of the distance between the viewpoint positions and the size of the eye and eyebrows, for example, a statistically determined value can be used.

When the child template has been created, the template creating section 23 creates a parent template (step S15). As described above, the parent template is a template including the two viewpoint positions with the middle point between the two viewpoint positions for the right eye and the left eye as a base point. The size of the parent template is determined from the following equation based on the distance between the right-eye and left-eye viewpoint positions. Average distance between human right-eye and left-eye viewpoint positions: measured distance between right-eye and left-eye viewpoint positions = size enough to include an average human face: parent template size When creating child template Similarly to the above, a statistically determined value can be used for the average value. The template created by the template creation unit 23 is supplied to the pattern matching determination unit 22.

When the creation of the template is completed, pattern matching is performed using the visible image fetched into the visible image storage unit 21 and the template created by the template creation unit 23. The pattern matching determination unit 22 first performs pattern matching between the parent template and the visible image (step S16). Pattern matching can be performed using, for example, a normalized correlation function. The pattern matching using the normalized correlation function is described in, for example, “Matrox Imaging Library Version 5.1.
User Guide (Matrox Imaging Library Version 5.1
User Guide) ”, pp. 154-155. The value obtained by the normalized correlation function is 0 to 100
(%), 100% means perfect match.

In the present embodiment, for example, if a correlation degree exceeding 85% is obtained, it is determined that the pattern matching has succeeded. In the pattern matching immediately after the template is created, since the image on which the template is based and the image data to be subjected to the pattern matching are the same, basically a correlation of almost 100% should be obtained.

If the result of pattern matching between the parent template and the visible image satisfies a predetermined degree of correlation, it is determined in step S17 that the pattern matching has been successful.
Move to step S18. On the other hand, if the degree of correlation is less than the predetermined value, it is considered that the template needs to be recreated, so the process returns to step S11, and the process is again performed from the acquisition of the infrared image.

If the pattern matching using the parent template succeeds, the pattern matching determination section 22 sets a search area for the viewpoint position (step S18). That is, the left half in the parent template is set as the search area for the right eye viewpoint position, and the right half in the parent template is set as the search area for the left eye viewpoint position. Then, pattern matching between the child template and the visible image is performed based on the set search area (step S19). In this way, by performing pattern matching hierarchically, narrowing down the search range in stages, and restricting the right eye left eye viewpoint position, detection of erroneous viewpoint position (not failure) is prevented, and accurate tracking is performed. Can be.

If the maximum correlation value satisfies the predetermined correlation value as a result of the pattern matching, it is determined that the pattern matching is successful in step S20, and the process proceeds to step S21.
Move to. On the other hand, if the maximum correlation value is less than the predetermined correlation value, the process returns to step S11 and starts again from the infrared image acquisition operation. At this time, the pattern matching determination unit 22 instructs the infrared light emission control unit 26 to emit infrared light again.

If it is determined in step S20 that the pattern matching is successful, the pattern matching determination unit 22 sends the finally obtained viewpoint position information (viewpoint position coordinates on the image) to the image display unit 3 for the right eye and the left eye. Output every time. In step S21, it is determined whether or not to end the system. If the end of the operation of the system is not instructed, a visible image is fetched from the visible image photographing unit 11, stored in the visible image storage unit 21, and then returns to step S16.

Thereafter, the pattern matching for the visible image is continued, and if the pattern matching fails, the infrared light is automatically emitted and the template is recreated. When the end processing is instructed in step S21,
After performing a predetermined end process (step S22), a series of processes is ended.

As described above, by using the retinal reflection image by infrared light, which can obtain high-precision information as viewpoint position information at the time of template creation, the irradiation time of infrared light can be suppressed to a minimum, and the observer can be observed. Concerns can also be reduced. Also, if template matching is unsuccessful, the template is automatically recreated,
Stable and accurate pattern matching results are obtained,
It can be suitably used for display control of a stereoscopic image display device as shown in FIG.

(Second Embodiment) In the first embodiment, when the pattern matching result is unsuccessful, the pattern matching is performed with high accuracy and good tracking by re-starting from the infrared image acquisition. As described above, in an environment where the subject does not move much, the same effect can be expected by periodically recreating the template. In the present embodiment, this configuration will be described.

FIG. 5 is a block diagram showing a configuration of a viewpoint position detecting device according to a second embodiment of the present invention. This figure also shows a case where the image display unit 3 is connected to form a stereoscopic image display system, similarly to FIG. 5, the same reference numerals are given to the same components as those in FIG. 1 showing the first embodiment, and the description thereof will be omitted.

The present embodiment differs from the first embodiment in that the viewpoint position detecting section 20 has a time measuring section 27. The time measuring unit 27 is a so-called timer means, and generates a signal at predetermined time intervals. This output signal is input to the infrared light emission control unit 26, and in response, the infrared light emission control unit 26 controls the infrared light emission unit 13 to emit infrared light. As a result, the same processing as when the system is started is started.

The specific operation will be described with reference to FIG. FIG. 6 is a flowchart showing the operation of the present embodiment.
6, steps that perform the same operations as those in FIG. 2 are denoted by the same reference numerals, and detailed descriptions thereof will be omitted.

As is clear from FIG. 6, the present embodiment
The processing step shown in FIG. 2 is different from the processing steps shown in FIG. 2 in that the step of determining the success of the pattern matching (steps S17 and S20) is removed, and the step S23 of determining whether or not the time measuring unit 27 outputs is added.

From step S10 for starting the system,
The processing is performed in the same manner as in FIG. 2 up to step S16 for performing pattern matching between the parent template and the visible image.
Upon completion of the pattern matching, a search area setting process for the viewpoint position is immediately performed (step S18), and a pattern matching process between the child template and the visible image is performed (step S19).

When the pattern matching between the child template and the visible image is completed in step S19, it is determined in step S23 whether or not a signal has been output from the time measuring unit 27. If no output is detected, the process proceeds to step S21, and depending on whether or not there is an instruction to end the system, the next visible image capture and pattern matching processing are repeated, or the system end processing (step S22) is performed. Do. On the other hand, when the signal output of the time measuring unit 27 is detected, the process returns to the infrared light emission step (step S11), and the processing is performed again from the acquisition of the infrared image, thereby updating the child template and the parent template.

If there is a signal output from the time measuring unit 27 during the pattern matching process in steps S16 to S19, the fact that the infrared light emission control unit 26 has detected the signal output is stored in, for example, an internal storage area. In advance, pattern matching between the child template and the visible image (step S
After the end of 19), the infrared light emitting unit 13 is instructed to emit infrared light, and the pupil position detection from the infrared image is instructed to the viewpoint position detection processing unit 24.

The signal generation cycle of the time measurement unit 27 is
It may be appropriately determined based on conditions such as the use environment of the image display unit 3 (whether the observer is easy to move, whether the observer changes frequently, and the like) and the processing time required for template creation. With such a configuration, in the present embodiment, it is possible to achieve both processing speed and detection accuracy with a simple configuration.

(Third Embodiment) In the present embodiment, a process in which the first embodiment and the second embodiment are combined is performed.
That is, in addition to the case where the result of the pattern matching is determined as failure, the template is forcibly updated at predetermined time intervals.

FIG. 7 is a block diagram showing a configuration of a viewpoint position detecting device according to the third embodiment of the present invention. This figure also shows a case where the image display unit 3 is connected to form a stereoscopic image display system, similarly to FIG. 7, the same components as those of FIG. 5 described above are denoted by the same reference numerals, and description thereof will be omitted. The components shown in FIG. 7 and the configuration shown in FIG. 5 have the same components and only differ in the connection relationship.
The operation will be described with reference to FIG.

FIG. 8 is a flowchart showing a viewpoint position detecting operation in the present embodiment. Steps having the same operations as those in FIGS. 2 and 6 are denoted by the same reference numerals, and detailed description of each step is omitted. As is clear from the comparison between FIGS. 2, 6, and 8, the present embodiment differs from the first embodiment (FIG. 2) in that a pattern matching result evaluation step S2 after pattern matching using a child template is performed.
A step S23 for determining whether or not a signal output of the time measuring unit has been detected is added between 0 and step S21 for determining the presence or absence of the end instruction.

With the addition of step S23, two pattern matching result evaluation steps S17 and S20 are performed.
In addition to the case where the pattern matching is evaluated as failed in any of the above, the template updating (creation) processing is performed for each signal output cycle of the time measuring unit 27 even if the pattern matching does not fail. As a result, the template is periodically optimized, and more accurate position detection can be realized.

(Fourth Embodiment) As described above, the viewpoint position detection using an infrared image utilizes the near-infrared reflection characteristics of the retina. Normally, the human face does not reflect infrared rays (like the retina), but an object that can reflect infrared rays, such as an observer wearing spectacles, is attached to the face or the forehead or other shooting area. In this case, three or more reflected images may be detected on the infrared image, or a very large reflected image may be detected. The viewpoint position may not be detected from such an infrared image, or even if the detection is possible, the error is large or the viewpoint position may be incorrect.

In each of the above embodiments, it is assumed that the two reflected images are clearly detected in the viewpoint position detection processing from the infrared image. Thus, the infrared image acquisition and the viewpoint position detection processing (steps S11 to S12 in FIG. 2, FIG. 6, or FIG. 8) from the acquired infrared image are repeated as many times as necessary.

However, in practice, it is possible that a normal viewpoint position cannot be detected from the infrared image even if the processing is performed many times due to reflection by eyeglasses or the like. Therefore, in the present embodiment, when the viewpoint position detection from the infrared image fails consecutively for a predetermined number of times, the viewpoint position detection by the infrared light is normally performed due to the reason that the observer wears glasses. It is characterized in that it is determined that the possibility of performing the template is low, and the template is created only from the visible image information.

The configuration of the viewpoint position detecting device according to the present embodiment may be the same as the configuration shown in FIG. 1, so that the description of each component is omitted, and the operation will be described with reference to FIG. FIG.
5 is a flowchart illustrating an operation of the viewpoint position detecting device according to the present embodiment. In the figure, the same reference numerals are given to the steps of the same processing as in FIG. 2, and the detailed description of the steps is omitted. In the following description, a case will be described as an example where the viewpoint position detection from the infrared image is abandoned when the viewpoint position detection from the infrared image has failed four consecutive times.

As shown in FIG. 9, this embodiment differs from the operation of the first embodiment shown in FIG.
4 to S26 are added. The same processing as in the first embodiment is performed from the activation of the system to the evaluation step of the viewpoint position detection result based on the infrared image (step S13).

If the detection of the viewpoint position from the infrared image has succeeded in step S13, the process proceeds to step S14, and the same processing as in the first embodiment is performed. on the other hand,
If it is determined in step S13 that the viewpoint position detection has failed, it is determined whether the number of consecutive failures has reached a predetermined number (four in this case) (step S2).
4). For example, the viewpoint position detection processing unit 24 stores the number of consecutive failures in the internal storage device. If the number of consecutive failures is less than the predetermined number, the process returns to the infrared image acquisition step (step S11), and the viewpoint position detection from the infrared image is attempted again.

On the other hand, when the number of consecutive failures reaches a predetermined number (four in this embodiment), the viewpoint position detection from the infrared image is abandoned, and a template is created only from the visible image. That is, the control unit (not shown) instructs the template creation unit 23 to perform image processing on the image data taken into the visible image storage unit 21 to extract a region and create a template. In this case, since the viewpoint position information cannot be obtained from the infrared image, the template is created first from the parent template. Specifically, first, the outline of the face is detected, and a parent template is created (step S2).
5). Next, an eye area is searched in the right area and the left area of the parent template to create a child template (step S).
26). Thereafter, the process proceeds to step S16, and the processes after the pattern matching are performed in the same manner as in the first embodiment.

In this embodiment, the step of evaluating the pattern matching result (steps S17 and S2)
In (0), the case where it is evaluated that the pattern matching has failed has been described, and the case where the processing is started again from the acquisition of the infrared image has been described. Instead of returning to S11, the system may be configured to start over from step S25, or may be configured to count the number of consecutive failures in pattern matching and start over from the acquisition of the infrared image only when the number exceeds a predetermined number.

As described above, according to the present embodiment, it is possible to appropriately deal with a problem that may actually occur, and to avoid the worst case where the viewpoint position cannot be detected.

(Fifth Embodiment) In the fourth embodiment,
In the case where it is difficult to detect the viewpoint position from the infrared image, for example, because the subject wears glasses, two templates are created from only the visible image, but in the present embodiment, It is characterized in that one template is created using the information of the infrared image. Since the configuration of the viewpoint position detecting device according to the present embodiment may be the same as the configuration shown in FIG. 1, the description of each component will be omitted, and the operation will be described with reference to FIG.

FIG. 10 is a flowchart showing the operation of the viewpoint position detecting device according to this embodiment. In the figure,
Steps of the same processing as in FIG. 2 are denoted by the same reference numerals, and detailed description of the steps is omitted.
Further, in the following description, as in the fourth embodiment, a case will be described as an example where the viewpoint position detection from the infrared image is abandoned when the viewpoint position detection from the infrared image has failed four consecutive times. . As shown in FIG. 10, the present embodiment differs from the operation in the fourth embodiment shown in FIG.
33 is added. The same processing as in the first embodiment is performed from the activation of the system to the evaluation step of the viewpoint position detection result based on the infrared image (step S13).

If the detection of the viewpoint position from the infrared image is successful in step S13, the process proceeds to step S14, and the same processing as in the first embodiment is performed. on the other hand,
If it is determined in step S13 that the viewpoint position detection has failed, it is determined whether the number of consecutive failures has reached a predetermined number (four in this case) (step S2).
4). For example, the viewpoint position detection processing unit 24 stores the number of consecutive failures in the internal storage device. If the number of consecutive failures is less than the predetermined number, the process returns to the infrared image acquisition step (step S11), and the viewpoint position detection from the infrared image is attempted again.

On the other hand, when the number of consecutive failures reaches a predetermined number (four in this embodiment), the viewpoint position detection from the infrared image is abandoned, and the visual image is detected using information obtained from the infrared image. Create a template from. This means, for example, that the subject is wearing spectacles, and even if the viewpoint position cannot be detected from the infrared image due to the reflection of the spectacle frame or lens, the spectacles are present at the position of the eyes,
Because there is a high possibility that the viewpoint exists in the reflection image, it is considered that it is useful as base point position information when creating a template.

For this reason, in the present embodiment, the viewpoint position detection processing section 24 calculates the center of gravity of the area including all the reflected lights included in the infrared image, and the template creating section 23 uses the center of gravity as a base point for the visible image. (Step S30). At this time, both the parent template and the child template are created in the fourth embodiment, but one template is created in the present embodiment. The size of the template is determined, for example, from information such as the horizontal width and the vertical width of the area including all the reflected light. After the template is created, a position in the template where the viewpoint is considered to be present is estimated, and the viewpoint positions of the left eye and the right eye are set.

When the creation of the template is completed, pattern matching with the visible image is performed using the template (step S32). If it is determined that the pattern matching is successful and there is no instruction for the end processing in step S33, the process returns to step S31 to continue the processing. on the other hand,
If it is determined in step S32 that the pattern matching has failed, the process returns to step S11 to start again from the acquisition of the infrared image. In the case of the present embodiment, since a highly accurate pattern matching result cannot be expected from the beginning, the evaluation criterion of the pattern matching result in step S32 is set somewhat looser than the criterion in the first embodiment that enables accurate viewpoint position detection from an infrared image. Etc. may be changed.

In this embodiment, in the step of evaluating the pattern matching result (step S32), when it is evaluated that the pattern matching has failed, the case where the processing is started again from the acquisition of the infrared image is shown. If the template is created using the information of the infrared image due to a failure, the process does not return to step S11 but proceeds to step S11.
Alternatively, the system may be configured to start over from 30 or to start from the acquisition of the infrared image only when the number of consecutive failures in pattern matching is counted and the number of times exceeds a predetermined number.

As described above, according to the present embodiment, it is possible to appropriately deal with a problem that can actually occur, and to avoid the worst case where the viewpoint position cannot be detected.

[0087]

[Other Embodiments] In the above embodiments, the case where the viewpoint position detection initiative of the present invention has the initiative of viewpoint position detection has been described. When the image display unit 3 or the remote controller is used, a button or the like that can be operated by the user is provided, and when the user has difficulty in recognizing the stereoscopic display, the user presses this button to start the infrared image acquisition operation. It is also possible to configure so that the viewpoint position detecting operation is redone. With such a configuration, the template is updated at an accurate timing, and the viewpoint position detection can be performed with higher accuracy. As a result, a stereoscopic image display system with a wide stereoscopic view range can be realized. .

When the infrared light is emitted, the amount of the emitted light may be changed in accordance with the brightness of the image display unit 3 or the distance to the observer. In (2), the light emission amount can be changed (increased or decreased) from the previous time. Such control of the light emission amount
The probability of successful viewpoint detection from the infrared image is increased, which results in obtaining a highly accurate viewpoint position detection result.

Further, in the above-described embodiment, the case where the detection result of the viewpoint position detecting device according to the present invention is supplied to the stereoscopic image display device has been described as an example. However, the viewpoint position detecting device according to the present invention is applicable to any application. Can be used.

Further, the specific methods described in the above embodiments, such as the pattern matching method and the template creation method, are not limited to those described in the embodiments, but may be equally applicable. It is needless to say that may be used.

In the above-described embodiment, the viewpoint position, which is the coordinate of the pinpoint, is configured to be output. However, for example, as in the above-described embodiment, the viewpoint position finally obtained is determined by the three-dimensional image display device. In the case of using for control, since the minimum control is possible if the center position of the viewpoint positions of the right eye and the left eye is known, the center position of the template may be output to the image display unit 3. In particular, when accurate viewpoint position information cannot be obtained from an infrared image as described in the fourth and fifth embodiments, such a configuration is effective.

Further, when it is difficult to detect the viewpoint position from the infrared image, in the case of Embodiments 4 and 5 in which the method is switched to another method, for example, it is assumed that the subject is wearing glasses. A means (such as a button) for instructing the viewpoint position detecting device may be provided so that when this button is pressed, the process shifts to the substitution step from the beginning. In this case, in the fourth embodiment, step S10 in FIG.
May be configured to directly acquire the visible image from step S <b> 25 and proceed to step S <b> 25, or proceed to step S <b> 25 when it is determined in step S <b> 13 that the viewpoint position detection from the infrared light has first failed. Similarly, in the fifth embodiment, an infrared image is acquired in step S11, and the process directly proceeds to step S30, or proceeds to step S30 when it is determined that the viewpoint position detection from infrared light has failed for the first time in step S13. It may be configured as follows.

The present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), and can be applied to a single device (for example, a copier, a facsimile). Device).

Further, an object of the present invention is to supply a storage medium (or a recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or an apparatus, and to provide the computer (or computer) of the system or the apparatus It is needless to say that the present invention can also be achieved by a CPU or an MPU) reading and executing the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. By executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written into the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, the program code is read based on the instruction of the program code. Needless to say, the CPU included in the function expansion card or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the above-described flowcharts (shown in FIGS. 2, 6, 8 to 10). Will be done.

[0098]

As described above, according to the present invention,
In a viewpoint position detection device that detects a viewpoint position of an observer and outputs viewpoint position information, an infrared image is used for initial template creation, and thereafter, a pattern matching of a visible image is performed. It is not necessary to continuously irradiate light to the observer, minimizing concerns about the influence on the observer, and creating a template using highly accurate position detection information from an infrared image in a short time Therefore, the template creation process can be reduced, and a template with high accuracy can be created.

Further, when the pattern matching of the visible image fails, such as when the observer changes or moves greatly, or by updating the template periodically, a template with high accuracy is always maintained. As a result, there is an effect that highly accurate viewpoint position information can be stably obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a viewpoint position detecting device according to a first embodiment of the present invention.

FIG. 2 is a flowchart illustrating the operation of the first embodiment of the present invention.

FIG. 3 is a diagram illustrating an example of an image obtained by infrared light.

FIG. 4 is a diagram illustrating a configuration of a template used in the embodiment of the present invention.

FIG. 5 is a block diagram illustrating a configuration of a viewpoint position detecting device according to a second embodiment of the present invention.

FIG. 6 is a flowchart illustrating the operation of the second embodiment of the present invention.

FIG. 7 is a block diagram illustrating a configuration of a viewpoint position detecting device according to a third embodiment of the present invention.

FIG. 8 is a flowchart illustrating the operation of the third embodiment of the present invention.

FIG. 9 is a flowchart illustrating the operation of the fourth embodiment of the present invention.

FIG. 10 is a flowchart illustrating the operation of the fifth embodiment of the present invention.

FIG. 11 is a perspective view showing a configuration of a rear cross lenticular type stereoscopic image display device.

FIG. 12 is a perspective view illustrating an example of a stereoscopic image display device that performs display control according to a viewpoint position.

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] March 24, 2000 (200.3.2.
4)

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Full text

[Correction method] Change

[Correction contents]

[Document Name] Statement

Patent application title: Viewpoint position detecting device, method and stereoscopic image display system

[Claims]

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a viewpoint detection apparatus and a viewpoint detection method for detecting a viewpoint position of a person to be measured, and more particularly to a viewpoint detection apparatus and a viewpoint detection method that achieve both high-speed processing and high detection accuracy. .

[0002]

2. Description of the Related Art Conventionally, there is a so-called stereoscopic image display device capable of stereoscopic viewing with the naked eye. There are various types of stereoscopic image display devices. For example, JP-A-09-31294 discloses a rear cross lenticular type. FIG. 11 is a perspective view of an essential part showing an example of a rear cross lenticular type stereoscopic image display device. In the figure, reference numeral 6 denotes a display device for displaying images, which is constituted by, for example, a liquid crystal element (LCD). In the figure, a polarizing plate, a color filter, an electrode, a black matrix, an antireflection film, and the like are omitted.

[0003] Reference numeral 10 denotes a backlight (surface light source) serving as an illumination light source. Display device 6 and backlight 1
Between 0, a mask substrate (mask) 7 on which a mask pattern having a checkered opening 8 is formed is arranged. The mask pattern is manufactured by patterning a metal deposited film such as chromium or a light absorbing material on a mask substrate 7 made of glass or resin. The backlight 10, the mask substrate 7, and the like constitute one element of the light source.

A first lenticular lens 3 and a second lenticular lens 4 made of transparent resin or glass are arranged between the mask substrate 7 and the display device 6. The first lenticular lens 3 is a vertical cylindrical lens array formed by arranging vertically long vertical cylindrical lenses in the left-right direction, and the second lenticular lens 4 is formed by arranging vertically long horizontal cylindrical lenses in the horizontal direction. It is a cylindrical lens array.

As shown in the figure, the image displayed on the display device 6 divides the left and right parallax images R and L into a large number of horizontal stripe pixels R and L in the vertical direction, and divides them into, for example, LRLLRLR from the screen. ...
And a horizontal stripe image alternately arranged to form one image.

The light from the backlight 10 is applied to the mask substrate 7
Then, the display device 6 is illuminated through each of the openings 8, and the left and right stripe pixels R and L are separately observed by both eyes of the observer.

That is, the mask substrate 7 is
0 illuminates, and light exits from the aperture 8. The first lenticular lens 3 is arranged on the observer side of the mask substrate 7, and the lens curvature is designed so that the mask substrate 7 is almost at the focal position of each cylindrical lens. In this cross section, the second lenticular lens 4 has no optical effect, so that the light beam emitted from one point on the opening 8 is converted into a substantially parallel light beam in this cross section.

The pair of openings and light-shielding portions of the mask pattern are set to substantially correspond to one pitch of the first lenticular lens 3.

Further, based on the relationship between the optical distance from a predetermined position of the observer to the first lenticular lens 3 and the optical distance from the first lenticular lens 3 to the mask pattern, the first lenticular lens 3 is formed. By determining the pitch of the mask pattern and the pitch of the pair of openings and the light-shielding portion of the mask pattern, light from the openings 8 can be uniformly collected to the left eye or the right eye over the entire width of the screen.
In this way, the left and right stripe pixels on the display device 6 are observed separately in the horizontal direction into the left eye and right eye regions.

The second lenticular lens 4 condenses all light beams emitted from each point on the opening 8 of the mask 7 on the right-eye or left-eye stripe pixel of the display device 6, illuminates and transmits the same. It diverges only in the vertical direction according to the NA at the time of condensing, and provides an observation area in which the left and right stripe pixels are uniformly separated from the predetermined eye height of the observer over the entire vertical width of the screen. .

However, the viewing angle of such a stereoscopic image display device is narrow, and if the observer's viewpoint deviates from the viewing angle, the stereoscopic display cannot be recognized. Therefore, there has been proposed a technique of detecting a viewpoint position of an observer and controlling image display in response to the movement of the viewpoint position, thereby enlarging a region where stereoscopic viewing is possible. For example, Japanese Patent Application Laid-Open No. H10-2323
No. 67 discloses a technique for expanding a stereoscopically visible area by moving a mask pattern or a lenticular lens in parallel to a display surface.

FIG. 12 is a diagram of a stereoscopic image display device disclosed in Japanese Patent Application Laid-Open No. 10-232367. 12, the same reference numerals are given to the same components as those in FIG. 11, and the description will be omitted. Since the three-dimensional image display device in FIG. 12 has a single lenticular lens, it does not include the second lenticular lens 4 in FIG.

In the three-dimensional image display device having such a configuration, control according to the movement of the observer 54 is performed as follows. First, the position sensor 51 detects a horizontal displacement of the observer 54 from a preset reference position,
The information is sent to the control unit 52, and when the control unit 52 outputs an image control signal to the display drive circuit 50 according to the shift information, the display drive circuit 50 displays the first or second horizontal stripe image on the display 6. I do. At the same time, the control unit 52 generates an actuator drive signal based on the displacement information, and drives the actuator 53 that moves the mask pattern 7 in the horizontal direction. Move to
As a result, even if the viewpoint position of the observer 54 changes, the range that can be stereoscopically viewed is expanded.

As described above, when the display is controlled in accordance with the viewpoint position of the observer, an image suitable for the observer's viewpoint can be displayed even if the detection accuracy is low or the processing time for detection is long. Absent. Therefore, it is very important to detect the viewpoint position of the observer with high accuracy and in a short time in the performance of the display device.

As the method of detecting the viewpoint position of the observer (measured person), 1) a method of irradiating the observer with infrared light and detecting the reflection of the retina (for example, see Banno "Pupil optics for viewpoint detection" System Design Method, IEICE Transactions D-II, Vol.J74-D-
II, No.6, pp.736-747, described in June 1991) 2) A method of processing visible images to detect the eyes of the observer (for example, Sakaguchi et al. Real-Time Face Expression Recognition Using "IEICE Transactions
D-II, Vol.J80-D-II, No.6, pp.1547-1554 1997/6
3) A method of detecting an observer's eyes by image processing using an infrared image and a visible image (for example, see Japanese Patent Application Laid-Open No. 8-28721).
No. 6).

The method 1) utilizes the fact that the human pupil has the property of retroreflecting near-infrared light (returning light in the same direction as the incident direction). Pupil reflected light is obtained as a sharp reflection peak, and usually shows a higher reflectance than the face, etc., so that only the pupil part is captured by imaging the observer using an infrared imaging device whose optical axis is coaxial with the light source. Bright images can be taken. If this photographed image is binarized with an appropriate threshold, an accurate viewpoint position can be detected from the extracted pupil position.

In the method 2), the position of the observer in the photographing range is limited in advance, the observer blinks in that state, and the eye region is extracted by the inter-frame image of the visible image. are doing. Created based on the eye area range
Viewpoint by pattern matching using template
Perform position detection.

In the method 3), the infrared image and the visible
Images at the same time and extract facial regions from these images
After that, a characteristic region such as an eye is detected using pattern matching . The infrared image is used for extracting a person candidate area and determining a temperature threshold used when extracting a skin color area from a color image.

[0019]

However, in the method (1), since it is necessary to continuously irradiate the observer with relatively strong infrared light, the influence of the infrared light on the observer is small. In addition to the problem of concern, there is also a problem that detection cannot be performed when the observer blinks because reflection by the retina is used. In addition, depending on the pupil enlargement or reduction due to the illuminance of the environment,
There is also a problem that it is difficult to follow the pupil reflection image.

In the method 2), the observer is required to perform operations such as adjustment of the observation position and blinking, which is troublesome for the observer. It requires time for operations such as adjustment of the observation position and blinking, which is not practical.

In the method 3), the irradiation intensity of infrared light may be lower than that in the method 1). However, after the intermediate processing result of the infrared image is obtained, the visible image is obtained by using the processing result. Is processed, the face area is detected using the processing result of the infrared image and the processing result of the visible image, and further pattern matching is performed. Therefore, the processing is very complicated. There is also a problem that it is not easy to create a template used for pattern matching. In addition, since the position of the face required to create a template for pattern matching is detected only from the visible image, there is a problem that the position accuracy is not very good.

Accordingly, it is an object of the present invention to provide a viewpoint position detecting apparatus and method which have high accuracy and followability in a short time, with a simple configuration and while suppressing the influence on the human body. It is another object of the present invention to provide a stereoscopic image display system having a stereoscopic image display device that controls display using viewpoint position information obtained by using the viewpoint position detecting device or method of the present invention. .

[0023]

That is, the gist of the present invention is a viewpoint position detecting device for detecting a viewpoint position of a person to be measured and outputting viewpoint position information, and acquiring an infrared image of the person to be measured. Infrared image acquiring means for acquiring, a visible image acquiring means for acquiring a visible image of the subject, detecting means for detecting a pupil position of the subject from the infrared image acquired by the infrared image acquiring means, A template creation unit that creates a template for pattern matching using the pupil position from the visible image acquired by the image acquisition unit, and a visible image acquired by the visible image acquisition unit using the template created by the template creation unit and pattern matching And a matching means for detecting the viewpoint position of the person to be measured and outputting the result as viewpoint position information.

Another aspect of the present invention is a viewpoint position detecting apparatus for detecting a viewpoint position of a person to be measured and outputting viewpoint position information. Acquisition means, visible image acquisition means for acquiring a visible image of the subject, from the infrared image acquired by the infrared image acquisition means,
Detecting means for detecting the pupil position of the subject, template creating means for creating a template for pattern matching using the pupil position from the visible image acquired by the visible image acquiring means, and a template created by the template creating means. Using the visible image acquired by the visible image acquiring means using
A template is created again using a matching unit that performs pattern matching , detects the viewpoint position of the subject, and outputs the detection result as viewpoint position information, and a detection unit and a template creation unit when a predetermined condition is satisfied. And a control means for controlling the visual point position.

Another aspect of the present invention is a stereoscopic image display system including the above-described viewpoint position detecting device according to the present invention and a stereoscopic image display device connected to the viewpoint position detecting device. There is provided a stereoscopic image display system that controls a stereoscopic image display device using viewpoint position information received from a detection device.

Another aspect of the present invention is a viewpoint position detecting method for detecting a viewpoint position of a person to be measured and outputting viewpoint position information, the method comprising detecting an infrared image of the person to be measured. An acquiring step, a visible image acquiring step of acquiring a visible image of the subject, a detecting step of detecting a pupil position of the subject from the infrared image acquired in the infrared image acquiring step, and a visible image acquiring step. From the acquired visible image,
A template creating step for creating a template for pattern matching using the pupil position, and using the template created in the template creating step, performing pattern matching with the visible image acquired in the visible image acquiring step to measure the subject. And a matching step of detecting the viewpoint position of the image and outputting the result as viewpoint position information.

Another subject of the present invention is a viewpoint position detecting method for detecting a viewpoint position of a person to be measured and outputting viewpoint position information. An acquiring step, a visible image acquiring step of acquiring a visible image of the subject, a detecting step of detecting a pupil position of the subject from the infrared image acquired in the infrared image acquiring step, and a visible image acquiring step. From the acquired visible image,
Performs a template creating step of creating a template for pattern matching with a pupil position, the visible image and the pattern matching <br/> for obtaining a visible image acquisition step using a template created by the template creation step, the person to be measured A template is created again using the matching step of detecting the viewpoint position of the object and outputting the detection result as viewpoint position information, and the detection step and the template creation step when a predetermined condition is satisfied. The present invention resides in a viewpoint position detection method characterized by repeatedly performing an image acquisition step and a matching step.

Another aspect of the present invention resides in a computer-readable storage medium storing a viewpoint position detecting method according to the present invention as a computer-executable program.

[0029]

Embodiments of the present invention will be described below in detail with reference to the drawings. In the following description, a stereoscopic image display system in which the viewpoint position detecting device of the present invention is connected to a stereoscopic image display device will be described, but the use of the viewpoint position detecting device of the present invention is limited to the stereoscopic image display device system. is not.

Also, in the present invention, the viewpoint position means the coordinates of a certain point indicating the position of the eyes of the observer. The viewpoint position information output by the viewpoint position detecting device of the present invention is not necessarily the coordinate value of one point. It does not have to be, and may be information indicating a certain area. Depending on the use, the position of the entire eye may be roughly known, and may be appropriately selected depending on the use.

(First Embodiment) FIG. 1 is a block diagram showing a configuration of a viewpoint position detecting device according to the present invention. The viewpoint position detection device according to the present embodiment includes an imaging unit 1 and a viewpoint position detection unit 2, and the image display unit 3 is the above-described stereoscopic image display device. Hereinafter, the viewpoint position detecting device and the image display unit 3 may be collectively referred to as a stereoscopic image display system.

The photographing section 1 comprises a visible image photographing section 11, an infrared image photographing section 12, and an infrared light emitting section 13, and photographs a visible image and an infrared image of an observer. Each of the visible image photographing section 11 and the infrared image photographing section 12 can be constituted by a video camera, and the infrared image photographing section 12 causes only infrared light to enter the internal light receiving element by a filter or the like. The infrared light emitting unit 13 can be composed of an infrared light emitting element such as an LED, for example.
The light emission amount necessary for obtaining retinal reflection can be obtained at a distance within a range which can be normally taken with the above.

The viewpoint position detecting section 2 includes a visible image storing section 21
, A pattern matching determination unit 22, a template creation unit 23, a pupil position detection processing unit 24, an infrared image storage unit 25, and an infrared light emission control unit 26. The viewpoint position detecting section 2 can be constituted by a general-purpose personal computer capable of storing an image signal output from the photographing section 1, for example.

The visible image storage unit 21 and the infrared image storage unit 2
5 is used as a means for storing image data photographed by the corresponding photographing units 11 and 12, and may be constituted by a semiconductor memory such as a RAM or by using a storage device such as a magnetic disk or an optical disk. Is also good.

The pattern matching determination unit 22 uses the template supplied from the template creation unit 23 to display position information of an area having the highest correlation with the template in the image stored in the visible image storage unit 21 on the image display unit. Output to 3. In addition, when the pattern matching fails, the infrared light emission control unit 26 outputs the infrared light emission unit 13 to emit light.

The template creating section 23 detects a pupil position.
Using the position information supplied from the processing unit 24 , a template for pattern matching used by the pattern matching determination unit 22 is created from the image data stored in the visible image storage unit 21.

The pupil position detection processing unit 24 receives a signal indicating that the infrared light emission control unit 26 has emitted light from the infrared light emission unit 13 and stores an infrared image based on the signal information. The pupil position is detected from the infrared image stored in the unit 25, and the position information is supplied to the template creating unit 23. Further, the position information may be supplied to the image display unit 3.

The infrared light emission control unit 26 controls the lighting of the infrared light emission unit 13 in accordance with the control of a control unit (not shown), a pupil position detection processing unit 24 , a pattern matching processing unit, and the like. These components of the viewpoint position detection unit 2 operate under the control of a control unit (not shown).

Next, a specific operation of the system shown in FIG. 1 will be described with reference to FIGS. FIG.
5 is a flowchart illustrating an operation of the viewpoint position detecting device in the present embodiment.

First, the respective units are started, such as turning on the power of the photographing unit 1, the viewpoint position detecting unit 2, and the image display unit 3, and initializing (step S10). Next, the control unit (not shown) instructs the infrared light emission control unit 26 to emit infrared light. In response to this, the infrared light emission control unit 26 turns on the infrared light emission unit 13 (step S11), and the pupil position detection processing unit
24 is notified that infrared light has been emitted. Then, the visible image captured by the visible image capturing unit 11 and the infrared image captured by the infrared image capturing unit 12 are loaded into the visible image storage unit 21 and the infrared image storage unit 25, respectively.
The infrared light emitting unit 13 is turned off. At this time, in order to create an accurate template and perform pattern matching,
It is preferable that the images taken into the respective image storage units have substantially the same timing (frame).

The emission of the infrared light is started, for example, by providing a sensor for detecting the presence or absence of an observer, automatically after the existence of the observer is confirmed, or by pressing the button through the observer. Any method can be used.

Next, the pupil position detection processing section 24 detects a pupil position from the infrared image fetched into the infrared image storage section 25 (step S12). The detection of the pupil position is basically performed by binarizing the image. However, in the infrared image, the reflected image that appears to the left is the right eye of the observer, and the reflected image that appears to the right is the left eye of the observer. FIG. 3 is an example of an infrared image, in which the pupil portion becomes much brighter than other portions due to retinal reflection of near-infrared light.

Since the reflection image due to retinal reflection is a partial area in the image, the position of the reflection image detected in step S12 is not a pinpoint coordinate value indicating one point in the image. The process of obtaining the pupil position , which is the coordinate value of the pinpoint image, from the reflection image may be appropriately performed before the template matching stage of pattern matching.

In step S13, if two points can be detected, the process proceeds to step S14. If the image cannot be detected normally due to the blinking of the observer, the process returns to step S11 to acquire an image again. Do.

If the detection of the pupil position is normally completed in step S13, the process of creating a template is started.
The templates for pattern matching used in the present embodiment are two child templates and one parent template. Each type of template will be further described with reference to FIGS. 4 (a) and 4 (b).

FIGS. 4A and 4B are diagrams for explaining a child template and a parent template used in the present embodiment, respectively. As shown in the figure, the two child templates each have a viewpoint based on the viewpoint positions of the right eye and the left eye (indicated by X in the figure), and the parent template includes the viewpoint positions of the right eye and the left eye. One template is based on a point. Here, the viewpoint position in the template is a coordinate indicating one point of the coordinates in the image.

In the present embodiment, template creation processing is performed from the creation of a child template. The template creating unit 23 uses the pupil positions (coordinate values on the image) of the right eye and the left eye detected by the pupil position detection processing unit 24 from the infrared image to extract the visible image stored in the visible image storage unit 21. , Right
Child template 1 based on eye pupil position , left eye pupil position
A child template 2 based on the position is created (step S14). The size of the child template is the pupil position of the right eye and the left eye
The distance is determined from the following equation according to the distance between the units . Average human
Distance between pupil positions of right eye and left eye : measured pupil position of right eye and left eye
Distance between positions = large enough to include average human eyes and eyebrows: size of child template Here, the distance between pupil positions and the average value of the size between eyes and eyebrows are: For example, a value obtained statistically can be used.

When the child template has been created, the template creating section 23 creates a parent template (step S15). As described above, the parent template has two pupil positions based on the midpoint of the two pupil positions of the right eye and the left eye
This is a template that includes the location. The size of the parent template is determined from the following equation based on the distance between the pupil positions of the right and left eyes . Average distance between pupil positions of right and left eyes of human: measured distance between pupil positions of right and left eyes = size enough to include average human face: size of parent template When creating child template Similarly to the above, a statistically determined value can be used for the average value. The template created by the template creation unit 23 is supplied to the pattern matching determination unit 22.

When the creation of the template is completed, pattern matching is performed using the visible image fetched into the visible image storage unit 21 and the template created by the template creation unit 23. The pattern matching determination unit 22 first performs pattern matching between the parent template and the visible image (step S16). Pattern matching can be performed using, for example, a normalized correlation function. The pattern matching using the normalized correlation function is described in, for example, “Matrox Imaging Library Version 5.1.
User Guide (Matrox Imaging Library Version 5.1
User Guide) ”, pp. 154-155. The value obtained by the normalized correlation function is 0 to 100
(%), 100% means perfect match.
Also, pattern matching is template matching
Sometimes called.

In the present embodiment, for example, if a correlation degree exceeding 85% is obtained, it is determined that the pattern matching has succeeded. In the pattern matching immediately after the template is created, since the image on which the template is based and the image data to be subjected to the pattern matching are the same, basically a correlation of almost 100% should be obtained.

If the result of pattern matching between the parent template and the visible image satisfies a predetermined degree of correlation, it is determined in step S17 that the pattern matching has been successful.
Move to step S18. On the other hand, if the degree of correlation is less than the predetermined value, it is considered that the template needs to be recreated, so the process returns to step S11, and the process is again performed from the acquisition of the infrared image.

If the pattern matching using the parent template succeeds, the pattern matching determination section 22 sets a search area for the viewpoint position (step S18). That is, the left half in the parent template is set as the search area for the right eye viewpoint position, and the right half in the parent template is set as the search area for the left eye viewpoint position. Then, pattern matching between the child template and the visible image is performed based on the set search area (step S19). In this way, by performing pattern matching hierarchically, narrowing down the search range in stages, and restricting the right eye left eye viewpoint position, detection of erroneous viewpoint position (not failure) is prevented, and accurate tracking is performed. Can be.

If the maximum correlation value satisfies the predetermined correlation value as a result of the pattern matching, it is determined that the pattern matching is successful in step S20, and the process proceeds to step S21.
Move to. On the other hand, if the maximum correlation value is less than the predetermined correlation value, the process returns to step S11 and starts again from the infrared image acquisition operation. At this time, the pattern matching determination unit 22 instructs the infrared light emission control unit 26 to emit infrared light again.

If it is determined in step S20 that the pattern matching is successful, the pattern matching determination unit 22 sends the finally obtained viewpoint position information (viewpoint position coordinates on the image) to the image display unit 3 for the right eye and the left eye. Output every time. In step S21, it is determined whether or not to end the system. If the end of the operation of the system is not instructed, a visible image is fetched from the visible image photographing unit 11, stored in the visible image storage unit 21, and then returns to step S16.

Thereafter, the pattern matching for the visible image is continued, and if the pattern matching fails, the infrared light is automatically emitted and the template is recreated. When the end processing is instructed in step S21,
After performing a predetermined end process (step S22), a series of processes is ended.

As described above, the pupil position when the template is created
By using a retinal reflection image by infrared light that can obtain highly accurate information as the placement information , the irradiation time of infrared light can be minimized, and concerns about the influence on the observer can be reduced. In addition, when the pattern matching is unsuccessful, the template is automatically re-created, so that a stable and accurate pattern matching result can be obtained.
It can be suitably used for display control of a stereoscopic image display device as shown in FIG.

(Second Embodiment) In the first embodiment, when the pattern matching result is unsuccessful, the pattern matching is performed with high accuracy and good tracking by re-starting from the infrared image acquisition. As described above, in an environment where the subject does not move much, the same effect can be expected by periodically recreating the template. In the present embodiment, this configuration will be described.

FIG. 5 is a block diagram showing a configuration of a viewpoint position detecting device according to a second embodiment of the present invention. This figure also shows a case where the image display unit 3 is connected to form a stereoscopic image display system, similarly to FIG. 5, the same reference numerals are given to the same components as those in FIG. 1 showing the first embodiment, and the description thereof will be omitted.

The present embodiment differs from the first embodiment in that the viewpoint position detecting section 2 has a time measuring section 27. The time measuring unit 27 is a so-called timer means, and generates a signal at predetermined time intervals. This output signal is input to the infrared light emission control unit 26, and in response, the infrared light emission control unit 26 controls the infrared light emission unit 13 to emit infrared light. As a result, the same processing as when the system is started is started.

The specific operation will be described with reference to FIG. FIG. 6 is a flowchart showing the operation of the present embodiment.
6, steps that perform the same operations as those in FIG. 2 are denoted by the same reference numerals, and detailed descriptions thereof will be omitted.

As is clear from FIG. 6, the present embodiment
The processing step shown in FIG. 2 is different from the processing steps shown in FIG. 2 in that the step of determining the success of the pattern matching (steps S17 and S20) is removed, and the step S23 of determining whether or not the time measuring unit 27 outputs is added.

From step S10 for starting the system,
The processing is performed in the same manner as in FIG. 2 up to step S16 for performing pattern matching between the parent template and the visible image.
Upon completion of the pattern matching, a search area setting process for the viewpoint position is immediately performed (step S18), and a pattern matching process between the child template and the visible image is performed (step S19).

When the pattern matching between the child template and the visible image is completed in step S19, it is determined in step S23 whether or not a signal has been output from the time measuring unit 27. If no output is detected, the process proceeds to step S21, and depending on whether or not there is an instruction to end the system, the next visible image capture and pattern matching processing are repeated, or the system end processing (step S22) is performed. Do. On the other hand, when the signal output of the time measuring unit 27 is detected, the process returns to the infrared light emission step (step S11), and the processing is performed again from the acquisition of the infrared image, thereby updating the child template and the parent template.

If there is a signal output from the time measuring unit 27 during the pattern matching process in steps S16 to S19, the fact that the infrared light emission control unit 26 has detected the signal output is stored in, for example, an internal storage area. In advance, pattern matching between the child template and the visible image (step S
After the completion of step 19), the infrared light emission unit 13 is instructed to emit infrared light, and the pupil position detection processing unit 24 is instructed to detect the pupil position from the infrared image.

The signal generation cycle of the time measurement unit 27 is
It may be appropriately determined based on conditions such as the use environment of the image display unit 3 (whether the observer is easy to move, whether the observer changes frequently, and the like) and the processing time required for template creation. With such a configuration, in the present embodiment, it is possible to achieve both processing speed and detection accuracy with a simple configuration.

(Third Embodiment) In the present embodiment, a process in which the first embodiment and the second embodiment are combined is performed.
That is, in addition to the case where the result of the pattern matching is determined as failure, the template is forcibly updated at predetermined time intervals.

FIG. 7 is a block diagram showing a configuration of a viewpoint position detecting device according to the third embodiment of the present invention. This figure also shows a case where the image display unit 3 is connected to form a stereoscopic image display system, similarly to FIG. 7, the same components as those of FIG. 5 described above are denoted by the same reference numerals, and description thereof will be omitted. The components shown in FIG. 7 and the configuration shown in FIG. 5 have the same components and only differ in the connection relationship.
The operation will be described with reference to FIG.

FIG. 8 is a flowchart showing a viewpoint position detecting operation in the present embodiment. Steps having the same operations as those in FIGS. 2 and 6 are denoted by the same reference numerals, and detailed description of each step is omitted. As is clear from the comparison between FIGS. 2, 6, and 8, the present embodiment differs from the first embodiment (FIG. 2) in that a pattern matching result evaluation step S2 after pattern matching using a child template is performed.
A step S23 for determining whether or not a signal output of the time measuring unit has been detected is added between 0 and step S21 for determining the presence or absence of the end instruction.

With the addition of step S23, two pattern matching result evaluation steps S17 and S20 are performed.
In addition to the case where the pattern matching is evaluated as failed in any of the above, the template updating (creation) processing is performed for each signal output cycle of the time measuring unit 27 even if the pattern matching does not fail. As a result, the template is periodically optimized, and more accurate position detection can be realized.

(Fourth Embodiment) As described above, the pupil position detection using an infrared image utilizes the near-infrared reflection characteristics of the retina. Normally, the human face does not reflect infrared rays (like the retina), but an object that can reflect infrared rays, such as an observer wearing spectacles, is attached to the face or the forehead or other shooting area. In this case, three or more reflected images may be detected on the infrared image, or a very large reflected image may be detected. The pupil position cannot be detected from such an infrared image, or even if the pupil position can be detected, an error is large or an incorrect pupil position may be obtained.

In each of the above-described embodiments, it is assumed that two reflected images are clearly detected in the pupil position detection processing from an infrared image. And the pupil position detection processing (steps S11 to S12 in FIG. 2, FIG. 6, or FIG. 8) from the acquired infrared image is repeated any number of times.

However, in reality, even if the processing is performed many times, the normal pupil is obtained from the infrared image due to the reflection by the glasses.
Failure to detect the hole position may occur. Therefore, in the present embodiment, when pupil position detection from an infrared image fails consecutively for a predetermined number of times, pupil position detection by infrared light is performed normally due to reasons such as the viewer wearing glasses. It is characterized in that it is determined that the possibility of performing the template is low, and the template is created only from the visible image information.

The configuration of the viewpoint position detecting device according to the present embodiment may be the same as the configuration shown in FIG. 1, so that the description of each component is omitted, and the operation will be described with reference to FIG. FIG.
5 is a flowchart illustrating an operation of the viewpoint position detecting device according to the present embodiment. In the figure, the same reference numerals are given to the steps of the same processing as in FIG. 2, and the detailed description of the steps is omitted. In the following description, it will be described as an example the case of abandoned pupil position detection from the infrared image when the pupil position detection from the infrared image fails four consecutive times.

As shown in FIG. 9, this embodiment differs from the operation of the first embodiment shown in FIG.
4 to S26 are added. The same processing as in the first embodiment is performed from the activation of the system to the evaluation step of the pupil position detection result based on the infrared image (step S13).

In step S13, the infrared image
If the pupil position detection has succeeded, the process shifts to step S14, and the same processing as in the first embodiment is performed thereafter. on the other hand,
If it is determined in step S13 that the pupil position detection has failed, it is determined whether the number of consecutive failures has reached a predetermined number (four in this case) (step S2).
4). The number of consecutive failures can be determined by, for example, the pupil position detection process.
The processing unit 24 stores the information in the internal storage device. If the number of consecutive failures is less than the predetermined number, the process returns to the infrared image acquisition step (step S11), and the pupil position detection from the infrared image is performed again.
Try to get out .

On the other hand, if the number of consecutive failures reaches a predetermined number (four in this embodiment), the pupil position from the infrared image
Abandon position detection and create a template from visible images only. That is, the control unit (not shown) instructs the template creation unit 23 to perform image processing on the image data taken into the visible image storage unit 21 to extract a region and create a template. In this case, the pupil from the infrared image
Since position information cannot be obtained, the template is created first from the parent template. Specifically, first, the outline of the face is detected, and a parent template is created (step S2).
5). Next, an eye area is searched in the right area and the left area of the parent template to create a child template (step S).
26). Thereafter, the process proceeds to step S16, and the processes after the pattern matching are performed in the same manner as in the first embodiment.

In this embodiment, the step of evaluating the pattern matching result (steps S17 and S2)
In (0), the case where it is evaluated that the pattern matching has failed has been described, and the case where the processing is started again from the acquisition of the infrared image has been described. Instead of returning to S11, the system may be configured to start over from step S25, or may be configured to count the number of consecutive failures in pattern matching and start over from the acquisition of the infrared image only when the number exceeds a predetermined number.

As described above, according to the present embodiment, it is possible to appropriately deal with a problem that may actually occur, and to avoid the worst case where the viewpoint position cannot be detected.

(Fifth Embodiment) In the fourth embodiment,
In the case where it is difficult to detect the pupil position from the infrared image due to the reason that the subject wears glasses, two templates are created from only the visible image, but in the present embodiment, It is characterized in that one template is created using the information of the infrared image. Since the configuration of the viewpoint position detecting device according to the present embodiment may be the same as the configuration shown in FIG. 1, the description of each component will be omitted, and the operation will be described with reference to FIG.

FIG. 10 is a flowchart showing the operation of the viewpoint position detecting device according to this embodiment. In the figure,
Steps of the same processing as in FIG. 2 are denoted by the same reference numerals, and detailed description of the steps is omitted.
In the following description, as in the fourth embodiment, will be described as an example the case of abandoned pupil position detection from the infrared image when the pupil position detection from the infrared image fails four consecutive times . As shown in FIG. 10, the present embodiment differs from the operation in the fourth embodiment shown in FIG.
33 is added. The same processing as in the first embodiment is performed from the activation of the system to the evaluation step of the pupil position detection result based on the infrared image (step S13).

In step S13, the infrared image
If the pupil position detection has succeeded, the process shifts to step S14, and thereafter, the same processing as in the first embodiment is performed. on the other hand,
If it is determined in step S13 that the pupil position detection has failed, it is determined whether the number of consecutive failures has reached a predetermined number (four in this case) (step S2).
4). The number of consecutive failures can be determined by, for example, the pupil position detection process.
The processing unit 24 stores the information in the internal storage device. If the number of consecutive failures is less than the predetermined number, the process returns to the infrared image acquisition step (step S11), and the pupil position detection from the infrared image is performed again.
Try to get out .

On the other hand, if the number of consecutive failures reaches a predetermined number (four in this embodiment), the pupil position from the infrared image
Abandon position detection and create a template from the visible image using information obtained from the infrared image. This is, for example, even if the subject is wearing spectacles and the pupil position cannot be detected from the infrared image due to the reflection of the frames and lenses of the spectacles, the spectacles are present at the eye position,
Because there is a high possibility that the viewpoint exists in the reflection image, it is considered that it is useful as base point position information when creating a template.

Therefore, in the present embodiment, the pupil position
The location detection processing unit 24 calculates the center of gravity of the region including all the reflected light included in the infrared image, and the template creating unit 23 creates a template from the visible image using the center of gravity as a base point (step S30). At this time, both the parent template and the child template are created in the fourth embodiment, but one template is created in the present embodiment. The size of the template is determined, for example, from information such as the horizontal width and the vertical width of the area including all the reflected light. After the template is created, a position in the template where the viewpoint is considered to be present is estimated, and the viewpoint positions of the left eye and the right eye are set.

When the creation of the template is completed, pattern matching with the visible image is performed using the template (step S32). If it is determined that the pattern matching is successful and there is no instruction for the end processing in step S33, the process returns to step S31 to continue the processing. on the other hand,
If it is determined in step S32 that the pattern matching has failed, the process returns to step S11 to start again from the acquisition of the infrared image. In the case of the present embodiment, since a highly accurate pattern matching result cannot be expected from the beginning, the evaluation criterion of the pattern matching result in step S32 is set somewhat looser than the criterion in the first embodiment that allows accurate pupil position detection from an infrared image. Etc. may be changed.

In this embodiment, in the step of evaluating the pattern matching result (step S32), when it is evaluated that the pattern matching has failed, the case where the processing is started again from the acquisition of the infrared image is shown. If the template is created using the information of the infrared image due to a failure, the process does not return to step S11 but proceeds to step S11.
Alternatively, the system may be configured to start over from 30 or to start from the acquisition of the infrared image only when the number of consecutive failures in pattern matching is counted and the number of times exceeds a predetermined number.

As described above, according to the present embodiment, it is possible to appropriately deal with a problem that can actually occur, and to avoid the worst case where the viewpoint position cannot be detected.

[0087]

[Other Embodiments] In the above embodiments, the case where the viewpoint position detection initiative of the present invention has the initiative of viewpoint position detection has been described. When the image display unit 3 or the remote controller is used, a button or the like that can be operated by the user is provided, and when the user has difficulty in recognizing the stereoscopic display, the user presses this button to start the infrared image acquisition operation. The pupil position detection operation may be configured to be redone. With such a configuration, the template is updated at an accurate timing, and the viewpoint position detection can be performed with higher accuracy. As a result, a stereoscopic image display system with a wide stereoscopic view range can be realized. .

When emitting infrared light, the amount of light emission is changed according to the brightness of the image display unit 3 or the distance to the observer, or the pupil position detection from the infrared image fails and is repeated. At this time, the light emission amount can be changed (increased or weakened) from the last time. Such control of the light emission amount increases the probability of successfully detecting the pupil position from the infrared image, and as a result, leads to obtaining a highly accurate viewpoint position detection result.

Further, in the above-described embodiment, the case where the detection result of the viewpoint position detecting device according to the present invention is supplied to the stereoscopic image display device has been described as an example. However, the viewpoint position detecting device according to the present invention is applicable to any application. Can be used.

Further, the specific methods described in the above embodiments, such as the pattern matching method and the template creation method, are not limited to those described in the embodiments, but may be equally applicable. It is needless to say that may be used.

In the above-described embodiment, the viewpoint position, which is the coordinate of the pinpoint, is configured to be output. However, for example, as in the above-described embodiment, the viewpoint position finally obtained is determined by the three-dimensional image display device. In the case of using for control, since the minimum control is possible if the center position of the viewpoint positions of the right eye and the left eye is known, the center position of the template may be output to the image display unit 3. In particular, the accurate pupil position from the infrared image as described in Embodiments 4 and 5
Such a configuration is effective when the location information cannot be obtained.

Further, when it is difficult to detect the pupil position from the infrared image, in the case of Embodiments 4 and 5 in which the method is switched to another method, for example, it is assumed that the subject is wearing glasses. Means (a button or the like) for instructing the viewpoint position detecting device may be provided so that when this button is pressed, the process shifts to the substitution step from the beginning. In this case, in the fourth embodiment, step S10 in FIG.
May be configured to directly acquire the visible image from step S25 and proceed to step S25, or proceed to step S25 when it is determined that the detection of the pupil position from the infrared light has failed for the first time in step S13. Similarly, in the fifth embodiment, an infrared image is acquired in step S11, and the process directly proceeds to step S30, or the pupil position detection from infrared light is performed in step S13.
What is necessary is just to comprise so that it may shift to step S30 at the time when it is determined that the first delivery has failed.

The present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), and can be applied to a single device (for example, a copier, a facsimile). Device).

Further, an object of the present invention is to supply a storage medium (or a recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or an apparatus, and to provide the computer (or computer) of the system or the apparatus It is needless to say that the present invention can also be achieved by a CPU or an MPU) reading and executing the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. By executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written into the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, the program code is read based on the instruction of the program code. Needless to say, the CPU included in the function expansion card or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the above-described flowcharts (shown in FIGS. 2, 6, 8 to 10). Will be done.

[0098]

As described above, according to the present invention,
In a viewpoint position detection device that detects a viewpoint position of an observer and outputs viewpoint position information, an infrared image is used for initial template creation, and thereafter, a pattern matching of a visible image is performed. It is not necessary to continuously irradiate light to the observer, minimizing concerns about the influence on the observer, and creating a template using highly accurate position detection information from an infrared image in a short time Therefore, the template creation process can be reduced, and a template with high accuracy can be created.

Further, when the pattern matching of the visible image fails, such as when the observer changes or moves greatly, or by updating the template periodically, a template with high accuracy is always maintained. As a result, there is an effect that highly accurate viewpoint position information can be stably obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a viewpoint position detecting device according to a first embodiment of the present invention.

FIG. 2 is a flowchart illustrating the operation of the first embodiment of the present invention.

FIG. 3 is a diagram illustrating an example of an image obtained by infrared light.

FIG. 4 is a diagram illustrating a configuration of a template used in the embodiment of the present invention.

FIG. 5 is a block diagram illustrating a configuration of a viewpoint position detecting device according to a second embodiment of the present invention.

FIG. 6 is a flowchart illustrating the operation of the second embodiment of the present invention.

FIG. 7 is a block diagram illustrating a configuration of a viewpoint position detecting device according to a third embodiment of the present invention.

FIG. 8 is a flowchart illustrating the operation of the third embodiment of the present invention.

FIG. 9 is a flowchart illustrating the operation of the fourth embodiment of the present invention.

FIG. 10 is a flowchart illustrating the operation of the fifth embodiment of the present invention.

FIG. 11 is a perspective view showing a configuration of a rear cross lenticular type stereoscopic image display device.

FIG. 12 is a perspective view illustrating an example of a stereoscopic image display device that performs display control according to a viewpoint position.

[Procedure amendment 3]

[Document name to be amended] Drawing

[Correction target item name] Fig. 1

[Correction method] Change

[Correction contents]

FIG.

[Procedure amendment 4]

[Document name to be amended] Drawing

[Correction target item name] Figure 2

[Correction method] Change

[Correction contents]

FIG. 2

[Procedure amendment 5]

[Document name to be amended] Drawing

[Correction target item name] Fig. 5

[Correction method] Change

[Correction contents]

FIG. 5

[Procedure amendment 6]

[Document name to be amended] Drawing

[Correction target item name] Fig. 6

[Correction method] Change

[Correction contents]

FIG. 6

[Procedure amendment 7]

[Document name to be amended] Drawing

[Correction target item name] Fig. 7

[Correction method] Change

[Correction contents]

FIG. 7

[Procedure amendment 8]

[Document name to be amended] Drawing

[Correction target item name] Fig. 8

[Correction method] Change

[Correction contents]

FIG. 8

[Procedure amendment 9]

[Document name to be amended] Drawing

[Correction target item name] Fig. 9

[Correction method] Change

[Correction contents]

FIG. 9

[Procedure amendment 10]

[Document name to be amended] Drawing

[Correction target item name] FIG.

[Correction method] Change

[Correction contents]

FIG. 10

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C061 AA06 AA29 AB12 AB16 AB17 AB24

Claims (22)

[Claims] 1. A viewpoint position detecting device for detecting a viewpoint position of a person to be measured and outputting viewpoint position information, comprising: an infrared image acquiring means for acquiring an infrared image of the person to be measured; A visible image acquiring unit that acquires a visible image, from the infrared image acquired by the infrared image acquiring unit, a detecting unit that detects a pupil position of the subject, and a visible image acquired by the visible image acquiring unit. Template creating means for creating a template for pattern matching using the pupil position; and using the template created by the template creating means, the visible image acquiring means performs template matching with the visible image acquired by A matching means for detecting a viewpoint position and outputting a result as the viewpoint position information. 2. A viewpoint position detecting device for detecting a viewpoint position of a person to be measured and outputting viewpoint position information, comprising: an infrared image acquiring means for acquiring an infrared image of the person to be measured; A visible image acquiring unit that acquires a visible image, from the infrared image acquired by the infrared image acquiring unit, a detecting unit that detects a pupil position of the subject, and a visible image acquired by the visible image acquiring unit. A template creating unit that creates a template for pattern matching using the pupil position; and performing template matching with the visible image acquired by the visible image acquiring unit using the template created by the template creating unit. A matching unit that detects a viewpoint position and outputs a detection result as the viewpoint position information; and a detection unit and the template when a predetermined condition is satisfied. Control means for controlling to create the template again by using the port creation means. 3. The control unit evaluates the detection result of the matching unit, and when it is determined that the detection has failed a predetermined number of times, creates the template again using the detection unit and the template creation unit. 3. The viewpoint position detecting device according to claim 2, wherein the control is performed in such a manner. 4. The viewpoint position detecting apparatus according to claim 2, wherein said control means controls the generation of the template again using the detection means and the template generation means at predetermined time intervals. . 5. The control means evaluates the detection result of the matching means, and when it is determined that the detection has failed a predetermined number of times, and again at predetermined time intervals, using the detection means and the template creation means. 3. The method according to claim 2, wherein the control is performed to create the template.
The viewpoint position detecting device according to the above. 6. A receiving means for receiving an instruction from a person to be measured, wherein said control means, when receiving a predetermined instruction via said receiving means, re-uses said detecting means and said template creating means. The viewpoint position detecting device according to claim 2, wherein control is performed to create the template. 7. The viewpoint according to claim 2, wherein the template creating unit creates a plurality of templates having different sizes, and the matching unit performs the template matching a plurality of times using the plurality of templates sequentially. Position detection device. 8. When the pupil detection by the detection means has failed a predetermined number of times, a pattern matching template is created using only the visible image or information obtained from the infrared image, and the template creation means creates the pattern matching template. 3. The viewpoint position detecting device according to claim 2, further comprising a second template creating unit that supplies the template to the matching unit. 9. A receiving means for receiving an instruction from a person to be measured, wherein the control means receives a predetermined instruction via the receiving means and uses the second template creating means to perform template matching. 3. The viewpoint position detecting device according to claim 2, wherein the control is performed to perform the following. 10. A viewpoint position detecting method for detecting a viewpoint position of a person to be measured and outputting viewpoint position information, comprising: an infrared image acquiring step of acquiring an infrared image of the person to be measured; A visible image acquisition step of acquiring a visible image, from the infrared image acquired in the infrared image acquisition step, a detection step of detecting a pupil position of the subject, and from the visible image acquired in the visible image acquisition step, A template creation step of creating a template for pattern matching using the pupil position, and using the template created in the template creation step, performing visible image acquisition in the visible image acquisition step and template matching to perform measurement of the subject. A matching step of detecting a viewpoint position and outputting a result as the viewpoint position information. That the viewpoint position detection method. 11. A viewpoint position detecting method for detecting a viewpoint position of a person to be measured and outputting viewpoint position information, comprising: an infrared image acquiring step of acquiring an infrared image of the person to be measured; A visible image acquisition step of acquiring a visible image, from the infrared image acquired in the infrared image acquisition step, a detection step of detecting a pupil position of the subject, and from the visible image acquired in the visible image acquisition step, A template creation step of creating a template for pattern matching using the pupil position, and performing template matching with the visible image acquired in the visible image acquisition step using the template created in the template creation step, and A matching step of detecting a viewpoint position and outputting a detection result as the viewpoint position information. The viewpoint is characterized in that the template is created again using the detection step and the template creation step when the condition is satisfied, and otherwise the visible image acquisition step and the matching step are repeatedly performed. Position detection method. 12. An evaluation step for evaluating a detection result of the matching step, wherein when it is determined that the detection has failed a predetermined number of times, the template is created again using the detection step and the template creation step. The viewpoint position detecting method according to claim 11, wherein: 13. A time detecting step for detecting a predetermined time, wherein the template is generated again by using the detecting step and the template generating step every time the predetermined time is detected by the time detecting step. The method according to claim 11, further comprising: 14. An evaluation step for evaluating a detection result of the matching step, and a time detection step for detecting a predetermined time, wherein the evaluation step determines that the detection has failed a predetermined number of times, and 12. The template is created again by using the detecting step and the template creating step each time the predetermined time is detected in the time detecting step.
The described viewpoint position detection method. 15. A receiving step for receiving an instruction from a person to be measured, wherein the step of receiving the predetermined instruction in the receiving step includes generating the template again by using the detecting step and the template generating step. The method of detecting a viewpoint position according to claim 11, wherein: 16. The viewpoint according to claim 11, wherein the template creating step creates a plurality of templates having different sizes, and performs the template matching a plurality of times by sequentially using the plurality of templates in the matching step. Position detection method. 17. When the pupil detection in the detecting step has failed a predetermined number of times, a pattern matching template is created using only the visible image or information obtained from the infrared image, and the template creating step creates the template. The method according to claim 11, further comprising a second template creation step of supplying the template as a template to the matching step. 18. The method according to claim 18, further comprising a receiving step of receiving an instruction from the subject, wherein in the receiving step, when a predetermined instruction is received, template matching is performed using the second template creating step. The viewpoint position detecting method according to claim 11, wherein 19. A stereoscopic image display system comprising: the viewpoint position detecting device according to claim 1; and a stereoscopic image display device connected to the viewpoint position detecting device. A stereoscopic image display system, wherein the stereoscopic image display device is controlled using the viewpoint position information received from the device. 20. A stereoscopic image display system comprising: the viewpoint position detecting device according to claim 1; and a stereoscopic image display device connected to the viewpoint position detecting device. A stereoscopic image display device having a mask pattern in which a display device controls an emission optical path, wherein the position or pattern of the mask pattern is changed using viewpoint position information received from the viewpoint position detection device. Stereoscopic image display system. 21. A computer-readable storage medium storing the method according to claim 10 as a computer-executable program. 22. A computer-readable storage medium storing the method according to claim 11 as a computer-executable program.