JP2001195582A - Device and method for detecting image, device and system for three-dimensional display, display controller, and program storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accelerate the detecting speed of face part by enhancing the probability of the existence of a face part within a search range to be set at the time of template matching. SOLUTION: During the time when a template matching is successful, the position of the search range is made to almost match with the moving direction of the face part up to the moment and further, the search range is set at a separate position corresponding to a moving speed. When the template matching is failed, the size of the next search range is set to a size corresponding to the moving speed up to the moment.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image detecting apparatus and an image detecting method for detecting a position of a specific part such as a predetermined face part such as a viewpoint, and more particularly to a viewpoint detecting method which achieves both high-speed processing and high detection accuracy. The present invention relates to an apparatus and a viewpoint detection method.

2. Description of the Related Art Conventionally, there is a so-called stereoscopic image display device capable of stereoscopic viewing with the naked eye. There are various types of stereoscopic image display devices. For example, JP-A-09-31294 discloses a rear cross lenticular type. FIG. 1 is a perspective view of an essential part showing an example of a rear cross lenticular type stereoscopic image display device. In the figure, reference numeral 6 denotes a display device for displaying images, which is constituted by, for example, a liquid crystal element (LCD). In the figure, a polarizing plate, a color filter, an electrode, a black matrix, an antireflection film, and the like are omitted. Reference numeral 10 denotes a backlight (surface light source) serving as an illumination light source. Display device 6 and backlight 1
Between 0, a mask substrate (mask) 7 on which a mask pattern having a checkered opening 8 is formed is arranged. The mask pattern is manufactured by patterning a metal deposited film such as chromium or a light absorbing material on a mask substrate 7 made of glass or resin. The backlight 10, the mask substrate 7, and the like constitute one element of the light source.

[0002] A first lenticular lens 3 and a second lenticular lens 4 made of transparent resin or glass are arranged between a mask substrate 7 and a display device 6. The first lenticular lens 3 is a vertical cylindrical lens array formed by arranging vertically long vertical cylindrical lenses in the left-right direction, and the second lenticular lens 4 is formed by vertically arranging horizontal cylindrical lenses long in the horizontal direction. It is a cylindrical lens array.

As shown in the figure, the image displayed on the display device 6 divides the left and right parallax images R and L into a large number of horizontal stripe pixels R and L in the vertical direction, and divides them into, for example, LRLLRLR from the screen. ...
And a horizontal stripe image alternately arranged to form one image.

The light from the backlight 10 is applied to the mask substrate 7
Then, the display device 6 is illuminated through each of the openings 8, and the left and right stripe pixels R and L are separately observed by both eyes of the observer.

That is, the mask substrate 7 is
0 illuminates, and light exits from the aperture 8. The first lenticular lens 3 is arranged on the observer side of the mask substrate 7, and the lens curvature is designed so that the mask substrate 7 is almost at the focal position of each cylindrical lens. In this cross section, the second lenticular lens 4 has no optical effect, so that the light beam emitted from one point on the opening 8 is converted into a substantially parallel light beam in this cross section.

[0006] A pair of openings and a light shielding portion of the mask pattern are set so as to substantially correspond to one pitch of the first lenticular lens 3.

Further, based on the relationship between the optical distance from a predetermined position of the observer to the first lenticular lens 3 and the optical distance from the first lenticular lens 3 to the mask pattern, the first lenticular lens 3 is formed. By determining the pitch of the mask pattern and the pitch of the pair of openings and the light-shielding portion of the mask pattern, light from the openings 8 can be uniformly collected to the left eye or the right eye over the entire width of the screen.
In this way, the left and right stripe pixels on the display device 6 are observed separately in the horizontal direction into the left eye and right eye regions.

The second lenticular lens 4 condenses all light beams emitted from each point on the opening 8 of the mask 7 on the right-eye or left-eye stripe pixel of the display device 6, illuminates and transmits the same. It diverges only in the vertical direction according to the NA at the time of condensing, and provides an observation area in which the left and right stripe pixels are uniformly separated from the predetermined eye height of the observer over the entire vertical width of the screen. .

However, the viewing angle of such a stereoscopic image display device is narrow, and if the observer's viewpoint deviates from the viewing angle, the stereoscopic display cannot be recognized. Therefore, there has been proposed a technique of detecting a viewpoint position of an observer and controlling image display in response to the movement of the viewpoint position, thereby enlarging a region where stereoscopic viewing is possible. For example, Japanese Patent Application Laid-Open No. 10-2323
No. 67 discloses a technique for expanding a stereoscopically visible area by moving a mask pattern or a lenticular lens in parallel to a display surface.

FIG. 2 is a diagram of a three-dimensional image display device disclosed in Japanese Patent Application Laid-Open No. 10-232367. 2, the same components as those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted. Since the three-dimensional image display device in FIG. 2 has a single lenticular lens, it does not have the second lenticular lens 4 in FIG.

In the three-dimensional image display device having such a configuration, control according to the movement of the observer 54 is performed as follows. First, the position sensor 51 detects a horizontal displacement of the observer 54 from a preset reference position, and sends the information to the control unit 52. The control unit 52 responds to the displacement information by using the display drive circuit. When the image control signal is output to the display 50, the display drive circuit 50 displays the first or second horizontal stripe image on the display 6. At the same time, the control unit 52 generates an actuator drive signal based on the displacement information and drives the actuator 53 that moves the mask pattern 7 in the horizontal direction, so that the mask pattern 7 can be separated from the left and right stripe images by the observer 54 in the best position. Move to As a result, even if the viewpoint position of the observer 54 changes, the range that can be stereoscopically viewed is expanded.

As described above, when the display is controlled according to the viewpoint position of the observer, an image suitable for the observer's viewpoint can be displayed even if the detection accuracy is low or the processing time for detection is long. Absent. Therefore, it is very important to detect the viewpoint position of the observer with high accuracy and in a short time in the performance of the display device.

As a method of detecting the viewpoint position of the observer (measured person), 1) a method of irradiating the observer with infrared light and detecting reflection of the retina (for example, Banno "Pupil optics for viewpoint detection" System Design Method, ”IEICE Transactions D-II, Vol. J74-D
-II, No.6, pp.736-747, listed in June 1991)

2) A method of image processing of a visible image to detect an observer's eyes (for example, Sakaguchi et al. “Real-time facial expression recognition using two-dimensional discrete cosine transform of image”, IEICE Transactions D) -II, Vol. J80-D-II, No.6, pp. 1547-1
554 June 1997)

3) A method of detecting an observer's eyes by image processing using an infrared image and a visible image (for example, see
-287216).

The method 1) utilizes the fact that the human pupil has the property of retroreflecting near-infrared light (returning light in the same direction as the incident direction). Pupil reflected light is obtained as a sharp reflection peak, and usually shows a higher reflectance than the face, etc., so that only the pupil part is captured by imaging the observer using an infrared imaging device whose optical axis is coaxial with the light source. Bright images can be taken. If this photographed image is binarized with an appropriate threshold, an accurate viewpoint position can be detected from the extracted pupil position.

In the method 2), the position of the observer in the photographing range is limited in advance, the observer blinks in that state, and the eye region is extracted by the inter-frame image of the visible image. are doing. Using a template created based on the extracted eye region, viewpoint position detection is performed by template matching.

In the method 3), an infrared image and a visible color image are simultaneously photographed, a face region is extracted from these images, and then a characteristic region such as an eye is detected by using template matching. The infrared image is used to extract a person candidate area,
It is used to determine a temperature threshold used when extracting a skin color region from a color image.

[0019]

However, in the method (1), since it is necessary to continuously irradiate the observer with relatively strong infrared light, the influence of the infrared light on the observer is small. In addition to the problem of concern, there is also a problem that detection cannot be performed when the observer blinks because reflection by the retina is used. In addition, depending on the pupil enlargement or reduction due to the illuminance of the environment,
There is also a problem that it is difficult to follow the pupil reflection image.

In the method 2), the observer is required to perform operations such as adjustment of the observation position and blinking, which is troublesome for the observer. It requires time for operations such as adjustment of the observation position and blinking, which is not practical.

In the method 3), the irradiation intensity of infrared light may be lower than that in the method 1). However, after the intermediate processing result of the infrared image is obtained, the visible image is obtained by using the processing result. Is processed, the face area is detected using the processing result of the infrared image and the processing result of the visible image, and further pattern matching is performed. Therefore, the processing is very complicated. There is also a problem that it is not easy to create a template used for pattern matching. In addition, since the position of the face required to create a template for pattern matching is detected only from the visible image, there is a problem that the position accuracy is not very good.

To solve the above problem, the applicant has
Japanese Patent Application No. 11-82455 on March 25, 1999 discloses a method of initially determining a search area in a face image to which template matching is applied, although template matching is used to detect a viewpoint position. Alternatively, when the template matching fails and a new search area position is determined when the search area is changed, the search is performed centering on the pupil position roughly determined using the reflection image from the pupil by near-infrared light. A method of determining the area was proposed.

However, this method requires irradiating the pupil with infrared light. Therefore, the size of the apparatus is increased due to an infrared irradiation apparatus or the like. Therefore, in order to reduce the size of the apparatus, the update position of the search area must be determined without the infrared light irradiation mechanism.

As a conventional technique for updating a search area, there is Japanese Patent Laid-Open No. Hei 8-287216. In this conventional technique, when the search area is moved without being able to detect a viewpoint in the search area up to that time, the size of the search area is gradually increased. That is, if the viewpoint cannot be detected even when the search area is enlarged, the search area is further expanded. If the viewpoint is detected, the size of the search area is reduced.

However, Japanese Patent Application Laid-Open No. Hei 8-28721
In the method of No. 6, since the search area is determined without considering the moving speed and the moving direction of the user's face, the determined search area is not necessarily optimal, and therefore, the expanded search area is used. , The probability that the viewpoint can be detected is reduced, and when the moving speed of the face is relatively high, there is a high possibility that the size of the search area must be enlarged again. Due to the re-expansion of the search area, an increase in the amount of calculation that occurs with the re-expansion further tends to cause a vicious cycle in which the search speed is further reduced, resulting in a failure in the re-detection.

Accordingly, an object of the present invention is to provide an apparatus and a method for detecting a face part which can track a moving face part (for example, a viewpoint position) reliably and with high accuracy while having a simple structure. Another object of the present invention is to provide a three-dimensional display device using them, a display controller, a display system, and the like.

[0027]

According to a first aspect of the present invention, there is provided a face part detecting apparatus, comprising: an imaging unit; an optional search area and a template set in a predetermined image captured by the imaging unit; By performing the matching, template matching means for detecting a position of a predetermined specific part in the predetermined image, speed detecting means for detecting a position change speed of the specific part, and the template matching means A change unit that changes the size of the search area when the search area is not detected in the search area, the change unit sets a larger search area as the position change speed detected for the specific part is larger. By setting, the detection speed of the specific part in the search area is improved.

The fact that the position change speed of a specific part is high means that the part is likely to have moved to an area within a larger range. Therefore, by setting a larger search area, template matching can be performed. Is reduced, and as a result, the detection speed of the specific site position can be improved.

According to a preferred aspect of the present invention, the specific part is a facial part in a facial image, and the speed detecting means detects the position of the facial part detected by the template matching means. The position change speed is detected based on the change. By diverting the result of the template matching for speed detection, hardware for speed detection becomes unnecessary.

According to a preferred aspect of the present invention, the specific portion is a viewpoint position of the facial portion.

If the tracking of the face part succeeds after the template matching has failed, there is no need to set a large search, so that the changing means is a preferable aspect of the present invention. The template matching means reduces the size of the search range when the specific part is detected in the search range changed by the change means. The search speed after resuming tracking of a specific part is improved.

[0032] In particular, the changing means of the device according to claim 5 includes:
It is characterized in that the size of the search range is returned to the initial value instead of simply reducing the search range.

According to a preferred embodiment of the present invention, the search range can be changed from the minimum value size to the screen size. By setting the maximum size to the screen size, it is extremely effective when the face part comes in again after leaving the display screen.

As long as the template matching is successful, the current search range should be working effectively. Therefore, according to claim 7, in such a case,
The search range is maintained at the minimum size.

As long as the template matching is successful, it is more efficient to move the search range without changing the size of the search range. Therefore, according to claim 8, the changing means has a moving means for moving the position of the search range,

The moving means moves the position of the search range to the matching position detected by the template matching means. For example, the matching position is matched with the matching position detected by the template matching means.

When the search range is moved, the probability that a face part exists in such a search range is higher if the amount of movement depends on the moving speed of the specific part up to that time. Therefore, according to claim 10, further comprising means for detecting a change in the viewpoint position based on a change in the matching position detected by the template matching means,

The moving means moves the search range by setting the next search range from the previously detected matching position to a position corresponding to the amount of change in the currently detected viewpoint position.

According to claim 11 which is a preferred aspect of the present invention, the changing means includes: as long as the template matching by the template matching means is successful.
Maintaining means for maintaining the size of the search range at the minimum initial value;
When the template matching fails for the first time, means for setting the size of the search range to a speed equivalent size corresponding to the position change speed detected by the speed detecting means,
Means for increasing the size of the search range by a predetermined value further than the speed equivalent size when template matching has repeatedly failed. That is, when the template matching fails, the search range is increased in accordance with the moving speed of the specific part, and thereafter, when the template matching continues to fail, the size of the search range is increased by a predetermined value.

Conversely, when the template matching has failed at the beginning, the search range is increased in accordance with the moving speed of the specific part. The size of the range may be made larger than the speed-equivalent size by an increment corresponding to a speed corresponding to the moving speed detected by the speed detecting means.

Increasing the search range in template matching may increase the possibility of erroneous determination. Therefore, in order to prevent erroneous determination, the threshold value for determining the success of matching is made variable in the template matching means, and when the changing means increases the size of the search range, the threshold value for determining the success of matching is increased.

The method (for example, claims 14 to 26) applied to the apparatus according to claims 1 to 13 achieves the above object as it is.

The above-mentioned problem of detecting a specific portion at a high speed can also be achieved by the device according to claim 27 having the following configuration. That is, the image detecting apparatus according to claim 27 performs a predetermined identification in the predetermined image by matching the template with an arbitrary search area set in the predetermined image captured by the imaging unit. Template matching means for detecting the position of the part, direction detecting means for detecting the moving direction of the specific part from a change in the detection position of the specific part, and detecting the position of the search range for the next template matching. Setting means for setting the moving direction of the specific part detected by the means. If the position of the search range is set so as to match the moving direction up to that time, the probability that the specific part is included in the search range set as such increases.

According to a preferred embodiment of the present invention, the apparatus further comprises a speed detecting means for detecting a position changing speed of the specific portion, wherein the setting means is provided for a next template matching. A search range is set in a moving direction of the specific part detected by the direction detecting means and at a position separated by a distance corresponding to the position change speed of the specific part detected by the speed detecting means. That is, the position of the next search range is set at a position on the extension of the moving direction of the specific part, which is separated by a distance corresponding to the position changing speed of the specific part.

According to a preferred aspect of the present invention, the direction detecting means detects the moving direction based on a change in the position of the specific part detected by the template matching means. The output result of the template matching can be used for detecting the moving direction, which contributes to downsizing of the apparatus.

According to a preferred embodiment of the present invention, the speed detecting means detects a position change speed based on a change in the position of the specific part detected by the template matching means. The output result of the template matching can be used for detecting the moving speed, and hardware is not required for detecting the moving speed.

According to a preferred aspect of the present invention, the specific part is a viewpoint position of a face part.

When a function for changing the size of the search range is added to the invention for changing the position of the search range, the speed of detecting a specific part is further improved. Therefore, as in claim 32, further comprising changing means for changing the size of the search area when the template matching means cannot detect the specific part in the search area,

The setting means sets the position of the search range for the next template matching in the moving direction of the specific part detected by the direction detecting means according to the size enlarged by the changing means.

According to a preferred aspect of the present invention, as long as the template matching means succeeds in the template matching, the setting means includes:
Keep the search range at a fixed size.

The method applied to the devices of claims 27 to 33 (for example, claims 34 to 40) can achieve the above object as it is.

The above object is further provided, for example, in claim 41, further comprising means for detecting the line of sight.

If the template matching means corrects the deformation of the template image in accordance with the detected line-of-sight direction, the accuracy of the template matching can be further improved.

The above object can also be achieved by a program storage medium storing a computer program for realizing the above-described image detection method.

Further, it is preferable that the viewpoint position detected by the image detecting device according to any one of claims 1 to 13 or 27 to 33 is applied to a stereoscopic display device.

Furthermore, it is preferable that the viewpoint position detected by the image detecting device according to any one of claims 1 to 13 or 27 to 33 is applied to a controller for controlling a stereoscopic display device.

The viewpoint position detected by the image detecting device according to any one of claims 1 to 13 or 27 to 33 is preferably applied to a stereoscopic display system.

[0058]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment in which the detection of a specific part of the present invention is applied to the detection of a face part, particularly to the detection of a viewpoint position will be described in detail with reference to the drawings. In the following description, a stereoscopic image display system in which the viewpoint position detecting device of the embodiment is connected to a stereoscopic image display device will be described, but the application of the face part detection of the present invention is limited to the stereoscopic image display device system. Not something.

In the following, a rear cross lenticular type stereoscopic display device is described.
The application of the face part detection of the present invention is not limited to the rear cross lenticular type stereoscopic display device, but may be applied to, for example, a parallax barrier type stereoscopic display device.

Further, in the present invention, the face part is not limited to the viewpoint position. Furthermore, the viewpoint position means the coordinates of a certain point indicating the position of the eyes of the observer. However, the viewpoint position information output by the viewpoint position detecting device of the present invention is not necessarily required to be the coordinate value of one point, and may be a certain region. May be indicated. Depending on the use, the position of the entire eye may be roughly known, and may be appropriately selected depending on the use.

First, a description will be given of a rear cross lenticular type stereoscopic display apparatus to which the viewpoint position detection of the embodiment is applied.

<Rear Cross Lenticular Method> The principle of the three-dimensional display of the cross lenticular method will be described with reference to the drawings. FIG. 3 is a view for explaining the principle of the cross lenticular system used in the present embodiment and a perspective view showing the configuration of the stereoscopic display device 1000.

Reference numeral 60 denotes a liquid crystal display device for displaying a parallax image. Reference numeral 61 denotes a display pixel portion formed of a liquid crystal layer or the like, which is formed between glass substrates 62. In the drawing, a polarizing plate, a color filter, an electrode, a black matrix, an antireflection film, and the like are not shown. Reference numeral 90 denotes a backlight serving as an illumination light source. A liquid crystal display device 80 for a barrier functioning as a spatial light modulator is placed in front of it. When a stereoscopic image is displayed, a checkered pattern (a black portion does not pass through a backlight item, Is displayed on the display screen of the display device 80, and when displaying a normal two-dimensional image, the corresponding area is displayed in white. The barrier liquid crystal display device 80 is controlled by the controller 100 via the memory 101.

Between the barrier liquid crystal display device 80 and the parallax image display liquid crystal display device 60, two lenticulars 71 and 72 made of a transparent resin or glass and perpendicular to each other are arranged as microlenses. The parallax image display liquid crystal display device 60 includes illustrations (61R is an image for the right eye, 6R).
The left and right parallax images are alternately arranged in the vertical direction and displayed in the form of horizontal stripes as in the case of 1L (image for left eye). The light from the backlight 90 passes through each light transmitting portion 81 of the barrier liquid crystal display device 80, and the lenticules 72 and 71
Illuminates the image liquid crystal display device 60 through the camera, and the left and right parallax images are separately observed by both eyes of the user.

FIG. 4 is a diagram for explaining the principle that the left and right parallax images are observed in a horizontally separated manner with both eyes of the user. This figure shows a cross section of the stereoscopic image display device described in FIG. 3 as viewed from above. When the barrier liquid crystal display device 80 is illuminated by the backlight 90, the light transmitting portion 81
The light is emitted from. A lenticular 72 is arranged on the user side of the barrier liquid crystal display device 80, and the lens curvature is designed so that the mask pattern on the barrier liquid crystal display device 80 is located substantially at the focal position of each cylindrical lens.

The light transmitting portion and the light shielding portion of the mask pattern shown in FIG. 4 correspond to the left image of the horizontally striped left and right images alternately arranged on the parallax image display liquid crystal display device 60. Yes, it is. Therefore, the light emitted from the light transmitting portion 81 passes through the lenticular 72 and illuminates the left image of the parallax image display liquid crystal display device 60 with directivity in a range indicated by a solid line in the drawing. In the figure, E _L denotes the left eye of the user and E _R denotes the right eye, and the pitch of the lenticular 72 is masked so that the light from the light transmission unit 81 is uniformly collected on the left eye over the entire width of the image. The pitch is slightly smaller than the pitch between each pair of light transmitting portions and light blocking portions of the pattern. As a result, transverse stripes in the left image displayed on the liquid crystal display device for displaying a parallax image is observed only in the range around the left eye E _L. In addition, with respect to the right eye E _R, the display device 8
The pattern of the light transmitting part and the light shielding part of the pattern of 0 is opposite to that of FIG. That is, the parallax image display liquid crystal display device 6
0 to now correspond to the stripe of the right image in the displayed vertically alternating arrangement with lateral stripes of left and right images, the right image through the lenticular 72 is illuminated directionally to range around the right eye E _R. Thus, the lateral stripes on the right image displayed on the liquid crystal display device for displaying a parallax image is observed only in the range around the right eye E _R. In this way, the left and right images on the parallax image display liquid crystal display device are observed separately in the horizontal direction to the left and right eyes.

Next, the observation region in the vertical direction will be described.
FIG. 5 is a schematic side view of a cross section in the vertical direction of the three-dimensional image display device based on the cross lenticular system shown in FIG. In FIG. 5, the lenticular 72 having no optical action and the flat glass substrate not directly related to the optical action are omitted in this cross section, and the lenticular 71 is also conceptually expressed. The openings of the mask pattern of the barrier liquid crystal display device 80 are in a checkered pattern as shown in FIG. 3, and in the vertical direction, horizontal stripes are displayed on the parallax image display liquid crystal display device 60 and arranged alternately in the upper and lower directions. It supports left and right images.

In FIG. 5, it is assumed that the opening pattern of the light transmitting portion 81 illuminates an image line for one of the eyes of the user.

[0069] Here, pitch V _m in the vertical direction of the mask pattern, the pitch V _L of the lenticular 71, the focal length of the direction in the plane of FIG. 5 of each cylindrical lens constituting the lenticular 71 and f _V The vertical pixel pitch of the parallax image display liquid crystal display device is V _d , the distance from the display surface of the parallax image display liquid crystal display device to the lenticular 71 is L ₁ , and the distance from the lenticular 71 to the mask pattern is L ₂ . and when,

V _d : V _m = L ₁ : L ₂ (1)

V _d : V _L = (L ₁ + L ₂ ) / 2: L ₂ (2)

1 / f _V = 1 / L ₁ + 1 / L ₂ (3)

The relationship is set so as to satisfy the following relationship. At this time, the light transmitting portions of the mask pattern converge on the corresponding pixel lines as focal lines perpendicular to the plane of FIG. Paying attention to one of the checkered openings, the light is emitted from the center point A of the central light transmitting portion 81-1 and the lenticular 7
The light beam incident on the corresponding one of the cylindrical lenses 71-1 is condensed on a focal line on the center point A 'of the corresponding pixel column 61-1 of the liquid crystal display device. Central light transmitting portion 81-
A light beam emitted from the center point A of 1 and incident on the cylindrical lens constituting the lenticular 71 other than 71-1 is condensed as a focal line at the center of the pixel line for displaying the image for the left eye of the liquid crystal display device. .

Light beams emitted from the points B and C at the ends of the light transmitting portion 81-1 and incident on the corresponding cylindrical lens 71-1 of the lenticular 71 are points B 'and C' at the ends of the pixel row 61-1. Each is focused on the upper focal line. Similarly, a light beam emitted from another point of the light transmitting portion 81-1 and incident on the cylindrical lens 71-1 is transmitted to the pixel column 6 of the liquid crystal display device.
The light is focused on 1-1 as a focal line. Also, the light transmitting portion 81-
All of the light fluxes that have emitted 1 and entered the cylindrical lenses other than 71-1 are also collected on the pixel line of the liquid crystal display device 60 that displays the image for the left eye.

Similarly, all of the light fluxes emitted from the light transmitting portions other than the light transmitting portion 81-1 of the checkered pattern shown in FIG. 5 are located on the pixel line for displaying the image for the left eye of the liquid crystal display device 60. The light is condensed, illuminates and transmits the pixel line for the left eye of the liquid crystal display device 60, and diverges only in the vertical direction according to the numerical aperture at the time of light condensing. An observation area is obtained in which the image for the left eye appears to be uniformly separated over the entire width in the direction. Here, the image for the left eye of the user has been described, but the same operation is performed for the image for the right eye of the user.

FIG. 6 is a side view of a vertical cross section of the stereoscopic image display device shown in FIG. FIG. 5 also shows members omitted in FIG. Here, the pitch of the opening of the mask pattern in the vertical direction is V _m , the pitch of the lenticular 71 is V _L , the vertical pixel pitch of the liquid crystal display device for displaying parallax images is V _d , the display of the liquid crystal display device for displaying parallax images. Distance from the surface to the user's main plane of the lenticular 71
L ₁ , the distance from the main plane of the lenticular 71 on the mask substrate side to the mask pattern is L ₂ , and the focal length f _{V of} each of the cylindrical lenses forming the lenticular 71 in the direction in the plane of FIG. (1), (2), (3)
V _d = V _m = V _L , L ₁ = L ₂ , f _V = L ₁ /
2 is set. That is, as already described with reference to FIG. 5, it is possible to obtain an observation area in which the right and left images appear to be uniformly separated from the predetermined eye height of the user over the entire vertical width of the screen. ing.

In the above description, as viewed from the user side,
Liquid crystal display device 60, lenticular 71, lenticular 7
2. The three-dimensional image display device is configured by arranging the checkered light transmitting portions 81 in this order.
May be interchanged. In this case, lenticular 7
2. By resetting the pitch and focal length of the lenticular 71 and the vertical and horizontal pitches of the checkered aperture so as to satisfy all of the above-described conditions, a stereoscopic image display device can be configured in the same manner as the above-described configuration.

Next, a method of displaying a stereoscopic image by the spatial light modulator 80 will be described. FIG. 7 is a diagram for explaining a stereoscopic image display method by the stereoscopic display device shown in FIG. FIG. 7A shows a pattern of a light transmitting portion and a light shielding portion of the spatial light modulator 80, and FIGS. 7B and 7C show a display pixel portion of the liquid crystal display device 60 for screen display. Similar to the above-described stereoscopic image display, the left and right parallax images are alternately combined in a horizontal stripe form and displayed on the liquid crystal display device 60.

In FIG. 7A, when the light transmitting portion of the spatial light modulator 80 is formed in each of the rectangular portions 65 shown by solid lines and the rectangular portion 66 is formed as a light shielding portion, As shown in FIG. 7B, a right parallax image R ₁ on the first scanning line, a left parallax image L _{2 on} the second scanning line, a right parallax image R _{3 on} the third scanning line, and so on.
The left and right parallax images are separately observed by the left and right eyes of the user, respectively. On the other hand, in FIG. 7A, a light-transmitting portion is formed in each of the rectangular portions 66 indicated by dotted lines, and when the rectangular portion 65 is a light-shielding portion, the first scanning line as shown in FIG. 7C. , A second parallax image L ₁ , a second scanning line, a right parallax image R ₂ , a third scanning line, a left parallax image L ₃ ,... The parallax images are separately observed by the left and right eyes of the user.

Here, by alternately displaying this state in a time division manner, the resolution of the left and right parallax images, which has been reduced to half due to the stripe composition, can be displayed at a high resolution without lowering the resolution.

At this time, if there is a difference between the rewriting speed of the display pixel portion of the image display liquid crystal display device 60 and the rewriting speed of the spatial light modulator, the rewriting timing of the image and the rewriting of the aperture pattern are made coincident with each other to the user. In order to make the boundary invisible, an image display controller and a spatial light modulator controller can be added for synchronization. At that time, the display pixel portion of the parallax image display liquid crystal display device and the corresponding scanning line of the spatial light modulator may be rewritten in a synchronized manner for each pixel, or may be rewritten in a synchronized manner for each corresponding scanning line. Is also good.

In a normal stereoscopic display method in which two parallax images of a right-eye image and a left-eye image are displayed in a time-division manner for each screen, the frame frequency is set to 120 Hz in order to prevent flicker.
Need to be raised. However, in the method of the present embodiment, since the left and right parallax images are combined and displayed in the form of a horizontal stripe, it is possible to observe with high resolution without feeling flicker even at a frame frequency of 60 Hz.

In the present embodiment, the opening pitch of the mask pattern in the vertical direction is V _m , the pitch of the lenticular 71 is V _L , and the focal length f _{V of} each of the cylindrical lenses constituting the lenticular 71 in the direction in the plane of FIG. The vertical pixel pitch of the parallax image display liquid crystal display device is V _d , the distance from the display surface of the parallax image display liquid crystal display device to the lenticular 71 is L ₁ , the distance from the lenticular 71 to the mask pattern is L ₂ , to, and is configured so as to satisfy _{V d = V m = V L} , L 1 = L 2, f V = L 1/2 relationship. Here, since V _d = V _m , a liquid crystal display device having a pixel pitch similar to that of the parallax image display liquid crystal display device can be used as the spatial light modulator.

Next, a description will be given of a mixed display means for switching between two-dimensional display and three-dimensional display in this configuration. In the above configuration using the transmissive spatial light modulator 80 such as a transmissive liquid crystal element, a predetermined area can be displayed as a stereoscopic image by controlling the aperture pattern, and the other part can be displayed as a two-dimensional image. it can. This will be described with reference to FIG. FIG. 8 is a diagram for explaining a method of displaying a three-dimensional image and a two-dimensional image in a mixed manner by the stereoscopic display device shown in FIG.
In the figure, (A) shows a light transmitting part of the spatial light modulator 80.
3B illustrates a pattern of a light shielding unit, and FIG. 3B illustrates an image pattern of a display unit of the liquid crystal display device 60 for displaying a parallax image. As shown in (B), the left and right parallax images R, L, and L in a region 67 of the liquid crystal display device 60 for displaying a stereoscopic image are displayed.
A horizontal stripe image in which R,.
A normal two-dimensional image S is displayed in other areas.

The pattern of the light transmitting portion and the light shielding portion of the spatial light modulator 80 corresponding to FIG. 8B is as shown in FIG. 8A. A checkered opening pattern is formed in the area 87 of the spatial light modulator 80 corresponding to the area 67 of the liquid crystal display device 60 for displaying a stereoscopic image. As a result, area 6
In the portions corresponding to 7 and 87, the left and right parallax images having directivity to the transmitted light separately reach the left and right eyes. In other areas, all the apertures are in a light-transmitting state so that the two-dimensional image S reaches both the left and right eyes. Thus, a stereoscopic image can be displayed only in the area 67. Further, as described above, the resolution of the stereoscopic image can be increased by alternately displaying the first combined stripe image and the second combined stripe image and changing the aperture pattern in synchronization with the first combined stripe image and the second combined stripe image.

<Display system> The stereoscopic display device 1 shown in FIG.
FIG. 9 shows a configuration of a stereoscopic display system that realizes an optimal stereoscopic view by tracking the viewpoint position using 000. The viewpoint position detection device according to the present embodiment includes an imaging unit 300 and a viewpoint position detection unit 400, and includes a viewpoint position detection unit 40.
The controller 100 connected to the barrier mask display device 8 of the display device 1000 generates an optimal barrier mask according to the viewpoint position detected by the detection unit 400.
Display at 0. Hereinafter, the viewpoint position detection device 400, the controller 100, and the display device 1000 may be collectively referred to as a stereoscopic image display system.

The photographing section 300 is constituted by, for example, a video camera for photographing a visible image of the observer.

The viewpoint position detecting section 400 includes the image storing section 40
1, a template matching determination unit 402, a template creation unit 406, a moving speed detection unit 403, a search range determination unit 404, and a search base point determination unit 405. The viewpoint position detecting unit 400 is, for example, the photographing unit 3
A general-purpose personal computer capable of storing the image signal output by the 00 can be used.

The image storage unit 401 is used as a means for storing image data captured by the image capturing unit 300. Even if the image storage unit 401 is configured by a semiconductor memory such as a RAM, it is configured by using a storage device such as a magnetic disk or an optical disk. Is also good.

The template matching judgment unit 402
Using the template supplied from the template creation unit 406, position information of an area having the highest correlation with the template in the image stored in the image storage unit 401 is output to the display controller 100. The controller 100 generates an optimal barrier mask using the position information. The judging unit 402 uses the detected position information as the moving speed detecting unit 4
03 and the search base point determination unit 405 are also output. This is an opportunity to change the search range for template matching when template matching fails,
This is for giving to the moving speed detecting unit 403 and the search base point determining unit 405.

The template creation unit 406 creates a template for pattern matching used by the template matching determination unit 402 from the image data stored in the image storage unit 401. The template is a predetermined pattern. Is also good.

The creation of a template in the template creation section will be described. The templates for template matching used here are, for example, two child templates and one parent template. Each type of template will be further described with reference to (a) and (b) of FIG.

FIGS. 10 (a) and 10 (b) are diagrams for explaining a child template and a parent template used in the present embodiment, respectively. As shown in the figure, the two child templates each have a viewpoint based on the viewpoint positions of the right eye and the left eye (indicated by X in the figure), and the parent template includes the viewpoint positions of the right eye and the left eye. One template is based on a point. Here, the viewpoint position in the template is a coordinate indicating one point of the coordinates in the image.

In the present embodiment, template creation processing is performed from the creation of a child template. The template creation unit 406 includes a child template 1 based on the right eye viewpoint position from the visible image stored in the image storage unit 401,
The child template 2 based on the left eye viewpoint position is created. The size of the child template is determined from the following equation based on the distance between the right-eye and left-eye viewpoint positions.

Average distance between the right and left eye viewpoint positions of the human: Measured distance between the right and left eye viewpoint positions = size enough to include the average human eye and eyebrows: size of child template Sa

Here, as the average value of the distance between the viewpoint positions and the size of the eye and eyebrows, for example, a statistically determined value can be used. Here, the template creation unit 406
The left eye viewpoint position and the right eye viewpoint position, which are the base points for template creation, are stored with default values. That is, the user's face image for determining the template is
At the time of template creation, to capture 0 correctly,
The user places his or her face at a predetermined position, and the right and left eye images in a predetermined range are shot by the shooting unit 300. As a result, the image captured by the image capturing unit 300, which is the basis for creating the template, and the image stored in the storage unit 401 accurately includes the images of the left and right eyes of the user.

Once the template is created, even if the user moves the viewpoint position, the template matching determination unit 402 tracks the viewpoint position, or the position where the left and right eyes will be located even if the template matching fails. The search range determination unit 404 and the search base point determination unit 405 operate so that the search range is moved to the range shown in FIG.

When a template is created, it is necessary to accurately measure the viewpoint position. For this purpose, the viewpoint may be modified as follows. That is, as shown in FIG. 11, an image of the user's face taken by the photographing unit 300 is displayed on the display device 10.
Display at 00. On the other hand, the creating unit 406 also displays the frame 450 of the template image at a predetermined position on the display device 1000 via the controller 100. The user moves his or her face position within the displayed frame so that his or her left and right eyes enter this frame 450. After confirming that the left and right eyes are within the frame, the template creation unit 406 is notified by operating the OK icon 451 using a keyboard or a mouse. The template creation unit 406 creates a template image according to the image data in the storage unit 401 at that time.

Continuing with the description of the template image, the parent template is a template that includes two viewpoint positions, starting from the middle point between the two viewpoint positions for the right eye and the left eye. The size of the parent template is determined from the following equation based on the distance between the right-eye and left-eye viewpoint positions. Average distance between the right and left eye viewpoint positions of a human: Measured distance between the right and left eye viewpoint positions = size enough to include an average human face: size of parent template

As in the case of creating the child template, a statistically determined value can be used for the average value. The template created by the template creation unit 406 is supplied to the pattern matching determination unit 402. As described above, the template creation unit 406 creates a template.

FIG. 12 illustrates a template matching method employed in the embodiment. In the figure, a template 410 is a template generated by the creation unit 406. It is assumed that the user (observer) has moved from position 413 to position 414. Search range determining unit 404
Determines the size of the search range 411. The search base point determination unit 414 determines the base point 412 of the search range. The template matching determination unit 402 performs template matching while moving the template 410 within a search range 411 having the base point position 412 as a base point.

In this embodiment, the search base point is determined first,
The search range is arranged so that the determined search base point is located at the center. FIG. 12 shows template matching using a parent template, but the same applies to a child template.

The template matching can be performed using, for example, a normalized correlation function. For the pattern matching using the normalized correlation function, for example, see “Matrox Imaging Library Version 5.1 User Guide (Matrox ImagingLibrary Version 5.1 User Guide)”.
Guide) "on pages 154-155. The value obtained by the normalized correlation function is 0 to 100
(%), 100% means perfect match.

In this embodiment, if a degree of correlation exceeding, for example, 85% is obtained, it is determined that the pattern matching has succeeded. In the pattern matching immediately after the template is created, since the image on which the template is based and the image data to be subjected to the pattern matching are the same, basically a correlation of almost 100% should be obtained.

<Following of Viewpoint Position> FIG. 13 illustrates a method of determining a base point of a search range employed in the present embodiment. In the figure, 420, 421, and 422 are, respectively,
Indicates the base point position of a template for which template matching was successful in the past. That is, in the (n-3) th template matching, matching was obtained at the point 420, in the (n-2) th template matching, matching was obtained at the point 421, and in the (n-1) th template matching, matching was performed at the point 420. , And where to set the base point of the search range for the n-th template matching will be described. In this embodiment, a point at which template matching is obtained is called a “matching point”. In the example of FIG. 13, when the matching point 421 moves to the matching point 422,
The speed vector V _n-1 is obtained. Then, a search base point 423 of a search range (referred to as a “basic search range” for convenience) for the n-th template matching to be performed is shifted from the matching point 422 in the same direction as the vector V _n−1 ,

[Equation 1] Distance k.V_n-1  Is set to a point position that is separated only by Here, k is a predetermined constant.
Is a number.

Normally, the user's viewpoint position often does not move. If the viewpoint position has not moved from the point 421 (that is, the points 421 and 422 match), it is expected that the viewpoint position in the n-th template matching has not moved. In this case, since V _n−1 = 0, the search base point 423 is set to overlap the matching point 421. Therefore, the viewpoint position is surely grasped within the basic search range that has not been moved.

In the example of FIG. 13, if a match is obtained within this basic search range, the position is set as the n-th matching point 424.

FIG. 14 illustrates a method of expanding the search range when no matching is obtained within the basic search range. In the present embodiment, if no matching is obtained in the basic search range, a search range larger than the basic search range is set. If no matching is obtained within the enlarged search range, a larger search range is set. In the example of Figure 14, except for the basic search range (E _0), p pieces of expanded search range in total _{(E 1, E 2, E} 3, E 4,
..., E _p-1 , E _p ). Generally, larger search range E _p is set associated with the value of the most recent viewpoint position movement velocity V _p which has been measured during the matching was 0.00. That is, various values of the moving speed V

V₁<V_Two<V_Three<V_Four<... <V_p-1<V_p  In this order, for that speed value, E₁<E_Two<E_Three<E_Four<... <E_p-1<E_p  The expanded search range is set in the order of magnitude.
If the measurement is done just before the template matching fails
The value of the specified moving speed is V_ThreeIf it was
The search range initially set as the search range is E_FourIt is. Also
Enlarged search range E_FourIn template matching within
If you lose, the next expanded search range is E_FiveIs set to General
Before the failure of template matching
The value of the moving speed was V_kIf it was, expanded search
The search range initially set as the range is E_kAnd the following
Extended search range is E_{k + 1}It is.

FIG. 16 is an example of FIG. 14, where template matching has failed in the basic search range.
This indicates that the search range has been expanded, template matching has been successful within the expanded search range, and a matching point 424 has been obtained.

FIGS. 17 and 18 are flowcharts showing the operation of the search range determining unit 404 and the search base point determining unit 405 in FIG. First, in step S100, the photographing unit 300, the viewpoint position detecting unit 400, and the controller 10
Start each unit such as turning on the power of 0 and initializing. next,
In step S102, the template creation unit 406
Create a template as shown in FIG. As described above, the template image is created by a method in which an image at a predetermined position on the face when the user's head is at a predetermined position is used as a template image (hereinafter, referred to as a “first method”). A method in which the user's head is moved so that the image of the user's face displayed on the display device 1000 falls within a predetermined frame on the display screen, and the image obtained at that time is used as a template image (hereinafter, referred to as “template image”). The second method is called)
There is.

In step S104, a search base point is set. In the first method, the search base point is a default coordinate position. On the other hand, in the second method, it is set at the center position of the frame. Since search base point set in step S104 is the initial search base point, and S _0.

In step S106, the counter i is initialized to 0, and in step S108, the search range is set to E ₀ (that is, the basic search range). In step S110, the search base point set in step S104 is used as a base point,
As the search range the basic search range E ₀ set in step S108, performs the template matching. Step S1
At 12, it is determined whether the target template image is found within the search range set in this way.

Assuming that the position where the template image is found is P ₀ , the next search base point S ₁ is determined in step S 114. When the elapsed time from the step S106 up to the step S114 to Delta] t, the velocity vector V ₀ where the viewpoint position moves from S ₀ to P _0, the

[Equation 2] According to FIG. 13, the next search base point S ₁ is

[Equation 3]

Determined according to the following. That is, the new search base point is parallel to the movement vector V of the viewpoint position calculated from the last recognized matching position and further based on the change in the recognized matching position where the template matching has succeeded so far. The direction is set at a position shifted by a distance proportional to the moving speed V. The reason for shifting is based on the assumption that if the viewpoint position has been shifted by then, the next viewpoint position will also be found at the shifted position. Conversely, when there is no movement, V is zero,
The new search base point is set to the original search base point.

Further, steps S116 to S10
Returning to 6, a counter i is initialized (this counter i is used when failing to template matching), in step S108, using a search range E _0. That is, unless there is a failure in template matching,
E ₀ is used. Then it is carried out template matching in step S110, it is determined whether find a template image within the search range E ₀ at the new search base point position.

It is assumed that the operations from step S106 to step S116 are repeated and a matching position is found at point 422 as shown in FIG. In this case, since the moving speed is V _n−1 , as shown in FIG.

[Equation 4]

The magnitude E ₀ is set at the search base point. In the example of FIG. 16, since the actual viewpoint position is at the point 424, template matching in the basic search range fails.
Go to 200. Step S200 to step S300
Compares the moving speed V _n-1 with the threshold speed in FIG. 15 to determine the size of the search range that matches each speed threshold. That is, for example, if the moving speed V _n-1 exceeds the maximum speed V _p , the argument p is stored in the counter i in step S202 and the eighteenth
The process proceeds to step S500 in the figure. The moving speed V _n-1 is the threshold speed
If V _p-1 is exceeded, the argument p-1 is stored in the counter i and the
When the moving speed V _n-1 exceeds the threshold speed V ₂ , the process proceeds to step S500 in FIG.
The process proceeds to step S500 of FIG. 18 contains the argument 2, if the moving velocity V _n-1 is less than the threshold speed V _2, in step S400, step 18 Figure stores argument 1 for a counter i Proceed to S500.

[0123] At step S500 of FIG. 18, to determine the search range E _I with the value of the counter i. Step S5
The template matching 02 the search range E in _I determined in step S500. A successful template matching in the search range E _I, the process returns to step S106, the size of the search range in order to return to the basic search range,
The counter i is returned to “0”.

[0124] On the other hand, if it was found that we failed to template matching in step S504, the operation of steps S500 to step S502, expanded search range is repeated until more than E _p.

[0125] expanding the search range E _p is set to the size of the screen of the display device 60. For this purpose, step S506
In YES, that, when the enlarged search range matches the size of the screen, by returning from step S506 to step S500, expanded search range size of the E _p, i.e., be maintained in the size of the screen You. By maintaining the size of the screen, when the viewpoint position goes out of the screen, the tracking can be resumed immediately when it returns to the inside of the screen again.

<Following Direction of Viewpoint Position> The generation order of the enlarged search range shown in FIG. 14 can be dealt with regardless of the moving direction of the viewpoint position in any of up, down, left and right directions. The enlarged search range in FIG. 14 is easy to cope with when the user does not move in a large range on the screen.

The order of the expanded search range is not limited to that shown in FIG. 14, and may be, for example, as shown in FIG. High followability when moving at high speed in an oblique direction. Further, different patterns shown in FIGS. 20 and 21 may be properly used. That is, when the moving direction of the viewpoint position is the right direction (for example, the sign of the moving speed vector V is positive), the moving direction of the viewpoint position is set to the left direction (for example, the moving speed vector V When the sign is negative, the search range expansion pattern shown in FIG. 21 is used. In this way, when the viewpoint position reciprocates left and right, the viewpoint position can be followed at the highest speed.

<Following of Viewpoint Position> The first embodiment (ninth embodiment)
Fig.) Shows that the search base point is [number
According to [4], each was determined based on the matching position by template matching.

In the second embodiment shown in FIG. 22, a search base point is determined based on a matching position by template matching. That is, the search base point according to the second embodiment shown in FIG. 23 is determined based on the last matching position (422 in the example of FIG. 23) as a base point, a basic search range, and a base search range. Sets an enlarged search range.

<Consideration of Matching Level> As described in the first embodiment, the template matching determination can output a matching degree. On the other hand, in the first embodiment, if the template matching fails once, the search range is expanded. Generally, when the search range is expanded, the probability of erroneous determination increases. Therefore, in the modification of the first embodiment, as shown in FIG. 24, an acceptance level determination unit 410 is provided to change the acceptance level according to the size of the set enlarged search range. Here, the acceptance level is a degree of coincidence serving as a criterion when the template matching determination unit determines that a match has been obtained, and is represented by% in the case of using the above-described normalized correlation function. Thus, for example, the search range determining section 404, when trying to set the search range of the maximum magnitude E _P is receiving level determining unit 410, the template matching determination unit 90 is larger than for example 85% 402 Output%. Upon receiving this, the determination unit 402 determines that matching has been achieved only when the degree of correlation is 90% or more. 25th
The figure shows a modification of the second embodiment in which the above-mentioned acceptance level determination unit is provided.

<Modification of Search Range Setting Method> In the control procedure shown in FIGS. 17 and 18 of the first embodiment, the size of the enlarged search range is determined by one step each time template matching fails. It was something to raise. This modification is such that the higher the moving speed of the viewpoint position detected immediately before is, the higher the enlargement speed is.

FIGS. 26 and 27 show a control procedure corresponding to this modification, and correspond to FIGS. 17 and 18 of the first embodiment. The control procedure of FIG. 26 is different from the control procedure of FIG. 17 in the latter steps S202 and S30.
2. Step S400 is the former step S202 ′,
In steps S302 'and S400', only i = p, j = p, i = 2, j = 2, i = 1, j = 1, respectively. With this setting, the second
If it is detected in step S504 in FIG. 7 that the template matching has failed, while i does not exceed p,
In step S512, i = i + j is performed, and using this i, the search range is determined in step S500.
_{Ei is} to be searched.

For example, steps S200 and S3
00, etc., if V less than ₆ moving speed V ₅ or the viewpoint position, i = 5, j = 5 is set, in step S512, i = 5 + 5 = 10 is set, the E ₁₀ The search range. Next, when the process fails and returns to step S512, the enlarged search range is expanded by five steps so that i = 10 + 5 = 15.

[0134] Thus, according to this modified example, if the moving speed is detected as V _k, the size of the search range, will be incremented by k stages. By such processing, the speed of enlargement of the search range is set according to the moving speed, so that the followability of the search range is improved.

[0135]

[Other Embodiments] In the above-described embodiments, the resetting of the search range is limited to the case where template matching for detecting a viewpoint position has failed. When used as a display system, a button or the like that can be operated by a user is provided on the image display unit 60 or a remote controller (not shown), and when the user has difficulty in recognizing the stereoscopic display, the user can press this button to perform a search. An operation for resetting the base position and size of the range may be performed again. With such a configuration, the template is updated at an accurate timing, and the viewpoint position detection can be performed with higher accuracy. As a result, a stereoscopic image display system with a wide stereoscopic view range can be realized. . In the first and second embodiments and the modification,
The next search base point position was determined by [Equation 3] or [Equation 4]. In this case, the value of k has a great influence on the search base point position. Therefore, a modified example in which the user can set the value of k through, for example, a keyboard is proposed.

If the value of k is set to a small value, such a setting is suitable for an application in which the user's head does not move so much. Conversely, if the value of k is set to a large value, such a setting is suitable for an application in which the user's head often moves.

In the above-described embodiment, the case where the detection result of the viewpoint position detecting device according to the present invention is supplied to the stereoscopic image display device has been described as an example. Can be used.

Further, the specific methods described in the above embodiments, such as the pattern matching method and the template creation method, are not limited to those described in the embodiments, but may be equally applicable. It is needless to say that may be used.

In the above-described embodiment, the viewpoint position, which is the pinpoint coordinate, is configured to be output. However, for example, as in the above-described embodiment, the finally obtained viewpoint position is determined by the three-dimensional image display device. In the case of using for control, since the minimum control is possible if the center positions of the viewpoint positions of the right eye and the left eye are known, the center positions of the right eye template and the left eye template are output. Is also good.

Further, in the above embodiment, as a means for detecting the moving speed of the viewpoint position, the calculation is performed based on the change of the viewpoint position detected by the template matching, but an external head position / posture measurement sensor is used. It is also possible to replace it with the detected moving speed of the viewpoint position.

In the above embodiment or example, the template image is fixed once obtained. However, if the user's line of sight changes, the face image captured by the image capturing apparatus 300 changes from the time when the template image was generated. Therefore, in this modified example, as shown in FIG.
An eye-gaze detection sensor 461 is provided, a template memory 461 is provided in the viewpoint position detection unit 400, and the template creation unit is changed to a template calculation creation unit 460. In this modification, the template image initially created is not different from the embodiment or the example. That is,
The template image is stored in the memory 461, and the image is sent to the template matching determination unit 402.

Thereafter, when the line of sight direction changes, the change is sent to the calculation creating section 460 as a change in camera parameters, and the calculation creating section 460 sends the template memory 46
1 is corrected (for example, FIG. 29) by performing reverse perspective transformation using camera parameters by the amount of change in the line of sight direction, and stores the corrected image in the memory 461. In this way, the matching accuracy does not decrease even if the line of sight changes.

Even if the present invention is applied to a system composed of a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), an apparatus (for example, a copying machine, a facsimile machine, etc.) Device).

An object of the present invention is to supply a storage medium (or a recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or an apparatus, and to provide a computer (a computer) of the system or the apparatus. It is needless to say that the present invention can also be achieved by a CPU or an MPU) reading and executing the program code stored in the storage medium.

In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. By executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written into the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, the program code is read based on the instruction of the program code. Needless to say, the CPU included in the function expansion card or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above.

[0148]

As described above, according to the present invention,
When the next search range is determined after the template matching has failed, by setting the size according to the moving speed up to that time, it is possible to increase the probability that a face part exists in the search range. That is, the detection speed of the face part can be increased. In addition, not only when the template matching has failed, but also when the template matching has succeeded, the position of the search range is substantially matched with the moving direction of the face part so far, so that the face part is within the search range. Can be increased. That is, the detection speed of the face part can be increased.

[Brief description of the drawings]

FIG. 1 is a basic configuration diagram of a conventional stereoscopic image display device.

FIG. 2 is a basic configuration diagram of a conventional stereoscopic image display device.

FIG. 3 is a configuration diagram of a stereoscopic display device to which the viewpoint position detecting device of the embodiment is to be applied.

FIG. 4 is a diagram illustrating the principle that left and right parallax images are horizontally separated and observed by both eyes of a user.

FIG. 5 is a schematic side view of a cross section in a vertical direction of a three-dimensional image display device based on a cross lenticular system.

FIG. 6 is a side view of a vertical cross section of the stereoscopic image display device of the embodiment.

FIG. 7 is a diagram illustrating a stereoscopic image display method by the stereoscopic display device shown in FIG.

FIG. 8 is a diagram illustrating a method of displaying a three-dimensional image and a two-dimensional image in a mixed manner by the stereoscopic display device shown in FIG. 3;

FIG. 9 is a block diagram showing a configuration of a first example of the viewpoint position detecting device of the embodiment.

FIG. 10 is a view for explaining a method of acquiring a template image used for detecting a viewpoint position according to the embodiment.

FIG. 11 is a view for explaining another example of a method for acquiring a template image used for viewpoint position detection according to the embodiment.

FIG. 12 is a view for explaining the principle of template matching employed in the apparatus of the embodiment.

FIG. 13 is a view for explaining the principle of generating a basic search range in the apparatus according to the embodiment.

FIG. 14 is an exemplary view for explaining an example of a method of generating an enlarged search range in the apparatus according to the embodiment.

FIG. 15 is a view for explaining the relationship between the moving speed of the viewpoint position and the size of the search range in an example of the method for generating the enlarged search range in FIG. 14;

FIG. 16 is a view for explaining an example of a method of generating an enlarged search range in the apparatus according to the embodiment.

FIG. 17 is a flowchart showing a control procedure of the first embodiment.

FIG. 18 is a flowchart illustrating a control procedure of the first embodiment apparatus.

FIG. 19 is a diagram illustrating another example of a method for generating an enlarged search range in the apparatus according to the embodiment.

FIG. 20 is a view for explaining still another example of a method of generating an enlarged search range in the apparatus according to the embodiment.

FIG. 21 is a view for explaining still another example of a method of generating an enlarged search range in the apparatus according to the embodiment.

FIG. 22 is a block diagram showing a configuration of an apparatus according to a second embodiment.

FIG. 23 is a view for explaining a method of determining a search base point according to the second embodiment.

FIG. 24 is a block diagram showing a configuration of a modification of the first embodiment.

FIG. 25 is a block diagram showing a configuration of a modification of the second embodiment.

FIG. 26 is a flowchart illustrating a control procedure according to a modification.

FIG. 27 is a flowchart illustrating a control procedure according to a modification.

FIG. 28 is a block diagram illustrating a configuration of a modification example device in which a line-of-sight direction is considered.

FIG. 29 is a diagram showing a change in a template image before and after correction when a line-of-sight direction changes.

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] February 7, 2001 (2001.2.7)

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Claim 1

[Correction method] Change

[Correction contents]

[Procedure amendment 2]

[Document name to be amended] Statement

[Correction target item name] Claim 5

[Correction method] Change

[Correction contents]

[Procedure amendment 3]

[Document name to be amended] Statement

[Correction target item name] Claim 9

[Correction method] Change

[Correction contents]

[Procedure amendment 4]

[Document name to be amended] Statement

[Correction target item name] Claim 14

[Correction method] Change

[Correction contents]

[Procedure amendment 5]

[Document name to be amended] Statement

[Correction target item name] Claim 18

[Correction method] Change

[Correction contents]

[Procedure amendment 6]

[Document name to be amended] Statement

[Correction target item name] Claim 22

[Correction method] Change

[Correction contents]

[Procedure amendment 7]

[Document name to be amended] Statement

[Correction target item name] Claim 27

[Correction method] Change

[Correction contents]

[Procedure amendment 8]

[Document name to be amended] Statement

[Correction target item name] Claim 32

[Correction method] Change

[Correction contents]

[Procedure amendment 9]

[Document name to be amended] Statement

[Correction target item name] Claim 34

[Correction method] Change

[Correction contents]

[Procedure amendment 10]

[Document name to be amended] Statement

[Correction target item name] Claim 39

[Correction method] Change

[Correction contents]

[Procedure amendment 11]

[Document name to be amended] Statement

[Correction target item name] 0027

[Correction method] Change

[Correction contents]

[0027]

According to a first aspect of the present invention, there is provided a face part detecting apparatus, comprising: an image pickup means; and an optional part set in a predetermined image picked up by the image pickup means.
Template matching means for detecting a position of a predetermined specific part in the predetermined image by matching a search range with a template; speed detecting means for detecting a position change speed of the specific part; and the template matching means There comprising a changing means for changing the size of the search range if it can not detect the specific portion within the search range, the changing means, as a position change speed detected for the particular site is large, By setting a larger search range , the speed of detecting a specific part within the search range is improved.

[Procedure amendment 12]

[Document name to be amended] Statement

[Correction target item name] 0028

[Correction method] Change

[Correction contents]

The fact that the position change rate of the specific portion large, since it is the likely to be moving the site in the region of correspondingly the greater range, greater search
By setting the range , the possibility of failure of template matching is reduced, and as a result, the detection speed of the specific site position can be improved.

[Procedure amendment 13]

[Document name to be amended] Statement

[Correction target item name] 0043

[Correction method] Change

[Correction contents]

The above-mentioned problem of detecting a specific portion at a high speed can also be achieved by the device according to claim 27 having the following configuration. That is, the image detection device according to claim 27, by performing matching between the template and the arbitrary search range set in the predetermined image captured by the imaging unit, the predetermined detection in the predetermined image. Template matching means for detecting the position of the part, direction detecting means for detecting the moving direction of the specific part from a change in the detection position of the specific part, and detecting the position of the search range for the next template matching. Setting means for setting the moving direction of the specific part detected by the means. If the position of the search range is set so as to match the moving direction up to that time, the probability that the specific part is included in the search range set as such increases.

[Procedure amendment 14]

[Document name to be amended] Statement

[Correction target item name] 0048

[Correction method] Change

[Correction contents]

When a function for changing the size of the search range is added to the invention for changing the position of the search range, the speed of detecting a specific part is further improved. Therefore, as in claim 32, wherein the search range when the template matching unit can not find the specific portion within the search range
Further comprising changing means for changing the size of the enclosure ,

Continued on the front page F term (reference) 5B057 AA20 BA02 CA12 CA16 DA06 DB02 DC32 5C061 AA07 AB12 AB14 5L096 BA18 GA08 HA03 HA08 JA09 KA03 9A001 HH21 HH34 KK31

Claims (46)

[Claims] 1. A position of a specific part in a predetermined image is detected by matching a template with an arbitrary search area set in a predetermined image picked up by the image pickup means. A template matching unit, a speed detecting unit that detects a position change speed of the specific region, and a change that changes a size of the search region when the template matching unit cannot detect the specific region in the search region. Means for setting, the larger the position change speed detected for the specific part, the larger the search area, thereby improving the detection speed of the specific part in the search area. Image detecting device. 2. The method according to claim 1, wherein the specific part is a facial part in a facial image, and the speed detecting unit detects a position change speed based on a change in the facial part position detected by the template matching unit. The image detection device according to claim 1. 3. The image detection apparatus according to claim 1, wherein the specific part is a viewpoint position of the face part. 4. The method according to claim 1, wherein the change unit decreases the size of the search range when the template matching unit detects the specific part in the search range changed by the change unit. Claims 1 to 3
The image detection device according to any one of the above. 5. The method according to claim 1, wherein the template matching unit returns the size of the search range to an initial value when the specific part is detected in the search range changed by the change unit. The image detection device according to any one of claims 1 to 4, wherein 6. The image detection apparatus according to claim 1, wherein the search range can be changed from a minimum value size to a screen size. 7. The image detection apparatus according to claim 6, wherein the search range is maintained at the minimum value size as long as the template matching unit succeeds in template matching. 8. The method according to claim 1, wherein the changing unit includes a moving unit that moves a position of the search range, and the moving unit moves the position of the search range to a matching position detected by the template matching unit. The image detection device according to claim 1. 9. The method according to claim 1, wherein the changing unit includes a moving unit that moves a position of the search range, and the moving unit matches the position of the search range with the matching position detected by the template matching unit. The image detection apparatus according to claim 8, wherein 10. The apparatus further comprises means for detecting a change in the viewpoint position based on a change in the matching position detected by the template matching means, wherein the moving means comprises:
10. The image detection apparatus according to claim 8, wherein the search range is moved by setting a next search range at a position corresponding to the amount of change in the viewpoint position detected this time. 11. The change unit includes: a maintenance unit that maintains a size of a search range at a minimum initial value as long as the template matching by the template matching unit is successful; Means for setting the size of the range to a speed-equivalent size corresponding to the position change speed detected by the speed detecting means; and, when template matching is repeatedly failed, the size of the search range is set to be smaller than the speed-equivalent size. 11. The image detecting apparatus according to claim 1, further comprising: means for increasing the size by a predetermined value. 12. The change unit includes: a maintenance unit that maintains a size of a search range at a minimum initial value as long as the template matching by the template matching unit is successful; and a search unit when the template matching fails for the first time. Means for setting the size of the range to a size corresponding to the speed corresponding to the position change speed detected by the speed detecting means; and, when template matching fails repeatedly, setting the size of the search range to be smaller than the speed equivalent size. The apparatus according to any one of claims 1 to 10, further comprising: means for increasing the size by an increment corresponding to a speed corresponding to the moving speed detected by said speed detecting means. 13. The template matching means according to claim 1, wherein a threshold value for determining a matching success is variable.
When the changing unit increases the size of the search range,
The image detection device according to claim 1, wherein a threshold value for determining the success of the matching is increased. 14. An image capturing step, and a position of a predetermined specific portion in the predetermined image is detected by matching a template with an arbitrary search area set in the predetermined image captured by the image capturing unit. A template matching step, a speed detection step of detecting a position change speed of the specific part, and a change of changing the size of the search area when the template matching step cannot detect the specific part in the search area. The method further comprises: setting a larger search area as the position change speed detected for the specific part is higher, thereby improving the detection speed of the specific part in the search area. Image detection method. 15. The method according to claim 15, wherein the specific part is a facial part in a facial image, and the velocity detecting step detects a velocity based on a change in the facial part position detected in the template matching step. Item 15. The image detection method according to Item 14. 16. The image detection method according to claim 14, wherein the specific part is a viewpoint position of the face part. 17. The method according to claim 17, wherein the template matching step reduces the size of the search range when the specific part is detected within the search range changed by the change step. The image detection method according to claim 14. 18. The method according to claim 18, wherein the template matching step returns the size of the search range to an initial value when the specific part is detected within the search range changed by the change step. The image detection method according to any one of claims 14 to 17, wherein 19. The image detection method according to claim 14, wherein the search range can be changed from a minimum value size to a screen size. 20. The method according to claim 19, wherein the search range is maintained at the minimum size as long as the template matching step succeeds in template matching. 21. The changing step includes a moving step of moving a position of a search range, wherein the moving step moves a position of the search range to a matching position detected by the template matching step. The image detection method according to any one of claims 14 to 20, wherein 22. The changing step includes a moving step of moving a position of a search range, wherein the moving step matches a position of the search range with a matching position detected by the template matching step. The image detection method according to claim 21, wherein 23. The method according to claim 23, further comprising the step of detecting a change in a viewpoint position based on a change in the matching position detected by the template matching step.
23. The image detection method according to claim 21, wherein the search range is moved by setting a next search range at a position corresponding to the amount of change in the viewpoint position detected this time. 24. The changing step includes a maintaining step of maintaining the size of the search range at the minimum initial value as long as the template matching in the template matching step is successful, and a searching step when the template matching fails for the first time. Setting the size of the range to a size corresponding to the speed corresponding to the position change speed detected in the speed detection step; and, if the template matching is repeatedly failed, set the size of the search range to be smaller than the speed equivalent size. 24. The image detecting method according to claim 14, further comprising: increasing a size by a predetermined value. 25. The changing step includes a maintaining step of maintaining the size of the search range at a minimum initial value as long as the template matching in the template matching step is successful, and a searching step when the template matching fails for the first time. Setting the size of the range to a size corresponding to the speed corresponding to the position change speed detected in the speed detection step; and, if the template matching fails repeatedly, set the size of the search range to be smaller than the speed equivalent size. The image according to any one of claims 14 to 23, further comprising a step of increasing a size by a speed corresponding increment according to the moving speed detected in the speed detecting step. Detection method. 26. In the template matching step, a threshold value for determining a successful match is variable,
When the change step increases the size of the search range,
The image detection method according to any one of claims 14 to 25, wherein a threshold value for determining the success of the matching is increased. 27. A position of a predetermined specific part in the predetermined image is detected by matching the template with an arbitrary search area set in the predetermined image captured by the imaging unit. Template matching means, direction detecting means for detecting the moving direction of the specific part from a change in the detection position of the specific part, and specifying the position of the search range for the next template matching by the direction detecting means. An image detecting apparatus, comprising: setting means for setting a moving direction of a part. 28. The apparatus according to claim 28, further comprising speed detection means for detecting a position change speed of the specific part, wherein the setting means determines a search range for the next template matching by the specific part detected by the direction detection means. 28. The image detecting apparatus according to claim 27, wherein the moving direction is set at a position separated by a distance corresponding to the position change speed of the specific part detected by the speed detecting means. 29. The method according to claim 29, wherein the specific part is a facial part in a facial image, and the direction detecting means detects a moving direction based on a change in the facial part position detected by the template matching means. An image detection device according to claim 27 or claim 28. 30. The specific part is a facial part in a facial image, and the velocity detecting means detects a position change velocity based on a change in the facial part position detected by the template matching means. Claim 28
The image detection device according to claim 1. 31. The image detection apparatus according to claim 27, wherein the specific part is a viewpoint position of the face part. 32. The image processing apparatus according to claim 32, further comprising: changing means for changing a size of the search area when the template matching means cannot detect the face part in the search area; 32. The position of a search range is set in the moving direction of the facial part detected by the direction detecting means according to the size enlarged by the changing means. Item 2. The image detection device according to item 1. 33. The apparatus according to claim 27, wherein the setting unit maintains the search range at a fixed size as long as the template matching unit succeeds in the template matching. The image detection device according to claim 1. 34. An image capturing step, and a position of a predetermined specific part in the predetermined image is detected by matching a template with an arbitrary search area set in the predetermined image captured by the image capturing unit. A template matching step, a direction detecting step of detecting a moving direction of the specific part from a change in a detection position of the specific part, and a position of a search range for the next template matching is detected by the direction detecting step. An image detecting method, comprising a setting step of setting a moving direction of a specific part. 35. A speed detecting step for detecting a position change speed of the face part, wherein the setting step sets a search range for a next template matching to a specific part detected by the direction detecting step. 35. The image detecting method according to claim 34, wherein the moving direction is set to a position separated by a distance corresponding to the position change speed of the specific part detected in the speed detecting step. 36. The method according to claim 36, wherein the specific part is a facial part in a facial image, and the direction detecting step detects a moving direction based on a change in the position of the facial part detected in the template matching step. The image detection method according to claim 34 or 35. 37. The specific part is a facial part in a facial image, and the velocity detecting step detects a position change velocity based on a change in the facial part position detected in the template matching step. Claim 35
2. The image detection method according to 1., 38. The image detection method according to claim 34, wherein the specific part is a viewpoint position of the face part. 39. The method according to claim 39, further comprising a changing step of changing a size of the search area when the face matching part cannot be detected in the search area in the template matching step. 39. The position of a search range for the moving object is set in the moving direction of the facial part detected by the direction detecting step according to the size enlarged by the changing step. 2. The image detection method according to claim 1. 40. The method according to claim 34, wherein the setting step maintains the search range at a fixed size as long as the template matching step succeeds in template matching. The image detection method described in the above. 41. The apparatus according to claim 1, further comprising: means for detecting a line-of-sight direction, wherein the template matching means corrects the deformation of the template image according to the detected line-of-sight direction. Or 1 or 2
The image detection device according to any one of claims 7 to 33. 42. The method according to claim 14, further comprising a step of detecting a line-of-sight direction, wherein the template matching step deforms and corrects the template image in accordance with the detected line-of-sight direction. 41. The image detection method according to claim 1 or any one of claims 34 to 40. 43. A program storage medium for storing a computer program for realizing the image detection method according to any one of claims 14 to 26 or 34 to 40. 44. A means for generating a spatially modulated image based on a viewpoint position detected by the image detection device according to any one of claims 1 to 13 or 27 to 33; A stereoscopic display device comprising: a barrier display unit that displays a modulated image; and an image display unit that is in front of the barrier display unit and that displays left and right parallax images. 45. A means for generating a spatially modulated image based on a viewpoint position; a barrier display means for displaying the spatially modulated image; and an image display means in front of the barrier display means for displaying left and right parallax images. A display controller connected to a three-dimensional display device comprising:
3. A display controller, which is connected to or built in the image detection device according to any one of 3. 46. A means for generating a spatially modulated image based on a viewpoint position, a barrier display means for displaying the spatially modulated image, and an image display means in front of the barrier display means for displaying left and right parallax images. And a stereoscopic display device comprising: a stereoscopic display device comprising:
3. A three-dimensional display system, comprising: a display controller connected to or incorporated in the image detection device according to any one of 3.