JP3351386B2 - Observer observation position detection method and apparatus - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for detecting an observer's observation position. The present invention relates to an observation position detection method used for a stereoscopic display device for presenting an image and the like, and an apparatus therefor. What
In this specification, the “observation position of the observer” is “viewing position”.
The position of the observer himself.

[0002]

2. Description of the Related Art In recent years, there has been an increasing demand for stereoscopic display, and various types of stereoscopic display devices have been studied. In particular, recently, an image having binocular parallax has been produced using a lenticular screen. 2. Description of the Related Art A stereoscopic display device that realizes stereoscopic vision by spatially separating and presenting eyes has attracted attention. This device has the advantage that no special glasses need to be worn. However, this device has a problem that the region in which stereoscopic vision is possible is limited to about the distance between human eyes, and the observer has to take an unnatural posture of always keeping his head fixed for observation. There was an inconvenience.

In order to solve such a problem, a viewpoint-following type stereoscopic display device which constantly detects an observer's observation position and always presents a correct stereoscopic image to the observer according to the detected observation position. Are known (for example, see
107247, JP-A-6-217348, etc.).

In the viewpoint-following three-dimensional display device according to the prior art, it is necessary to always attach a magnetic sensor or the like to the observer in order to detect the observer's observation position, which gives the observer an uncomfortable feeling. This is a factor that halves the advantage of a stereoscopic display device using a lenticular screen that does not require wearing special glasses.

In addition, when used by an unspecified number of observers, the use of a wearable observation position detecting device causes a problem in hygiene. Therefore, an apparatus and a method that can detect the observation position of the observer in a non-contact manner are desired.

Here, as a method of detecting the observer's observation position in a non-contact manner, the observer's face is imaged with a TV camera or the like, and only the observer's pupil is extracted from the captured image, and the pupil position is extracted. A method of detecting the observation position from the image can be considered.
In order to extract only the pupil from the captured image of the observer, for example, an awake state detecting device as disclosed in Japanese Patent Application Laid-Open No. 6-270711 can be used. In this apparatus, pupil extraction is performed using the reflection characteristics of the human pupil. Since the human pupil has a higher reflectance in the infrared region than other parts of the face, the human pupil tends to be brightly imaged when illuminated by an infrared light illuminating device having an optical axis coaxial with the imaging device.

FIG. 14 shows the above-mentioned JP-A-6-270711.
FIG. 2 is a diagram in which the pupil extracting means of the awake state detecting device in the Japanese Patent Publication is extracted. In this pupil detection device, the camera 901
And a light source 903 for illuminating the subject 900. The light source 903 is arranged so that its optical axis is coaxial with the optical axis of the camera 901.

The light source 903 is turned on by a light emission command signal from a light source light emission control unit 907 in response to a measurement start signal from the overall control unit 908. At this time, the image signal captured by the camera 901 is converted into a digital image by the A / D converter 904 and stored in the image memory 905. After that, the pupil of the subject 900 is extracted by calculating the image of the subject 900 stored in the image memory 905 by the pupil extracting unit 906. The pupil extraction unit 906 performs a pupil extraction process by performing a binarization process assuming that a pixel having a specific luminance between the thresholds Th1 and Th2 in the image stored in the image memory 905 is pupil reflected light. . In addition, a labeling process is performed on the binarized image to specify a pupil portion from a noise component.

[0009]

However, when the observation position of an observer moving in a wide area is detected by using the pupil extraction means used in the conventional arousal state detection device, the spatial position of the infrared light source is limited. If the illuminance distribution is uneven or the distance between the observer and the infrared light source fluctuates, the infrared illuminance on the observer's face changes. There is a disadvantage that binarization extraction cannot be performed. Further, in the conventional awake state detecting device, complicated image processing such as labeling processing is always required to specify a pupil portion from a noise component other than the pupil portion.

[0010]

SUMMARY OF THE INVENTION The object of the present invention is to improve the disadvantages of the prior art. In particular, the present invention is intended to determine the position of a moving observer when the distribution of the face illuminance of the observer is not uniform or when the observer and the infrared light source It is an object of the present invention to provide a method and an apparatus for detecting an observer's observation position that can be accurately detected in a non-contact manner even when the distance fluctuates.

[0011]

In order to achieve the above object, according to the first aspect of the present invention, a face for imaging an observer's face by an imaging means while illuminating the face of the observer with a predetermined illumination means. An imaging step, and an observation position calculation step of temporarily storing image information of the face imaged in the face imaging step in an image memory and calculating an observation position of an observer from the stored image information by predetermined image processing. A first information processing step of performing an operation of emphasizing image information of a pupil portion of a face image stored in an image memory in the above-described observation position calculation step; A second information processing step of binarizing the image information calculated in the first information processing step and emphasizing the pupil portion with a predetermined binarization threshold, and observing an observer from the binarized image information Detect position A structure in which a third information processing step.

[0012] Then, between the first information processing step and the second information processing step, predetermined binarization is performed based on the face image information in which the pupil portion has been enhanced by the first information processing step. A configuration is provided in which a binarization threshold value calculation step for determining a threshold value in real time is provided.

Therefore, according to the first aspect of the present invention,
In particular, since it is possible to obtain information on the face image in which the pupil portion has been emphasized by the first information processing step, it is easy to determine the binarization threshold value in the second information processing step,
In the information processing step, the observation position of the observer can be determined quickly and with high accuracy. Further, the threshold value for binarization can be set in real time based on the information of the captured face image. Since the determination is made, even if the illuminance of the face is small, an appropriate binarization threshold can be set in real time according to the illuminance of the captured face image, and the observer moves at the point where the Even if the illuminance changes, there is an advantage that image information in which the pupil portion is enhanced can be obtained sufficiently in response to the change.

According to the second aspect of the present invention, the face of the observer is illuminated by the predetermined illuminating means while the face of the observer is imaged by the imaging means, and the face is imaged by the face imaging step. An observation position calculating step of temporarily storing the image information of the observer in an image memory and calculating the observation position of the observer by predetermined image processing from the stored image information. A first information processing step of performing the above-described observation position calculating step by performing an operation of enhancing image information of a pupil portion of a face image stored in an image memory; and a pupil portion calculated by the first information processing step. A second information processing step of binarizing the face image in which is emphasized with a predetermined binarization threshold, and a third information processing step of detecting an observer's observation position from the binarized image information. Structure with To.

[0015] Between the first information processing step and the second information processing step, the signal level of the pupil portion in the face image is always constant even if the observation position of the observer changes. A light source light amount control step of driving and controlling the above-described illumination means is provided so as to make the lighting unit light.

Therefore, according to the second aspect of the present invention,
In particular, image information in which the pupil portion is emphasized can be obtained by the first information processing step, so that it is easy to determine the binarization threshold value in the second information processing step, and to observe It is possible to quickly and accurately determine the observer's observation position, and furthermore, when the observer moves and the face illuminance changes, the light source light amount control step immediately functions and the illumination means is driven. Under the control, the face illuminance of the observer is set to the state before the movement. For this reason, there is no need to calculate the threshold value for binarization, and accordingly, there is an advantage that the calculation processing time for determining the observation position can be shortened and the configuration can be simplified.

According to a third aspect of the present invention, in the method for detecting an observer's observation position according to the first or second aspect,
In the first information processing step, the operation of enhancing the pupil portion of the face image is performed by image differentiation processing.

Therefore, in the invention according to claim 3,
In addition to functioning equivalently to the above-described claim 1 or 2, the pupil portion of the face image is sharply enhanced by the image differentiation process. The binarization operation based on this can be performed extremely accurately.

According to a fourth aspect of the present invention, in the method for detecting an observer's observation position according to the first aspect, the binarization threshold value in the binarization threshold value calculation step is calculated in advance by emphasizing a pupil portion of the face image. , The image standard deviation of the face image subjected to the calculation is calculated, and then the image standard deviation is determined based on the image standard deviation.

Therefore, in the invention according to claim 4,
In addition to having the same function as the first aspect of the present invention, an appropriate binarization threshold value for the pupil portion can be reliably set, and in such a point, the reliability of the binarized information is increased. be able to.

According to a fifth aspect of the present invention, in the method of detecting an observer's observation position according to the second aspect, the light source light amount of the illuminating means, which is a control reference in the light source light amount control step, is determined in advance by using the pupil portion of the face image. The image standard deviation of the image on which the emphasis calculation has been performed is calculated, and then the image standard deviation is determined based on the image standard deviation.

Therefore, in the invention according to claim 5,
In addition to functioning equivalently to the above-described invention, there is an advantage that the control amount of the light amount of the illumination means can be more appropriately specified in real time.

According to each of the sixth and seventh aspects of the present invention, the method for detecting an observer's observation position according to any one of the first to fifth aspects includes first to third information processing steps. One of the information processing steps is provided with a series of information processing steps of another system.

Then, a series of information processing steps of the other systems is performed by a marker determining step of determining the presence or absence of a marker previously attached to the face of the observer from the face information stored in the image memory. A marker image binarization step of operating in preference to the one information processing step and binarizing a marker image in the image memory based on a predetermined threshold value when it is determined that a marker is present in the marker determination step; An observation position calculating step of specifying the position of the center of gravity of the observer based on the binarized information of the marker image obtained in the marker image binarizing step, and calculating the observation position of the observer based on the center of gravity position And a method of constructing the common configuration.

For this reason, each of the inventions according to claims 6 and 7 functions in the same manner as the invention according to any one of claims 1 to 5 described above, and further includes an observer wearing glasses or the like. However, there is an advantage that the observation position during and after the movement can be reliably captured and specified in real time.

Here, the predetermined threshold value in the above-described marker image binarization step is configured to use a predetermined threshold value which is smaller than the reflection luminance of the reflection marker and larger than the luminance of the pupil reflection light of the observer. May be.

In each of the inventions described in claims 8 to 10, the image pickup means for picking up the face of the observer, the lighting means arranged on the image pickup means side for illuminating the face of the observer, and the image picked up by the image pickup means An image processing means for calculating the observation position of the observer based on the image information of the face of the observer, wherein the image processing means comprises:
An image memory for storing image information of the observer's face obtained by the above-mentioned imaging means, and a pupil image to be subjected to a predetermined calculation for enhancing the pupil portion of the observer in the image information stored in the image memory An emphasis unit, a binarization processing unit that binarizes pupil image data output from the pupil image emphasis unit based on a predetermined binarization threshold, and a binarization processing unit based on a calculation result of the binarization processing unit. And an observation position calculation unit for calculating the observation position of the observer.

Then, a binarization threshold for a pupil image is determined in real time on the basis of the image information output from the pupil image enhancement unit and sent to the binarization processing unit. A configuration in which a binarization threshold setting unit is provided is adopted as a common basic configuration.

Therefore, in each of the inventions according to the eighth to tenth aspects, it is possible to obtain face image information in which the pupil portion is enhanced by the action of the pupil image enhancing portion, and thus the pupil portion in the binarization processing portion is obtained. Since the binarization of the information can be executed with high accuracy and certainty, and the threshold for binarization is determined in real time based on the image information of the imaged face, Even if the illuminance is low, an appropriate binarization threshold can be set in real time in accordance with the imaged face image information. Correspondingly, image information of the face in which the pupil portion is emphasized can be obtained. Therefore, the observation position calculation unit can quickly and accurately calculate the observation position of the observer, and externally outputs the position information. Towards It is possible to force.

Here, it is preferable to dispose the above-mentioned illumination means so that its optical axis is substantially coaxial with the optical axis of the imaging means. A light source illuminance calculating circuit for calculating an image standard deviation of image information output from the pupil image enhancing means in advance and calculating a light source illuminance based on the standard deviation; It may be constituted by a binarization threshold setting circuit that sets a binarization threshold based on the image standard deviation calculated by the calculation circuit.

In each of the inventions described in claims 11 to 13, the imaging means for imaging the face of the observer, the illumination means arranged on the imaging means side for illuminating the face of the observer, and the image taken by the imaging means An image processing means for calculating the observation position of the observer based on the face image information of the observer. An image memory for storing image information of the face, a pupil image enhancing unit for performing a predetermined operation for enhancing a pupil of an observer in the image information stored in the image memory, and an output from the pupil image enhancing unit A configuration including: a binarization processing unit that binarizes pupil image data based on a predetermined threshold; and an observation position calculation unit that calculates an observation position of the observer based on a calculation result of the binarization processing unit. When That.

The output of the illumination means is variably controlled so that the output level of the image information output from the pupil image enhancement section is always constant even if the observation position of the observer changes. A light source light quantity control unit to be used is also provided.
Further, a configuration in which a light source light quantity setting unit that regulates the operation of the light source light quantity control unit is provided in the above-described image processing unit is adopted as a common basic configuration.

Therefore, in each of the eleventh to thirteenth aspects, the face image information in which the pupil portion is enhanced by the operation of the pupil image enhancement part is the same as in each of the eighth to tenth aspects. Can be obtained, the binarization of the information of the pupil part in the binarization processing unit can be executed with high accuracy and certainty. Further, when the observer moves and the face illuminance changes, Immediately, the light source light quantity control section functions to drive and control the output of the illuminating means to set the face illuminance of the observer to the state before the movement. For this reason, a fixed value can be used as the threshold value for binarization, so that the calculation of the threshold value for binarization becomes unnecessary, and the calculation processing time of the calculation processing system when determining the observation position is accordingly reduced. There is an advantage that it can be shortened and the configuration can be simplified.

Here, the above-mentioned light source light amount control unit is provided with a light source illuminance calculation circuit for calculating the light source illuminance of the illuminating means based on image information output from the pupil image enhancement unit, and an output from the light source illuminance calculation circuit. A predetermined calculation based on the light source illuminance information to be performed, and a light source light amount that variably sets the light source light amount of the above-described illumination means in real time so that the output level of the image information output from the pupil image enhancing unit is always constant. It is preferable to use a setting circuit.

In this way, the output of the illumination means can be quickly and smoothly variably set in accordance with the movement of the observer, and even when the observer moves, the binarized information of the pupil portion can be more reliably obtained. Obtainable.

The above-described light source illuminance calculation circuit calculates the image standard deviation of the image information output from the pupil image enhancing section and calculates the light source illuminance of the illumination means based on the calculated standard deviation. A configuration having a function may be adopted.

According to a fourteenth aspect of the present invention, in the observer observation position detecting apparatus according to the eighth, ninth, tenth, eleventh, twelfth, or thirteenth aspect, the pupil image enhancing section is stored in the image memory. A configuration having an image differentiation processing function of differentiating the selected image is adopted as a common basic configuration.

Therefore, according to the invention of claim 14, in the above-mentioned claim 8, 9, 10, 11, 12 or 13
In addition to having the same function as the invention described in the above, furthermore, the pupil portion of the face image is more clearly emphasized by the image differentiation processing, and in this regard, the threshold value for binarization is specified and the threshold value based on this is determined. The valuation operation can be performed very accurately and quickly.

According to each of the inventions described in claims 15 and 16, the above-described claim 8, 9, 10, 11, 12, 13, or 14 is provided.
In the observer's observation position detecting device described in (1), a reflection marker is attached to the face of the observer in advance, and the presence or absence of the reflection marker is determined in the image memory based on the face image information stored in the image memory. A marker judgment unit is also provided.

In the output stage of the reflection marker determination unit, when the reflection marker determination unit determines that there is a reflection marker, the reflection marker determination unit operates in preference to the pupil image enhancement unit and stores the marker image in the image memory. A marker image binarizing means for binarizing is provided, and a center of gravity of the image information of the face is calculated from the calculation result of the marker image binarizing means, and a marker position calculation for specifying this as an observation position of an observer is provided. The configuration in which the function is provided in the above-described observation position calculation unit is adopted as a common basic configuration.

Therefore, in each of the inventions according to claims 15 and 16, the above-mentioned claims 8, 9, 10, 11, 1
In addition to having the same functions as those of the invention described in 2, 13, or 14, furthermore, even if the observer is wearing glasses or the like, even if there is reflected light from the glasses or the like, reflection during and after movement There is an advantage that the marker can be reliably captured, and thus the observation position can be reliably captured and specified in real time.

Here, the reflection luminance of the above-mentioned reflection marker is set in advance to be larger than the reflection luminance of the pupil reflection light of the observer, and the threshold for binarization in the above-mentioned reflection marker determination means is set to the above-mentioned reflection marker. The predetermined threshold value may be smaller than the reflection luminance and larger than the luminance of the pupil reflection light.

Thus, in each of the above-mentioned claims, the pupil portion is enhanced in the infrared image of the observer face, and the binarization threshold value for pupil extraction is determined from the standard deviation of the enhanced infrared image of the observer face. By variably controlling (or the amount of light of the illumination means) in real time, pupil information of the observer can be stably extracted, thereby enabling accurate non-contact observation position detection.

[0044]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS (First Embodiment) A first embodiment of the present invention will be described below with reference to FIGS. First, an observation position detection device according to the present embodiment includes an imaging system 1 that images the face of an observer 100 as illustrated in FIG. 1, and performs face processing on the face information imaged by the imaging system 1 to perform image processing on the observer 100. And an image processing system 11 as image processing means for calculating an observation position.

The imaging system 1 has a CC as an imaging means.
It comprises a D camera 2 and an infrared illuminating means 3 as an illuminating means for illuminating the face 101 of the observer 100. Here, the infrared illuminating means 3 includes an infrared output light source 3A for outputting an infrared ray for illumination, and an illumination direction of the infrared ray for illumination output from the infrared output source 3A substantially corresponding to the optical axis incident on the CCD camera 2 described above. And a half mirror 2B set on the same line. That is, the optical axis of the CCD camera 2 and the optical axis of the infrared output source 3A are combined using the half mirror 2B so as to be coaxial with each other as shown in FIG.

When the face 101 of the observer 100 is imaged using the imaging system 1, the human retina has a characteristic of reflecting infrared light more strongly than other face parts. A face image with a particularly bright pupil 101A is obtained.

In this embodiment, a light emitting diode having a wavelength of 850 [nm] is used as the infrared output light source 3A, which is the light source of the infrared illuminating means 3. In addition, the CCD camera 2 as an imaging unit has a built-in wavelength filter that blocks light having a wavelength of 760 [nm] or less.
The CCD camera 2 is not affected by external light such as a room light during operation.

The CCD camera 2 outputs a monochrome analog video signal of the NTSC standard. The CCD camera 2 is driven at a frequency of 15.75 [kHz] in the horizontal direction (horizontal direction) and at a frequency of 60 [Hz] in the vertical direction (vertical direction). Thereby, the face 101 of the observer 100 is imaged by the CCD camera 2 for each field, that is, 60 times per second under the illumination of the infrared illuminating means 3.

On the other hand, an image processing system 11 as an image processing means is provided with an image memory 12 for storing face image information of the observer 100 obtained by the above-described imaging system 1.
A pupil image enhancing unit 13 for performing a predetermined operation for enhancing the pupil portion 101a of the observer in the image information stored in the image memory 12, and pupil image data output from the pupil image enhancing unit 13 A binarization processing unit 14 for performing a binarization process based on a predetermined binarization threshold, an observation position calculation unit 15 for calculating an observer's observation position based on a calculation result of the binarization processing unit 14, Is configured. The input stage of the binarization processing unit 14 determines a binarization threshold for the pupil image in real time based on the image information output from the pupil image enhancement unit 13 described above. A binarization threshold setting unit 16 for sending the data to 14 is also provided.

In this embodiment, the binarization threshold setting unit 16 calculates the image standard deviation of the image information output from the pupil image enhancing means 13 in advance and calculates the light source illuminance based on the standard deviation. A calculation circuit 16A and a binarization threshold setting circuit 16 for setting a binarization threshold based on the image standard deviation calculated from the light source illuminance calculation circuit 16A
B.

Here, in the above-mentioned image memory 12, C
Facial image 10 of observer 100 captured by CD camera 2
1 has a resolution of 640 × 240 pixels for each field,
It is stored as an 8-bit (256 gradation) monochrome digital image. Also, CCD camera 2
Is 2 [m], and the size of one pixel of the field image stored in the image memory 12 is 1 on the face 101 of the observer 100 in the horizontal direction.
The lens magnification is adjusted to correspond to [mm] and 2 [mm] in the vertical direction.

In the pupil image enhancing section 13, an image differentiation operation is performed on the face image of the observer 100 stored in the image memory 12 described above (execution of the differentiation processing function of the pupil image enhancing section 13). By this image differentiation operation, the pupil portions 101a of both eyes of the observer 100 are particularly emphasized with respect to other portions of the face 101. After that, the light source illuminance calculation circuit 16A calculates the standard deviation of the image after the image differentiation operation processing, and subsequently sets the binarization threshold in the binarization threshold setting circuit 16B based on the standard deviation. After setting the binarization threshold, the above-described binarization processing unit 14 binarizes and extracts only the pupil portions 101a of both eyes of the observer 100 as white pixels. Then, the binarized and extracted white pixels (pupil part 1) in the binarization processing unit 14
01a), the observation position calculation unit 15 sets the observer 10
An observation position of 0 is calculated.

Here, all processing in the image processing system 11 is performed by, for example, 1/60 second by a predetermined computer, and the observation position of the observer 100 is synchronized with the vertical synchronization signal of the CCD camera 2. Image processing system 11
Is output 60 times per second.

Next, the operation principle of each component of the first embodiment will be described in further detail according to the operation procedure. FIG.
FIG. 1 shows an example of an image captured by the imaging system 1.
Here, FIG. 2A shows an image of the observer 100 obtained by the CCD camera 2 in FIG. 1, and FIG. 2B shows a horizontal luminance distribution of an image between line segments AB in FIG. Show. As shown in FIG. 2B, the pupil portion 101a
Is imaged brighter than the face portion 101 (face image capturing step).

FIG. 3 shows an operation example of the pupil image enhancing section 13 in the image processing system. The face image obtained by the CCD camera 101 in this face imaging step is sequentially recorded in the image memory 12 as an image I (x, y), and thereafter, based on the image information of the face stored in the image memory 12, the observer Is calculated (observation position calculation step). Here, x and y represent the horizontal and vertical coordinate positions of the captured image. FIG. 3A shows the horizontal luminance distribution of the image of the observer recorded in the image memory 12 and corresponds to the one shown in FIG.

The pupil image enhancing section 13 performs an image differentiation operation on the image I (x, y) recorded in the image memory 12 described above. This image differentiation operation is an example of an operation that particularly emphasizes the pupil portion 101a in the image of the observer (first information processing step).

This image differentiation operation is, as shown in FIG. 3 (a), a second differentiation process based on a difference between pixels separated by the number of pixels dx corresponding to the pupil diameter D, 2 × I (x, y) − I (x + dx, y) −I (x−d
x, y). By performing this image differentiation operation,
As shown in FIG. 3B, only the pupil portion 101a can be emphasized with respect to the face portion 101.

FIGS. 4A and 4B show the calculation range of the standard deviation in the light source illuminance calculation unit 16A described above. The light source illuminance calculation unit 16A measures the illuminance of the infrared illumination of the observer's face 101 by calculating the standard deviation σ of the image after the differential operation processing. The calculation range of the standard deviation σ is shown in FIG.
(A) is an area (calculation range S) of 60 [mm] × 150 [mm] in the vertical direction including the pupil portions 101a of both eyes of the observer, and is 30 pixels × 150 pixels (4500) in the image memory 12. Pixel).

The calculation range S of the standard deviation σ is, as shown in FIG. 4B, the immediately preceding field (1
/ 60 seconds ago) and the observation position F11 of the field just before (2/60 seconds ago) and the movement vector V15 calculated from them are constantly updated.

FIGS. 5A and 5B are diagrams showing histograms of the image after the differential operation processing. Among them, FIG.
Then, the distribution of the pupil component 201a is distributed as a value having a large absolute value on the positive side and the negative side, and the distribution of the component 202 other than the pupil is distributed about three times the standard deviation σ around 0. For this reason, the binarization threshold setting unit 16 sets the binarization threshold to:
For example, by setting to four times the standard deviation, the pupil component 201a on the positive side and the component 202 other than the pupil can be separated.

As shown in FIG. 5B, even when the illuminance of the infrared illumination is small, the components other than the pupil 20
The distribution of 2 is distributed about three times the standard deviation σ with 0 as the center. If the binarization threshold is set to four times the standard deviation, only the pupil image can be extracted, and it is possible to cope with the illuminance fluctuation of the infrared illumination.

When calculating the standard deviation σ, the pupil portion 20
Although the luminance of 1a is also included, the area of the pupil portion 201a of both eyes is about 39 [mm ² ] (20 pixels) when calculated using the average pupil diameter (5 [mm]) of a human, and the calculation range (450
Since it is only 1/225 of (0 pixel), the calculated standard deviation σ can be considered to be equivalent to the standard deviation σ of only components other than the pupil.

After calculating the standard deviation σ, the binarization threshold setting unit 16
In B, a value four times the standard deviation σ, that is, 4σ, is set as a binarization threshold (a binarization threshold setting step). As described above, the threshold value for binarization in the binarization threshold value setting step calculates the image standard deviation of the face image subjected to the operation of enhancing the pupil portion of the face image, and based on the calculated image standard deviation. Is determined.

FIGS. 6A and 6B are diagrams for explaining the operation of the above-described binarization processing section 14. FIG. The binarization processing unit 14 performs an image binarization process on the differentially-calculated image processed by the pupil image enhancement unit 13 (second information processing step).

The binarization threshold in the binarization processing unit 14 may be a value specified in advance, but in this embodiment, the binarization threshold setting unit 16 sets the binarization threshold for each time-series image. A value four times the standard deviation σ is used.

Here, when the illuminance of the infrared illumination is high, the threshold value Th (= 4σ) is set high as shown in FIG. 6A, and when the illuminance of the infrared illumination is low, it is shown in FIG. 6B. As described above, the threshold value Th (= 4σ) is set low. Therefore, even if the illuminance of the infrared illumination fluctuates, the pupil portion 101a alone can always be obtained as the pupil extraction image (extracted pupil portion) 105 sufficiently in response to the fluctuation.

FIG. 7 is a diagram for explaining the operation of the observation position calculation unit 15 described above. In FIG. 7, in order to detect the observation position of the observer, for example, an observer image 10
7, a position corresponding to the midpoint of the pupil portion 101a of both eyes may be calculated. In the present embodiment, the observation position 107a is calculated by calculating the midpoint between the horizontal minimum coordinate 112 and the horizontal maximum coordinate 113 of the pupil extraction image 105 extracted in the pupil extraction image 107 (third information processing step). .

Then, by operating each of the systems 1 and 11 having the above contents, the face 101 of the observer 100 was photographed, and the observer's observation position was detected.

In the chart shown in FIG. 8, (a) and (b) show that the illuminance of the infrared illumination is 60 [mW / m ² ] and 20 [mW / m ² ], respectively.
It is a figure which shows the brightness distribution of the observer's face image image | photographed in the case of [mW / m < ² >]. Here, (a) in FIG.
The image shown in (b) includes a pupil portion 101a and a face portion 1
01 and 01 are respectively imaged. Then, for the image I (x, y) shown in (a) and (b) in FIG. 8, a second differential operation is performed to obtain a difference between pixels separated by 5 pixels corresponding to the diameter of the pupil portion 101a. , 2 × I (x, y) -I (x + 5, y) -I (x-5,
y) was performed.

The results after the image differentiation operation are shown in FIG.
(D). By this image differentiation operation, the pupil portion 1
01a can be emphasized. Here, the standard deviations σ in the images (c) and (d) in FIG. 8 are “8” and “3”, respectively, and “3” which is four times the standard deviation σ
(C), (c) in FIG.
When the binarization processing is performed on each of the images shown in FIG.
(E) and (f), the pupil portion 10 in FIGS. 8 (a) and (b) is always irrespective of the illuminance of the infrared illumination.
Only 1a was binarized and extracted as the pupil extraction image 105.

FIG. 9 shows the observer 10 in the above embodiment.
FIG. 9 is an explanatory diagram showing a result of detecting an observation position of 0. In the present embodiment, the horizontal minimum coordinate 112 and the horizontal maximum coordinate 113 of the pupil extraction image 105 are respectively detected 60 times per second, and the observation position 107a can be stably detected 60 times per second from these middle points.

Next, an example in which the observation position detecting device in the above embodiment is applied to a viewpoint tracking type stereoscopic display device will be described with reference to FIG.

In the viewpoint-following three-dimensional display device shown in FIG.
A stereoscopic image is observed through 05. Reference numeral 501 indicates the mounting location of the observation position detection device in FIG. 1 described above. The observation position detection device 501 images the face of the observer from 20 degrees below (with reference to the horizontal) in order to prevent the pupil of the observer 100 from being blocked by the eyelids. An optical system control unit 506 is connected to the observation position detection device 501.
The optical system control unit 506 controls the viewpoint following optical system mechanism unit 503 in the liquid crystal projector 504 according to the observation position of the observer 100.

When the above-described observation position detecting device is applied to a viewpoint-following type three-dimensional display device, the observation position can be accurately detected in a non-contact manner, and a three-dimensional display device with less burden on the observer can be obtained. Was completed.

(Second Embodiment) FIG. 11 shows a second embodiment of the present invention. In the second embodiment, as in the case of the above-described first embodiment, the imaging system 1
An image processing system 21 is provided that inputs face image information obtained by the imaging system 1 and performs predetermined signal processing to detect an observation position of an observer. The second embodiment is characterized in that a light source light amount control unit 20 for controlling the output of the infrared output light source 3A is provided in addition to the infrared output light source 3A included in the imaging system 1.

Further, in the second embodiment, the image processing system 21 is provided with a light source light quantity setting section 26 instead of the binarization threshold value setting section 16 in the first embodiment, and the light source light quantity The present embodiment is characterized in that a configuration in which the light source light amount control unit 20 described above is driven based on output information from the setting unit 26 is employed.

Here, the light source light amount setting unit 26 calculates the image standard deviation based on the pupil image data output from the pupil image enhancing unit 13 in the first embodiment to calculate the light source illuminance. A calculation circuit 26A (functioning equivalently to the light source illuminance calculation circuit 16A in the first embodiment); and a light source light amount setting circuit 26B for setting a control amount of the light source light amount based on output information from the light source illuminance calculation circuit 26A. It consists of. Other configurations are the same as those of the above-described first embodiment (FIG. 1), as shown in FIG.

Next, the operation of the second embodiment will be described. First, also in the second embodiment, CC
The image of the observer 100 obtained by the D camera 2 is the same as the image shown in FIG. 2 as in the case of the above-described first embodiment.

The image obtained by the CCD camera 2 is recorded in the image memory 12 as an image I (x, y), as in the case of the first embodiment. here,
x and y represent the horizontal and vertical coordinate positions of the captured image, respectively. The observer 100 recorded in the image memory 12
The horizontal luminance distribution of the image is similar to that of the first embodiment described above (see FIG. 3).

The image I recorded in the image memory 12
The pupil image enhancement unit 112 performs an image differentiation operation on (x, y). The image differentiation operation is, as shown in FIG. 3A, a secondary differentiation process based on a difference between pixels separated by the number of pixels dx corresponding to the pupil diameter D, 2 × I (x, y) −I (x + dx, y). ) -I (xd
x, y). By performing this image differentiation operation, only the pupil portion 101a can be emphasized with respect to the face portion 100 as shown in FIG.

The light source illuminance calculation unit 26A measures the illuminance of the infrared illumination on the observer's face by calculating the standard deviation σ of the image after the differential operation as in the first embodiment. The light source light quantity setting circuit 26B includes a light source illuminance calculation circuit 26.
A control signal (predetermined control amount information) corresponding to the value of the standard deviation σ calculated by A is sent to the light source light amount control unit 20 to control the light amount output from the infrared output light source 3A (light source light amount control step).

In the second embodiment, when the illuminance on the observer's face is 60 [mW / m ² ], the standard deviation σ calculated by the light source illuminance calculation circuit 26A is adjusted to be “8”. It is. Therefore, if the light source light amount is controlled so that the standard deviation σ calculated by the light source illuminance calculation circuit 26A is always “8”, the illuminance on the observer's face is always 60 [m
W / m ² ]. That is, the control amount of the light source light amount of the above-mentioned illuminating means (infrared illuminating means) 3 which is a control reference in the light source light amount controlling step is determined by the
The standard deviation of the image subjected to the operation for emphasizing 01a is calculated and determined based on this.

Even when the observer 100 approaches the CCD camera 2, the illuminance of the infrared output light source 3A on the face 101 of the observer 100 is always maintained at 60 [mW / m ² ]. At this point, the observer 100 is
It is also possible to secure safety when the user looks into the camera 2 or the like. After adjusting the light source light amount by the light source light amount control unit 20, the binarization processing unit 14 sets "32" which is four times the standard deviation σ (= 8) under 60 [mW / m ² ] illumination.
Is used as a threshold to perform a binarization process.

By operating the apparatus having the above contents, the face 101 of the observer 100 is photographed.
0 observation positions were detected.

As in the first embodiment, the luminance distribution of an image obtained by photographing the observer's face has the same result as (1) in the chart of FIG. 8, and corresponds to the diameter of the pupil portion 101a. A second derivative operation that calculates a difference between pixels separated by 5 pixels, 2 × I (x, y) −I (x + 5, y) −I (x−5,
By performing y), the same result as (c) in FIG. 8 could be obtained.

As described above, in the second embodiment, since the illuminance on the face 101 of the observer 100 is constantly controlled to be constant, the luminance level is the same as that shown in FIG. Can always be maintained, and a pupil extraction image 205 as shown in FIG. 8E can be stably obtained. In the second embodiment, the horizontal minimum coordinates and the horizontal maximum coordinates of the pupil extraction image are detected 60 times per second, as in the first embodiment described above. As a result, the observation position 107a could be detected 60 times per second.

As described above, even in the present embodiment, it is possible to obtain substantially the same operation and effect as in the first embodiment described above. In particular, the standard deviation σ calculated by the light source illuminance calculation circuit 26A is, for example, always equal to “1”. 8 ", the illuminance on the observer's face can be constantly maintained at, for example, 60 [mW / m ² ]. Therefore, the threshold value for binarization is always a constant value. Therefore, the calculation of the threshold value becomes unnecessary, and the image processing time of the image processing system can be shortened accordingly.

(Third Embodiment) Next, a third embodiment will be described with reference to FIG. In the third embodiment, as in the case of the first embodiment described above, the imaging system 1 and face image information obtained by the imaging system 1 are input and subjected to predetermined signal processing to observe the image by the observer. And an image processing system 31 for detecting a position. In the third embodiment shown in FIG. 12, a reflection marker for detecting a face position is attached to the face 101 of the observer 100 in advance, and this is used as an imaging system equivalent to the imaging system in the first embodiment. 1 so that the observer 10
A method is provided for identifying the 0 face position. Furthermore, even in the case of the observer 100 without the reflection marker, the face position is determined by the image processing system 21 having the same function as the image processing system 11 disclosed in the above-described first embodiment. It is calculated and specified.

Hereinafter, this will be described in more detail. The reflection marker is attached by the observer when it is difficult to detect the observation position by pupil extraction due to a noise component caused by reflected light of the wearable object (glasses, earrings, hairpins, etc.) of the observer. .

Here, as the reflection marker, one having a reflection surface such as spherical glass particles (glass beads) or a corner cube is used. These reflection markers reflect infrared rays emitted from the infrared illuminating means 3 in a direction opposite to the incident direction. This type of reflective marker is attached to the observer's eyebrow position (middle point of both eyes) and the middle point of both eyes in the frame of the spectacles by using an adhesive tape or metal fittings.

Further, as this reflection marker, for example, 1
[Cm] x 1 [cm] (or diameter 1
[Cm]), the size of which is such that the reflected image from the reflective marker is always brightly captured as an image larger than the reflected light from the observer's pupil reflected light or the wearing object. If there is, it is not particularly limited.

The reflection marker is imaged by the imaging system 1 together with the face image of the observer 100. The image information is processed by the image processing system 31. The image processing system 31 determines the presence or absence of a reflection marker from the image memory 22 that stores the face image information of the observer 100 obtained by the above-described imaging system 1 and the face image information stored in the image memory 22. A reflection marker determination unit 32A that detects the marker information; and a first binarization processing unit 3 as a marker image binarization unit that binarizes the marker information detected by the reflection marker determination unit 32A.
2B and the center of gravity of the image is calculated based on the marker information binarized by the first binarization processing unit 32B, and the observation position of the observer 100 is calculated based on the image centroid.
And an observation position calculation unit 35 that outputs the position information as 00.

Further, the image processing system 31 includes a pupil image emphasizing section 23 for performing a predetermined operation for emphasizing the pupil portion 101a of the observer in the face image information stored in the image memory 22 described above. A second binarization processing unit 24 that binarizes the pupil image data output from the pupil image enhancement unit 23 based on a predetermined binarization threshold, and an operation of the second binarization processing unit 24 The above-described observation position calculation unit 35 that calculates the observation position of the observer based on the result is provided.

Here, as described above, the observation position calculation unit 35 calculates the image center of gravity based on the marker information binarized by the first binarization processing unit 32, and calculates the observation position of the observer 100. And a second observation position calculation function for calculating the observation position of the observer based on the calculation result of the second binarization processing unit 24.

The image emphasizing section 23 operates when the information e indicating that there is no reflective marker is input from the reflective marker determining section 32A, and the information f indicating that the reflective marker is present is input from the reflective marker determining section 32A. If it does, it will not work. Then, the above-described observation position calculation unit 3
In 5, the first observation position calculation function is configured to operate prior to the second observation position calculation function.

The input stage of the second binarization processing section 24 further determines a binarization threshold for the pupil image in real time based on the image information output from the pupil image enhancement section 23 described above. Then, a binarization threshold value setting unit 36 for sending the data to the second binarization processing unit 24 is provided.

In the third embodiment, the binarization threshold setting unit 36 calculates the image standard deviation of the image information output from the pupil image enhancing means 23 in advance and calculates the light source illuminance based on this. A light source illuminance calculation circuit 16A having an illuminance calculation function to perform the threshold value setting, and a binarization threshold setting circuit 16B for setting a binarization threshold based on the image standard deviation calculated from the light source illuminance calculation circuit 16A. Have been. That is, the binarization threshold value setting unit 36
It has exactly the same configuration and the same function as the binarization threshold setting unit 16 in the embodiment.

For this reason, as in the case of the above-described first embodiment, the binarization threshold value setting circuit 16B responds to the change in the illuminance of the above-described face image even if the illuminance of the face image changes. A predetermined threshold can be set, and in such a point,
There is an advantage that the arithmetic processing at the time of binarization can be performed accurately and smoothly even if the illuminance of the face image changes.

Next, the components of the third embodiment and the overall operation will be described.

An image obtained by the CCD camera 101 is recorded in the image memory 22 as an image I (x, y). Here, x and y represent the horizontal and vertical coordinate positions of the captured image. For the image I (x, y) recorded in the image memory 22, the reflection marker determination unit 32A determines whether or not the observer 100 is wearing the reflection marker (marker determination step).

As described above, the reflection marker is attached to the observer's eyebrow position (middle point of both eyes) or the midpoint position of both eyes in the frame of the spectacles by using an adhesive tape or metal fittings, and the reflection surface is, for example, The size is 1 [cm] × 1 [cm], and therefore, it is always brightly picked up as an image larger than the pupil reflected light of the observer or a reflected image of the wearing object.

Therefore, the image stored in the image memory 22 is subjected to the binarization processing, and then the center of gravity of the image is calculated, whereby the center position of the reflection marker is quickly and reliably detected as the observation position. .

That is, the above-described reflection marker determination unit 32A
Determines the presence or absence of a reflective marker based on a threshold Th1 that is higher than the luminance of the pupil reflected light of the observer in the captured image and lower than the reflected light luminance of the reflective marker. If the reflection marker determination unit 32A determines that there is a reflection marker, the first binarization processing unit 32B executes the image memory 22
Is binarized with the threshold Th1 (binarization processing step), and the result is stored in the observation position calculation unit 3.
5, the observation position calculation unit 35 performs an image centroid operation on the image after the binarization processing, and outputs the image centroid position as an observation position (observation position calculation step).

On the other hand, when the reflection marker determination section 32A determines that there is no reflection marker, the pupil image enhancement section 23 which has input such information immediately starts operating. At the same time, the light source illuminance calculation circuit 16A and the binarization threshold setting circuit 1
6B, the second binarization processing unit 24, and the observation position calculation unit 35 operate, and perform the same operations as in the first embodiment.

As described above, even in the third embodiment, the horizontal minimum coordinate and the horizontal maximum coordinate of the pupil extraction image are detected 60 times per second, as in the case of the first embodiment. The observation position was stably detected 60 times per second from these midpoints. When the observer wears the reflective marker, the presence or absence of the reflective marker is automatically determined, and the position of the reflective marker is detected 60 times per second from the binarized reflected marker image. did it.

Further, also when the observation position detecting device of the present embodiment is applied to a viewpoint-following stereoscopic display device, the observation position can be accurately detected in a non-contact manner as in the first embodiment. This makes it possible to provide a three-dimensional display device with a small burden on the observer.

(Fourth Embodiment) FIG. 13 shows a fourth embodiment of the present invention. In the fourth embodiment, as in the case of the above-described third embodiment, the imaging system 1
An image processing system 41 is provided which receives face image information obtained by the imaging system 1 and performs predetermined signal processing to detect an observation position of an observer. The fourth embodiment is characterized in that a light source light amount control unit 20 for controlling the output of the infrared output light source 3A is provided in addition to the infrared output light source 3A included in the imaging system 1.

Further, in the fourth embodiment, the image processing system 41 is provided with a light source light quantity setting section 46 instead of the binarization threshold value setting section 26 in the third embodiment, and the light source light quantity setting section 46 is provided. The present embodiment is characterized in that a configuration in which the light source light amount control unit 20 described above is driven based on output information from the setting unit 46 is employed.

Here, the light source light quantity setting section 46 calculates an image standard deviation based on pupil image data output from a pupil image enhancing section 43 described later to calculate a light source illuminance. And a light source light quantity setting circuit 26B for setting a control amount of the light source light quantity based on output information from the light source illuminance calculation circuit 26A.

Further, in the fourth embodiment, the second binarization processing unit 44 differs from the third embodiment (FIG. 12) in that the The pupil image enhancing unit 43, the light source illuminance calculation circuit 26A, and the light source light amount setting circuit 26B are operated independently of the determination of the presence or absence of the reflection marker by the reflection marker determination unit 32A. Thus, the illuminance on the observer's face is constantly set to, for example, 60 [mW /
m ² ] is controlled. Other configurations are the same as those of the third embodiment (FIG. 12) as shown in FIG.

Hereinafter, this will be described in detail. First, as the imaging system 1, the imaging system in the third embodiment described above is used as it is. And this imaging system 1
Image information captured by the image processing system 41
Processed by

The image processing system 41 includes an image memory 22 for storing face image information of the observer 100 obtained by the above-described imaging system 1, and an image memory 22 for storing the image information.
Marker determining unit 32A that determines the presence or absence of a reflective marker from the face image information stored in the memory and detects the marker information
A first binarization processing unit 32B as a marker image binarization processing unit for binarizing the marker information detected by the reflection marker determination unit 32A; and a first binarization processing unit 32B And an observation position calculation unit 35 which calculates the image center of gravity based on the binarized marker information to calculate the observation position of the observer 100 and outputs it as position information of the observer 100. The detection of the reflection marker and the signal processing thereof are configured in the same manner as in the third embodiment.

Further, the image processing system 41 includes a pupil image emphasizing section 43 for performing a predetermined operation for emphasizing the pupil portion 101a of the observer in the face image information stored in the image memory 22 described above. A second binarization processing unit 44 that binarizes the pupil image data output from the pupil image enhancement unit 43 based on a predetermined binarization threshold, and an operation of the second binarization processing unit 44 The above-described observation position calculation unit 45 that calculates the observation position of the observer based on the result is provided.

Here, the observation position calculation unit 45 calculates the center of gravity of the image based on the marker information binarized by the first binarization processing unit 32B as described above, and determines the observation position of the observer 100. It has a first observation position calculation function for calculating and a second observation position calculation function for calculating the observation position of the observer based on the calculation result of the second binarization processing unit 44. Then, the observation position calculation unit 45 is configured so that the first observation position calculation function operates prior to the second observation position calculation function.

Further, the above-described pupil image enhancing section 43 is configured to operate at a predetermined timing regardless of the result of the determination of the presence or absence of a reflective marker determined by the reflective marker determining section 32A.

Further, in the fourth embodiment, the face 10 of the observer 100 is
1 is controlled so that the illuminance at 1 is always constant irrespective of its position.
In, the threshold value of the binarization process is set to a predetermined threshold value in advance, and the binarization process is executed based on the threshold value. That is, in the fourth embodiment, the first
Since the binarization threshold setting unit 26 required in the third embodiment is not provided, the calculation speed is increased as compared with the third embodiment.

The other structures, functions and effects are almost the same as those of the third embodiment (FIG. 12).

In the fourth embodiment (FIG. 13) as well, the horizontal minimum coordinate and the horizontal maximum coordinate of the pupil extraction image are detected 60 times per second as in the first embodiment described above. In this state, the observation position could be detected 60 times per second. In addition, when the observer wears the reflective marker, the reflective marker is automatically determined to be mounted, and the position of the reflective marker can be detected 60 times per second from the binarized reflective marker image 60 times per second. .

When the observation position detecting device according to the present embodiment is applied to a viewpoint-following three-dimensional display device, the observation position can be accurately detected in a non-contact manner as in the above-described embodiments. A three-dimensional display device with less burden on the user can be obtained.

As described above, in the first to fourth embodiments, the position of the illuminating device may not be exactly coaxial, but may be arranged near the optical axis as long as the pupil of the observer can be imaged brightly. Also, for the purpose of reducing noise due to reflected light from specular objects such as glasses,
A polarizing plate, a quarter-wave plate, or the like may be inserted on the optical axis of the illumination device or the imaging device.

As the imaging system, in addition to the NTSC standard, various systems such as the PAL standard and the SECAM standard may be used, or a system having higher horizontal and vertical frequencies may be used. The image memory may have various resolutions.

Pupil image enhancement sections 13, 23, 43, binarization threshold setting sections 16, 36, light source light quantity setting sections 26, 46, binarization processing sections 14, 24, 32B, 44, observation position calculation section 15, The arithmetic processing in the units 35 and 45 and the reflection marker determination unit 32A and the like may use a dedicated digital logic circuit or an analog arithmetic circuit using an operational amplifier other than using an electronic computer.

Further, the image differential operation in the pupil image enhancement units 13, 23, 43 is given by 2 × I (x, y) −I (x, y + dx) −I (x, y−
dx), may also be performed in the vertical direction, such as
In addition, the image differential operation in both the horizontal and vertical directions, 4 × I (x, y) −I (x + dx, y) −I (x−d
(x, y) -I (x, y + dx) -I (x, y-dx).

The pupil image enhancing sections 13, 23, 43
The image differential operation in is a first order such as I (x-dx / 2, y) -I (x + dx / 2, y), I (x, y-dx / 2) -I (x, y + dx / 2) Differential operation may be used. This pupil image enhancement unit 1
The pupil enhancement unit calculations in 3, 23, and 43 may use template matching using a pupil-shaped template.

The calculation of the light source illuminance in the light source illuminance calculation unit 16A can be performed by using the sum of the absolute values of the luminances of the output images described above, in addition to calculating the standard deviation of the output images in the pupil image enhancement units 13, 23, and 43. Good. When the image differentiation operation is performed only in the horizontal direction, the image memories 12 and 2
A pupil image may be extracted for each scanning line by using a line memory 2 as a line memory, or a method of scanning with a galvanomirror or the like in the vertical direction using an imaging device (CCD sensor) as a line sensor may be used.

[0126]

As described above, according to the present invention, the pupil portion is enhanced in the infrared image of the observer's face, and the binarization threshold value (or the pupil extraction threshold value) is extracted from the standard deviation of the enhanced infrared image of the observer's face. By controlling the amount of light of the illuminating means), the observer moving in a wide area has a non-uniform spatial illuminance distribution of the infrared light source and a change in the distance between the observer and the infrared light source. Even if there is a change or the like, only the pupil portion of the observer can be constantly and stably extracted, and the predetermined arithmetic processing can be performed quickly and with high accuracy. And a device for detecting an observation position of an observer, which is superior to the related art, that can detect and output an image in real time. Furthermore, according to the present invention, when a predetermined reflection marker is attached to the face of the observer in advance, even if there are many noise components other than the pupil portion, the position of the reflection marker is easily detected by detecting the position of the reflection marker. The observation position of the observer can be accurately detected by the image processing.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

2A and 2B are diagrams illustrating images captured by the imaging system in FIG. 1; FIG. 2A is an explanatory diagram illustrating a face image of an observer; FIG. 2B is a diagram illustrating A- in FIG. 2A; It is a diagram showing detection information along the B line.

3A and 3B are explanatory diagrams showing the operation principle of a pupil image enhancing unit disclosed in FIG. 1, wherein FIG. 3A is a diagram showing a specific example of FIG. 2B, and FIG. It is a diagram showing an example in the case of performing differential operation processing of (a).

4A and 4B are diagrams illustrating a calculation range of a standard deviation in a light source illuminance calculation unit. FIG. 4A is an explanatory diagram illustrating an example of a range including two pupil portions, and FIG. It is an explanatory view showing that the calculation range of the standard deviation is the same even in the case.

5A and 5B are diagrams showing histograms of images after differential operation processing. FIG. 5A is a diagram showing a histogram when the illuminance is large, and FIG. 5B is a diagram showing a histogram when the illuminance is small. It is.

FIG. 6 is a diagram for explaining the binarization processing of the binarization processing unit in the image processing apparatus according to the present invention. FIG. 6A is an explanatory diagram showing the binarization process when the illuminance is large. FIG. 6B is an explanatory diagram showing a binarization process when the illuminance is small.

FIG. 7 is an explanatory diagram for explaining a function of an observation position calculation unit in the observation position detection device of the present invention.

FIG. 8 is a chart showing the state of image processing of an image of the face of the observer, in which (a), (c) and (e) show cases where the illuminance is large, and (b) (d) (f) ) Shows the case where the illuminance is small.

FIG. 9 is an explanatory diagram showing a result of detecting a change in an observation position of an observer using the observation position detection device of the present invention.

FIG. 10 is an explanatory diagram showing an overall configuration of a viewpoint tracking type stereoscopic display device using the observation position detection device of the present invention.

FIG. 11 is a block diagram showing a second embodiment of the observation position detection device of the present invention.

FIG. 12 is a block diagram showing a third embodiment of the observation position detection device of the present invention.

FIG. 13 is a block diagram showing a fourth embodiment of the observation position detection device of the present invention.

FIG. 14 is a block diagram showing an example of a pupil extracting means in a conventional example (awake state detecting device).

[Explanation of symbols]

Reference Signs List 1 imaging system 2 CCD camera as imaging means 3 infrared illuminating means as illuminating means 3A infrared output light source 11, 21, 31, 41 image processing system as image processing means 12, 22 image memory 13, 23 pupil image enhancing section 14 Binarization processing section 15, 35, 45 Observation position calculation section 16 Binarization threshold setting section 16A, 26A Light source illuminance calculation circuit 16B Binarization threshold setting circuit 20 Light source light quantity control section 24, 44 Second binarization processing Units 26, 46 Light source light amount setting unit 26B Light source light amount setting circuit 32A Reflection marker determination unit 32B First binarization processing unit as marker image binarization unit 100 Observer 101 Face 101A Pupil 101a Pupil portion 105, 205 Pupil extraction Image 107a 111 Observation position 201a Pupil component

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-8-297019 (JP, A) JP-A-8-271221 (JP, A) JP-A-6-21165 (JP, A) JP-A-2- 133885 (JP, A) JP-A-64-82181 (JP, A) JP-A-64-73330 (JP, A) JP-A-2000-312671 (JP, A) JP-A-7-13105 (JP, A) 63-31311 (JP, U) (58) Fields investigated (Int. Cl. ⁷ , DB name) G01B 11/00-11/30 G06T 7/00

Claims (16)

(57) [Claims] 1. A face imaging step of imaging an observer's face by an imaging means while illuminating a face of the observer with a predetermined illumination means, and temporarily storing image information of the face imaged in the face imaging step. The image information is stored in the image memory and the observer's own
In the viewer's viewing position detection method and a viewing position calculation step of calculating the position, the observation position calculation step, the performing emphasizing operation image information of the pupil portion of the image information stored in said image memory (1) an information processing step, a second information processing step of binarizing the image information calculated in the first information processing step and emphasizing the pupil portion with a predetermined binarization threshold, and A third information processing step of detecting the position of the observer from the obtained image information, and a third information processing step between the first information processing step and the second information processing step. 1. An observation position detection of an observer, wherein a binarization threshold value calculation step of determining the predetermined binarization threshold value in real time based on the image information in which a pupil portion is enhanced by the information processing step is provided. Method. 2. A face imaging step in which the face of the observer is imaged by an imaging means while the face of the observer is illuminated by a predetermined illumination means, and image information of the face imaged in the face imaging step is temporarily stored. The image information is stored in the image memory and the observer's own
In the viewer's viewing position detection method and a viewing position calculation step of calculating the position, the observation position calculation step, the performing emphasizing operation image information of the pupil portion of the stored facial image in the image memory A first information processing step, a second information processing step of binarizing the image information in which the pupil portion has been emphasized in the first information processing step with a predetermined binarization threshold value, and the binarized image A third information processing step of detecting the position of the observer himself from the information, and between the first information processing step and the second information processing step , An observation position detection method for an observer, wherein a light source light amount control step of driving and controlling the illumination means is provided so that a signal level of a pupil portion in the face image is always constant even if the position changes. . 3. The observer's observation position detecting method according to claim 1, wherein the operation of enhancing the pupil portion of the face image in the first information processing step is performed by image differentiation processing. Observation position detection method of the observer. 4. The face image in which the binarization threshold value in the binarization threshold value calculation step is calculated by emphasizing a pupil portion of the face image according to the observer's observation position detection method according to claim 1. A method for calculating an image standard deviation of the image and determining based on the image standard deviation. 5. The method for detecting an observer's observation position according to claim 2, wherein the light source light amount of the illuminating means, which is a control reference in the light source light amount control step, is calculated by emphasizing a pupil portion of the face image. A method for detecting an observation position of an observer, which calculates an image standard deviation of the obtained image and determines the image standard deviation based on the image standard deviation. 6. The method according to claim 1, wherein one of the first to third information processing steps includes another system. A series of information processing steps comprising: a series of information processing steps consisting of the other systems are determined based on the face information stored in the image memory to determine the presence or absence of a reflective marker attached to the face of the observer in advance. A marker judging step, and a marker which operates prior to the one of the information processing steps and binarizes a marker image in the image memory based on a predetermined threshold when it is judged that the marker is present in the marker judging step An image binarization step, and the center of gravity of the observer is specified based on the binarization information of the marker image obtained in the marker image binarization step, and the position of the observer itself is determined based on the center of gravity position. Calculate The observation position of the observer detection method is characterized in that is constituted by the observation position calculation step. 7. The method for detecting an observer's observation position according to claim 6, wherein the predetermined threshold value in the marker image binarization step is:
An observation position detection method for an observer, comprising using a predetermined threshold value in an area smaller than the reflection luminance of the reflection marker and larger than the luminance of the pupil reflected light of the observer. 8. An image pickup means for picking up an image of the face of the observer, an illuminating means arranged on the image pickup means side for illuminating the face of the observer, and image information of the face of the observer picked up by the image pickup means. An image processing means for calculating the position of the observer based on the image processing means, wherein the image processing means stores image information of the face of the observer obtained by the imaging means. Memory, a pupil image enhancing unit for performing a predetermined operation for enhancing a pupil portion of the observer in the image information stored in the image memory, and a pupil image output from the pupil image enhancing unit A binarization processing unit that performs a binarization process on the data based on a predetermined binarization threshold, and based on a calculation result of the binarization processing unit,
A configuration including an observation position calculation unit that calculates the position of the observer itself, and the binarization processing unit includes a binarization unit for a pupil image based on image information output from the pupil image enhancement unit. An observation position detection device for an observer, further comprising a binarization threshold setting unit that determines a threshold in real time and sends the threshold to the binarization processing unit. 9. The observation position detecting device according to claim 8, wherein the illuminating means is disposed so that an optical axis thereof is substantially coaxial with an optical axis of the imaging means. Observation position detector for observer. 10. The observation position detection device according to claim 8, wherein the binarization threshold value setting unit calculates an image standard deviation of image information output from the pupil image enhancement unit, and based on the calculated standard deviation. A light source illuminance calculation circuit for calculating a light source illuminance, and a binarization threshold setting circuit for setting a binarization threshold based on an image standard deviation calculated from the light source illuminance calculation circuit. Observation position detection device for the observer. 11. An imaging means for imaging an observer's face,
Illuminating means arranged on the imaging means side for illuminating the face of the observer; and image processing means for calculating the position of the observer itself based on image information of the face of the observer captured by the imaging means. In the observer observation position detecting apparatus, the image processing means includes: an image memory for storing image information of the face of the observer obtained by the imaging means; and an image memory for storing the image information in the image memory. A pupil image enhancement unit that performs a predetermined operation to enhance the pupil portion of the observer, and a binarization processing unit that binarizes pupil image data output from the pupil image enhancement unit based on a predetermined threshold.
The observer himself based on the calculation result of this binarization processing unit
And an observation position calculation unit that calculates the position of the observer, and the illumination unit always has a constant output level of image information output from the pupil image enhancement unit even if the observation position of the observer changes. A light source light quantity control unit for variably controlling the output of the lighting means is provided so that the light source light quantity setting unit for driving and controlling the light source light quantity control unit is
An observation position detection device for an observer, which is provided in the image processing means. 12. The observer's observation position detecting device according to claim 11, wherein the light source light amount setting unit calculates a light source illuminance of the illumination unit based on image information output from the pupil image enhancing unit. A light source illuminance calculation circuit that performs a predetermined calculation based on the light source illuminance information output from the light source illuminance calculation circuit so that the output level of the image information output from the pupil image enhancement unit is always constant. A light source light quantity setting circuit for variably setting a light source light quantity of the means in real time. 13. The observer's observation position detection apparatus according to claim 12, wherein the light source illuminance calculation circuit calculates and based on an image standard deviation of image information output from the pupil image enhancement unit. A light source illuminance calculation function for calculating a light source illuminance of the illumination means. 14. The method of claim 8, 9, 10, 11, or 12.
Or the observer's observation position detection device according to item 13, wherein the pupil image enhancement unit has an image differentiation processing function of differentiating an image stored in the image memory. Position detection device. 15. The method of claim 8, 9, 10, 11, 1.
The observation position detection device for an observer according to 2, 13, or 14, wherein a reflection marker is mounted on a face of the observer in advance,
The image memory further includes a reflection marker determination unit that determines the presence or absence of a reflection marker from the face image information stored in the image memory. A marker is provided at the output stage of the reflection marker determination unit by the reflection marker determination unit. When it is determined that the marker image binarizing means is operated prior to the pupil image enhancing section and binarizes the marker image in the image memory, the marker image binarizing means calculates An observation position detection device for an observer, wherein the observation position calculation unit has a marker position calculation function of calculating a center of gravity position of a face image and specifying the calculated position as an observer's own position . 16. The observer's observation position detecting device according to claim 15, wherein the reflection luminance of the reflection marker is set to be larger than the reflection luminance of the pupil reflection light of the observer in advance, and the reflection marker determination is performed. An observation position detection apparatus for an observer, wherein a threshold for binarization in the unit is a predetermined threshold that is smaller than the reflection luminance of the reflection marker and larger than the luminance of the pupil reflection light.