KR20070061091A - Method for deciding the position of an observer, and an observer tracking 3d display system and method using that - Google Patents

Abstract

A method for determining the position of an observer, and an observer-tracking 3-dimensional display system and method using the method are provided to show an observation position to an observer by using an indicator when the observer is not located at a 3-dimensional stereo image observation position. An observer-tracking 3-dimensional display system includes an image acquisition unit(31), an observer position tracking unit(32), and a guide unit(33). The image acquisition unit acquires an image of an observer. The observer position tracking unit extracts 3-dimensional position information of the observer from the image of the observer and generates an observer position indication control signal. The guide unit shows the best observation position to the observer according to the observer position indication control signal.

Description

Method for deciding the position of an observer, and an observer tracking 3D display system and method using that}

1 is a block diagram of an embodiment of an observer tracking type 3D stereoscopic image display system according to the present invention;

2 is a configuration diagram of another embodiment of the observer tracking type 3D stereoscopic image display system according to the present invention;

3 is a configuration diagram of an embodiment of an observer position tracking device according to the present invention;

FIG. 4 is an exemplary view illustrating observation position changes with respect to the image display apparatus and the 3D stereoscopic image display mask positions of FIG.

5 is an exemplary view showing a relationship between a camera and a display device and a camera coordinate system according to the present invention;

6 is a detailed configuration diagram of an embodiment of the 3D position information extraction unit of the observer position tracking apparatus according to the present invention;

7 is a view showing a general face posture looking at the display device according to the present invention (a two-dimensional view of the x-z plane at y = −∞),

8 is a view showing the relationship between the camera and the left / right eyes according to the present invention.

* Explanation of symbols for the main parts of the drawings

11: CCD camera 12: indicator

13: 3D video display device 21: motors (M1 to M4)

22: three-dimensional stereoscopic image display mask 23: video display device

31: Image signal input unit 32: 3D location information extraction unit

33: control signal generator

The present invention relates to an observer positioning method and an observer tracking type 3D display system using the same, and more particularly, to three-dimensional display of an observer after recognizing the observer's position within a certain range from a fixed observation position. Positioning the observer to control a three-dimensional stereoscopic image display mask such as a lenticular or parallax barrier to guide the observer to the stereoscopic image viewing position or to observe the three-dimensional stereoscopic image from the observer's position. A method, and an observer tracking type 3D display system using the same, and a method thereof.

The parallax barrier method uses a parallax that causes an image or an image to appear three-dimensional by placing a barrier filter with the image panel at a distance from the front and then observing different images or images through the barrier filter. parallax).

In general, the biggest factor that humans feel in three dimensions is the effect of spatial differences on the left and right retinas caused by the left and right eyes looking at an object in one direction. Using this effect, a method of displaying different images, that is, stereoscopic images (three-dimensional images), in both left and right eyes was used. The stereoscopic image (3D image) has been developed into a method of wearing glasses and a method of wearing glasses according to a display method, and the most representative method of viewing without glasses is a lenticular plate representing only vertical parallax. It is a method of viewing a stereoscopic image by attaching it to an image display device, and furthermore, using a multiview image display mask such as a multi-view or ultra-multi-view IP (Integral Photography) plate, a cross lenticular plate, or a rectangular lens plate to express vertical and horizontal parallax. It has been developed.

As such, the 3D display method is divided into a method of observing a 3D image with glasses and a method of observing a 3D image without glasses.

A method of observing a 3D image by wearing glasses is a method of observing a 3D image in a large space such as a theater and uses a binocular stereoscopic image (3D image) display method. As a method of viewing without wearing glasses, a small number of observers mainly observe a three-dimensional image at a position where an observation position is determined.

By the way, as a representative conventional method for observing a 3D image without wearing glasses, a parallax barrier method is used, but when displaying a 3D stereoscopic image using a conventional parallax barrier method, a 3D image is displayed. Since the scope of observation is limited, there is a problem that the observer must observe the 3D image at a fixed position.

The present invention has been proposed to solve the above problems, the observer positioning method and the viewer tracking using the same to guide the observer by the indicator when the observer is not in the 3D stereoscopic image observation position It is an object of the present invention to provide a three-dimensional display system and a method thereof.

In addition, the present invention controls a three-dimensional stereoscopic image display mask when observing a three-dimensional stereoscopic image by a parallax barrier or a lenticular method, thereby observing the three-dimensional stereoscopic image even if the observer is out of a certain range from the three-dimensional stereoscopic image observation position. Another object of the present invention is to provide an observer positioning method, an observer tracking type 3D display system using the same, and a method thereof.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

In accordance with an aspect of the present invention, there is provided a method of determining an observer, the method comprising: obtaining an image of an observer through a camera; And extracting three-dimensional positional information extracting three-dimensional positional information of the observer using the positional information of both eyes of the observer and the parameter information of the camera in the acquired image.

On the other hand, the observer tracking three-dimensional display system, the observer tracking three-dimensional display system, the image acquisition means for obtaining an image of the observer; Observer position tracking means for generating observer position guidance control signals by extracting three-dimensional position information of the observer from the image of the observer acquired by the image acquisition means; And means for guiding the observer to an observation position by displaying a signal for guiding the observer to an optimal observation position according to the observer position guidance control signal from the observer position tracking means.

In addition, the observer tracking three-dimensional display method of the present invention, the observer tracking three-dimensional display method, comprising the steps of: obtaining an image of the observer; An observer position tracking step of extracting three-dimensional position information of the observer from the acquired observer image to generate an observer position guide control signal; And displaying a signal for guiding the observer to an optimal observation position according to the observer position guide control signal to guide the observer to the observation position.

On the other hand, another observer tracking three-dimensional display system of the present invention, the observer tracking three-dimensional display system, the image acquisition means for obtaining an image of the observer; Observer position tracking means for generating an image display mask control signal by extracting three-dimensional position information of the observer from the image of the observer acquired by the image acquisition means; And means for moving the image display mask according to the image display mask control signal from the observer position tracking means to display a three-dimensional stereoscopic image adapted to the observer position.

In addition, another observer tracking three-dimensional display method of the present invention, the observer tracking three-dimensional display method, comprising the steps of: obtaining an image of the observer; An observer position tracking step of generating an image display mask control signal by extracting three-dimensional position information of the observer from the acquired observer image; And moving the image display mask according to the image display mask control signal to display a 3D stereoscopic image adjusted according to the position of the observer.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is an exemplary configuration diagram of an observer tracking type 3D stereoscopic image display system according to an exemplary embodiment of the present invention, and recognizes and observes an observer position through an observer position tracking device 30, thereby indicating an observation position to an observer. 12) is shown.

As shown in FIG. 1, an observer tracking type 3D stereoscopic image display system according to an embodiment of the present invention includes a CCD camera 11 and a CCD camera 11 for capturing an image of an observer. An observer position tracking device (see FIG. 3 to be described later) 30 for generating an observer position guidance control signal by receiving an observer's image and extracting three-dimensional position information of the observer, and an observer from the observer position tracking device 30. Display a signal to guide the observer to the optimum observation position according to the position guidance control signal to guide the observer to the observation position (for example, by controlling the LED (Light Emitting Diode) corresponding to the up / down / left / right / front / back) And an indicator 12 for guiding the observer to the viewing position.

That is, FIG. 1 is an embodiment of an observer tracking type 3D stereoscopic image display system including the above function, and the image information of the observer is recognized by the 3D image display device 13 to recognize and track the position of the observer. An input CCD camera 11 and an indicator 12 for guiding a viewing position to an observer are included.

In FIG. 1, when the 3D image display apparatus 13 displays a 3D image at an arbitrary observer position without guiding the observer's position to the observation position, the 3D image display apparatus may be configured as shown in FIG. 2. have. In general, when displaying a 3D stereoscopic image, the 3D stereoscopic image display mask 22 such as a parallax barrier or a lenticular is fixedly positioned on the image display apparatus 23. However, in the present invention, the image display apparatus 23 and the 3D image are fixed. By separating the 3D stereoscopic image display mask 22 and then moving the 3D stereoscopic image display mask 22, the 3D stereoscopic image can be observed even if the observer is at a position deviated to some extent from the observation position (see equation described later). To be configured.

2 is a configuration diagram of another embodiment of the observer tracking type 3D stereoscopic image display system according to an embodiment of the present invention, and through the observer position tracking device 30 through recognition and tracking of the observer position, based on the 3D stereoscopic image display mask By controlling (22), it is shown how the observer can observe a three-dimensional stereoscopic image at that position without moving even if the observer is at a position that is somewhat out of the observation position (see equation described later).

As shown in FIG. 2, an observer tracking type 3D stereoscopic image display system according to another embodiment of the present invention includes a CCD camera 11 for capturing (photographing) an image of an observer, and a CCD camera 11. An observer position tracking device (see FIG. 3 to be described later) 30 for generating an image display mask control signal by receiving an observer's image and extracting three-dimensional position information of the observer, and an image from the observer position tracking device 30. The motors M1 to M4 21 and the image display device 23 are spaced apart from each other to generate the up / down / left / right / front / rear rotational driving force according to the display mask control signal. It is moved according to the rotational driving force of the M4 (21), and includes a three-dimensional image display mask 22 for allowing the image display device 23 to display a three-dimensional stereoscopic image at the observer position.

That is, in FIG. 2, four motors (M1, M2, M3, M4) 21 and a driving means are configured to control the position of the three-dimensional stereoscopic image display mask 22, and each motor 21. The 3D stereoscopic image display mask 22 can be moved up / down / left / right and forward / backward from the image display device 23 by the operation of.

3 is a configuration diagram of an observer position tracking apparatus according to an embodiment of the present invention, in which “31” is an image signal input unit, “32” is a three-dimensional position information extractor, “33” is a control signal generator; &Quot; 34 " represents a three-dimensional stereoscopic image display panel (same as the three-dimensional stereoscopic image display mask 22 in FIG. 2) / indicator (same as the indicator 12 in FIG. 1), respectively.

As shown in FIG. 3, the observer position tracking apparatus 30 according to the present invention includes an image signal input unit 31 and an image signal input unit 31 for acquiring an image of an observer from an external camera. A three-dimensional position information extractor 32 for extracting three-dimensional position information of the observer by using the position information of the left and right eyes of the observer and the camera internal parameter information in the image, and the three-dimensional position information extractor ( And a control signal generator 33 for generating a control signal (image display mask position control signal or observer position guide control signal) based on the three-dimensional position information extracted in FIG.

In this case, the image display mask position control signal is a signal for controlling a motor capable of controlling the position of the image display mask. The image display mask position control signal is a signal which is converted into any one of voltage, current, frequency, or pulse according to the type of motor and controlled.

In addition, the observer position guide control signal is an up / down / left / left / input input from the 3D position information extracting unit 32 to guide the observer to a fixed observation position when the observer is out of the 3D stereoscopic image observation position. It is a signal that guides the observer to the observation position by controlling LEDs corresponding to up / down / left / right and front / back according to the right rotation angle and distance value.

Here, the image signal input unit 31 receives the observer's image in real time and transmits it to the 3D location information extracting unit 32. The 3D location information extracting unit 32 processes the image signal by the following steps in order to recognize and track the position of the observer.

The camera is arranged in the upper center portion of the video display device 23 as shown in FIG. 5, and the x and y axes of the camera coincide with the horizontal and vertical planes of the display device.

FIG. 4 is an exemplary view illustrating changes in observation position with respect to the image display device and the 3D stereoscopic image display mask position of FIG. 2, and illustrates the relationship between the image display device 23 and the 3D stereoscopic image display mask 22. will be.

In general, the observation position is fixed at a predetermined distance as shown in FIG. However, when the observer approaches the image display apparatus 23, the stereoscopic image is placed at the observer position by making the distance between the image display apparatus 23 and the 3D stereoscopic image display mask 22 close as shown in FIG. 4B. In addition, when the viewer moves left and right, the 3D stereoscopic image display mask 22 is moved left and right as shown in FIGS. can do.

6 is a detailed configuration diagram of an embodiment of the 3D location information extraction unit of the observer location tracking apparatus according to the present invention.

As shown in FIG. 6, the 3D location information extracting unit 32 of the observer position tracking apparatus according to the present invention receives an image input signal from a camera in real time through the image signal input unit 31, and the left and right eyes. Two-eye detection unit 61 for detecting the center position of the camera and the focal length, camera internal parameter information (principal point, aspect ratio, etc.) are calculated to calculate the focal length information in pixels. based on the camera calibration unit 62 and the position information of the two eyes in the image acquired by the binocular detector 61 and the camera calibration unit 62 and the internal camera parameter information, respectively. Distance information calculator (Camera to eye dist) for converting the three-dimensional position information (Pan / Tilt angles θ, φ and distance values z ) required by the control signal generator 33 to the camera and the observer ance computation unit) (63).

Here, the eye detection unit 61 may detect the center point of each eye by a known method (A. Haro et al. "Detecting and Tracking Eyes By Using Their Physiological Properties, Dynamics, and Appearance", CVPR2000).

In addition, the camera calibration unit 62 may be performed using the "Tsai calibration" method (conventional paper cited) commonly used in computer vision.

For reference, the papers related to the detection and tracking of both eyes are as follows.

The technology of searching and tracking the position of the eyes in the human face is an important technology in the field of Human Computer Interaction (HCI), which has been published by numerous researchers and is actively being published. Among them, "A. Haro et al." Is titled "Detecting and Tracking Eyes By Using Their Physiological Properties, Dynamics, and Appearance" (CVPR vol. 1, pp. 163-168). He published a paper on Pattern Recognition. This paper is about the technology to detect and track both eyes by receiving the image signal in real time. We detected two eyes stably using infrared rays to detect the two eyes, and then, after stochastic modeling of the change in the pixel value brightness of the eyes, a method that can robustly track both eyes based on Kalman filter Suggested. This technique can be used in the binocular detector 61 of the present invention to robustly detect and track the position of both eyes in the image.

In addition, the papers related to camera calibration are described in detail.

"RY Tsai" was published in 1987 in "IEEE Journal of Robotics and Automation" by "Metrology Using Off-the-Shelf TV Cameras and Lenses" (IEEE Journal of Robotics and Automation, Vol. 3, No. 4, pp. 323- 344, August 1987.) is a representative algorithm that is still commonly used in computer vision because of its high accuracy. In the camera calibration unit 62 of the present invention, an internal parameter of the camera may be extracted using a calibration algorithm proposed by "Tsai".

In addition, when looking at related patents, there are many patents related to face detection and tracking, and there are not many patents related to eye detection and tracking. In particular, a lot of patents related to the iris recognition technique, and a patent for detecting the blinking of the eyes to prevent the driver's drowsiness while driving a car have been applied. Searching for a patent related to detection and tracking of both eyes in the present invention searches for a patent registered in 1998 under the title of "eye image tracking device" (registration number 10-0296215-0000). This patent is largely divided into an eye detection part and a tracking part, and for detecting the eye, the input image is binarized to track not only the eye but also the area around the eye.

The operation of the observer tracking type 3D stereoscopic image display system using the observer position tracking device according to the present invention having the structure as described above will be described in detail as follows.

FIG. 7 is a view illustrating a general face posture looking at a display device according to the present invention (a two-dimensional view of the xz plane at y = −∞), and FIG. 8 is a view of a camera and left / right eyes according to the present invention. A diagram showing a relationship.

In general, the viewer's display viewing posture is displayed as shown in FIG. 7 (the line connecting the two eyes from the camera is orthogonal). In FIG. 8, images of two eyes are formed in an image plane and center positions E and F of the two eyes in the image plane are obtained from both eyes detector 61. In FIG. 8, f denotes a focal length in pixel unit obtained by the camera calibration unit 62. The distance information calculation unit 63 between the camera and the observer calculates the distances and angles θ and φ in the Z- axis direction from the camera center to the position C corresponding to the center of the left and right eyes, respectively, to the control signal generator 33. to provide.

를 하기의 [수학식 1]을 이용하여 구할 수 있다.First of all, using the center position and focal length of both eyes,

It can be obtained using Equation 1 below.

Therefore, the angle θ can be obtained as shown in Equation 2 below.

와

는 합동 관계와 사람의 좌/우 두 눈의 거리에 해당하는 거리 d 를 이용하면 카메라 센터와 점 C 까지의 거리

는 하기의 [수학식 3]과 같이 구해진다. And right triangle

Wow

Is the distance between the camera center and point C using the distance d , which is the joint relationship and the distance between the left and right eyes of the person.

Is obtained as shown in Equation 3 below.

와 두 눈의 중앙점이 Y 축 방향으로 b 만큼 떨어져 있다고 했을 때 하기의 [수학식 4]와 같은 식으로 구할 수 있다. At this time, the calculation of the angle φ is the distance

Assuming that the center point of the eyes and b are separated by b in the Y- axis direction, Equation 4 can be obtained.

The distance in the Z- axis direction from the center of the camera to the center point C of the left and right eyes can be obtained using Equation 5 below.

Accordingly, by the above method, the 3D position information extracting unit 32 may calculate the observer left and right rotation angle θ , the up and down rotation angle φ , and the distance z between the observer and the camera, and generate the control signal. Transfer to section 33. In this case, the 3D position information extracting unit 32 uses the 3D position information (left and right rotation angle θ , up and down rotation angle φ and the distance value z between the observer and the image display device). The indicator 12 may be enabled by an error of a reference value to move to a fixed observation position, or the 3D stereoscopic image display mask 22 may be controlled to display a 3D stereoscopic image at a corresponding observer position. .

Then, the control signal generator 33 converts the control signal into a control signal using the information input from the three-dimensional position information extractor 32. In the three-dimensional position information extracting unit 32, the position information of the observer, that is, the left and right rotation angle θ , the vertical rotation angle φ , and the space between the observer and the image display apparatus 23 based on the center of the image display apparatus 23. Provide the distance z value.

Therefore, the control signal generator 33 converts the 3D position information into a control signal capable of controlling the 3D stereoscopic image display mask 22 or the indicator 12 and provides the input signal to each input. For example, the control signal is a signal for controlling the motor 21 capable of controlling the position of the 3D stereoscopic image display mask 22. The control signal may be any one of voltage, current, frequency, or pulse according to the type of the motor 21. Control by converting to one.

As another example, in the case of the indicator 12, the up / down / left / right rotation angles and distances input from the 3D location information extractor 32 to guide the observer to a fixed observation position when the observer is out of the observation position. According to the value, the observer is guided to the observation position by using the indicator 12 including light emitting diodes (LEDs) corresponding to up / down / left / right and front / back.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

As described above, when the observer is out of the 3D stereoscopic image viewing position, the 3D stereoscopic image display apparatus uses the indicator to guide the observer to the observation position or the observer tracking type 3D stereoscopic image display apparatus. By displaying a three-dimensional stereoscopic image at any observer position, the observer can observe the three-dimensional stereoscopic image without any movement of the observer, even if the observer deviates somewhat from the observation position.

Claims (15)

In the method of determining the position of the observer, Obtaining an image of an observer through a camera; And 3D position information extraction step of extracting the 3D position information of the observer using the position information of both eyes of the observer and the parameter information of the camera in the obtained image Observer positioning method comprising a. The method of claim 1, The three-dimensional location information extraction step, Detecting both eye positions of the observer's eyes from the acquired image; And 3D position information calculation step of calculating the 3D position information for the observer using the position information of the two eyes in the detected image and the parameter information of the camera Observer positioning method comprising a. The method of claim 2, The both eye position detection step, An observer positioning method, characterized in that the position of both eyes is detected on an image plane. The method of claim 2, Parameter information of the camera, And a focal length of the camera. The method according to any one of claims 2 to 4, The three-dimensional position information calculation step, Observer positioning method characterized by expressing the three-dimensional position information for the observer by the left / right rotation angle ( θ ), the up / down rotation angle ( φ ) and the distance value ( z ) between the observer and the image display device. . The method of claim 5, The three-dimensional position information calculation step, The left / right rotation angle θ is calculated using Equation a below, and the up / down rotation angle φ is calculated using Equation b below. The distance value ( z ) of the observer is calculated using the following Equation c. Equation a

[Equation b]

[Equation c]

(Where f is the focal length, d is the distance between the left and right eyes,

Is the distance to the camera center and point C, b is the distance

The distance between the center point of the two eyes is in the Y- axis direction, θ is the left and right rotation angle of the observer, φ is the vertical rotation angle of the observer, and z is the distance between the observer and the camera.) In the observer tracking type 3D display system, Image acquisition means for acquiring an image of the observer; Observer position tracking means for generating observer position guidance control signals by extracting three-dimensional position information of the observer from the image of the observer acquired by the image acquisition means; And Means for guiding the observer to an observation position by displaying a signal for guiding the observer to an optimal observation position according to the observer position guidance control signal from the observer position tracking means; Observer tracked three-dimensional display system comprising a. The method of claim 7, wherein The observer position tracking means, Means for extracting positional information of both eyes of an observer on an image plane from an image of the observer acquired by the image obtaining means; Means for calculating three-dimensional position information of the observer by using the extracted position information of both eyes of the observer and internal camera parameter information; And Means for generating the observer position guidance control signal based on the calculated 3D position information of the observer Observer tracked three-dimensional display system comprising a. In the observer tracking type 3D display system, Image acquisition means for acquiring an image of the observer; Observer position tracking means for generating an image display mask control signal by extracting three-dimensional position information of the observer from the image of the observer acquired by the image acquisition means; And Means for displaying a three-dimensional stereoscopic image adjusted according to the observer position by moving the image display mask in accordance with an image display mask control signal from the observer position tracking means; Observer tracked three-dimensional display system comprising a. The method of claim 9, The observer position tracking means, Means for extracting positional information of both eyes of an observer on an image plane from an image of the observer acquired by the image obtaining means; Means for calculating three-dimensional position information of the observer by using the extracted position information of both eyes of the observer and internal camera parameter information; And Means for generating the image display mask position control signal based on the calculated 3D position information of the observer Observer tracked three-dimensional display system comprising a. The method of claim 10, The image display mask, An observer tracking type three-dimensional display system, characterized in that spaced apart from the image display device. In the observer tracking type three-dimensional display method, Obtaining an image of the observer; An observer position tracking step of extracting three-dimensional position information of the observer from the acquired observer image to generate an observer position guide control signal; And Guiding the observer to an observation position by displaying a signal for guiding the observer to an optimal observation position according to the observer position guide control signal; Observer tracked three-dimensional display method comprising a. The method of claim 12, The observer location tracking step, Extracting location information of both eyes of an observer on an image plane from the acquired observer image; Calculating three-dimensional position information of the observer using the extracted position information of both eyes of the observer and internal camera parameter information; And Generating the observer position guidance control signal based on the calculated 3D position information; Observer tracked three-dimensional display method comprising a. In the observer tracking type three-dimensional display method, Obtaining an image of the observer; An observer position tracking step of generating an image display mask control signal by extracting three-dimensional position information of the observer from the acquired observer image; And Displaying a 3D stereoscopic image adjusted according to the viewer position by moving the image display mask according to the image display mask control signal; Observer tracked three-dimensional display method comprising a. The method of claim 14, The observer location tracking step, Extracting location information of both eyes of an observer on an image plane from the acquired observer image; Calculating three-dimensional position information of the observer using the extracted position information of both eyes of the observer and internal camera parameter information; And Generating the image display mask position control signal based on the extracted 3D position information; Observer tracked three-dimensional display method comprising a.

Images