WO2010095770A1

WO2010095770A1 - Method for automatically adjusting depth of field to visualize stereoscopic image

Info

Publication number: WO2010095770A1
Application number: PCT/KR2009/000876
Authority: WO
Inventors: 신병석; 강동수
Original assignee: 인하대학교 산학협력단
Priority date: 2009-02-19
Filing date: 2009-02-24
Publication date: 2010-08-26
Also published as: KR100956453B1

Abstract

The present invention relates to a method for adjusting the depth of field of a stereoscopic image according to the focal plane of a user. More specifically, the present invention comprises the following steps of: (a) finding the distance between the pupils of a user by a setup time interval to calculate a focal distance with which a user sees a stereoscopic image; (b) calculating a focal plane through the calculated focal distance; and (c) automatically adjusting the depth of field of a sampling of pixels, included in a circle of confusion of pixels, which are within a set number around a stereoscopic pixel corresponding to the calculated focal plane. The method for automatically adjusting depth of field to visualize a stereoscopic image according to the present invention calculates a focal distance, with which a user sees a stereoscopic image, by a setup time interval and automatically adjusts the depth of field of pixels of a circle of confusion of pixels, which are within a set number around a pixel corresponding to the focal plane of the calculated stereoscopic image, by a setup time interval, thereby guaranteeing minimum rendering speed of depth of field of a stereoscopic image with minimum distortion.

Description

[Correction 28.10.2009 by Rule 26] Automatic Depth Control Method for Stereoscopic Visualization

The present invention relates to a method for adjusting the depth of field of a stereoscopic image according to the focal plane of the user, and more specifically, to the pixel of the stereoscopic image corresponding to the focal plane of the stereoscopic image calculated through the focal length at which the user views the stereoscopic image. The present invention relates to a method for automatically adjusting the depth of field of sampling pixels included in a confusion circle of pixels located within a set number based on the reference.

Stereoscopy expression technology is a technology that can be immersed by three-dimensional representation of the scene of the virtual or real world. In other words, the stereoscopic image allows the user to feel a stereoscopic feeling from two two-dimensional images by using binocular disparity (binocular disparity) such as the human eye.

Such stereoscopic images provide depth-of-focus (depth of the gaze of the stereoscopic image) so that a person can feel a real sense of distance. Therefore, watching stereoscopic images for a long time may cause dizziness or vomiting because the user's focus is continuously forced.

As a conventional technique for solving such a problem, a system for applying a depth of field of a stereoscopic image in real time by moving a pointer to a position viewed by a user using a device such as a mouse has been proposed.

However, the conventional depth-of-field application system of the stereoscopic image has a problem in that it is possible to know only where the user is looking at the screen and cannot calculate which depth of focus the user is looking at in the generated scene.

The present invention has been proposed to solve the above-mentioned problems, and calculates a focal plane by calculating a focal length at which a user views a stereoscopic image at a set time interval, and calculates a focal plane corresponding to the calculated focal plane of the stereoscopic image. It is an object of the present invention to provide a method for automatically adjusting the depth of field of sampling pixels included in a confusion circle of pixels located within a set number based on pixels of an image according to the focal length.

Another object of the present invention is to adjust the depth of field by sampling a smaller number of pixels as the confusion circles of pixels closer to the focal plane among the confusion circles, and to adjust the depth of field among the confusion circles. It is to provide a control range of the depth of field in the stereoscopic image to adjust the depth of field by sampling a larger number of pixels as the confusion source of the pixels of the confusion circle in the far place.

Automatic depth of field control method for stereoscopic image visualization according to the characteristics of the present invention for achieving the above object,

(a) calculating a focal length at which a user views a stereoscopic image by determining a distance between pupils of two eyes of a user at a predetermined time interval;

(b) calculating a focal plane based on the calculated focal length; And

and (c) automatically adjusting the depth of field of sampling pixels included in a confusion circle of pixels located within a set number based on the calculated pixels of the stereoscopic image corresponding to the focal plane. do.

Preferably, step (a) is

(a-1) identifying the center positions of the pupils of the two eyes of the user at predetermined time intervals; And

(a-2) calculating the distance between the two eyes using the center position of the pupil determined at the set time interval and calculating the focal length of the user using the calculated distance.

More preferably, the step (a-1),

(1) irradiating and reflecting electromagnetic waves to the two eyes to determine positions of the two eyes through reflection points;

(2) separating the pupil and the cornea at the identified two eye positions;

(3) zoning the separated pupils; And

(4) extracting the central position of the pupil using the search window.

More preferably, before step (a),

The method may further include measuring a focal length for each distance between the two eyes of the user so as to be used when calculating the focal length of the user according to the distance between the two eyes in the step (a-2).

More preferably, the distance between the two eyes in the step (a-2) is characterized in that calculated through the following equation.

Equation

Where M _x is the sum of the x coordinates of the pupil area, M _y is the sum of the y coordinates of the pupil area, P _cx and P _cy are the center coordinates of the pupil, and n is the number of pixels having the pupil area value in the search window, respectively. Indicates.

Preferably, in step (c),

The closer to the pixel corresponding to the focal plane among the confusion circles, the depth of field of the smaller number of sampling pixels is adjusted, and the farther from the pixel corresponding to the focal plane among the confusion circles, the depth of field of the larger number of sampling pixels is adjusted. It characterized in that to adjust.

Preferably, in step (c),

A constellation circle including a pixel closest to the pixel corresponding to the focal plane is adjusted for depth of field by sampling six pixels, and a constellation circle including a second pixel closest to the pixel corresponding to the focal plane is 12 pixels. The depth of field is adjusted by sampling, and a confusion circle including a pixel corresponding to the focal plane and a third to set number of pixels is configured to adjust the depth of field by sampling 24 pixels.

More preferably, the focal length is divided into three stages according to the distance.

The automatic depth of field adjustment method for visualizing a stereoscopic image according to the present invention includes calculating a focal length at which a user views a stereoscopic image in real time, and a confusion circle of pixels located within a set number of pixels corresponding to the calculated focal plane of the stereoscopic image. By automatically adjusting the depth of field of the sampling pixels at set time intervals, it is possible to ensure the minimum rendering speed of the depth of field of the stereoscopic image with minimal distortion.

1 is a block diagram of an automatic depth of field control system for stereoscopic visualization of the present invention.

2 is a conceptual diagram illustrating an off-axis technique and an on-axis technique.

3 is a conceptual diagram showing the appearance of confusion circle in the image plane of the point B farther than the focal plane.

4 is a block diagram showing a sampling method for implementing a depth of field in the stereoscopic image of the present invention.

Figure 5 is a flow chart of the automatic depth of field control method for stereoscopic visualization of the present invention.

Figure 6 is a graph comparing the experimental results of the experimenters for the three-dimensional image of the general three-dimensional image of the present invention.

10: tracking device

12: LED lights

14: camera

16: control unit

18: calculation unit

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an automatic depth of field control system for stereoscopic visualization of the present invention. As shown in FIG. 1, the automatic depth of field control system 100 for stereoscopic visualization of the present invention includes an eye-tracking device 10 and a rendering module 20.

The eye tracking device 10 may include an LED light 12, a camera 14, a controller 16, and a calculator 18.

The LED light 12 consists of infrared LEDs for irradiating and reflecting electromagnetic waves, such as infrared rays, which do not directly burden the eyes of the user. The camera 14 preferably consists of a pair of infrared cameras to photograph the reflection surfaces of infrared rays reflected from the pupils of both eyes of the user. Here, the LED light 12 is preferably disposed in parallel with the pair of cameras 14 so as to be able to detect the pupil reflection points of the two eyes of the user.

In an embodiment of the present invention, the controller 16 distinguishes the pupil and the cornea using a double threshold method at the position of the eyes of the user identified by the camera 14, and uses a labeling method. Pupils can be zoned. Since the double threshold method and the labeling method are already known techniques, detailed description thereof will be omitted.

In addition, in one embodiment of the present invention, the controller 16 moves the search window by means of a mean shift method to accurately extract the position of the pupil, (after the search window determines the initial position from the infrared reflecting point). Using this, it is possible to extract the center of the pupil of the eyes of the user.

The calculation unit 18 calculates the distance between the two eyes using the center position of the pupil through the center position of the pupil of both eyes of the user. The distance between the eyes of the user is calculated through Equation 1 below.

Equation 1

In the present invention, by measuring the focal length for each distance between the eyes of the user in advance, it is possible to recognize the focal length for the stereoscopic image of the user in advance by calculating only the distance between the eyes of the user. Accordingly, the calculator 18 may calculate the focal length of the user using the calculated distance between the two eyes.

On the other hand, the calculation unit 18 divides the focal length into three steps of distance range values, for example, the farthest, the middle, and the closest parts, to the rendering module. The reason for dividing the focal length into three stages is that it is difficult to divide it into several stages due to the limitation of the resolution (640 x 80) of the camera 14 as a result of experimenting the distance change of the pupil according to the depth of eye. to be.

The rendering module 20 automatically adjusts the depth of field of a confusion circle of pixels located within a set number of pixels corresponding to the focal plane of the stereoscopic image calculated by the eye tracking device at a set time interval according to the focal length of the user. .

Here, the principle of generating the stereoscopic image by the rendering module 20 is as follows.

2 is a conceptual diagram illustrating an off-axis technique and an on-axis technique. As shown in FIG. 2, generally, there are an off-axis technique and an on-axis technique for generating a stereoscopic image. Here, the off-axis technique is a method of setting the camera 14 such that the left camera and the right camera face the same target point by a predetermined distance e as shown in FIG. On the other hand, the on-axis technique is a method of setting the distance between the cameras 14 by a predetermined distance e and setting the two cameras 14 in parallel as shown in FIG.

In the visualization part of the stereoscopic image, the two methods are not significantly different, but in the implementation process, the difference in the stereoscopic effect is different depending on how the target point is taken in the off-axis method. On the other hand, the on-axis technique calculates the visual direction of the camera 14 in parallel axes so that the position of the camera 14 can be easily calculated in one parallel movement operation. The effect of stereoscopic images does not change. Therefore, the present invention uses a relatively simple on-axis technique in consideration of the execution speed.

Meanwhile, the principle of controlling the depth of field of the stereoscopic image by the rendering module 20 is as follows.

In general, when the aperture is very small, such as the pinhole camera 14, all three-dimensional points correspond one-to-one to one point in the image plane. However, the larger the size of the aperture that accepts light like the lens of the human eye, the three-dimensional points do not correspond one-to-one.

3 is a conceptual diagram illustrating a confusion circle appearing at an image plane of a point B farther from the focal plane. As shown in FIG. 3, points at focused distances, such as point A, are mapped to the image plane one-to-one, such as A ', but larger than one point if closer or farther away, such as point B. You can see that it is mapped to the circle (B '). Thus, a point of an object that corresponds to pixels forming a circle on an image is called a confusion circle.

The rendering module 20 of the present invention adjusts the depth of field of the pixels of the confusion circle. When the depth of field of all the pixels of the confusion circle affects the pixels corresponding to the focal plane of the calculated stereoscopic image, the depth of focus is adjusted. The points of all locations except the larger the relative distance, the larger the size of the confusion circle has a problem that the amount of calculation is too large. That is, in order to calculate the color of the desired sampling point, the rendering module 20 needs to calculate the weight according to the distance by loading the color of the numerous points around and the calculation amount increases exponentially.

On the contrary, when the rendering module 20 does not consider the confusion circle around the pixel corresponding to the calculated focal plane of the stereoscopic image, the image quality is deteriorated and serious color-bleeding occurs in the resultant image.

Therefore, the rendering module 20 of the present invention automatically adjusts the depth of field of the confusion circle of pixels located within a set number of pixels corresponding to the calculated focal plane of the stereoscopic image at set time intervals. That is, the rendering module 20 of the present invention adjusts the depth of field by sampling a smaller number of pixels as pixels closer to the pixel corresponding to the focal plane among the confused circles, and pixels far from the pixel corresponding to the focal plane among the confused circles. As the number of pixels is increased, the depth of field is adjusted by adjusting a depth of field by sampling a larger number of pixels.

According to this principle, the rendering module 20 of the present invention is replaced with an appropriately blurred image according to a circle of confusion located within a set number of images to be visualized for both eyes of a user. Since it can be expressed by the human eye, it is possible to create an effect in a three-dimensional image that is seen by the human eye.

4 is a block diagram illustrating a sampling method for implementing depth of field in a stereoscopic image of the present invention. As shown in Fig. 4, for example, the first confusion circle including the pixel closest to the pixel corresponding to the focal plane selects six (60 ° interval) pixels as sampling pixels to adjust the depth of field. The pixel closest to the pixel corresponding to the focal plane is a pixel of confusion circle, preferably corresponding to three pixels away from the focal plane.

The second confusion circle including the second closest pixel, that is, four pixels away from the pixel corresponding to the focal plane, selects 12 pixels (30 ° intervals) as sampling pixels to adjust the depth of field.

And the third, fourth, and fifth confusion circles, including the eighth pixel, from the third pixel closest to the pixel corresponding to the focal plane (5 pixels away from the pixel corresponding to the focal plane). A depth of field is adjusted by selecting a pixel of (15 ° intervals) as the sampling pixel. That is, the sampling pixels of the first to fifth congestion sources are commonly located in the virtual radiation when the virtual radiation is drawn in the direction of the fifth confusion circle in the focal plane.

By this principle, the rendering module 20 may adjust the depth of field of pixels corresponding to the focal plane of the stereoscopic image as effectively as possible while minimizing the deterioration of image quality for real-time stereoscopic image generation. If you sample too many confusion circles, the rendering time will be longer. If you sample too few confusion circles, the rendering time will be shorter but the image quality will be lower. Therefore, the optimal number of confusion circles and the number of sampling pixels are selected. .

Hereinafter, an automatic depth of field control method for stereoscopic visualization of the present invention will be described.

5 is a flowchart illustrating an automatic depth of field adjustment method for stereoscopic visualization of the present invention. As shown in FIG. 5, first, the LED lighting 12 of the eye tracking device generates a reflection point by irradiating and reflecting infrared rays to both eyes of a user (S100). Subsequently, the camera 14 of the eye tracking device captures the reflection points at set time intervals to determine the positions of two eyes of the user (S102), and the controller 16 of the eye tracking device detects the pupil and the cornea at the positions of the two eyes. Separate and zone the separated pupil (S104). Next, the control unit 16 of the eye tracking device extracts the center position of the pupil using the search window (S106), and the calculation unit 18 of the eye tracking device uses the center position of the pupil determined at a predetermined time interval. The distance between the two eyes is calculated (S108). Subsequently, the calculation unit 18 of the eye tracking device calculates a focal length of the user using the calculated distance (S110) and transfers it to the rendering module 20. Here, when the calculation unit 18 calculates the focal length of the user according to the distance between the two eyes, the focal length for each distance between the two eyes of the user measured in advance is used.

Next, the rendering module 20 grasps a confusion circle of pixels located within a set number of pixels corresponding to the focal plane of the stereoscopic image calculated by the calculation unit 18 of the eye tracking device (S112). 20) samples the set number of pixels from the identified confusion circle and adjusts the depth of field according to the focal length (S114). At this time, the rendering module 20 adjusts the depth of field by sampling a smaller number of pixels closer to the pixel corresponding to the focal plane among the confused circles, and increasing the number of pixels farther from the pixel corresponding to the focal plane among the confused circles. Adjust the depth of field by sampling the pixels.

Hereinafter, with reference to Figure 6 will be described the experimental results of the automatic depth of field control method for stereoscopic image visualization of the present invention.

A 3.0 GHz Intel Pentium PC equipped with an nVIDIA GeForce7600GS ^™ graphics card was used to experiment with a depth-of-field automatic adjustment system for GPU-based stereoscopic images.

The depth of field image quality was compared with different sampling numbers. First, if the depth of field is adjusted only for 12 sampling pixels around the pixel corresponding to the focal plane, the rendering speed exceeds 100 fps on average, but severe color-bleeding occurs like a border around the mesh. It was confirmed that a white circle was formed.

On the other hand, when the depth of field of the confusion circle is adjusted for all points around the pixel corresponding to the focal plane, it was confirmed that the expression of the depth of focus was very smooth. However, considering all confusion circles for all points, the computational amount of pixels increases and the rendering speed of the rendering module 20 is less than 1 fps due to the limitation of the number of instructions of the pixels.

Lastly, when the depth of field is adjusted for a sampling pixel of a confusion circle in the set number range around the pixel using the method proposed by the present invention, a smooth image of excellent image quality is guaranteed while guaranteeing an average speed of about 30 fps. I could confirm it.

Figure 6 is a graph comparing the experimental results of the experimenters for the three-dimensional image according to the present invention. As shown in FIG. 6, in the present invention, in order to measure the degree of side effects reduction when viewing 3D images, a survey was performed after 10 users experienced the system proposed in the general 3D image and the invention, respectively. As a result, most of the experimenters felt more dizziness when not using the system of the present invention, and because of this, when using the method of the present invention, more immersive feelings than general stereoscopic images could be felt.

The present invention described above may be variously modified or applied by those skilled in the art, and the scope of the technical idea according to the present invention should be defined by the following claims.

Claims

In the automatic depth of field control method for stereoscopic visualization,

(a) calculating a focal length at which a user views a stereoscopic image by determining a distance between pupils of two eyes of a user at a predetermined time interval;

(b) calculating a focal plane based on the calculated focal length; And

(c) automatically adjusting the depth of field of sampling pixels included in a confusion circle of pixels located within a set number based on the calculated pixels of the stereoscopic image corresponding to the focal plane;

Automatic depth of field control method for stereoscopic image visualization comprising a.
According to claim 1, wherein step (a) is,

(a-1) identifying the center positions of the pupils of the two eyes of the user at predetermined time intervals; And

(a-2) calculating the distance between the two eyes using the center position of the pupil determined at the set time interval and calculating the focal length of the user by using the calculated distance; Automatic depth of field adjustment method for.
The method of claim 2, wherein step (a-1) comprises:

(1) irradiating and reflecting electromagnetic waves to the two eyes to determine positions of the two eyes through reflection points;

(2) separating the pupil and the cornea at the identified two eye positions;

(3) zoning the separated pupils; And

And (4) extracting the center position of the pupil using the search window.
The method of claim 2, wherein before step (a),

And measuring a focal length for each distance between the two eyes of the user so that the step (a-2) can be used to calculate the focal length of the user according to the distance between the two eyes. How to automatically adjust the depth of field for visualization.
The method of claim 2,

In step (a-2), the distance between the two eyes is calculated by the following equation, the automatic depth of field control method for stereoscopic image visualization.

Equation

Where M x is the sum of the x coordinates of the pupil area, M y is the sum of the y coordinates of the pupil area, P cx and P cy are the center coordinates of the pupil, and n is the number of pixels having the pupil area value in the search window, respectively. Indicates.
The method of claim 1, wherein in step (c),

The closer to the pixel corresponding to the focal plane among the confusion circles, the depth of field of the smaller number of sampling pixels is adjusted, and the farther from the pixel corresponding to the focal plane among the confusion circles, the larger depth of field Automatic depth of field adjustment method for stereoscopic image visualization, characterized in that for adjusting the.
The method of claim 1, wherein in step (c),

The constellation circle including the pixel closest to the pixel corresponding to the focal plane is sampled six pixels to adjust the depth of field, and the constellation circle including the second pixel closest to the pixel corresponding to the focal plane is 12 pixels. The depth of field is adjusted by sampling, and a confusion circle including a pixel corresponding to the focal plane and a third to set number of pixels automatically adjusts the depth of field by sampling 24 pixels. How to adjust the depth of field.
The method of claim 1,

The focal length is divided into three steps according to the distance range Automatic depth control method for stereoscopic image visualization, characterized in that.