KR100439341B1 - Depth of field adjustment apparatus and method of stereo image for reduction of visual fatigue - Google Patents

Abstract

1. TECHNICAL FIELD OF THE INVENTION

The present invention relates to an apparatus and method for adjusting depth of focus of a stereo image for reducing visual fatigue.

2. The technical problem to be solved by the invention

The present invention obtains the disparity information from the images acquired by two cameras, and filters the region out of the depth of field range of the human being based on the disparity information, thereby visualizing the time generated when viewing the stereoscopic image. To provide a device for adjusting the depth of focus of a stereo image for reducing fatigue and a method thereof and a computer-readable recording medium recording a program for realizing the method.

3. Summary of Solution to Invention

According to an aspect of the present invention, there is provided an apparatus for adjusting a depth of field of a stereo image for reducing visual fatigue, the apparatus comprising: image input means for receiving a stereo left and right image captured by a camera; A disparity map generating means for selecting a reference image to perform depth of focus filtering among the stereo left and right images, and calculating a disparity of the target image based on the selected reference image to generate a disparity map of the reference image; Image filtering means for performing filtering to blur an area outside the depth of focus of the reference image based on the disparity map; And image output means for synthesizing the reference image filtered through the image filtering means and the unfiltered target image transmitted through the image input means and outputting the three-dimensional image.

4. Important uses of the invention

The present invention is used in the depth of focus adjustment device for three-dimensional image display.

Description

Depth of field adjustment apparatus and method of stereo image for reduction of visual fatigue

The present invention relates to an apparatus for adjusting the depth of field of a stereo image for reducing visual fatigue, and a method and a computer readable recording medium recording a program for realizing the method. An apparatus for adjusting the depth of focus of a stereo image, and a method and a program for realizing the method, for reducing visual fatigue caused by a large value variation when displaying a stereo movie acquired by a camera in three dimensions. The present invention relates to a recording medium which can be read by.

Due to the development of computer technology and digital broadcasting technology, research on 3DTV (3D TV) broadcasting technology has been conducted recently. The configuration and principle of the three-dimensional broadcasting system will be briefly described by taking an eyeglass type 3DTV broadcasting system as an example.

First, images are usually acquired using two cameras. At this time, each camera may be arranged to arrange the two cameras in parallel or to converge the two cameras to one point on the three-dimensional according to the type of the camera device. The two images captured at two different positions are recorded on a video cassette recorder (VCR), or multiplexed through a multiplexer, encoded, transmitted, and then received through a receiver. Display sequentially or overlapping on one monitor or projector screen so that viewers can feel the depth when wearing glasses and watching images. Depth information felt by humans is disparity, which is a difference between the positions of one point on the left and right stereo images, which are generated when the same point on the 3D is photographed by the left and right stereo cameras and the positions projected on the left and right images are different from each other. It is synthesized in the human brain and feels three-dimensional depth.

However, in a real environment, as shown in FIG. 1, the human visual system has a limited depth of field, which is a range that can be naturally seen without degrading the quality of the scene image observed. When the observer is out of a certain distance in the forward or backward direction from another object's three-dimensional space position, the human visual system will blur the object. In addition, the pupil of human vision sees the angle adjusted toward the center of the object being fixed.

On the other hand, the current binocular stereoscopic stereoscopic display technology requires the viewer to watch a three-dimensional image by separating the focus and gaze point. That is, as shown in FIG. 1, the viewer's eyes focus on the display screen to view the focused image regardless of the position currently being viewed. As a result, when the displayed 3D image is an image in which all regions are not obscured at all, an image of an object having a position (depth) in a 3D space outside the depth of focal depth that can be viewed without deterioration in the viewer's retina. Even a haze is not formed, which is not suitable for the natural human visual environment, causing visual fatigue. This is called accommodation-convergence correspondence breakdown.

In order to solve this problem and to view comfortable 3D images, a method of limiting the range of change of depth information or keeping the viewer away from the screen has been proposed, but there are limitations in applying them to various 3D image situations. Then in 1995 The research presented by the study shows that this problem can be solved by mimicking the human visual system described above, and based on this, the study of depth of field simulation by Blohm et al.

Their research has shown that stereoscopic fatigue problems can be improved by adjusting the depth of field, but there is a new problem of average image quality deterioration.

The present invention has been proposed to solve the above problems, and obtains variation information from an image acquired by two cameras, and based on the variation information, an area outside the depth of field of a human being. The present invention provides a device for adjusting the depth of focus of a stereo image for reducing visual fatigue caused by viewing a stereoscopic image, and a computer readable recording medium having recorded thereon a program for implementing the method. There is this.

1 is an exemplary diagram showing a difference between a gaze position and a focusing position of a conventional human visual system.

2 is a configuration diagram of an apparatus for adjusting the depth of focus of a stereo image for reducing visual fatigue according to the present invention;

3 is an exemplary diagram illustrating a logistic function graph of a function representing a change in a filter size and a coefficient according to a variation value used in FIG. 2.

* Explanation of symbols for the main parts of the drawings

21: stereo image input unit 22: camera correction unit

23: disparity map generator 24: image filtering unit

25: video output unit 26: gaze detection unit

According to an aspect of the present invention, there is provided an apparatus for adjusting a depth of field of a stereo image for reducing visual fatigue, the apparatus comprising: image input means for receiving stereo left and right images captured by a camera; A disparity map generating means for selecting a reference image to perform depth of focus filtering among the stereo left and right images, and calculating a disparity of the target image based on the selected reference image to generate a disparity map of the reference image; Image filtering means for performing filtering to blur an area outside the depth of focus of the reference image based on the disparity map; And image output means for synthesizing the base image filtered through the image filtering means and the unfiltered target image transmitted through the image input means and outputting the three-dimensional image.

The present invention may further include image correction means for correcting the stereo left and right images through epipolar line alignment, and transmitting the corrected stereo left and right images to the disparity map generation means.

The present invention automatically detects a region of interest (viewing region) of an observer and inputs a detection result to the image filtering means to change the region to be adaptively filtered as the object of interest changes. The apparatus further comprises a gaze detection means.

The present invention also provides a method for adjusting the depth of focus of a stereo image for reducing visual fatigue applied to an apparatus for adjusting a depth of field of a stereo image, the method comprising: receiving a stereo left and right image captured by a camera; ; Selecting a reference image to perform depth of focus filtering among the left and right stereo images, and calculating a disparity of the target image based on the selected reference image to generate a disparity map of the reference image; A third step of performing filtering to blur an area outside the depth of focus of the reference image based on the disparity map; And a fourth step of synthesizing the reference image filtered through the third step and the unfiltered target image transmitted through the first step and outputting the three-dimensional image.

The present invention may further include a fifth step of correcting the stereo left and right images through epipolar line alignment and transferring the corrected stereo left and right images to the disparity map generator.

The present invention provides a sixth method for automatically detecting a region of interest (viewing region) of an observer and inputting a detection result to an image filtering unit to change the region to be adaptively filtered as the object of interest changes. Characterized in that further comprises the step.

The present invention also provides a device for adjusting a depth of field of a stereo image having a processor, comprising: a first function of receiving a stereo left and right image captured by a camera; A second function of selecting a reference image to perform depth of focus filtering among the left and right stereo images and calculating a variation of the target image based on the selected reference image to generate a disparity map of the reference image; A third function of performing filtering to blur an area out of a depth of focus of the reference image based on the disparity map; And a program for realizing a fourth function of synthesizing the reference image filtered through the third function and the unfiltered target image transmitted through the first function and outputting the result as a 3D image. Provide the medium.

The present invention further provides a computer-readable recording medium having a program for further realizing a fifth function of correcting the stereo left and right images through epipolar line alignment and transferring the corrected stereo left and right images to the disparity map generator. Provide the medium.

The present invention provides a sixth method for automatically detecting a region of interest (viewing region) of an observer and inputting a detection result to an image filtering unit to change the region to be adaptively filtered as the object of interest changes. A computer readable recording medium having recorded thereon a program for further realizing the function is provided.

The stereo image input unit 21 according to the present invention directly inputs images acquired from two cameras or receives images recorded in a VCR. The disparity map generator 23 calculates and records the disparity information between the left and right stereo images as described above using a stereo image matching algorithm. At this time, if the epipolar lines are aligned, the left and right images are matched as they are. Otherwise, the epipolar line alignment is performed through the camera correction unit 22 to calculate the disparity information.

Here, in the calculation of the disparity information, two types of disparity information are obtained according to which image is selected as the reference image, and the image to be used as the input image by the image filtering unit 24 of the next step is also determined according to the reference image.

That is, when the variation of the right image is calculated based on the left image, the disparity map of the left image is obtained, and thus the image to be filtered becomes the left image. On the contrary, when the variation of the left image is calculated based on the right image, the right image is filtered. The selection of the reference image is freely determined by the user. The image filtering unit 24 filters the pixels whose disparity values of the corresponding pixels do not satisfy a predetermined criterion or range based on the detection result and the disparity information from the observer gaze detection unit 26 of the image previously selected by the user. If you exclude the object from the target and perform the disparity value-based filtering on the remaining pixels, and the disparity value is out of range of the human detectable disparity, the corresponding pixel is hazy depending on the deviation. To reduce the

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a configuration diagram of an apparatus for adjusting the depth of focus of a stereo image for reducing visual fatigue according to the present invention.

As shown in FIG. 2, an apparatus for adjusting depth of focus of a stereo image for reducing visual fatigue according to the present invention includes a stereo image input unit 21 for receiving stereo left and right images captured by a camera, and a stereo left and right image. A variation map generator 23 for generating a variation map of the reference image by selecting a reference image to perform focus depth filtering and calculating a variation of the target image based on the selected reference image, and based on the variation map An image filtering unit 24 for performing filtering that blurs an area out of the depth of focus of the image, and the reference image filtered through the image filtering unit 24 and the stereo image input unit 21 are transmitted. And an image output unit 25 for synthesizing the unfiltered target image and outputting the 3D image.

In addition, the apparatus for adjusting depth of focus of a stereo image for reducing visual fatigue according to the present invention corrects the stereo left and right images through epipolar line alignment, and transmits the corrected stereo left and right images to the disparity map generator 23. It further comprises a camera correction unit 22.

In addition, the apparatus for adjusting the depth of focus of the stereoscopic image for reducing visual fatigue according to the present invention automatically detects a region of interest (viewing region) of an observer, inputs a detection result to the image filtering unit 24, and observes. The apparatus further includes a gaze detection unit 26 configured to change an area to be adaptively filtered as an object changes.

Here, the disparity map generator 23 uses stereo matching among the stereo left and right images, and selects a reference image to be used as a reference for the disparity map to be calculated initially by the user only once, and then uses the disparity map for all the reference images only. Create

In addition, the disparity map generator 23 generates a disparity map by alternately changing the reference image to be used as a reference of the disparity map to be calculated in left and right at frame intervals determined by the user using stereo matching among stereo left and right images.

The image filtering unit 24 also determines an image to be used as an input image according to the reference image. That is, when the variation of the right image is calculated based on the left image, the disparity map of the left image is obtained, and thus the image to be filtered becomes the left image. On the contrary, when the variation of the left image is calculated based on the right image, the right image is filtered. The selection of the reference image is freely determined by the user.

In addition, the image filtering unit 24, based on the detection result and the variation information from the observer gaze detection unit 26 for the image previously selected by the user, the pixel whose variation value does not satisfy a certain reference or range Excludes from the filtering target and performs variation-based filtering on the remaining pixels so that if the variation is out of range of human-detectable variation, the corresponding pixel is dim according to the deviation. It will reduce fatigue.

That is, the image filtering unit 24 determines the filter parameter by using the following filtering function using the variation information as the independent variable and the filter size and the coefficient of the filter as the dependent variable.

(a) using the linear function y = ax + b; or

(b) using the quadratic function y = ax ² + bx + c; or

(c) using the exponential function y = a * exp (bx) + c; or

(d) logistic function y = a + (b / (1+ (x / c) ^d ))

(e) square root function

In addition, the image filtering unit 24 uses a disparity-based filtering method that depends on four types of pixel disparity information as follows.

(a) How to filter all pixels

(b) a method of filtering only pixels having a variation smaller than a variation threshold given by a user

(c) a method of filtering only pixels having a variation greater than a variation threshold given by the user

(d) A method of filtering only pixels that are smaller than the variation threshold 1 or greater than the variation threshold 2 given by the user (variation threshold 1 <transition threshold 2).

The operation of the apparatus for adjusting the depth of focus of the stereo image according to the present invention having the above structure will be described in detail as follows.

First, assuming that left and right stereo images captured by two cameras are recorded in the VCR, the analog video output signal from the VCR is received and converted into digital image data through an analog / digital converter, followed by the stereo image input unit 21. Enter each as Alternatively, when the VCR supports digital output, the VCR directly inputs the output signal to the stereo image input unit 21, respectively. At this time, flag data indicating whether or not camera correction is necessary is input together with image data.

Then, the stereo image input unit 21 analyzes the flag data and sends the input signal to the camera correction unit 22 when the camera correction is necessary. Otherwise, the stereo image input unit 21 directly sends the left and right images to the disparity map generator 23. The image transferred to the disparity map generator 23 calculates disparity information for each pixel according to a disparity extraction algorithm stored in advance. The disparity extraction algorithm is an area-based matching method that finds the two most similar pixels by calculating similarity between two regions in a window frame having a specific size and shape, and calculates the disparity therefrom and features of the image such as edges. The feature-based matching method finds and finds the variation between the features.Either method verifies the reliability of the obtained matching points, removes the incorrect matching points, and interpolates the pixels for which the mutations are not obtained. This is necessary. The disparity map thus obtained is sent to the image filtering unit 23 together with the original image.

As a result of the detection of the gaze detection unit 26, the image filtering unit 24 receives an image selected as the disparity map and the image to be filtered, and receives filtering conditions determined by the user.

For example, assuming that the variation range of the input stereo video ranges from -20 pixels to 20 pixels, and that the variation range that can be comfortably viewed by a real human eye ranges from -10 pixels to 10 pixels, comfortable viewing is possible. Pixels out of range should be filtered. Alternatively, when the variation range of the stereo video has a range of 0 to 20 pixels, pixels having more than 10 pixels of variation should be filtered. In the opposite case, if the range is -20 to 0, pixels having a range of -10 or less should be filtered out.

On the other hand, the size or coefficient of the filter to be filtered depends on the variation information, and the size or coefficient of the filter should be determined so that the pixel having a large difference from the reference variation value is further deteriorated. Therefore, it is necessary to define a function that expresses a change in filter size and coefficient according to the variation value. These functions can be linear functions, quadratic functions, exponential functions, square root functions, and logistic functions. In the present invention, a logistic function as shown in FIG. 3 will be described as an example.

Logistic functions are defined as follows.

y = y_min + y_max / (1+ (x / middle)) ^slope

y_min and y_max determine the range of y-values that y-values have within a given range of x. In the present invention, the size of the filter to be filtered and the range of coefficients are determined. Middle and slope are parameters that determine the rate of increase or decrease of the logistic function. Middle is an x value where y is an average value of y_min and y_max. In the present invention, the reference variation value and the corresponding pixel Use x as the absolute value of the difference from the variation. The slope is the slope of the logistic function, where a large absolute value causes a sharp increase or decrease, while a small absolute value causes a gentle increase or decrease. If the sign of the slope is positive, y decreases as x increases, and if the sign of the slope is negative, y increases as x increases.

Filtering is performed for each pixel by using the filter size and the filter coefficient thus determined, and the resultant image is transferred to the image output unit 25 and displayed as a 3D image through the display unit along with the other unfiltered image. It is possible to enjoy a three-dimensional screen with less visual fatigue.

On the other hand, when it is difficult to extract the disparity map in real time, the disparity map may be obtained in advance for each frame offline, stored separately in a separate buffer or storage device, and then used for filtering by reading each frame.

In summary, two stereo images input from two cameras are epipolar line aligned in advance or through a camera correction unit 22, and an image to be subjected to depth depth filtering is selected. The disparity map generator 23 is used to extract disparity information for each pixel by using the image to be filtered as a reference image and the remaining image as a target image. The disparity map generator 23 finds the pixel position of the other image corresponding to each pixel on one image using a stereo image matching algorithm using the two stereo images, and then positions the two pixels. Compute the phosphorus variation. Next, blurring a pixel or an area having a deviation out of a predetermined range by using a disparity map extracted using the image as a reference image for a previously selected depth of field filtering. It is delivered to the image output unit 25 so that the viewer can observe the three-dimensional image.

As such, the difference between the prior art and the present invention is that the stereoscopic image is viewed by using a filtering method in which the object region of interest (the region currently observed by the observer) is left as it is and the remaining region is further hazy as the variation value increases. By reducing the visual fatigue phenomenon of the poetry, and by illuminating only one image, the image quality is not felt when observing the actual 3D image. In addition, the present invention further provides a gaze detection unit 26 capable of automatically detecting a region of interest or gaze of an observer, thereby inputting the detection result into the image filtering unit 24 so that an object to be gazed at. As you change, it will change the area that is adaptively filtered.

The method of the present invention as described above may be implemented as a program and stored in a computer-readable recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.).

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes are possible in the art without departing from the technical spirit of the present invention. It will be clear to those of ordinary knowledge.

As described above, the present invention obtains disparity map information from an image captured by two cameras, and filters an area that is outside the depth of field range of a human being based on the disparity map information. There is an effect that can reduce the visual fatigue that occurs during viewing.

In addition, the present invention, by filtering only one of the two images, there is an effect that can solve the conventional resolution degradation problem caused by filtering both images.

Claims (17)

In the apparatus for adjusting the depth of field of the stereo image for reducing visual fatigue, Image input means for receiving stereo left and right images captured by a camera; A disparity map generating means for selecting a reference image to perform depth of focus filtering among the stereo left and right images, and calculating a disparity of the target image based on the selected reference image to generate a disparity map of the reference image; Image filtering means for performing filtering to blur an area outside the depth of focus of the reference image based on the disparity map; And Image output means for synthesizing the filtered base image filtered through the image filtering means and the unfiltered target image transmitted through the image input means and outputting the 3D image. Depth adjustment device of the stereo image including a. The method of claim 1, Image correction means for correcting the stereo left and right images through epipolar line alignment, and transmitting the corrected stereo left and right images to the disparity map generation means. Device for adjusting the depth of focus of the stereo image further comprising. The method according to claim 1 or 2, Observation region detection means for automatically detecting a region of interest (viewing region) of an observer and inputting a detection result to the image filtering means to change the region to be adaptively filtered as the object of interest changes. Device for adjusting the depth of focus of the stereo image further comprising. The method of claim 3, wherein The variation map generating means, The stereo image is characterized by generating a disparity map over all frames only for the reference image after selecting a reference image to be used as a reference for the disparity map to be calculated, using stereo matching among stereo left and right images. Depth adjustment device. The method of claim 4, wherein The variation map generating means, An apparatus for adjusting the depth of focus of a stereo image, wherein a stereo map is generated by alternately changing a reference image to be used as a reference for the disparity map to be calculated, left and right at every frame interval determined by the user. The method of claim 5, wherein The image filtering means, An apparatus for adjusting the depth of focus of a stereo image, characterized in that the filter parameter is determined by using the following filtering function using the variation information as an independent variable and the filter size as a dependent variable. (a) using the linear function y = ax + b; or (b) using the quadratic function y = ax ² + bx + c; or (c) using the exponential function y = a * exp (bx) + c; or (d) logistic function y = a + (b / (1+ (x / c) ^d )) (e) square root function The method of claim 6, The image filtering means, An apparatus for adjusting depth of focus of a stereo image, comprising using a disparity-based filtering method that depends on four types of pixel disparity information as follows. (a) How to filter all pixels (b) a method of filtering only pixels having a variation smaller than a variation threshold given by a user (c) a method of filtering only pixels having a variation greater than a variation threshold given by the user (d) A method of filtering only pixels that are smaller than the variation threshold 1 or greater than the variation threshold 2 given by the user (variation threshold 1 <transition threshold 2). In the depth of field adjustment method for reducing visual fatigue applied to the depth of field adjustment device of the stereo image, A first step of receiving stereo left and right images captured by a camera; Selecting a reference image to perform depth of focus filtering among the left and right stereo images, and calculating a disparity of the target image based on the selected reference image to generate a disparity map of the reference image; A third step of performing filtering to blur an area outside the depth of focus of the reference image based on the disparity map; And A fourth step of synthesizing the filtered reference image through the third step and the unfiltered target image transmitted through the first step, and outputting the three-dimensional image; Method of adjusting the depth of focus of the stereo image comprising a. The method of claim 8, A fifth step of correcting the stereo left and right images through epipolar line alignment and transferring the corrected stereo left and right images to the disparity map generator; Focus depth adjustment method of the stereo image further comprising. The method according to claim 8 or 9, A sixth step of automatically detecting an observer's region of interest (viewing region) and inputting the detection result to the image filtering unit so as to change the region to be adaptively filtered as the object of interest changes; Focus depth adjustment method of the stereo image further comprising. The method of claim 10, The variation map generation process, The stereo image is characterized by generating a disparity map over all frames only for the reference image after selecting a reference image to be used as a reference for the disparity map to be calculated, using stereo matching among stereo left and right images. How to adjust the depth. The method of claim 11, The variation map generation process, A method of adjusting the depth of focus of a stereo image, characterized by generating a disparity map by alternately changing a reference image to be used as a criterion of the disparity map to be calculated as a reference for the disparity map to be calculated from the left and right images. The method of claim 12, The process of performing the filtering, A method of adjusting the depth of focus of a stereo image, characterized in that the filter parameter is determined using the following filtering function using the variation information as an independent variable and the filter size as a dependent variable. (a) using the linear function y = ax + b; or (b) using the quadratic function y = ax ² + bx + c; or (c) using the exponential function y = a * exp (bx) + c; or (d) logistic function y = a + (b / (1+ (x / c) ^d )) (e) square root function The method of claim 13, The process of performing the filtering, A method of adjusting the depth of focus of a stereo image, characterized by using a disparity-based filtering method that depends on four types of pixel disparity information as follows. (a) How to filter all pixels (b) a method of filtering only pixels having a variation smaller than a variation threshold given by a user (c) a method of filtering only pixels having a variation greater than a variation threshold given by the user (d) A method of filtering only pixels that are smaller than the variation threshold 1 or greater than the variation threshold 2 given by the user (variation threshold 1 <transition threshold 2). In the depth of field adjusting device of the stereo image having a processor, A first function of receiving stereo left and right images captured by a camera; A second function of selecting a reference image to perform depth of focus filtering among the left and right stereo images and calculating a variation of the target image based on the selected reference image to generate a disparity map of the reference image; A third function of performing filtering to blur an area out of a depth of focus of the reference image based on the disparity map; And A fourth function of synthesizing the filtered reference image through the third function and the unfiltered target image transmitted through the first function and outputting the synthesized image into a 3D image; A computer-readable recording medium having recorded thereon a program for realizing this. The method of claim 15, A fifth function of correcting the stereo left and right images through epipolar line alignment and transferring the corrected stereo left and right images to the disparity map generator; A computer-readable recording medium that records a program for further realization. The method according to claim 15 or 16, Sixth function for automatically detecting the region of interest (viewing region) of the observer and inputting the detection result to the image filtering unit to change the region to be adaptively filtered as the object of interest changes. A computer-readable recording medium that records a program for further realization.