KR101709974B1 - Method and System for Generating Depth Contour of Depth Map - Google Patents

Abstract

A method and system for creating a depth map fillet line is provided. The depth map isomorphism generation method according to an embodiment of the present invention quantizes the depth values of a depth map into a plurality of quantization steps and generates an isomorphism in a quantized depth map. Accordingly, in the process of converting the 2D image contents to the 3D image contents, the depth map isomorphism can be automatically generated, so that it is not necessary to manually draw the curve of the depth map boundary in the graphic tool. This makes it possible to greatly reduce the required manpower and to greatly improve the working speed.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and system for generating a depth map,

The present invention relates to 3D image processing, and more particularly, to a method of generating and utilizing a depth map necessary for converting a 2D image content to a 3D image content.

In the present situation where there is a serious lack of 3D image contents due to insufficient supply compared to the demand, the importance of 3D image conversion technology is increasing in terms of securing 3D image contents.

3D image conversion technology converts existing 2D images captured by analog or digital into content into 3D image content. Although the techniques of the conversion technique are various, in either method, the basic principle is to generate a depth map that estimates the depth of the monocular image by analyzing the monocular stereoscopic information included in the input 2D image and output the left and right images according to the time difference.

In order to maintain high image quality, the 3D image conversion technique is performed manually every frame, which requires a lot of manpower and long working time. As a result, the productivity problem of 3D image contents is a major obstacle to the formation of the market .

Also, 3D image conversion work is very labor intensive. Precise conversion requires a lot of manpower and long working time because every frame of the image has to be manually manipulated during the work process. The more intricate 3D images the higher the intensity of these tasks.

3D image conversion of 2D image consists of the process of analyzing image information and separating object and background, foreground and background, and giving stereoscopic values to each object and background to produce stereoscopic image.

In the automatic method, separation of objects and backgrounds and assignment of depth values are performed automatically, whereas in manual work, manual work is performed based on know-how of the technician. There are three ways to do this: auto-converting, semi-auto, and fully manual.

Autoconverting is cost-effective because it converts automatically when a conversion device or software is installed, but the stereoscopic quality is very low. The manual method, which is a manual method, can obtain a very good stereoscopic effect, but it has a disadvantage that it requires a lot of manpower, time and cost.

The conversion of the existing movie into the 3D image is performed in a completely manual manner, in which most of the operations are performed frame by frame. There are many conversion companies that work with semi-automatic or hybrid conversion system, which is the middle between auto conversion and manual mode, and it is time to convert differently depending on whether the ratio of workload is more automatic or manual .

In the future, a system that automatically converts 3D images will develop, but the auto-conversion method is expected to disappear only as a temporary measure to stop the absence of 3D image contents. A high-quality conversion process requires the development of a variety of efficient technologies that can take advantage of the manual approach and overcome the drawbacks.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method and apparatus for generating a bezel from a depth map having a continuous depth value in a process of converting a 2D image content to a 3D image content, .

According to an aspect of the present invention, there is provided a method of generating a depth map of a depth map, the method comprising: quantizing depth values of a depth map into a plurality of quantization steps; And generating a bezel in the quantized depth map.

The quantization step may linearly or nonlinearly quantize the depth values of the depth map.

According to another aspect of the present invention, there is provided a method of generating a depth map of a depth map, the method including removing a feather at a boundary of a quantized depth map.

According to another aspect of the present invention, there is provided a depth map creating method for a depth map, the method comprising: setting a region having a region size smaller than a critical size in a quantized depth map as a feather region; And converting the feather region to a depth value of a surrounding region having a smallest depth value difference.

According to another aspect of the present invention, there is provided a method of generating a depth map isometric line including extracting inflection points of the bezel; And converting the curve between the inflection points into another type of curve.

According to another aspect of the present invention, there is provided a computer-readable recording medium having a plurality of quantization steps for quantizing depth values of a depth map, And generating an isomorphism in the quantized depth map. A program for performing a depth map isomorphism generation method is recorded.

As described above, according to the embodiments of the present invention, it is possible to automatically generate a depth map beautifying line in the process of converting a 2D image content to a 3D image content, so that a curve of a depth map boundary It is not necessary to draw the image. This makes it possible to greatly reduce the required manpower and to greatly improve the working speed.

In addition, according to the embodiments of the present invention, all intervals of the depth map is converted into Bezier curves, which facilitates fine correction.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a flow chart provided in the description of a 3D content production method to which the present invention is applicable; Fig.
Fig. 2 is a diagram provided in a side view of Fig. 1,
Figure 3 is a flow chart provided in the description of a depth map isokinetic line generation method, according to an embodiment of the present invention;
4 is a diagram illustrating the result of depth map linear quantization,
5 is a diagram illustrating a result of a depth map nonlinear quantization,
6 is a diagram showing a feather of a quantized depth map,
7 is a diagram illustrating a result of extracting a feather region using a labeling technique in a quantized depth map,
8 is a diagram provided in the description of a technique for removing a feather region,
9 is a diagram illustrating a quantized depth map in which a feather region is removed,
FIG. 10 is a diagram showing masks generated for each quantization step in a quantized depth map,
FIG. 11 is a view showing isometric lines for each quantization step,
12 is a diagram showing a quantized depth map isosbene,
13 is a diagram showing a method of calculating the degree of bend in the isosbene,
14 is a diagram showing the result of extracting a final bending point from bending point candidates,
FIG. 15 is a view showing a depth map in which a bezier curve is reproduced,
16 is a block diagram of a 3D content production system according to another embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings.

1. 3D Content Production

1 is a flow chart provided in the description of a 3D content production method to which the present invention is applicable. The illustrated 3D content production method is a process of generating 3D content from 2D content.

As shown in FIG. 1, first, the 2D image content is input (S105) and divided into shots (S110). In step S110, the 2D image content is divided into a plurality of shots. Frames included in the same shot have the same / similar background.

The shot division in step S110 may be performed manually by a professional artist, or may be performed automatically using a program for dividing similar frames into shot units.

Thereafter, one shot is selected (S115), and a key-frame is selected from the selected shot (S120). The shot selection in step S115 can be automatically selected sequentially (i.e., the first shot is selected first and then the next shot is selected) according to the time order of the shot in the 2D image content, The order of selection may be determined in a different order. Hereinafter, it is assumed that the shot selection is sequentially performed in accordance with the time sequence of the shot.

In addition, it is possible to automatically select the temporally preceding frame (i.e., the first frame) among the frames constituting the key-frame selection shot in step S120 as a key-frame, Other out-of-frame frames may be selected as key-frames. Hereinafter, it is assumed that the key-frame is the first frame of the shot.

Next, in step S 120, a depth map for the selected key-frame is generated (S 125). The depth map generation in step S125 can be performed not only by the automatic generation using the program, but also by the manual operation of the professional artist.

Thereafter, the depth map of the depth map generated in step S125 is generated (S130). In step S130, the depth map is generated by quantizing the depth map using a plurality of quantization steps according to the depth using a program.

Next, the isoform created in step S130 is matched to the next-frame (step S135). If there is a motion object, the angle of view is shifted, and there is a portion of the keyframe that is different from the keyframe in which the next-frame is the previous-frame, the filler of the keyframe is not exactly matched to the next-frame.

Accordingly, the isosbian is partially moved (matched) so as to perfectly match the next-frame (S140). The isochronous moving process in step S140 is also automatically performed using the program. By step S140, a fully matched iso-line is generated in the next-frame.

Thereafter, in step S 140, a depth map of the next-frame is generated based on the matched bezel (S 145). The depth map of the next-frame generated in step S145 is a depth map of the quantized state.

Then, the depth map is interpolated to complete the depth map of the next-frame (S150). The depth map interpolation process in step S150 is also automatically performed using a program.

Steps S135 to S150 are repeated until all frames constituting the shot are completed (S155). That is, when the depth map for the second frame of the shot is completed, the isoquadrature of the second frame is matched to the third frame (S135), the isochronous match move is performed (S140), and based on the match- The depth map of the third frame is generated (S145), and the depth map is completed through the interpolation (S150), and the same operation is performed on the frames after the third frame.

When the depth map generation for all the frames constituting the shot is completed (S155-Y), steps S115 to S155 are repeated for the second shot, and depth map generation is performed for all the frames constituting the second shot Once completed, the same operation is performed for subsequent shots.

When the above procedure is completed for all the shots constituting the 2D image content (S160-Y), the 3D images of the frames constituting the 2D image content are converted using the depth maps generated so far (S165). Then, in step S165, the conversion processing result is output as the 3D image content (S170).

Fig. 2 is a view provided in a side view of Fig. 1. Fig. 2,

1) The first frame among the frames constituting the shot selected in step S115 is selected as a key frame (S120)

2) generating a depth map for the selected key-frame (S125)

3) Extract the isomorphism of the depth map (S130)

4) Match the extracted isosceles to the next-frame, perform match move (S135, S140)

5) A depth map of the next-frame is generated based on the match-moved bezel (S145)

6) interpolating the generated depth map and completing the depth map of the next-frame (S150).

2. Depth map  Crease

The process of generating the depth map of the depth map in step S130 shown in FIGS. 1 and 2 will be described in detail with reference to FIG. 3 is a flow chart provided in the description of a depth map isokinetic line generation method according to an embodiment of the present invention.

As shown in FIG. 3, first, a depth map of a key frame is received (S131), and linear quantization (S132) or non-linear quantization (S133) is performed on the input depth map. Depth values in the depth map input in step S131 are continuous while depth values of the depth map are quantized into a plurality of quantization steps by step S132 or S133.

The number of quantization steps to be applied to step S132 or step S133 may be specified according to requirements and specifications.

Thereafter, the faders existing in the boundary portions of the quantized depth map are removed (S134), and the flesh lines are extracted from the quantized depth map from which the feather is removed (S135). The isosceles extracted in step S135 are lines connecting pixels having the same depth value in the depth map.

Next, the bending points are extracted from the isosceles of the depth map (S136), and the curve between the bending points is converted into a Bezier curve (S137) to complete the depth map fillet. The inflection points extracted in step S136 refer to points having a large change in direction from the bezel.

In the conversion in step S137, the control points of the cubic Bezier curve are calculated using the total of four points by calculating the two inflection points and the midpoint between the 1/4 and 3/4 points therebetween, And transforming it into a Bezier curve using the calculated control points.

Hereinafter, the steps constituting Fig. 3 will be described in more detail.

3. Key-frame Depth map  Linear quantization

The key-frame depth map linear quantization is a technique of quantizing the depth map by dividing the depth value range of the depth map into the same size according to the number of quantization steps set by the user, and can be represented by the following equation (1).

(1)

(One)

Here, x represents the depth value of the input key-frame depth map, and x _Max and x _Min represent the maximum depth value and the minimum depth value of each depth map, respectively. And, step represents the number of quantization steps and? Represents the size of the quantization step. Finally, Q (x) represents the quantized depth value.

As expressed by the above equation (1), the size of the quantization step is calculated by grasping the maximum depth value and the minimum depth value in the depth map and dividing the difference by the number of quantization steps set by the user .

Using the above equation (1), it is possible to obtain a quantization depth map in which the range of the depth values that can be represented by the input key-frame depth map is divided into quantization steps of the same size. FIG. 4 shows a quantization depth map obtained by linear quantization and a histogram thereof. 4, (a) is an input key-frame depth map, (b) is a depth map in which the size of the quantization step is set to 10, and (c) It is a linear quantized depth map.

4. Key-frame Depth map  Nonlinear quantization

The key-frame depth map nonlinear quantization is a technique for quantizing the key-frame depth map by calculating the size of each quantization step according to the number of quantization steps set by the user, to an optimum size for the key-frame depth map.

A K-means clustering technique can be used to calculate the size of the optimized quantization step. The following (2) shows the expression of the key-frame depth map nonlinear quantization using the K-Means clustering technique.

(2)

(2)

Here, k represents the number of quantization steps and μ _i represents an average value of the i-th quantization step. Also, S _i denotes a set of pixels belonging to the i-th quantization step, and x _j denotes a depth value of a pixel belonging to the set. V represents the total dispersion.

The key-frame depth map nonlinear quantization divides the quantization step size equally according to the number of quantization steps set by the user, and sets the center value of each step as the initial average values of the quantization step. Then, each pixel of the depth map is compared with the initial average values, and the nearest value is found and classified into a set of the quantization steps. If the entire depth map is classified, the average value of the sets belonging to each quantization step is calculated again and updated to the average value of the quantization step. This process is repeated until the current average values and the updated average values do not change or the total variance value no longer decreases. When all the processes are completed, nonlinear quantization is performed by replacing the pixels belonging to the set of quantization steps with the average value of the set.

FIG. 5 shows a quantization depth map obtained by nonlinear quantization and a histogram thereof. 5, (a) is an input key-frame depth map, (b) is a non-linear quantized depth map by setting the size of the quantization step to 10, (c) It is a non-linear quantized depth map.

5. Quantized Depth map Feather  remove

At the boundary of the original depth map, there is a feather area for smooth change. The feather region appears as small regions at the boundary of the quantization depth map generated by the linear or nonlinear quantization technique.

FIG. 6 shows a feather of a quantized depth map. Fig. 6 (a) shows the original depth map, and Fig. 6 (b) shows the quantized depth map.

To extract the feather region from the quantized depth map, a labeling technique is used. The labeling technique is a technique for separating regions having the same value continuously into neighboring pixels in an image into one region. When the quantized depth map is divided into the respective regions using the labeling technique, the featherers are separated into one small region because they have different depth values from the surrounding depth values.

Since the feather region is a very small region unlike the other region than the feather region, the number of pixels belonging to the region is very small. Therefore, if the number of pixels belonging to the separated region by the labeling technique is smaller than a certain threshold value, it is extracted as a feather region.

FIG. 7 shows an example of extracting a feather region using a labeling technique in a quantized depth map. FIG. 7A shows a result of extracting the feather region by labeling, and FIG. 7B shows an enlarged view of the feather region.

8 is a diagram provided in the description of a technique for removing a feather region. In Fig. 8, the area A represents a feather area, and the area B and the area C represent non-feather areas (non-feather areas).

In order to remove the feather area, the depth value of the surrounding pixels of the area A, which is the feather area, is searched, and the difference value between the searched depth value and the depth value of the feather area is obtained. Then, the depth value of the feather region is replaced with the depth value of the neighboring pixel having the smallest difference value to remove the feather region.

A quantized depth map in which the feather region is removed by this technique is illustrated in FIG. FIG. 9A shows a quantized depth map in which a feathered area is not removed, and FIG. 9B shows a quantized depth map in which a feathered area is removed.

6. Quantization Depth map  Based bechle extraction

The quantization depth map based isilluminated line extraction is a process of generating a closed curve connecting a region having the same depth value in the quantization depth map from which the feather is removed.

To this end, as shown in FIG. 10, a mask is first generated for each quantization step in the quantized depth map. Next, as shown in FIG. 11, an outline is extracted from the mask generated for each quantization step. By doing this, it is possible to obtain the isochronous line for each quantization step.

Then, when the isomensis lines are integrated for each quantization step, a quantized depth map isomorphism can be obtained as shown in FIG.

7. Extraction of bending point in the bony line

The inflection point means a point where the change in direction from the bezoar line is large. Extraction points are extracted in order to distinguish each curve from the extracted bechymetry. Fig. 13 shows a method of calculating the degree of bend in the biceps, and Equation (3) below calculates the degree of bending of the reference pixels in the biceps.

(3)

(3)

Here, D represents the degree of bending of the reference pixel. And, FV represents the direction component distribution of the pixels before the reference pixel, and PV represents the direction component distribution of the pixels after the reference pixel. W _H and W _L represent the weights in the direction most similar to the direction of the reference pixel and the weight in the direction most different from the direction of the reference pixel, S denotes the number of pixels to calculate the direction distribution before and after the reference pixel, Represents the distance from the reference pixel. Finally, B _{i , d} has a value of 1 if the pixel at distance d is i-direction, or 0 otherwise. In order to obtain the degree of bending of one pixel of the bezel, it is possible to calculate the direction change distribution of the pixels before and after and calculate the difference of the direction of the reference pixel by obtaining the difference value of the distribution before and after. The degree of bend D is calculated for all the pixels in the bezel line, and when the value exceeds a certain threshold value, the pixel is determined as the bending point candidate.

The flexion point candidates appear consecutively from the beginning of the change in the wrist line. Therefore, the bending point candidate having the maximum bending degree in the continuous bending point candidates is extracted as the final bending point. Fig. 14 shows the result of extracting the final bending point from the bending point candidates. Specifically, Fig. 14 (a) shows the bending point candidates in green, and Fig. 14 (b) shows the final bending point in green.

8. Curve between inflection points Bezier  Curve transformation

The isosceles of each quantization step can be divided into respective curves with the extracted bending point as both ends, and these curves are converted into cubic Bezier curves, respectively. In the embodiment of the present invention, the bending point of the both ends and the midpoint between them are calculated, and the Bezier curve equation is estimated using these points. This gives the control points of the estimated Bezier curve. The following equation (4) represents the equation of the cubic Bezier curve.

(4)

(4)

Here, P represents the control point of the Bezier curve, and t represents the change period of the curve between 0 and 1. Finally, B (t) represents the point of the Bezier curve at time t. Equation (4) can be expressed in the form of a product of a T matrix representing t values, a P matrix representing control points, and M matrices representing coefficients of t as in equation (5). (6) can be expressed as the product of matrix multiplication.

(5)

(5)

(6)

(6)

As shown in the following equation (7), K, the matrix of the midpoints between the bending points of the both ends received by the input, can be considered as the x component and the y component separately. In the embodiment of the present invention, two values of 1/4 and 3/4 points are calculated as intermediate points, and a total of four points are used as inputs.

(7)

(7)

If the ratio is calculated using the values of the input points and the curved points at the beginning and end of the curve as shown in the following equation (8), the approximate t value of the input point can be obtained. The t values for all input points are obtained and expressed as a matrix, as shown in the following equation (9).

(8)

(8)

(9)

(9)

To create an optimal curve, we calculate the error of the Bezier curve by calculating the pixel position difference between the points of the actual isokinos and the restored points through the Bezier curve. Equation (10) shows the formula for obtaining the error with respect to the y component and also shows the formula summarized by the matrix calculation.

(10)

(10)

The value of the Py matrix when the error value is minimized in Eq. (10) becomes the y component of the control point of the optimal Bezier curve. Equation (11) shows the process of obtaining the value of Py based on the minimum error.

(11)

(11)

If we obtain the y component point of the optimal Bezier curve through calculation, we can obtain the optimal x component by changing the input to x component and repeating the same process. FIG. 15 shows a depth map regenerating the bezel using points of the converted Bezier curve.

9. 3D content production system

16 is a block diagram of a 3D content production system according to another embodiment of the present invention. 16, a 3D content production system 200 according to an exemplary embodiment of the present invention includes a 2D image input unit 210, a depth map generation unit 220, a 3D conversion unit 230, (240).

The depth map generator 220 generates a depth map of 2D image frames input through the 2D image input unit 210 using the key-frame depth map. The key-frame depth map is generated one by one for each shot in a 2D image, which can be generated both by a program and by a professional artist.

The depth map generator 220 extracts the isosceles of the key-frame depth map, matches the extracted isosceles to the next-frame and performs a match move, generates a depth map of the next-frame based on the match- Thereby generating depth maps for the 2D image frames by a procedure of completing the depth map of the next-frame.

The 3D conversion unit 230 performs 3D conversion processing on the 2D image frames using the depth maps generated by the depth map generation unit 220. The 3D image output unit 240 outputs the result of the conversion process performed by the 3D conversion unit 230 as a 3D image.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

200: 3D content production system
210: 2D image input unit
220: Depth map generation unit
230: 3D conversion unit
240: 3D image output unit

Claims (6)

Quantizing the depth values of the depth map into a plurality of quantization steps;
Setting a region having a region size smaller than a critical size in a quantized depth map as a feather region;
Converting the feather region into a depth value of a surrounding region having a smallest depth value difference;
Generating a bezel in the depth map transformed by the transforming step;
Extracting inflection points of the bezel; And
And converting a curve between the inflection points into another type of curve.
The method according to claim 1,
Wherein the quantization step comprises:
Wherein the depth map is generated by linearly quantizing or non-linearly quantizing the depth values of the depth map.
delete delete delete Quantizing the depth values of the depth map into a plurality of quantization steps; And
Setting a region having a region size smaller than a critical size in a quantized depth map as a feather region;
Converting the feather region into a depth value of a surrounding region having a smallest depth value difference;
Generating a bezel in the depth map transformed by the transforming step;
Extracting inflection points of the bezel; And
And converting the curved line between the curved points into another type of curved line. A computer-readable recording medium having recorded thereon a program for performing the method.

Images