TWI496110B - Method of Transforming 3D Image Video with Low Complexity Two - dimensional Video - Google Patents

Description

Method for converting 3D video video with low complexity 2D video

The invention relates to the technology of image processing, in particular to a method for converting 3D video images of low complexity 2D video.

Press, the biggest difference between 2D video and 3D video is whether there is a depth map. The depth map contains the relative distance between the objects in the 2D video and the viewer or the photographer, so the depth map is for 3D display. The presentation of the technology is very important.

Traditionally, there are two ways to generate image video depth information maps. One is to use the multi-lens camera to capture the same scene, and at the same time to obtain two or more scene view 2D video images, and then use the screen disparity to calculate The depth information map; the other is to generate the depth information map by the video image captured by the general camera, but generally the camera can only capture the single-view two-dimensional image at a time, so the method of calculating the depth of field using a single single-view two-dimensional image is used. The computational complexity is very high. For example, Battiato Sebastiano mentioned in the Depth map generation by image classification paper published in 2004 mainly includes several methods: Image Classification, Vanishing Point Detection, Mean-Shift Segmentation. ) and other technologies. These techniques all require high computational complexity. For example, Dorin Comaniciu mentioned the Mean-Shift operation in the Mean shift analysis and applications published in 1999, which requires the use of integrals and exponential and logarithmic operations, which also requires high computational complexity. Therefore, the implementation of traditional technology in real-time applications is very difficult.

The main purpose of the present invention is to provide a low-complexity two-dimensional video image conversion method for three-dimensional image video, which can generate depth information not required for two-dimensional video video by low-complexity image analysis and processing technology, and then two-dimensional Video video is converted to 3D video.

A further object of the present invention is to provide a low-complexity two-dimensional video image conversion method for three-dimensional image video, which can greatly reduce the computational complexity, and can be adapted to hardware or embedded system software to realize instant stereoscopic video generation. And can maintain the good quality of 3D video.

In order to achieve the foregoing objective, a low-complexity two-dimensional video image conversion method for converting three-dimensional video images according to the present invention is to sequentially convert each picture of a video video, and the method includes the following steps: a) determining Edge feature points: determine whether each pixel (pixel) in a picture is an edge feature point; b) find the vanishing line: perform operations on each pixel that is an edge feature point, and when computing a pixel, The pixel is centered, and the adjacent pixels which are also edge feature points around the pixel are found, and the respective line information after being connected with the pixel located at the center is stored, and then the line information is passed according to the edge The number of feature points is sorted, and at least the first two lines of the most edge feature points are defined as a vanishing line; c) the picture is classified: whether the edge feature point density in the picture is greater than a density threshold value, And determining whether the number of vanishing lines in the picture is greater than a threshold value, and if both are greater than the threshold value, determining that the picture is close to taking a picture The sign of the sign, then use the color to calculate the ratio of the sky and the distant mountain in the picture. If it is greater than a landscape threshold, it is determined that the picture is a picture with a landscape feature; if not, the picture is determined to have a disappearing area. a picture of the feature; d) finding the disappearing area and generating a preliminary depth map: if the picture in step c) is judged to be a picture with the feature of the disappearing area, the technique of disappearing area detection is used to find out the disappearance of the picture The area, which mainly uses the intersection of all the vanishing lines as the vanishing point, and finds the most densely populated area in the picture, that is, defined as the disappearing area; if the disappearing area is outside the picture, it disappears according to the same The trend of the line sets the boundary of the most consistent trend of the picture as the disappearing area; if the picture is judged to be the picture with the scenic feature, the definition of the disappearing area is directly at the top of the picture; and then, according to each of the pictures a distance between the pixel and the disappearing region to generate a Gradient Depth Map (GDM); and then, according to the edge feature point of the image, The depth map performs a predetermined operation to enhance the edge of the object in the picture to generate a final depth information map; e) detecting the contrast and defining the depth of field: if the picture in step c) is judged to have a close-up feature The picture defines a plurality of blocks for the picture, each block has a plurality of pixels, and each of the blocks detects the contrast, thereby distinguishing whether the block belongs to a foreground object or a background object in the picture. And define the depth of the foreground object to the nearest depth of field, and the depth of the background object is the farthest depth of field to produce the final depth map. In this way, the computational complexity can be greatly simplified, and the 2D video can be converted into 3D video, and can be applied to hardware or embedded system software to realize real-time stereoscopic video generation, and can maintain good 3D video. quality.

In order to clarify the technical features of the present invention in detail, the following preferred embodiments are described with reference to the following description. In the following, since the present invention belongs to the technology of image processing, in the representation of the schema, the difference in depth of field and display image It is difficult to express it by mechanical drawing, please express it by image.

As shown in the first figure, a low-complexity two-dimensional video image conversion method for three-dimensional video video is provided in a first embodiment of the present invention, which sequentially converts each picture of an image video, and the method mainly has The following steps:

a) Judging edge feature points: determining whether each pixel (pixel) in a picture is an edge feature point. In the embodiment, when determining the edge feature point, the average value of the RGB (red, green, blue) format of the pixel and the surrounding eight pixels or the Y of the YUV (brightness, chroma, density) format is used. Value, through the vertical and horizontal mask array to find the horizontal value and vertical value, and add the absolute value of the horizontal value and the absolute value of the vertical value, and then compare this result with an edge threshold value, if greater than The edge threshold value determines that the pixel belongs to an edge feature point. Since the conventional method is to calculate the square root after the sum of the squares of the horizontal value and the vertical value, such calculation complexity is high, so the calculation method of the present invention greatly simplifies the conventional computational complexity by about 65%, and still maintains Nearly the same results as the conventional calculation method.

b) finding the vanishing line: performing operations on each of the pixels that are edge feature points, and when computing the pixel, centering on the pixel, finding adjacent pixels that are also edge feature points around the pixel, and The line information after being connected with the pixel located at the center is stored, and the line information is sorted according to the number of edge feature points passing through it, and the two lines passing through the most edge feature points are defined as vanishing lines ( Vanishing line). In the present embodiment, as shown in the second figure, the adjacent pixels refer to eight pixels around the pixel and sixteen pixels around the outer layer, and the 5x5 block Hoff is used. The conversion method (5x5 Block Hough Transform) is used to find the vanishing line. In the actual operation, taking the second figure as an example, the black point S in the second figure is the currently processed pixel and is the edge feature point. First, whether the white area (A position) in the second picture has edge features is found. Point, then look for the gray area (X position and Y position) whether there are edge feature points, the figure shows that the A position, the X position and the Y position have edge feature points, and at this time, the black point S is transmitted. The Hough transform method calculates the straight line information from 147 degrees to 168 degrees and records it. This approach reduces computational complexity by approximately 56% compared to the unsimplified Hough transform method.

c) classifying the picture: determining whether the edge feature point density in the picture is greater than a density threshold value, and determining whether the number of vanishing lines in the picture is greater than a threshold value, and if both are greater than the threshold value, determining the The picture is a picture with a close-up feature, as shown in the third picture (C); then the color is used to calculate the ratio of the sky to the distant mountain in the picture. If it is greater than a landscape threshold, the picture is judged to have a landscape feature. The picture is as shown in the third figure (B); if it is not the above, it is judged that the picture is a picture having the feature of the disappearing area, as shown in the third figure (A).

d) finding the disappearing area and generating a preliminary depth map: if the picture in step c) is judged to be a picture with the characteristics of the disappearing area, the technique of disappearing area detection is used to find the disappearing area of the picture, which is mainly The intersection point of all the vanishing lines is used as the vanishing point, and the area with the most dense vanishing point is found in the picture, which is defined as the disappearing area; if the disappearing area is outside the screen, then according to the point of 8x8 square pixels The trend of the disappearance line is set as the disappearing area of the most suitable trend boundary of the picture; if the picture is judged to be the picture with the landscape feature, the directly defined disappearing area is located at the top of the picture; and then, according to the picture The distance between each pixel and the vanishing area produces a Gradient Depth Map (GDM). The fourth figure (A) shows a preliminary depth map corresponding to the third figure (A), and the fourth figure (B) shows a preliminary depth map corresponding to the third figure (B); wherein the picture with the landscape features is mostly The farthest place will fall at the top of the picture, for example: sky or distant mountains, so the disappearing area of the picture with landscape features will be at the top of the preliminary depth map, refer to the third picture (B) and fourth Figure (B); Next, a predetermined operation is performed on the preliminary depth map according to edge feature points of the picture, thereby enhancing the edge of the object in the picture to generate a final depth information map. The foregoing predetermined operation, in this embodiment, refers to the operation using Joint Bilateral Filtering (JBF), but does not use an edge stop function. Since the conventional joint bilateral filtering method uses the edge stop function to detect edges, the present invention does not need to use this function to detect edges, thereby reducing computational complexity by 26%. The final depth information map of the picture with the disappearing area feature and the picture with the landscape feature is as shown in the fifth figure (A) and the fifth figure (B).

The embodiment further includes a step d1) for cutting: the cutting is performed by a cutting method, which mainly groups pixels of similar colors, and generates a cutting action on the image to detect the object. And adjusting the preliminary depth map according to the result of the cutting, so that the depth values of the same object in the picture are consistent in depth information. The aforementioned cutting method is a 4-bit Removing Segmentation method, as shown in the sixth figure, which removes the four lower bits of the RGB octet value of each pixel, and The remaining four bits are used for color comparison, and the pixels with similar colors are grouped together. In the sixth figure, the upper part displays the octet value of each pixel RGB, and the lower part shows the removal of the four low level values. After that, the four-bit value is grouped; since the commonly used cutting algorithm is Mean-shift Segmentation, a very high amount of calculation is required, and the four-bit removal used in the present invention is removed. In addition to the effect of object segmentation, the cutting method can reduce the computational complexity of about 99.8% compared to the conventional cutting method described above. This step d1) may also first cut the picture before step a), and then adjust it after proceeding to step d) to generate the preliminary depth map.

e) detecting the contrast and defining the depth of field: if the picture in step c) is judged to be a picture having a near-photographing feature, a plurality of areas are defined for the picture, each of the blocks having a plurality of pixels, each of the blocks Detecting the contrast, and then distinguishing whether the block belongs to the foreground object or the background object in the picture, and defines the depth of the foreground object as the nearest depth of field, and the depth of the background object is the farthest depth of field to generate the final depth. Information map. The final depth information map of the picture with the near-photographing feature is as shown in the fifth figure (C). The aforementioned formula for calculating the contrast is expressed by the following formula (1).

Contrast=(I _max -I _min )/(I _max +I _min ) Equation (1)

Where I _max is the maximum brightness of the pixels in the block range, and I _min is the minimum value of the brightness.

It can be seen from the above steps that the present invention is simplified in many calculation methods, thereby greatly reducing the computational complexity. Thus, the present invention can generate two-dimensional images by low complexity image analysis and processing techniques. The depth information not available in the video and video, and then the 2D video is converted into 3D video.

In addition, the invention can greatly reduce the computational complexity, and can be adapted to hardware or embedded system software to realize real-time stereoscopic video generation, and can maintain good quality of 3D video.

The first figure is a flow chart of a preferred embodiment of the present invention.

The second drawing is a schematic view of a preferred embodiment of the invention showing the state of finding the vanishing line.

The third figure (A) is an image example diagram of a preferred embodiment of the present invention, showing a picture having a feature of a disappearing area.

The third figure (B) is an image example diagram of a preferred embodiment of the present invention, showing a picture with landscape features.

The third diagram (C) is an image example diagram of a preferred embodiment of the present invention, showing a picture with a near-photographing feature.

Figure 4 (A) is a schematic illustration of a preferred embodiment of the present invention showing a preliminary depth map having features of vanishing regions.

Figure 4 (B) is a schematic view of a preferred embodiment of the present invention showing a preliminary depth map with landscape features.

Figure 5 (A) is a schematic view of a preferred embodiment of the present invention showing a final depth information map having features of vanishing regions.

Figure 5 (B) is a schematic view of a preferred embodiment of the present invention showing a final depth information map having landscape features.

Figure 5 (C) is a schematic view of a preferred embodiment of the present invention showing a final depth information map with near-photographing features.

The sixth figure is a schematic diagram of the present invention in accordance with a preferred embodiment, showing the state in which the pixels are grouped.

Claims (7)

A low-complexity two-dimensional video video conversion method for three-dimensional video video is to sequentially convert each picture of a video video. The method includes the following steps: a) determining edge feature points: determining each of the pictures Whether the pixel is an edge feature point; b) finding the vanishing line: performing operations on each pixel that is an edge feature point, and when computing the pixel, centering on the pixel, finding the same around the pixel Is the adjacent pixel of the edge feature point, and stores each line information after the line is connected with the pixel located at the center, and then sorts the line information according to the number of edge feature points passed through it, and passes the most edge At least the first two lines of the feature point are defined as a vanishing line; c) classifying the picture: determining whether the edge feature point density in the picture is greater than a density threshold, and determining whether the number of vanishing lines in the picture is greater than A quantity threshold value, if both are greater than the threshold value, it is determined that the picture is a picture with a near-photographing feature, and then the color is used to calculate the picture If the ratio of the sky to the distant mountain is greater than a threshold value, the picture is judged to be a picture with a landscape feature; if not, the picture is determined to be a picture with a disappearing area feature; d) the missing area is found and generated Preliminary depth map: If the picture in step c) is judged to have a feature of the disappearing area feature, the technique of disappearing area detection is used to find the disappearing area of the picture, which is mainly to use the intersection of all the vanishing lines. Is the vanishing point, and finds the most densely populated area in the picture, that is, defined as the disappearing area; if the disappearing area is outside the picture, the most consistent boundary of the picture according to the trend of the vanishing line Set to the disappearing area; if the picture is judged to be a picture with a landscape feature, directly define the disappearing area at the top of the picture; and then generate a preliminary depth map according to the distance between each pixel in the picture and the disappearing area. (Gradient Depth Map, GDM); then, according to the edge feature points of the picture, a predetermined operation is performed on the preliminary depth map, thereby being in the picture The edge of the object is enhanced to generate a final depth information map; e) detecting the contrast and defining the depth of field: if the picture in step c) is judged to be a picture having a near-photographing feature, then a plurality of blocks are defined for the picture, Each of the blocks has a plurality of pixels, and each of the blocks detects the contrast, thereby distinguishing whether the block belongs to a foreground object or a background object in the picture, and defines a depth of the foreground object to the nearest depth of field, the background. The depth of the object is the farthest depth of field to produce the final depth map. The method for converting a three-dimensional image video of low complexity two-dimensional video image according to claim 1 of the patent application, wherein: in step a), when determining the edge feature point, a pixel and a surrounding pixel thereof are used The average value of the RGB (red, green, blue) format or the Y value of the YUV (brightness, chrominance, density) format, the horizontal and horizontal mask arrays are used to find the horizontal and vertical values, and the level is The absolute value of the value is added to the absolute value of the vertical value. If it is greater than the edge threshold value, it is determined that the pixel belongs to the edge feature point. The method for converting a three-dimensional video image of a low-complexity two-dimensional video image according to the first aspect of the patent application, wherein: in the step b), the adjacent pixel refers to eight pixels around the pixel. And sixteen pixels around the outer layer, and use the 5x5 Block Hough Transform to find the vanishing line. The method for converting a three-dimensional video image of a low-complexity two-dimensional image according to claim 1 of the patent application, wherein: in step d), when the disappearing region of the image is found by using a technique of disappearing region detection, In the range of 8x8 square pixels, the most densely populated area is found in the picture, which is defined as the disappearing area. The method for converting a three-dimensional video image of a low-complexity two-dimensional image according to claim 1 of the patent application, wherein: in the step d), the predetermined operation is performed by using a joint bilateral filtering method (Joint Bilateral Filtering, JBF) to perform the operation, but does not use the edge stop function. The method for converting a three-dimensional video image of a low-complexity two-dimensional image according to claim 1 of the patent application, wherein: further comprising: performing a step d1): cutting the image by a cutting method, the main The pixels with similar colors are grouped together, and a cutting action is performed on the image to detect the object, and then the preliminary depth map is adjusted according to the result of the cutting, so that the depth value of the same object in the image is in the depth information. It is consistent. The method for converting a three-dimensional image video of low complexity two-dimensional video image according to claim 6 of the patent application scope, wherein: in step d1), the cutting method is a four-bit removal cutting method (4-bit Removing) Segmentation) removes the four lower bits of the RGB octet value of each pixel, and compares the colors with the remaining four bits, and groups the pixels with similar colors accordingly.