TWI517091B - Method of 2d-to-3d depth image construction and device thereof - Google Patents

Description

Two-dimensional to three-dimensional depth image establishing method and device

The invention relates to a depth image establishing method, in particular to a two-dimensional to three-dimensional depth image establishing method and device.

Depth image refers to the distance information between an object and a viewer in a two-dimensional image. The image formed by projection onto a plane is usually represented by a gray scale diagram of 8 bits representing 256 levels. A value of 0 represents the farthest from the viewer, and a luminance value of 255 represents the closest to the viewer.

The current methods of generating depth images can be roughly divided into three main categories: 1) using a depth camera, 2) making a multi-view camera, and 2) performing a two-dimensional to three-dimensional image conversion of an image obtained using a single camera. The first two can only be used to capture new images. For existing old 2D images such as old photos and pictures, only the third category can be used.

Using the third category, the most important part is to perform edge detection, analyze the edge of the object, and give it the depth value. However, with the currently used edge detection method, it is applied to Chinese classical painting, due to the way of painting. It will make the blocks surrounded by the detected edges too broken and too small, so even the same object is often divided into multiple blocks, and accordingly Given the depth value, the effect is not very good.

In addition, the coloring methods and perspectives of Chinese classical paintings are different from those of Western paintings or modern photographs. The present invention will use the features commonly found in Chinese classical paintings to generate depth images to facilitate the reconstruction of virtual perspectives at the back end. Give users the visual shock of watching Chinese ancient paintings differently.

Accordingly, it is an object of the present invention to provide a two-dimensional to three-dimensional depth image creation method suitable for Chinese classical painting.

Therefore, another object of the present invention is to provide a two-dimensional to three-dimensional depth image establishing apparatus suitable for Chinese classical painting.

Therefore, the two-dimensional to three-dimensional depth image establishing method of the present invention is performed by a depth image establishing device and a computing device, the depth image establishing device comprising a memory and a processor connected to the memory, the memory storing one The code, the processor reads the code and performs the following steps included in the method:

(A) reading an original image from the computing device, and generating a predetermined background depth image of the same dimension and a result background depth image that has not been assigned, the original image comprising a plurality of pixels each having a brightness value, The preset background depth image includes a plurality of pixels each having a depth value.

(B) determining whether each luminance of the original image has a luminance value greater than a luminance threshold, and if so, performing the step (C) as a background pixel, otherwise performing the step (D) as a foreground pixel.

(C) making the position corresponding to the background pixel the background depth image of the pixel is given the pixel of the preset background depth image at the same position Depth value.

(D) performing a superpixel calculation on the original image, and dividing the original image into a plurality of superpixels, each of the superpixels including a plurality of pixels having similar luminance values and adjacent positions, and performing step (E) .

(E) first reading the depth values of the pixels corresponding to the same position on the preset background depth image, and then respectively averaging the depth values corresponding to all the foreground pixels of the same superpixel, respectively A superpixel depth value of each superpixel, and then assigning the superpixel depth value correspondingly to a pixel corresponding to the foreground pixel in the resulting background depth image.

Preferably, the step (F) performed after the step (E) is further included:

(F) performing a depth correction method based on the image edge, first performing edge detection and thinning on the original image to obtain a plurality of edge lines of the original image, and then foregrounding the background depth image according to the edge lines The pixels are scanned and replaced with depth values such that the depth values of the pixels surrounded by the same edge line are uniform.

Preferably, the method for replacing the scanned path and the depth value comprises:

(F1) From the lower right to the upper left, use the N-shaped scanning path of the left and right mirrors. If the depth value of the pixel is smaller than the right pixel during scanning, the depth value of the pixel will be the depth of the right pixel. The value is replaced.

(F2) The N-shaped scan path is used from the lower left to the upper right. If the depth value of the pixel is smaller than the left pixel during scanning, the depth value of the pixel is replaced by the depth value of the left pixel.

(F3) Use the Z-shaped scan path from top left to bottom right, if When the depth value of the pixel is smaller than the upper pixel during scanning, the depth value of the pixel is replaced by the depth value of the upper pixel.

(F4) From the upper right to the lower left, use the Z-shaped scanning path of the left and right mirrors. If the depth value of the pixel is smaller than the lower pixel during scanning, the depth value of the pixel will be the depth value of the lower pixel. Replaced.

Preferably, the preset background depth image is set in a stepwise manner from top to bottom to represent each pixel as a depth value that is far from the user.

Preferably, the method further comprises a step (G) performed before the step (D): performing label connectivity calculation on the foreground pixels, giving the same and unique number of the connected pixels, and obtaining non-adjacent regions. The number of pixels of the block, and the pixel whose number of tags is less than a certain threshold is regarded as the background pixel and step (C) is performed, and the remaining pixels perform step (D).

Preferably, the superpixel calculation is to first divide the original image into a plurality of square grids, each grid has a central pixel, and then calculate each central pixel and a predetermined range thereof. The plurality of pixels are related to the difference in brightness between the two and the feature distance of the difference in position, and then each of the pixels is classified into the super-pixel of the central pixel corresponding to the shortest feature distance.

Preferably, in the step (A), when the original image is a color image, the color image is further converted into a gray scale, so that the original image includes the pixels each having the brightness values.

Thus, the two-dimensional to three-dimensional depth image establishing device of the present invention performs the two-dimensional to three-dimensional depth image establishing method as described above.

The effect of the invention is that the foreground and the background are distinguished by judging whether the brightness value is greater than a brightness threshold, and the depth value is averaged by the super-pixel, and the depth value corresponding to the foreground pixel can be more accurately generated.

1‧‧‧Deep image creation device

11‧‧‧ memory

12‧‧‧ Processor

2‧‧‧ Computing device

S1-S8‧‧‧ steps

S11, S31‧‧ steps

Other features and effects of the present invention will be apparent from the following detailed description of the preferred embodiments of the invention. FIG. 1 is a block diagram illustrating a comparison of the two-dimensional to three-dimensional depth image creation methods of the present invention. 2 is a flow chart illustrating the preferred embodiment; FIG. 3 is an image diagram illustrating an original image; FIG. 4 is an image diagram illustrating the application of a brightness threshold; FIG. 5 is an image view. The result of removing the noise is illustrated; FIG. 6 is an image diagram illustrating that the background pixel is given a depth value; FIG. 7 is an image diagram illustrating the initialization of the superpixel calculation; FIG. 8 is an image diagram illustrating the superpixel The result of the calculation; FIG. 9 is an image diagram illustrating the depth value corresponding to the average superpixel; FIG. 10 is an image diagram illustrating the result of thinning; FIG. 11 is a scanning path schematically illustrating the depth correction method based on the image edge; And Figure 12 is an image diagram illustrating a resulting background depth image.

Referring to FIG. 1 and FIG. 2, a preferred embodiment of the two-dimensional to three-dimensional depth image establishing method of the present invention is provided by a depth image establishing device 1 The computing device 2 is configured to include a memory 11 and a processor 12 connected to the memory 11. The memory 11 stores a code, and the processor 12 reads the code to execute the method. The following steps are included:

Step S1 reads an original image from the computing device 2 (as shown in FIG. 3), and generates a preset background depth image of the same dimension and a resulting background depth image that has not been assigned. The assignment method of the preset background depth image is described later.

Step S11 - If the original image is a color image, convert it to gray scale. Thus, the original image to be analyzed includes a plurality of pixels, each of which has a luminance value. The conversion formula is Y=0.299×R+0.587×G+0.114×B, where Y is the brightness value, and R, G, and B are the values of the three primary colors of the color image, respectively.

Step S2: performing a binarization operation on the luminance value, and setting a pixel corresponding to a darker portion of the original image in the background depth image to 0 as the object portion of the image, that is, the foreground, and the image is brighter. The pixel is set to 255, which is regarded as the background information of the image, so that the foreground and the background are initially distinguished, and the resulting background depth image is generated (see FIG. 4). In terms of execution, when the luminance value of one pixel is greater than a luminance threshold, the pixel is represented as the background, and then proceeds to step S4, otherwise proceeds to step S3.

In addition, because it is for Chinese classical painting, and the Chinese classical painting is painted on a white background, the effect of binarization is to distinguish the foreground from the background. The effect is particularly good.

Step S3 - performing image tag connectivity, and binarizing the device The foreground pixel regarded as the same, unique number assigned to the connected pixels, and the pixel whose number of labels is less than a certain threshold is regarded as noise (step S31) and is changed as the background pixel, and the steps are entered. S4.

In the present embodiment, eight adjacent connections are used, and the foreground pixels in the image are tagged, and the eight adjacent regions are marked with the same number. The manner in which the labels are connected is well known to those skilled in the art and will not be described herein.

After the label is connected, the pixels with too small the number of labels are regarded as noise, and these pixels are filtered out and regarded as the background part of the image (as shown in Fig. 5), so that the corresponding depth information can be obtained. Completely fill in the image of the object and the background.

Step S4: After determining the foreground pixel and the background pixel, the step is to fill in the depth value of the pixel (the white portion in FIG. 5) of the result background depth image corresponding to the back pixel. The way to fill in is based on the preset background depth image.

After a related discussion on Chinese classical painting, it was found that Chinese classical painting is different from the fixed perspective of other paintings, and adopts the method of scatter perspective. The so-called scatter perspective is that there can be multiple viewpoints in one picture. Combine the images seen from different angles on the same picture, thus breaking the traditional space limitation. Due to this special painting technique, the style of Chinese classical painting will be a bottom-up movement method. According to the results of the discussion, this embodiment takes the preset background depth image as eight according to the above-mentioned characteristics of Chinese classical painting. The element indicates that the brightness value 0 is placed at the top of the preset background depth image, which represents the farthest from the viewer, and vice versa. 255 is placed under the preset background depth image, which is the closest to the viewer, and the middle part is set according to the number of pixels in the vertical direction of the image, and the pixels are set from top to bottom in a gradual manner to represent the user from the far side. Depth value.

Then, the pixel of the background depth image corresponding to the background pixel of the position is given the depth value of the pixel of the preset background depth image at the same position (as shown in FIG. 6).

Step S5: performing superpixel calculation on the original image, and dividing the original image into a plurality of superpixels, each of the superpixels including a plurality of pixels whose luminance values are similar and adjacent to each other. There are many algorithms for superpixels. In this embodiment, a simple linear iterative clustering (SLIC) algorithm using a modified K-means algorithm (k-means) is used.

Initially, the original image is divided into a plurality of square grids (as shown in Fig. 7), each grid has a central pixel, and then the label value of each pixel is calculated, if the i-th pixel is The feature distance D of the kth grid center pixel related to the difference in brightness and the position difference between the two is the smallest, and the label value of the pixel is updated to k, wherein the feature distance , the difference in brightness Location gap , 1 is the brightness value, x and y are the position coordinate values, and m is the normalization coefficient (constant). For the side length of the aforementioned mesh, N is the total number of pixels of the original image, and K is the preset number of meshes to be formed. It should be noted that, in order to speed up the calculation, the feature distance D of the picture and the central pixel is calculated only for a predetermined range of pixels of 2S×2S around each center pixel of the mesh.

After each pixel has a corresponding label value, the luminance values l _i and the position coordinate values x _i and y _{i of} all the pixels of the same label are averaged respectively, and a new grid center pixel luminance value is obtained. _k and the position coordinate values x _k and y _{k are} continuously repeated to repeat the above actions until the error converges to the set threshold value, and a superpixel close to the edge of the object is obtained (Fig. 8).

Step S6 - After the step S4, the portion of the result background depth image corresponding to the background pixel has been given a depth value, and the portion corresponding to the foreground pixel is now processed. First, the depth values of the pixels corresponding to the same position on the preset background depth image are read, and the depth values corresponding to all foreground pixels belonging to the same superpixel are respectively averaged to obtain the super-values respectively. A super-pixel depth value of the pixel, and then assigning the super-pixel depth value to the pixel corresponding to the foreground pixel in the resulting background depth image (see Figure 9).

So far, the result background depth image has been initially completed, the background part is the gradient depth value, the foreground part includes a plurality of super pixels, and each super pixel has only one depth value. Then it proceeds to step S7.

Step S7- Performing an image edge-based depth correction method, first performing edge detection on the original image to obtain a plurality of edge lines of the original image, and then performing foreground pixels on the resulting background depth image according to the edge lines The substitution of the scan and the depth value makes the depth values of the pixels surrounded by the same edge line uniform.

In this implementation, Sobel edge detection is used first, then the detection result is binarized, and then Z.S (Zhang and Suen, Zhang and Sun) thinning, the edge pixels whose width is larger than one pixel are refined into only one pixel width, and the edge lines are generated, and applied to the part of the result background depth image corresponding to the foreground pixel (Fig. 10) It means that the depth value of the foreground pixel located on the edge line is equal to zero.

Referring to FIG. 11 together, the following four scanning paths and depth values are replaced by pixels that are not edge lines:

(a) From the lower right to the upper left, use the N-shaped scanning path of the left and right mirrors. If the depth value of the pixel is smaller than the right pixel during scanning, the depth value of the pixel will be the depth of the right pixel. The value is replaced.

(b) The N-shaped scan path is used from the lower left to the upper right. If the depth value of the pixel is smaller than the left pixel during scanning, the depth value of the pixel is replaced by the depth value of the left pixel.

(c) A zigzag scan path is used from the upper left to the lower right. If the depth value of the pixel is smaller than the upper pixel during scanning, the depth value of the pixel is replaced by the depth value of the upper pixel.

(d) From the upper right to the lower left, use the Z-shaped scanning path of the left and right mirrors. If the depth value of the pixel is smaller than the lower pixel during scanning, the depth value of the pixel will be the depth value of the lower pixel. Replaced.

The above four scanning path and depth value substitution methods are continuously repeated until the depth values of the pixels surrounded by the same edge line are unified. Finally, the edge information is eliminated by making the depth value of the foreground pixel on the edge line equal to the minimum value of the four adjacent medium depth values.

Step S8 - after processing the portion corresponding to the foreground pixel in step S7, adding the result of step S4, the final background depth of the result is obtained. Degree image (Figure 12).

It is worth mentioning that if the results of sobel edge detection and ZS thinning are directly used to scan and replace the progressive background depth image, it is easy to cause some of the blocks surrounded by the same edge line to be too small. If the block is too small, the range of the depth value is uniform, so that the resulting background depth image is similar to the preset background depth image. However, the present invention is to scan and replace the background depth image (Fig. 9) which is initially completed by using superpixels in step S6, so that even if there are too small blocks, the preliminary use of superpixels is first performed. The action is unified, and the depth value corresponding to the foreground pixel can be generated more accurately.

In summary, the binarization action is used to distinguish the foreground from the background, and the super-pixels are used to initially average the depth values, and the depth values corresponding to the foreground pixels can be more accurately generated, and finally the image-based edges are executed. The depth correction method can better establish the depth image of Chinese classical painting, so it can achieve the purpose of the present invention.

The above is only the preferred embodiment of the present invention, and the scope of the present invention is not limited thereto, that is, the simple equivalent changes and modifications made by the patent application scope and patent specification content of the present invention, All remain within the scope of the invention patent.

S1-S8‧‧‧ steps

S11, S31‧‧ steps

Claims (9)

A two-dimensional to three-dimensional depth image creation method is performed by a depth image creation device and a computing device. The depth image creation device includes a memory and a processor connected to the memory, and the memory stores a code. The processor reads the code and performs the following steps included in the method: (A) reading an original image from the computing device, and generating a preset background depth image of the same dimension and a result background that has not been assigned a depth image, the original image includes a plurality of pixels each having a brightness value, the preset background depth image including a plurality of pixels each having a depth value; (B) determining whether each pixel of the original image is The brightness value is greater than a brightness threshold, if yes, the background pixel is treated as step (C), otherwise the foreground pixel is treated as step (D); (C) the position corresponds to the background pixel of the background pixel The pixel of the image is given a depth value of the pixel of the preset background depth image at the same position; (D) the superpixel calculation is performed on the original image, and the original image is divided into a plurality of super paintings Each of the superpixels includes a plurality of pixels having similar luminance values and adjacent positions, performing step (E); and (E) reading the foreground pixels corresponding to the same position on the preset background depth image. The depth value of the pixel is averaged, and the depth values corresponding to all foreground pixels of the same superpixel are respectively averaged to obtain a super-pixel depth value of each super-pixel, and then the super-pixel depth value is correspondingly assigned to The result is a pixel corresponding to the foreground pixel in the background depth image; wherein the super pixel calculation is to first divide the original image into a plurality of square grids, each grid having a central pixel for each painting The label value of the prime is calculated. If the i-th pixel is related to the k-th grid center pixel, the difference in luminance between the two and the feature distance D of the position difference is the smallest, then the label of the pixel is The value is updated to k, where the feature distance , the difference in brightness Location gap , l is the brightness value, x and y are the position coordinate values, and m is the normalization coefficient (constant). For the side length of the aforementioned mesh, N is the total number of pixels of the original image, K is the preset number of meshes to be formed, and for a predetermined range of pixels around the center pixel of each mesh, To calculate the feature distance D between the picture and the center pixel, and then classify each picture element into the super-pixel of the center pixel corresponding to the shortest feature distance D; when each pixel has a corresponding label value Then, the luminance values l _i and the position coordinate values x _i , y _{i of} all the pixels of the same label are averaged respectively, and a new grid center pixel luminance value l _k and position coordinate values x _k , y _{k are obtained} . The above actions are continuously repeated to update until the error converges to the set threshold value, thereby approaching the superpixel of the edge of the object. The two-dimensional to three-dimensional depth image establishing method according to claim 1, further comprising the step (F) performed after the step (E): (F) performing an image edge-based depth correction method, and performing edge processing on the original image Detecting and thinning to obtain a plurality of edge lines of the original image, and then performing a scan and depth value replacement method on the foreground pixels of the resulting background depth image according to the edge lines, so that the same edge is The depth values of the pixels surrounding the edge are uniform. The two-dimensional to three-dimensional depth image establishing method according to claim 2, wherein the method for replacing the path and the depth value of the scanning includes: (F1) using a left-right mirrored N-shaped scanning path from the lower right to the upper left If the depth value of the pixel is smaller than the right pixel during scanning, the depth value of the pixel is replaced by the depth value of the right pixel; (F2) the N-shaped scanning path is used from the lower left to the upper right. If the depth value of the pixel is smaller than the left pixel during scanning, the depth value of the pixel is replaced by the depth value of the left pixel; (F3) the z-shaped scanning path is used from the upper left to the lower right. If the depth value of the pixel is smaller than the upper pixel during scanning, the depth value of the pixel is replaced by the depth value of the upper pixel; and (F4) the left-right to the lower left using the left-right mirroring z-shaped The scan path, if the depth value of the pixel is smaller than the lower pixel during scanning, the depth value of the pixel is replaced by the depth value of the lower pixel. The two-dimensional to three-dimensional depth image establishing method according to claim 1, wherein the preset background depth image is set in a gradient manner from top to bottom to represent each pixel as a depth value that is far from the user. The two-dimensional to three-dimensional depth image establishing method according to claim 1, further comprising a step (G) performed before the step (D): performing label connectivity calculation on the foreground pixels, giving the connected pixels the same and A unique number, and a pixel whose number of labels is less than a threshold is regarded as a background pixel and step (C) is performed, and the remaining pixels perform step (D). 2D to 3D depth image as claimed in any one of claims 1 to 5 In the image creation method, in the case that the original image is a color image, the color image is also converted into a gray scale, so that the original image includes the pixels each having the brightness values. A two-dimensional to three-dimensional depth image establishing device includes a memory and a processor connected to the memory, the memory storing a code, the depth image establishing device is connected to a computing device, wherein the processor reads The program reads an original image from the computing device, and generates a preset background depth image of the same dimension and a result background depth image that has not been assigned. The original image includes a plurality of pixels each having a brightness value. The preset background depth image includes a plurality of pixels each having a depth value; the processor determines, for each pixel of the original image, whether the brightness value is greater than a brightness threshold; if the processor determines that the brightness value is greater than The brightness threshold is regarded as a background pixel, and the pixel corresponding to the background pixel of the background pixel is given a depth value of the pixel of the preset background depth image at the same position; if the processor determines If the brightness value is not greater than the brightness threshold, the foreground pixel is regarded as a foreground pixel and the super-pixel calculation is performed on the original image, and the original image is divided into multiple a superpixel, each of the superpixels includes a plurality of pixels having similar brightness values and adjacent positions, and reading depth values of pixels corresponding to the same position on the preset background depth image by the foreground pixels, and then And respectively, the depth values corresponding to all the foreground pixels of the same superpixel are respectively obtained, respectively, and a superpixel depth value of each superpixel is obtained, and then the superpixel depth value is correspondingly assigned to the corresponding background depth image. a pixel of the foreground pixel; wherein the superpixel calculation is to first divide the original image into a plurality of square grids, each grid having a central pixel, and calculating a label value for each pixel, If the i-th pixel is related to the k-th grid center pixel, the difference in luminance between the two and the feature distance D of the position difference is the smallest, the label value of the pixel is updated to k, wherein the feature distance , the difference in brightness Location gap , l is the brightness value, x and y are the position coordinate values, and m is the normalization coefficient (constant). For the side length of the aforementioned mesh, N is the total number of pixels of the original image, K is the preset number of meshes to be formed, and for a predetermined range of pixels around the center pixel of each mesh, To calculate the feature distance D between the picture and the center pixel, and then classify each picture element into the super-pixel of the center pixel corresponding to the shortest feature distance D; when each pixel has a corresponding label value Then, the luminance values l _i and the position coordinate values x _i , y _{i of} all the pixels of the same label are averaged respectively, and a new grid center pixel luminance value l _k and position coordinate values x _k , y _{k are obtained} . The above actions are continuously repeated to update until the error converges to the set threshold value, thereby approaching the superpixel of the edge of the object. The two-dimensional to three-dimensional depth image establishing device of claim 7, wherein the processor further performs edge detection and thinning on the original image to obtain a plurality of edge lines of the original image, and then according to the edges The line scans the foreground pixels of the background depth image to replace the depth values, so that the depth values of the pixels surrounded by the same edge line are unified. The preset background depth image is set in a stepwise manner from top to bottom to set each pixel as a depth value that is far from the user. The two-dimensional to three-dimensional depth image establishing method according to claim 7 or 8, wherein, in the case that the original image is a color image, the processor further converts the color image into gray scale, so that the original image includes The pixels each having the brightness values.