TW202201344A - Depth image based rendering method, electrical device and computer program product - Google Patents

Abstract

A depth image based rendering method includes: obtaining a two-dimensional image and a corresponding depth image, and warping the two-dimensional image according to the depth image to obtain a warped image which includes at least one hole region; filling the hole region from down to up to generate a first filled image; filling the hole region from up to down to generate a second filled image; classifying each pixel of the hole region as a first angle type or a second angle type, filling the pixel by adopting the first filled image if the pixel belongs to the first angle type or by adopting the second filled image if the pixel belongs to the second angle type.

Description

Depth-based image generation method, electronic device and computer program product

The present disclosure relates to a depth-based image generation method, which adopts a weighted 3D decimal warp calculation and a double-path gradient hole filling method.

Comfortable 3D imaging has become one of the hottest trends in technology. In recent years, entertainment industry, medical system, etc. have all applied 3D imaging in daily life. However, comfortable 3D image development requires multi-view images. Unfortunately, multi-view cameras are expensive and difficult to transmit, resulting in lack of content and quality degradation. Depth image based rendering (DIBR) can generate multi-view 3D images based on a single 2D image and depth map. The existing traditional depth image based rendering technology includes three main processing stages: depth map preprocessing; 3D warp shift (3D Warping); Hole Filling. Depth map preprocessing is to smooth the depth map to reduce horizontal depth variation and thus reduce voids after 3D warping. 3D warp shift uses the principle of optical projection. The pixels in the 2D image move linearly and horizontally according to the depth of field value. The pixel with a large depth of field value moves more, and vice versa. 3D warping will generate many holes, and the above-mentioned steps of filling the holes are to fill the holes produced by 3D warping through methods such as interpolation, extrapolation, and repairing. How to improve the above process is a topic of concern to those skilled in the art.

An embodiment of the present invention provides a depth-based image generation method, which is suitable for an electronic device. The depth-based image generation method includes: obtaining a two-dimensional image and a depth image corresponding to the two-dimensional image, and performing pixel processing on the two-dimensional image according to the depth image. warping to obtain a warped image, the warped image includes at least one hollow area; filling the hollow area from top to bottom to generate a first filled image; filling the hollow area from bottom to top to generate a second filled image; and classifying each pixel in the cavity area into a first angle class or a second angle class, if the first angle class is the first padded image, and if the second angle class is the second padded image, the pixel is filled with the second padded image.

In some embodiments, the step of filling the hole area from top to bottom to generate the first filled image includes: for the first pixel in the hole area, calculating a horizontal gradient value according to the upper pixel and the upper left pixel of the first pixel, and calculate a vertical gradient value according to the left pixel and the upper left pixel of the first pixel; if the horizontal gradient value is greater than the vertical gradient value, the upper pixel is used to fill the first pixel; and if the vertical gradient value is greater than the horizontal gradient value, the left gradient value is used. pixels to fill the first pixel.

In some embodiments, the step of filling the hole region from bottom to top to generate the second filled image includes: for the first pixel of the hole region, calculating a horizontal gradient value according to the lower pixel and the lower left pixel of the first pixel, And calculate a vertical gradient value according to the left pixel and the lower left pixel of the first pixel; if the horizontal gradient value is greater than the vertical gradient value, use the lower pixel to fill the first pixel; and if the vertical gradient value is greater than the horizontal gradient value, use the left gradient value. pixels to fill the first pixel.

From another perspective, the above-mentioned step of classifying each pixel in the hole area into the first angle class or the second angle class includes: for each pixel in the hole area, if the pixel is a hole pixel and the left pixel is the same as the upper one. If the pixel is a non-hole pixel, the pixel is classified as the first angle category; for each pixel in the hole area, if the pixel is a hole pixel and the left pixel belongs to the first angle category, the pixel is classified as the first angle category ; for each pixel in the hole area, if the pixel is a hole pixel, the pixel below belongs to the first angle class and the pixel to the left is a non-hole pixel, classify this pixel as the first angle class; and classify the rest of the hole pixels in the hole area Classified as the second angle category.

In some embodiments, the above-mentioned step of performing pixel warping on the two-dimensional image according to the depth image to obtain the warped image includes: calculating the decimal point position according to the depth value in the depth image, and filling the depth value to the decimal point position To generate a warped depth image, and fill the corresponding pixel value in the 2D image into the decimal point position to generate a warped 2D image; for an integer point position, calculate the maximum depth value in the vicinity of the integer point position , and delete the depth values in the adjacent range that differ from the maximum depth value by more than a critical value; and calculate the pixel value at the integer point position by weighting according to the pixel values corresponding to the remaining depth values in the adjacent range.

In some embodiments, the step of calculating the pixel value at the integer point position according to the pixel value corresponding to the remaining depth value in the adjacent range includes: calculating a Gaussian between the integer point position and the position of the remaining depth value in the adjacent range distance; and using the Gaussian distance as a weight to add up the pixel values corresponding to the remaining depth values to calculate the pixel value at the integer point position.

In some embodiments, the above-mentioned critical range includes a horizontal direction and a vertical direction.

From another perspective, an embodiment of the present invention provides an electronic device including a memory and a processor. The memory stores a plurality of instructions, and the processor is used for executing the instructions to complete the above-mentioned depth image-based generation method.

From another perspective, an embodiment of the present invention provides a computer program product, which is loaded and executed by an electronic device to implement the above-mentioned depth-based image generation method.

Using the above-mentioned depth-based image generation method can reduce holes and also reduce the amount of calculation.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, the following embodiments are given and described in detail with the accompanying drawings as follows.

The terms "first", "second", etc. used in this document do not mean a particular order or order, but are only used to distinguish elements or operations described in the same technical terms.

FIG. 1 is a schematic diagram illustrating an electronic device according to an embodiment. Referring to FIG. 1 , the electronic device 100 may be a smart phone, a tablet computer, a personal computer, a notebook computer, a server, an industrial computer, or various electronic devices with computing capabilities, etc., the invention is not limited thereto. The electronic device 100 includes a processor 110 and a memory 120, wherein the processor 110 can be a central processing unit, a microprocessor, a microcontroller, an image processing chip, a special application integrated circuit, etc., and the memory 120 can be a random access A memory, read-only memory, flash memory, floppy disk, hard disk, optical disk, pen drive, magnetic tape, or a database accessible through the Internet in which a plurality of instructions are stored and executed by the processor 110 These instructions implement a depth-based image generation method, which will be described in detail below.

FIG. 2 is a schematic diagram illustrating a method for generating a depth image based on an embodiment. Referring to FIG. 2 , the depth-based image generation method is used to generate two-dimensional images with different viewing angles according to a two-dimensional image 210 and a depth image 220 corresponding to the two-dimensional image 210 . For example, the viewing angle of the two-dimensional image 210 is set to viewing angle 0, and the depth image 220 includes the depth value measured from the viewing angle 0. In this embodiment, the closer to the lens, the greater the depth value, but the invention is not limited thereto. According to the 2D image 210 and the depth image, the 2D image 230 of view 1, the 2D image 240 of view 2, the 2D image 250 of view-1 can be generated, and so on. Here, the viewing angle 1 to the viewing angle 3 are used to represent the right viewing angle, and the viewing angle-1 to the viewing angle-3 are used to represent the left viewing angle, but the present invention is not limited thereto. For example, if there is a foreground object in the 2D image 210, the foreground object will be shifted to the left when viewed from the right perspective, and the foreground object will be shifted to the right when viewed from the left perspective, when the foreground object is closer to the lens. The displacement will be larger.

Here, the 2D image 210 and the depth image 220 may be obtained through an image sensor and a depth sensor, respectively, or may be obtained from an existing database or other devices. The above-mentioned image sensor may include a Charge-coupled Device (CCD) sensor, a Complementary Metal-Oxide Semiconductor (Complementary Metal-Oxide Semiconductor) sensor, or other suitable photosensitive elements. The above-mentioned depth sensor may include dual cameras, an artificial intelligence depth generation network, a structured light sensing device, or any device that can sense the depth of a scene.

FIG. 3 is a schematic flowchart illustrating a method for generating a depth image based on an embodiment. Referring to FIG. 3 , in step 310 , the depth image 220 is preprocessed, for example, smoothing is performed to reduce the variation of the depth value. Any preprocessing can be used here, and the present invention does not limit the content of step 310 . In step 320, a pixel warp is performed on the 2D image 210 according to the preprocessed depth image 220. In particular, this embodiment proposes a weighted 3D decimal warp calculation. After step 320 is executed, a warped image is obtained, and the warped image includes one or more hollow regions. These hollow regions are filled in step 330. In this embodiment, a dual-path gradient hole filling method is used to fill the hollow regions. A virtual perspective image, such as a two-dimensional image 230, can be generated later. Steps 320 and 330 will be described in detail below with examples.

，預處理過後的深度影像220的景深值表示為

，x與y分別代表X座標與Y座標。上述的步驟320包括了步驟401~405，在步驟401中，根據深度影像220中的每一個深度值計算一小數點位置。具體來說，根據深度值可以計算出一視差位移值，如以下數學式1所示。 [數學式1]

FIG. 4A and FIG. 4B are schematic flowcharts illustrating a weighted 3D decimal curvature calculation according to an embodiment. Please refer to FIG. 4A, where the pixels in the two-dimensional image 210 are represented as

, the depth of field value of the preprocessed depth image 220 is expressed as

, x and y represent the X coordinate and the Y coordinate, respectively. The above-mentioned step 320 includes steps 401 to 405 . In step 401 , the decimal point position is calculated according to each depth value in the depth image 220 . Specifically, a parallax displacement value can be calculated according to the depth value, as shown in the following equation 1. [Mathematical formula 1]

為視差位移值。

表示小數點位置的小數點精確度，K可為2、4、8或任意正整數。

表示視角，例如為圖2所示的-3~-1與1~3的其中之一。

為景深視差參數(depth disparity ratio)，可依照相機及立體顯示器的參數來決定。

表示捨去小數的整數。上述的數學式1是將傳統的視差位移值乘上K倍，經過四捨五入的運算以後再除以K倍，但值得注意的是視差位移值

可以是整數也可以有小數，若以K=4為例，小數值可為.00、.25、.50、.75等四個值。接下來，將x座標加上視差位移值

便可以得到一個小數點位置。in

is the parallax displacement value.

Indicates the decimal point precision of the decimal point position, K can be 2, 4, 8 or any positive integer.

Indicates the viewing angle, for example, one of -3 to -1 and 1 to 3 shown in FIG. 2 .

It is a depth disparity ratio, which can be determined according to the parameters of the camera and the stereoscopic display.

Represents an integer with rounded decimals. The above mathematical formula 1 is to multiply the traditional parallax displacement value by K times, and then divide it by K times after rounding, but it is worth noting that the parallax displacement value

It can be an integer or a decimal. If K=4 is used as an example, the decimal value can be four values such as .00, .25, .50, and .75. Next, add the x-coordinate to the parallax displacement value

You can get a decimal point position.

移動至上述的小數點位置以產生曲移後深度影像，此步驟可以表示為以下數學式2，

即是曲移後深度影像，

為小數點位置。值得注意的是Y座標不需要位移。 [數學式2]

In step 402, the original depth of field value is

Move to the above-mentioned decimal point position to generate the warped depth image, this step can be expressed as the following equation 2,

That is, the depth image after the warp shift,

for the decimal point position. It is worth noting that the Y coordinate does not need to be shifted. [Mathematical formula 2]

即是曲移後二維影像。 [數學式3]

In step 403, the corresponding pixel value in the two-dimensional image 210 is filled in the decimal point position to generate a two-dimensional image after warping. This step can be expressed as the following mathematical formula 3,

That is, the two-dimensional image after the warp shift. [Mathematical formula 3]

，則可以保留較大的深度值，捨棄較小的深度值。If the position of the decimal point overlaps, that is, different x corresponds to the same

, the larger depth value can be kept and the smaller depth value can be discarded.

可以包括K個小數，這等同於曲移後二維影像

的寬度會是二維影像

的K倍，如曲移後二維影像411所示。相同的，曲移後深度影像

的寬度會是深度影像

的K倍，如曲移後深度影像412所示。Since the parallax displacement value in Equation 1

Can include K decimals, which is equivalent to the 2D image after warp

will be the width of the 2D image

K times, as shown in the two-dimensional image 411 after the warp. The same, the depth image after warping

The width of will be the depth image

K times, as shown in the depth image 412 after the warp.

與曲移後深度影像

，在此繪示了局部(

)的像素值與深度值，值得注意的是較粗的標線表示整數點位置，如

、

、

、

等，而其他的標線表示小數點位置，如.25、.50、.75等。在此例子中，景深值“30”被填入

的位置，景深值“121”被填入

的位置，而景深值“120”被填入

。每個景深值都對應至一個像素值，景深值“30”是對應至像素值“68”，景深值“121”是對應至像素值“215”，景深值“120”是對應至像素值“212”。接下來要計算每個整數點位置上的像素值。5A and FIG. 5B are partial schematic diagrams illustrating a warped two-dimensional image and a warped depth image according to an embodiment. Please refer to FIG. 5A , since there is no displacement on the Y coordinate, here the X coordinate is used to represent the two-dimensional image after warping.

Depth image with warp

, where the partial (

) pixel value and depth value, it is worth noting that the thicker reticle indicates the integer point position, such as

,

,

,

etc., while other reticles indicate the decimal point position, such as .25, .50, .75, etc. In this example, the depth of field value "30" is filled in

position, the depth of field value "121" is filled in

position, and the depth of field value "120" is filled in

. Each depth value corresponds to a pixel value, the depth value "30" corresponds to the pixel value "68", the depth value "121" corresponds to the pixel value "215", and the depth value "120" corresponds to the pixel value "212". The next step is to calculate the pixel value at each integer point position.

Referring to FIGS. 4A and 5A , in step 404, for each integer point position in the depth image after warping, the maximum depth value in the adjacent range of the integer point position is calculated, and the difference between the adjacent range and the maximum depth value is deleted. A depth value that exceeds a threshold. If the integer point position is x, the adjacent range can be expressed as the following Mathematical formula 4. [Mathematical formula 4]

表示鄰近範圍，也表示在此範圍內所有

所形成的集合。在此實施例中K=4，因此鄰近範圍

可以設定為左方2個小數點位置到右方2個小數點位置，以整數點位置“100”為例，鄰近範圍

表示從

至

的範圍。接下來可以根據鄰近範圍

的像素值來決定整數點位置

上的像素值，如果有多個像素被位移至此鄰近範圍

，則保留與最大景深值相差一臨界值內的景深值，因為較大的景深值在前景，會遮蓋位於背景的像素值。具體來說，鄰近範圍

內的最大深度值為“121”，表示為

，如以下數學式5所示。鄰近範圍

內剩餘的景深值形成一個前景景深值集合，表示為以下數學式6。 [數學式5]

[數學式6]

Indicates the adjacent range, also means all within this range

formed collection. In this example K=4, so the adjacent range

It can be set from 2 decimal points on the left to 2 decimal points on the right. Taking the integer point position "100" as an example, the adjacent range

means from

to

range. Next, according to the adjacent range

pixel value to determine the integer point position

The pixel value on , if multiple pixels are shifted into this neighborhood

, the depth of field value within a critical value from the maximum depth of field value is retained, because the larger depth of field value is in the foreground and will cover the pixel value in the background. Specifically, the proximity

The maximum depth value inside is "121", which is expressed as

, as shown in Equation 5 below. Proximity

The remaining depth-of-field values within form a set of foreground depth-of-field values, which are expressed as Equation 6 below. [Mathematical formula 5]

[Math 6]

為前景景深值集合，在此例子中包括了“121”與“120”，而景深值“30”則被刪除。

為臨界值，可設定為任意合適的數值。

For the foreground depth value set, "121" and "120" are included in this example, and the depth value "30" is deleted.

is a critical value, which can be set to any suitable value.

中對應的像素值也會被刪除，剩餘的景深值與像素值如圖5B所示，剩餘的像素值可以表示為以下數學式7。在此例子中剩下的像素值為“215”與“212”。 [數學式7]

In addition, the 2D image after warping

The corresponding pixel values in are also deleted, the remaining depth of field values and pixel values are shown in FIG. 5B , and the remaining pixel values can be expressed as the following mathematical formula 7. The remaining pixel values in this example are "215" and "212". [Math 7]

只包括水平方向，但在一些實施例中也可包括垂直方向。舉例來說，上述的鄰近範圍

可以替換為

，表示如以下數學式8。上述的數學式5可以替換為以下數學式9。 [數學式8]

[數學式9]

It is worth noting that in Figure 5A and Figure 5B the adjacent range

Only the horizontal direction is included, but may also include the vertical direction in some embodiments. For example, the aforementioned proximity range

can be replaced with

, which is represented by the following Mathematical formula 8. The above-described mathematical formula 5 may be replaced with the following mathematical formula 9. [Math 8]

[Math 9]

內剩餘的深度值所對應的像素值計算整數點位置上的像素值。因為距離整數點位置越近的像素值越重要，在此實施例中可先計算整數點位置與鄰近範圍內剩餘的深度值的位置之間的高斯距離，然後以高斯距離作為權重來加總這些剩餘的深度值所對應的像素值以計算出整數點位置上的像素值，在其他實施例中上述高斯距離亦可以替換為其他線性或非線性距離權重。步驟405可以表示為以下數學式10與數學式11。 [數學式10]

[數學式11]

Referring to FIG. 4A, next in step 405, according to the proximity range

The pixel value corresponding to the remaining depth value within the calculation of the pixel value at the integer point position. Because the pixel value closer to the integer point position is more important, in this embodiment, the Gaussian distance between the integer point position and the position of the remaining depth values in the adjacent range can be calculated first, and then the Gaussian distance is used as a weight to sum these The pixel values corresponding to the remaining depth values are used to calculate the pixel values at the integer point positions. In other embodiments, the above-mentioned Gaussian distance can also be replaced with other linear or nonlinear distance weights. Step 405 can be expressed as the following Math 10 and Math 11. [Math 10]

[Math 11]

為參數，代表高斯分布的變異數。

為鄰近範圍

內剩餘像素值的X座標與Y座標所形成的集合。雖然圖5A與圖5B只採用了X方向的鄰近範圍，但上述的數學式10、11是更泛用(general)的表示，不只採用X方向上的相鄰像素值，也採用了Y方向上的相鄰像素值。在計算出整數點位置上的像素值以後可以得到曲移後影像

，例如圖4B的曲移後影像410，其中包括至少一個空洞區420。在此實施例中是要產生出左方的視角，因此影像中的物件會往右位移，在物件的左方會產生空洞區420。Where (x, y) is the X-coordinate and Y-coordinate of the integer point position.

is a parameter representing the variance of the Gaussian distribution.

for the adjacent range

The set of X and Y coordinates of the remaining pixel values within. Although only the adjacent range in the X direction is used in FIGS. 5A and 5B , the above-mentioned mathematical expressions 10 and 11 are more general expressions, not only using the adjacent pixel values in the X direction, but also using the adjacent pixel values in the Y direction. adjacent pixel values. After calculating the pixel value at the integer point position, the warped image can be obtained

For example, the warped image 410 in FIG. 4B includes at least one hollow region 420 therein. In this embodiment, a left viewing angle is to be generated, so the object in the image will be displaced to the right, and a hollow area 420 will be generated on the left side of the object.

Since the conventional pixel warping method only uses integer values, the decimal point has to be discarded when there is a decimal point, so many additional holes are generated. Conversely, the above approach preserves the decimal point, thus greatly reducing the generation of holes.

，可以根據上方像素與左上方像素計算水平梯度值，如以下數學式12所示，另外可根據左方像素與左上方像素計算垂直梯度值，如以下數學式13所示。 [數學式12]

[數學式13]

Next, the dual-path gradient hole filling method is performed, as shown in Figure 6. In this embodiment, the warped image 410 is filled with two directions. Specifically, in step 610 , the hollow area in the warped image 410 is filled from top to bottom to generate a first filled image 611 . When scanning the warped image 410 from top to bottom and left to right, it can ensure that the upper pixel, upper left pixel and left pixel of the hole have values (not holes), so for a pixel in the hole area

, the horizontal gradient value can be calculated according to the upper pixel and the upper left pixel, as shown in the following Equation 12, and the vertical gradient value can be calculated according to the left pixel and the upper left pixel, as shown in the following Equation 13. [Math 12]

[Math 13]

為左上方像素，

為上方像素，

為左方像素。若水平梯度值

大於垂直梯度值

，表示在像素

周圍存在垂直邊緣，因此採用上方像素

來填補像素

。另一方面，如果垂直梯度值

大於水平梯度值

，表示在像素

周圍存在水平邊緣，則採用左方像素

來填補像素

。換言之，步驟610是根據以下數學式14來進行由上往下填補。 [數學式14]

in

is the upper left pixel,

is the upper pixel,

is the left pixel. If the horizontal gradient value

greater than the vertical gradient value

, expressed in pixels

There is a vertical edge around it, so the pixel above is taken

to fill pixels

. On the other hand, if the vertical gradient value

greater than the horizontal gradient value

, expressed in pixels

There is a horizontal edge around it, the left pixel is used

to fill pixels

. In other words, step 610 is to perform top-to-bottom padding according to the following equation 14. [Math 14]

即是第一填補後影像611。在此實施例中當如果垂直梯度值

等於水平梯度值

時是採用左方像素，但在其他實施例也可以採用上方像素，本發明並不在此限。obtained image

That is, the first filled image 611 . In this example, if the vertical gradient value

equal to the horizontal gradient value

In other embodiments, the left pixel is used, but the upper pixel may also be used in other embodiments, and the present invention is not limited thereto.

、左方像素

與左下方像素

有值(不是空洞)，因此可以根據下方像素與左下方像素計算水平梯度值，如以下數學式15所示，並且根據左方像素與左下方像素計算垂直梯度值，如以下數學式16所示。 [數學式15]

[數學式16]

On the other hand, in step 620 , the hollow area in the warped image 410 is filled from bottom to top to generate a second filled image 621 . If scanning from bottom to top, left to top, the bottom pixel

, left pixel

with the bottom left pixel

has a value (not a hole), so the horizontal gradient value can be calculated from the lower pixel and the lower left pixel, as shown in the following Equation 15, and the vertical gradient value can be calculated according to the left pixel and the lower left pixel, as shown in the following Equation 16 . [Math 15]

[Math 16]

大於垂直梯度值

，表示像素

周圍存在垂直邊緣，因此採用下方像素

來填補像素

。如果垂直梯度值

大於水平梯度值

，採用左方像素

來填補像素

。換言之，步驟620是根據以下數學式17進行由下往上填補。 [數學式17]

If the horizontal gradient value

greater than the vertical gradient value

, representing pixels

There is a vertical edge around it, so the pixel below is taken

to fill pixels

. If the vertical gradient value

greater than the horizontal gradient value

, using pixels on the left

to fill pixels

. In other words, step 620 is to perform bottom-up padding according to the following mathematical formula 17. [Math 17]

即是第二填補後影像621。值得注意的是，當像素周圍有45度的邊緣時，第一填補後影像611的填補效果較好，當像素周圍有135度的邊緣時，第二填補後影像621的填補效果較好。因此，在步驟630中會將空洞區的每一個像素分為兩個角度類別，接下來可以根據角度類別來從第一填補後影像611與第二填補後影像621擇一以填補對應的像素。實作上我們採用一個遮罩M(x,y)，如果是空洞區則在對應的位置填入0，非空洞區則填入255，如以下數學式18所示。 [數學式18]

obtained image

That is, the second padded image 621 . It is worth noting that when there is an edge of 45 degrees around the pixel, the filling effect of the first filled image 611 is better, and when there is an edge of 135 degrees around the pixel, the filling effect of the second filled image 621 is better. Therefore, in step 630, each pixel in the cavity area is divided into two angle categories, and then one of the first padded image 611 and the second padded image 621 can be selected according to the angle categories to fill the corresponding pixel. In practice, we use a mask M(x,y), if it is a hole area, fill in 0 in the corresponding position, and fill in 255 in the non-hole area, as shown in the following equation 18. [Math 18]

For each pixel in the hole area, if the left pixel and the upper pixel are non-hole pixels, classify the corresponding pixel into the 45-degree angle category, and fill in 128 in the corresponding position in the mask M(x,y), This judgment can also be expressed as the following Mathematical formula 19. [Math 19]

Next, based on the concept of warp shift, the void pixel will extend horizontally, extending the 45-degree angle pixel category to the right, and this horizontal extension can be expressed as the following mathematical formula 20. [Math 20]

Finally, vertical extension is performed, as shown in the following Equation 21. [Math 21]

After the vertical extension, the remaining positions marked "0" in the mask M(x,y) are classified into the angle category of 135 degrees. If a pixel belongs to the angle category of 45 degrees, the result of the first padded image 611 is used for padding; if a pixel belongs to the angle category of 135 degrees, the result of the second padded image 621 is used for padding. This step can be expressed as is the following Mathematical formula 22. [Math 22]

例如為圖6的影像631，在各個角度類別都填補的很好。final image

For example, the image 631 in FIG. 6 is well filled in every angle category.

In this embodiment, the pixels are classified by the mask M(x, y), but in other embodiments, arbitrary data structures and arbitrary symbols can also be used for classification. From another point of view, for each pixel in the hole area, if this pixel is a hole pixel and the left pixel and the upper pixel are non-hole pixels, this pixel is classified as an angle class of 45 degrees; if this pixel is a hole pixel And the left pixel belongs to the 45 degree angle category, then this pixel is classified as the 45 degree angle category; if this pixel is a hole pixel, the lower pixel belongs to the 45 degree angle category, and the left pixel is a non-hole pixel, this is Pixels are classified into the 45 degree angle class; after the above steps the remaining empty pixels are classified into the 135 degree angle class. The present invention does not limit what numbers, symbols or letters are used to mark the above-mentioned angle categories.

From another perspective, the present invention also proposes a computer program product, which can be written in any programming language and/or platform. When the computer program product is loaded into a computer system and executed, the above-mentioned program can be executed. Based on the depth image generation method.

In the above-mentioned depth-based image generation method, a more natural and low-computation virtual perspective image can be obtained compared to existing methods.

Although the present invention has been disclosed above by the embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, The protection scope of the present invention shall be determined by the scope of the appended patent application.

100: Electronics 110: Processor 120: memory 210, 230, 240, 250: 2D images 220: Deep Image 310, 320, 330, 401~405: Steps 411: 2D image after warping 412: Depth image after warp 410: Image after warp 420: Hollow Zone 610, 620, 630: Steps 611: Image after first padding 621: Image after second padding 631: Video

[ FIG. 1 ] is a schematic diagram illustrating an electronic device according to an embodiment. [ FIG. 2 ] is a schematic diagram illustrating a method for generating a depth image based on an embodiment. [ FIG. 3 ] is a schematic flowchart illustrating a method for generating a depth image based on an embodiment. [ FIG. 4A ] and [ FIG. 4B ] are schematic flowcharts illustrating a weighted 3D decimal curvature calculation according to an embodiment. 5A and FIG. 5B are partial schematic diagrams illustrating a two-dimensional image after warping and a depth image after warping, according to an embodiment. [FIG. 6] A schematic diagram illustrating a method for filling dual-path gradient holes according to an embodiment.

210, 230: 2D Imagery

220: Deep Image

310, 320, 330: Steps

Claims (9)

A method for generating a depth image, suitable for an electronic device, the method for generating a depth image includes: obtaining a two-dimensional image and a depth image corresponding to the two-dimensional image, and performing pixel warping on the two-dimensional image according to the depth image to obtain a warped image, wherein the warped image includes at least one hollow area; filling the at least one void area from top to bottom to generate a first filled image; filling the at least one hole region from bottom to top to generate a second filled image; and classifying each pixel of the at least one hole region into a first angle class or a second angle class, if the first angle class is the first padded image, and if the second angle class is the second padded image image to fill that pixel. The depth image-based generation method according to claim 1, wherein the step of filling the at least one hole region from top to bottom to generate the first filled image comprises: For a first pixel of the at least one cavity region, a horizontal gradient value is calculated according to the upper pixel and the upper left pixel of the first pixel, and a vertical gradient value is calculated according to the left pixel and the upper left pixel of the first pixel. ; If the horizontal gradient value is greater than the vertical gradient value, use the upper pixel to fill the first pixel; and If the vertical gradient value is greater than the horizontal gradient value, the left pixel is used to fill the first pixel. The depth image-based generating method according to claim 1, wherein the step of filling the at least one hole region from bottom to top to generate the second filled image comprises: For a first pixel of the at least one cavity region, a horizontal gradient value is calculated according to the lower pixel and the lower left pixel of the first pixel, and a vertical gradient value is calculated according to the left pixel and the lower left pixel of the first pixel ; If the horizontal gradient value is greater than the vertical gradient value, use the lower pixel to fill the first pixel; and If the vertical gradient value is greater than the horizontal gradient value, the left pixel is used to fill the first pixel. The depth image-based generation method as claimed in claim 1, wherein the step of classifying each pixel of the at least one hollow region into the first angle class and the second angle class includes: For each of the pixels in the at least one hole region, if the pixel is a hole pixel, and the pixel to the left and the upper pixel of the pixel are non-hole pixels, classifying the pixel into the first angle class; For each of the pixels in the at least one hole region, if the pixel is a hole pixel and the pixel to the left of the pixel belongs to a first angle class, classifying the pixel as the first angle class; For each of the pixels in the at least one hole region, if the pixel is a hole pixel, the pixel below the pixel belongs to the first angle class, and the pixel to the left of the pixel is a non-hole pixel, the pixel is classified as the first angle class an angle category; and The remaining hole pixels of the at least one hole region are classified into the second angle class. The depth image-based generating method according to claim 1, wherein the step of performing pixel warping on the two-dimensional image according to the depth image to obtain the warped image comprises: Calculate the decimal point position according to a depth value in the depth image, fill in the depth value to the decimal point position to generate a depth image after warping, and fill in the decimal point with the corresponding pixel value in the 2D image point position to generate a shifted 2D image; for an integer point location, calculating a maximum depth value within a neighboring range of the integer point location, and deleting depth values within the neighboring region that differ from the maximum depth value by more than a threshold; and The pixel value at the position of the integer point is calculated according to the pixel value corresponding to the remaining depth value in the adjacent range. The method for generating a depth image according to claim 5, wherein the step of calculating the pixel value at the integer point position according to the pixel value corresponding to the remaining depth value in the adjacent range includes: calculating the Gaussian distance between the integer point location and the location of the remaining depth values in the neighborhood; and Using the Gaussian distance as a weight, the pixel values corresponding to the remaining depth values are summed to calculate the pixel value at the integer point position. The depth image-based generation method according to claim 6, wherein the critical range includes a horizontal direction and a vertical direction. An electronic device, comprising: a memory that stores a plurality of instructions; and a processor for executing the instructions to complete the steps: obtaining a two-dimensional image and a depth image corresponding to the two-dimensional image, and performing pixel warping on the two-dimensional image according to the depth image to obtain a warped image, wherein the warped image includes at least one hollow area; filling the at least one void area from top to bottom to generate a first filled image; filling the at least one hole region from bottom to top to generate a second filled image; and classifying each pixel of the at least one hole region into a first angle class or a second angle class, if the first angle class is the first padded image, and if the second angle class is the second padded image image to fill that pixel. A computer program product loaded and executed by an electronic device to complete the steps: obtaining a two-dimensional image and a depth image corresponding to the two-dimensional image, and performing pixel warping on the two-dimensional image according to the depth image to obtain a warped image, wherein the warped image includes at least one hollow area; filling the at least one void area from top to bottom to generate a first filled image; filling the at least one hole region from bottom to top to generate a second filled image; and classifying each pixel of the at least one hole region into a first angle class or a second angle class, if the first angle class is the first padded image, and if the second angle class is the second padded image image to fill that pixel.

Images