TWI736335B - Depth image based rendering method, electrical device and computer program product - Google Patents
Depth image based rendering method, electrical device and computer program product Download PDFInfo
- Publication number
- TWI736335B TWI736335B TW109121387A TW109121387A TWI736335B TW I736335 B TWI736335 B TW I736335B TW 109121387 A TW109121387 A TW 109121387A TW 109121387 A TW109121387 A TW 109121387A TW I736335 B TWI736335 B TW I736335B
- Authority
- TW
- Taiwan
- Prior art keywords
- pixel
- image
- depth
- gradient value
- value
- Prior art date
Links
Images
Landscapes
- Image Processing (AREA)
Abstract
Description
本揭露是關於一種基於深度影像生成方法,其中採用了加權式3D小數曲移計算與雙徑梯度空洞填補方法。 This disclosure relates to a depth-based image generation method, which uses a weighted 3D decimal warp calculation and a double-path gradient hole filling method.
舒適3D影像顯像已經是現今科技熱門的趨勢之一,近年來娛樂產業、醫療體系等等都將3D影像顯像應用在生活上。然而,舒適3D影像顯像需多視角影像,可惜因多視角攝影機拍攝成本昂貴且不易傳送,導致內容缺乏及品質下降。基於深度影像生成技術(depth image based rendering,DIBR)可根據單一2D影像與景深圖產生多視角的3D影像,現有傳統基於深度影像生成技術包括三個主要處理階段:深度圖預處理;3D曲移(3D Warping);空洞填補(Hole Filling)。深度圖預處理乃對深度圖進行平滑處理,以減少水平深度變化,進而降低3D曲移之後的空洞。3D曲移乃利用光學投射原理,2D影像中的像素依照景深值做線性水平移動,景深值大的像素移動較大,反之則移動較小。3D曲移會產生許多空洞, 上述空洞填補的步驟便是透過例如內插、外插、修復等方法來填補3D曲移產生的空洞。如何改進上述流程,為此領域技術人員所關心的議題。 Comfortable 3D imaging is already one of the hot trends in science and technology. In recent years, the entertainment industry, medical systems, etc. have all applied 3D imaging in daily life. However, comfortable 3D image display requires multi-view images. Unfortunately, multi-view cameras are expensive and difficult to transmit, resulting in lack of content and degradation of quality. Based on the depth image based rendering (DIBR) technology, a multi-view 3D image can be generated from a single 2D image and a depth map. The existing traditional depth image based rendering technology includes three main processing stages: depth image preprocessing; 3D curvature shift (3D Warping); Hole Filling. Depth map preprocessing is to smooth the depth map to reduce the horizontal depth variation, thereby reducing the voids after 3D warping. The 3D curve shift uses the principle of optical projection. The pixels in the 2D image move linearly and horizontally according to the depth of field value. The pixels with a larger depth of field move more, and vice versa. 3D warping will produce many holes, The above-mentioned hole filling step is to fill the holes generated by 3D warping through methods such as interpolation, extrapolation, and repair. How to improve the above process is a topic of concern to those skilled in the art.
本發明的實施例提出一種基於深度影像生成方法,適用於一電子裝置,此基於深度影像生成方法包括:取得二維影像以及對應二維影像的深度影像,並且根據深度影像對二維影像做像素曲移以得到曲移後影像,此曲移後影像包括至少一個空洞區;由上而下填補空洞區以產生第一填補後影像;由下而上填補空洞區以產生第二填補後影像;以及將空洞區的每一個像素分為第一角度類別或第二角度類別,若是第一角度類別則採用第一填補後影像,若是第二角度類別則採用第二填補後影像來填補像素。 An embodiment of the present invention provides a depth-based image generation method suitable for an electronic device. The depth-based image generation method includes: obtaining a two-dimensional image and a depth image corresponding to the two-dimensional image, and pixelizing the two-dimensional image according to the depth image Warping to obtain a warped image, the warped image including at least one hollow area; filling the hollow area from top to bottom to generate a first filled image; filling the hollow area from bottom to top to generate a second filled image; And each pixel in the cavity area is divided into a first angle category or a second angle category. If it is the first angle category, the first padded image is used, and if it is the second angle category, the second padded image is used to fill the pixels.
在一些實施例中,上述由上而下填補空洞區以產生第一填補後影像的步驟包括:對於空洞區的第一像素,根據第一像素的上方像素與左上方像素計算一水平梯度值,並且根據第一像素的左方像素與左上方像素計算一垂直梯度值;若水平梯度值大於垂直梯度值,採用上方像素來填補第一像素;以及若垂直梯度值大於水平梯度值,採用左方像素來填補第一像素。 In some embodiments, the above step of filling the cavity area from top to bottom to generate the first filled image includes: for the first pixel of the cavity area, calculating a horizontal gradient value according to the upper pixel and the upper left pixel of the first pixel, And calculate a vertical gradient value based on the left pixel and the upper left pixel of the first pixel; if the horizontal gradient value is greater than the vertical gradient value, the upper pixel is used to fill the first pixel; and if the vertical gradient value is greater than the horizontal gradient value, the left Pixels to fill the first pixel.
在一些實施例中,上述由下而上填補空洞區以產生第二填補後影像的步驟包括:對於空洞區的第一像素,根據第一像素的下方像素與左下方像素計算一水平梯度值, 並且根據第一像素的左方像素與左下方像素計算一垂直梯度值;若水平梯度值大於垂直梯度值,採用下方像素來填補第一像素;以及若垂直梯度值大於水平梯度值,採用左方像素來填補第一像素。 In some embodiments, the above step of filling the cavity area from bottom to top to generate the second filled image includes: for the first pixel of the cavity area, calculating a horizontal gradient value according to the lower pixel and the lower left pixel of the first pixel, And calculate a vertical gradient value based on the left pixel and the bottom left pixel of the first pixel; if the horizontal gradient value is greater than the vertical gradient value, use the bottom pixel to fill the first pixel; and if the vertical gradient value is greater than the horizontal gradient value, use the left Pixels to fill the first pixel.
以另一個角度來說,上述將空洞區的每一像素分為第一角度類別或第二角度類別的步驟包括:對於空洞區的每一個像素,如果此像素為空洞像素且左方像素與上方像素為非空洞像素,將此像素分類為第一角度類別;對於空洞區的每一個像素,如果此像素為空洞像素且左方像素屬於第一角度類別,則將此像素分類為第一角度類別;對於空洞區的每一個像素,如果此像素為空洞像素、下方像素屬於第一角度類別且左方像素為非空洞像素,將此像素分類為第一角度類別;以及將空洞區的其餘空洞像素分類為第二角度類別。 From another perspective, the above step of classifying each pixel in the cavity area into the first angle category or the second angle category includes: for each pixel in the cavity area, if the pixel is a hole pixel and the pixel on the left and above If the pixel is a non-hole pixel, classify this pixel as the first angle category; for each pixel in the hole area, if the pixel is a hole pixel and the left pixel belongs to the first angle category, then this pixel is classified as the first angle category ; For each pixel in the hole area, if the pixel is a hole pixel, the lower pixel belongs to the first angle category, and the left pixel is a non-hole pixel, classify this pixel as the first angle category; and classify the remaining hole pixels in the hole area Classified as the second angle category.
在一些實施例中,上述根據深度影像對二維影像做像素曲移以得到曲移後影像的步驟包括:根據深度影像中的深度值計算一小數點位置,將深度值填入至小數點位置以產生曲移後深度影像,並將二維影像中對應的像素值填入小數點位置以產生曲移後二維影像;對於一整數點位置,計算整數點位置的鄰近範圍內的最大深度值,並且刪除鄰近範圍內與最大深度值相差超過一臨界值的深度值;以及根據鄰近範圍內剩餘的深度值所對應的像素值加權計算整數點位置上的像素值。 In some embodiments, the step of performing pixel warping on the two-dimensional image according to the depth image to obtain the warped image includes: calculating a decimal point position according to the depth value in the depth image, and filling the depth value to the decimal point position To generate the depth image after the curvature shift, and fill the corresponding pixel value in the two-dimensional image into the decimal point position to generate the two-dimensional image after the curvature shift; for an integer point position, calculate the maximum depth value within the vicinity of the integer point position , And delete the depth values in the neighboring range that differ from the maximum depth value by more than a critical value; and weighting the pixel values at the integer point positions according to the pixel values corresponding to the remaining depth values in the neighboring range.
在一些實施例中,上述根據鄰近範圍內剩餘的深度 值所對應的像素值計算整數點位置上的像素值的步驟包括:計算整數點位置與鄰近範圍內剩餘的深度值的位置之間的高斯距離;以及以高斯距離作為權重來加總剩餘的深度值所對應的像素值以計算出整數點位置上的像素值。 In some embodiments, the above is based on the remaining depth in the adjacent range The step of calculating the pixel value at the position of the integer point corresponding to the pixel value of the value includes: calculating the Gaussian distance between the position of the integer point and the position of the remaining depth value in the adjacent range; and using the Gaussian distance as a weight to add up the remaining depth The pixel value corresponding to the value is used to calculate the pixel value at the integer point position.
在一些實施例中,上述的鄰近範圍包括水平方向與垂直方向。 In some embodiments, the aforementioned proximity range includes a horizontal direction and a vertical direction.
以另一個角度來說,本發明的實施例提出一種電子裝置,包括記憶體與處理器。記憶體儲存有多個指令,處理器用以執行這些指令以完成上述的基於深度影像生成方法。 From another perspective, an embodiment of the present invention provides an electronic device including a memory and a processor. The memory stores a plurality of instructions, and the processor executes these instructions to complete the above-mentioned depth-based image generation method.
以另一個角度來說,本發明的實施例提出一種電腦程式產品,由電子裝置載入並執行以完成上述的基於深度影像生成方法。 From another perspective, an embodiment of the present invention provides a computer program product that is loaded and executed by an electronic device to complete the above-mentioned depth-based image generation method.
採用上述的基於深度影像生成方法可以減少空洞也可以減少計算量。 Using the above-mentioned depth-based image generation method can reduce holes and also reduce the amount of calculation.
為讓本發明的上述特徵和優點能更明顯易懂,下文特舉實施例,並配合所附圖式作詳細說明如下。 In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail in conjunction with the accompanying drawings.
100:電子裝置 100: electronic device
110:處理器 110: processor
120:記憶體 120: memory
210,230,240,250:二維影像 210,230,240,250: 2D image
220:深度影像 220: Depth image
310,320,330,401~405:步驟 310, 320, 330, 401~405: steps
411:曲移後的二維影像 411: Two-dimensional image after warping
412:曲移後的深度影像 412: Depth image after warping
410:曲移後影像 410: Image after curve shift
420:空洞區 420: Hollow Area
I(x,y):像素 I(x,y): pixel
D(x,y):景深值 D(x,y): depth of field value
( ,y):曲移後深度影像 ( ,y ): Depth image after curve shift
( ,y):曲移後二維影像 ( ,y ): Two-dimensional image after warping
(x,y):曲移後影像 ( x,y ): image after shifting
R x :鄰近範圍 R x : Proximity range
(x,y):最大深度值 ( x,y ): maximum depth value
N( , ):集合 N ( , ):gather
610,620,630:步驟 610,620,630: steps
611:第一填補後影像 611: First Filled Image
621:第二填補後影像 621: second post-fill image
631:影像 631: image
[圖1]是根據一實施例繪示電子裝置的示意圖。 [Fig. 1] is a schematic diagram showing an electronic device according to an embodiment.
[圖2]是根據一實施例繪示基於深度影像生成方法的示意圖。 [Fig. 2] is a schematic diagram showing a depth-based image generation method according to an embodiment.
[圖3]是根據一實施例繪示基於深度影像生成方法的流程 示意圖。 [Fig. 3] is a flowchart of a depth-based image generation method according to an embodiment Schematic.
[圖4A]與[圖4B]是根據一實施例繪示加權式3D小數曲移計算的流程示意圖。 [FIG. 4A] and [FIG. 4B] are schematic diagrams illustrating the flow of weighted 3D decimal warp calculation according to an embodiment.
[圖5A與圖5B]是根據一實施例繪示曲移後二維影像與曲移後深度影像的局部示意圖。 [FIG. 5A and FIG. 5B] are partial schematic diagrams showing a two-dimensional image after a curve shift and a depth image after a curve shift according to an embodiment.
[圖6]是根據一實施例繪示雙徑梯度空洞填補方法的示意圖。 [Fig. 6] is a schematic diagram illustrating a method for filling a double-path gradient hole according to an embodiment.
關於本文中所使用之「第一」、「第二」等,並非特別指次序或順位的意思,其僅為了區別以相同技術用語描述的元件或操作。 Regarding the “first”, “second”, etc. used in this text, it does not particularly mean the order or sequence, but only to distinguish elements or operations described in the same technical terms.
圖1是根據一實施例繪示電子裝置的示意圖。請參照圖1,電子裝置100可以是智慧型手機、平板電腦、個人電腦、筆記型電腦、伺服器、工業電腦或具有計算能力的各種電子裝置等,本發明並不在此限。電子裝置100包括了處理器110與記憶體120,其中處理器110可為中央處理器、微處理器、微控制器、影像處理晶片、特殊應用積體電路等,記憶體120可為隨機存取記憶體、唯讀記憶體、快閃記憶體、軟碟、硬碟、光碟、隨身碟、磁帶或是可透過網際網路存取之資料庫,其中儲存有多個指令,處理器110會執行這些指令來完成一基於深度影像生成方法,以下將詳細說明此方法。
FIG. 1 is a schematic diagram illustrating an electronic device according to an embodiment. 1, the
圖2是根據一實施例繪示基於深度影像生成方法
的示意圖。請參照圖2,基於深度影像生成方法是用以根據一張二維影像210與對應此二維影像210的深度影像220來產生不同視角的二維影像。例如,二維影像210的視角設定為視角0,深度影像220包括從此視角0量測到的深度值,在此實施例中越靠近鏡頭的景深值越大,但本發明並不在此限。根據二維影像210與深度影像可以產生視角1的二維影像230、視角2的二維影像240以及視角-1的二維影像250,以此類推。在此用視角1~視角3來表示右方的視角,視角-1~視角-3來表示左方的視角,但本發明並不在此限。舉例來說,如果二維影像210中具有一前景物件,從右方視角觀察時此前景物件會向左位移,從左方視角觀察時此前景物件會向右位移,當前景物件越靠近鏡頭時位移會越大。
Fig. 2 illustrates a depth-based image generation method according to an embodiment
Schematic diagram. Please refer to FIG. 2, the depth-based image generation method is used to generate two-dimensional images with different viewing angles according to a two-
在此,二維影像210與深度影像220可以分別透過影像感測器與深度感測器取得,也可從既有的資料庫或其他裝置取得。上述的影像感測器可包括感光耦合元件(Charge-coupled Device,CCD)感測器、互補性氧化金屬半導體(Complementary Metal-Oxide Semiconductor)感測器或其他合適的感光元件。上述的深度感測器可包括雙攝影機、人工智慧深度生成網路、結構光感測裝置或任意可以感測場景深度的裝置。
Here, the two-
圖3是根據一實施例繪示基於深度影像生成方法的流程示意圖。請參照圖3,在步驟310會對深度影像220進行預處理,例如進行平滑處理以減少深度值變化,在此
可以採用任意的預處理,本發明並不限制步驟310的內容。在步驟320中,根據預處理以後的深度影像220對二維影像210做像素曲移,特別的是在此實施例提出一種加權式3D小數曲移計算。執行步驟320後會取得一個曲移後影像,此曲移後影像包括一或多個空洞區,在步驟330會填補這些空洞區,在此實施例會採用一個雙徑梯度空洞填補方法,填補空洞區以後可以產生虛擬視角影像,例如為二維影像230。以下將舉實施例詳細說明步驟320與步驟330。
FIG. 3 is a schematic flowchart of a depth-based image generation method according to an embodiment. Please refer to FIG. 3, in
圖4A與圖4B是根據一實施例繪示加權式3D小數曲移計算的流程示意圖。請參照圖4A,在此,二維影像210中的像素表示為I(x,y),預處理過後的深度影像220的景深值表示為D(x,y),x與y分別代表X座標與Y座標。上述的步驟320包括了步驟401~405,在步驟401中,根據深度影像220中的每一個深度值計算一小數點位置。具體來說,根據深度值可以計算出一視差位移值,如以下數學式1所示。
4A and 4B are schematic diagrams illustrating the flow of weighted 3D decimal warp calculation according to an embodiment. Please refer to FIG. 4A. Here, the pixels in the two-
其中(x,y)為視差位移值。K表示小數點位置的小數點精確度,K可為2、4、8或任意正整數。m表示視角,例如為圖2所示的-3~-1與1~3的其中之一。P c 為景深視差參數(depth disparity ratio),可依照相機及立體顯示器的參數來決定。表示捨去小數的整數。上述的數學
式1是將傳統的視差位移值乘上K倍,經過四捨五入的運算以後再除以K倍,但值得注意的是視差位移值(x,y)可以是整數也可以有小數,若以K=4為例,小數值可為.00、.25、.50、.75等四個值。接下來,將x座標加上視差位移值(x,y)便可以得到一個小數點位置。
in ( x,y ) is the parallax displacement value. K represents the decimal point accuracy of the decimal point position, K can be 2, 4, 8 or any positive integer. m represents the viewing angle, for example, one of -3 to -1 and 1 to 3 shown in FIG. 2. P c is the depth disparity ratio, which can be determined according to the parameters of the camera and the stereo display. Represents an integer with decimals rounded off. The above
在步驟402,將原本的景深值D(x,y)移動至上述的小數點位置以產生曲移後深度影像,此步驟可以表示為以下數學式2,( ,y)即是曲移後深度影像,x+(x,y)為小數點位置。值得注意的是Y座標不需要位移。
In
在步驟403,將二維影像210中對應的像素值填入小數點位置以產生曲移後二維影像,此步驟可以表示為以下數學式3,( ,y)即是曲移後二維影像。
In
如果小數點位置發生重疊的情形,即不同的x對應至相同的x+(x,y),則可以保留較大的深度值,捨棄較小的深度值。 If the position of the decimal point overlaps, that is, different x corresponds to the same x + ( x,y ), the larger depth value can be kept, and the smaller depth value can be discarded.
由於在數學式1中視差位移值(x,y)可以包括K個小數,這等同於曲移後二維影像( ,y)的寬度會是二維影像I(x,y)的K倍,如曲移後二維影像411所示。相同的,曲移後深度影像( ,y)的寬度會是深度影像D(x,y)的K倍,如曲移後深度影像412所示。
Since the parallax displacement value in Mathematical formula 1 ( x, y ) can include K decimals, which is equivalent to a two-dimensional image after warping ( , y ) will be K times the width of the two-dimensional image I ( x, y ), as shown in the two-
圖5A與圖5B是根據一實施例繪示曲移後二維影像與曲移後深度影像的局部示意圖。請參照圖5A,由於Y座標上沒有位移,在此以X座標來表示曲移後二維影像( ,y)與曲移後深度影像( ,y),在此繪示了局部(=98~101)的像素值與深度值,值得注意的是較粗的標線表示整數點位置,如=98、=99、=100、=101等,而其他的標線表示小數點位置,如.25、.50、.75等。在此例子中,景深值“30”被填入=99.50的位置,景深值“121”被填入=99.75的位置,而景深值“120”被填入=100.25。每個景深值都對應至一個像素值,景深值“30”是對應至像素值“68”,景深值“121”是對應至像素值“215”,景深值“120”是對應至像素值“212”。接下來要計算每個整數點位置上的像素值。 5A and 5B are partial schematic diagrams illustrating a two-dimensional image after a curvature shift and a depth image after a curvature shift according to an embodiment. Please refer to Figure 5A, since there is no displacement on the Y coordinate, the X coordinate is used to represent the two-dimensional image after the shift ( ,y ) and the depth image after shifting ( ,y ), here is a part ( =98~101) pixel value and depth value, it is worth noting that the thicker marking line indicates the position of the integer point, such as =98, =99、 =100, =101, etc., while other marking lines indicate the position of the decimal point, such as .25, .50, .75, etc. In this example, the depth of field value "30" is filled in =99.50 position, the depth of field value "121" is filled in =99.75 position, and the depth of field value "120" is filled in =100.25. Each depth value corresponds to a pixel value. The depth value "30" corresponds to the pixel value "68", the depth value "121" corresponds to the pixel value "215", and the depth value "120" corresponds to the pixel value "212". The next step is to calculate the pixel value at each integer point position.
請參照圖4A與圖5A,在步驟404中,對於曲移後深度影像中的每個整數點位置,計算整數點位置的鄰近範圍內的最大深度值,並且刪除鄰近範圍內與最大深度值相差超過一臨界值的深度值。若整數點位置為x,則鄰近範圍可以表示為以下數學式4。
4A and 5A, in
R x 表示鄰近範圍,也表示在此範圍內所有所形成的集合。在此實施例中K=4,因此鄰近範圍R x 可以設定為左方2個小數點位置到右方2個小數點位置,以整數點位置“100”為例,鄰近範圍R x 表示從=99.50至=100.50的範圍。 接下來可以根據鄰近範圍R x 的像素值來決定整數點位置=100上的像素值,如果有多個像素被位移至此鄰近範圍R x ,則保留與最大景深值相差一臨界值內的景深值,因為較大的景深值在前景,會遮蓋位於背景的像素值。具體來說,鄰近範圍R x 內的最大深度值為“121”,表示為(x,y),如以下數學式5所示。鄰近範圍R x 內剩餘的景深值形成一個前景景深值集合,表示為以下數學式6。 R x represents the adjacent range, and also represents all within this range The resulting collection. In this embodiment, K=4, so the adjacent range R x can be set from 2 decimal point positions on the left to 2 decimal point positions on the right. Taking the integer point position "100" as an example, the adjacent range R x means from =99.50 to = 100.50 range. Next, the integer point position can be determined according to the pixel value of the neighboring range R x =100 pixel value, if multiple pixels are shifted to this adjacent range R x , the depth value within a critical value from the maximum depth value is retained, because the larger depth value is in the foreground and will cover the pixels in the background value. Specifically, the maximum depth value in the adjacent range R x is "121", which is expressed as ( x, y ), as shown in Math 5 below. The remaining depth values in the adjacent range R x form a foreground depth value set, which is expressed as the following mathematical formula 6.
S(x,y)為前景景深值集合,在此例子中包括了“121”與“120”,而景深值“30”則被刪除。ε D 為臨界值,可設定為任意合適的數值。 S(x,y) is the foreground depth value set. In this example, "121" and "120" are included, and the depth value "30" is deleted. ε D is a critical value and can be set to any suitable value.
此外,曲移後二維影像( ,y)中對應的像素值也會被刪除,剩餘的景深值與像素值如圖5B所示,剩餘的像素值可以表示為以下數學式7。在此例子中剩下的像素值為“215”與“212”。 In addition, the two-dimensional image after the curve shift ( The corresponding pixel value in ,y ) will also be deleted. The remaining depth value and pixel value are shown in FIG. 5B, and the remaining pixel value can be expressed as the following mathematical formula 7. The remaining pixel values in this example are "215" and "212".
值得注意的是,在圖5A與圖5B中鄰近範圍R x 只包括水平方向,但在一些實施例中也可包括垂直方向。舉例來說,上述的鄰近範圍R x 可以替換為R x,y ,表示如以下數學式8。上述的數學式5可以替換為以下數學式9。 It should be noted that the adjacent range R x in FIGS. 5A and 5B only includes the horizontal direction, but in some embodiments, it may also include the vertical direction. For example, the aforementioned proximity range R x can be replaced with R x,y , which is expressed as the following mathematical formula 8. The above-mentioned Mathematical Formula 5 can be replaced with the following Mathematical Formula 9.
參照圖4A,接下來在步驟405,可以根據鄰近範圍R x 內剩餘的深度值所對應的像素值計算整數點位置上的像素值。因為距離整數點位置越近的像素值越重要,在此實施例中可先計算整數點位置與鄰近範圍內剩餘的深度值的位置之間的高斯距離,然後以高斯距離作為權重來加總這些剩餘的深度值所對應的像素值以計算出整數點位置上的像素值,在其他實施例中上述高斯距離亦可以替換為其他線性或非線性距離權重。步驟405可以表示為以下數學式10與數學式11。
4A, the
其中(x,y)為整數點位置的X座標與Y座標。σ為參數,代表高斯分布的變異數。N( , )為鄰近範圍R x,y 內剩餘像素值的X座標與Y座標所形成的集合。雖然圖5A與圖5B只採用了X方向的鄰近範圍,但上述的數學式10、11是更泛用(general)的表示,不只採用X方向上的相鄰像素值,也採用了Y方向上的相鄰像素值。在計算出整數點
位置上的像素值以後可以得到曲移後影像(x,y),例如圖4B的曲移後影像410,其中包括至少一個空洞區420。在此實施例中是要產生出左方的視角,因此影像中的物件會往右位移,在物件的左方會產生空洞區420。
Where (x, y) is the X coordinate and Y coordinate of the integer point position. σ is a parameter, which represents the variance of the Gaussian distribution. N ( , ) Is the set formed by the X and Y coordinates of the remaining pixel values in the adjacent range R x,y. Although Figures 5A and 5B only use the proximity range in the X direction, the above mathematical formulas 10 and 11 are more general expressions. Not only the adjacent pixel values in the X direction are used, but also the Y direction is used. The neighboring pixel value. After calculating the pixel value at the integer point position, the warped image can be obtained ( x, y ), for example, the
習知的像素曲移方法因為只採用整數值,當有小數點時只好捨去小數點,因此會額外產生許多孔洞。相反地,上述的做法保留小數點,因此可大量減少孔洞的產生。 Because the conventional pixel shift method only uses integer values, when there is a decimal point, the decimal point has to be discarded, so many additional holes are generated. On the contrary, the above method retains the decimal point, so the generation of holes can be greatly reduced.
接下來執行雙徑梯度空洞填補方法,如圖6所示。在此實施例中會用兩個方向來填補曲移後影像410。具體來說,在步驟610中,由上而下填補曲移後影像410中的空洞區以產生第一填補後影像611。當由上而下,由左而右地掃瞄曲移後影像410時,可以確保空洞的上方像素、左上方像素與左方像素有值(不是空洞),因此對於空洞區中的一個像素(x,y),可以根據上方像素與左上方像素計算水平梯度值,如以下數學式12所示,另外可根據左方像素與左上方像素計算垂直梯度值,如以下數學式13所示。
Next, perform the double-path gradient hole filling method, as shown in Figure 6. In this embodiment, two directions are used to fill the
其中(x-1,y-1)為左上方像素,(x,y-1)為上方像素,(x-1,y)為左方像素。若水平梯度值大於垂直梯度值,表示在像素(x,y)周圍存在垂直邊緣,因此採用上方像素(x,y-1)來填補像素(x,y)。另一方面,如果垂直梯度值 大於水平梯度值,表示在像素(x,y)周圍存在水平邊緣,則採用左方像素(x-1,y)來填補像素(x,y)。換言之,步驟610是根據以下數學式14來進行由上往下填補。 in ( x -1 ,y -1) is the upper left pixel, ( x,y -1) is the upper pixel, ( x -1 , y ) is the left pixel. If the horizontal gradient value Greater than the vertical gradient value , Expressed in pixels There are vertical edges around ( x,y ), so the upper pixels are used ( x,y -1) to fill the pixels ( x,y ). On the other hand, if the vertical gradient value Greater than the horizontal gradient value , Expressed in pixels If there is a horizontal edge around (x,y ), the left pixel is used ( x -1 , y ) to fill the pixels ( x,y ). In other words, step 610 is to fill in from top to bottom according to the following formula 14.
所得到的影像(x,y)即是第一填補後影像611。在此實施例中當如果垂直梯度值等於水平梯度值時是採用左方像素,但在其他實施例也可以採用上方像素,本發明並不在此限。
The resulting image ( x,y ) is the first
另一方面,在步驟620中,由下而上填補曲移後影像410中的空洞區以產生第二填補後影像621。如果從下而上,由左到上進行掃描,下方像素(x,y+1)、左方像素(x-1,y)與左下方像素(x-1,y+1)有值(不是空洞),因此可以根據下方像素與左下方像素計算水平梯度值,如以下數學式15所示,並且根據左方像素與左下方像素計算垂直梯度值,如以下數學式16所示。
On the other hand, in
若水平梯度值大於垂直梯度值,表示像素(x,y)周圍存在垂直邊緣,因此採用下方像素(x,y+1)來填補像素(x,y)。如果垂直梯度值大於水平梯度值,採用 左方像素(x-1,y)來填補像素(x,y)。換言之,步驟620是根據以下數學式17進行由下往上填補。 If the horizontal gradient value Greater than the vertical gradient value For pixels There are vertical edges around ( x,y ), so the pixels below are used ( x,y +1) to fill the pixels ( x,y ). If the vertical gradient value Greater than the horizontal gradient value , Using pixels on the left ( x -1 , y ) to fill the pixels ( x,y ). In other words, step 620 is to fill in from bottom to top according to the following formula 17.
所得到的影像(x,y)即是第二填補後影像621。值得注意的是,當像素周圍有45度的邊緣時,第一填補後影像611的填補效果較好,當像素周圍有135度的邊緣時,第二填補後影像621的填補效果較好。因此,在步驟630中會將空洞區的每一個像素分為兩個角度類別,接下來可以根據角度類別來從第一填補後影像611與第二填補後影像621擇一以填補對應的像素。實作上我們採用一個遮罩M(x,y),如果是空洞區則在對應的位置填入0,非空洞區則填入255,如以下數學式18所示。
The resulting image ( x, y ) is the second
對於空洞區的每一個像素,如果左方像素與上方像素為非空洞像素,將對應的像素分類為45度的角度類別,並且在遮罩M(x,y)中對應的位置填入128,此判斷也可表示為以下數學式19。 For each pixel in the hole area, if the left pixel and the upper pixel are non-hole pixels, the corresponding pixel is classified into a 45 degree angle category, and 128 is filled in the corresponding position in the mask M(x,y), This judgment can also be expressed as Equation 19 below.
[數學式19]M(x,y)=128,if M(x,y)=0 and M(x-1,y)=M(x,y-1)=255 [Math 19] M(x,y)=128,if M(x,y)=0 and M(x-1,y)=M(x,y-1)=255
接下來基於曲移概念,空洞像素會水平延伸,將45度的角素類別往右延伸,此水平延伸可以表示為以下數 學式20。 Next, based on the concept of warping, the hole pixels will extend horizontally, extending the 45-degree pixel category to the right. This horizontal extension can be expressed as the following number School formula 20.
[數學式20]M(x,y)=128,if M(x,y)=0 and M(x-1,y)=128 [Math 20] M(x,y)=128,if M(x,y)=0 and M(x-1,y)=128
最後再進行垂直延伸,如以下數學式21所示。 Finally, the vertical extension is performed, as shown in the following mathematical formula 21.
[數學式21]M(x,y)=128,if M(x,y)=0 and M(x,y+1)=128 and M(x-1,y)=255 [Math 21] M(x,y)=128,if M(x,y)=0 and M(x,y+1)=128 and M(x-1,y)=255
在垂直延伸以後,遮罩M(x,y)中剩餘標記為“0”的位置則分類為135度的角度類別。如果某像素屬於45度的角度類別,則採用第一填補後影像611的結果進行填補;如果某像素屬於135度的角度類別,則採用第二填補後影像621的結果進行填補,此步驟可以表示為以下數學式22。
After extending vertically, the remaining positions marked as "0" in the mask M(x,y) are classified into the angle category of 135 degrees. If a pixel belongs to the angle category of 45 degrees, the result of the first
最後得到的影像(x,y)例如為圖6的影像631,在各個角度類別都填補的很好。
The final image ( x, y ) is, for example, the
在此實施例中是以遮罩M(x,y)來對像素進行分類,但在其他實施例中也可以採用任意資料結構、任意符號來進行分類。以另一個角度來說,對於空洞區的每一個像素,如果此像素為空洞像素且左方像素與上方像素為非空洞像素,將此像素分類為45度的角度類別;如果此像素為空洞像素且左方像素屬於45度的角度類別,則將此像素分類為45度的角度類別;如果此像素為空洞像素,下方像素屬於45度的角度類別,並且左方像素為非空洞像素,將 此像素分類為45度的角度類別;在經過上述步驟以後將其餘的空洞像素分類為135度的角度類別。本發明並不限制用什麼數字、符號或字母來標記上述的角度類別。 In this embodiment, a mask M(x, y) is used to classify pixels, but in other embodiments, any data structure and any symbol can also be used for classification. From another perspective, for each pixel in the cavity area, if this pixel is a hole pixel and the left and upper pixels are non-hole pixels, classify this pixel as a 45 degree angle category; if this pixel is a hole pixel And the pixel on the left belongs to the angle category of 45 degrees, then the pixel is classified as the angle category of 45 degrees; if the pixel is a hole pixel, the pixel below belongs to the angle category of 45 degrees, and the pixel on the left is a non-hole pixel, then This pixel is classified into an angle category of 45 degrees; after the above steps, the remaining hole pixels are classified into an angle category of 135 degrees. The present invention does not limit what numbers, symbols or letters are used to mark the above-mentioned angle categories.
以另外一個角度來說,本發明也提出了一電腦程式產品,此產品可由任意的程式語言及/或平台所撰寫,當此電腦程式產品被載入至電腦系統並執行時,可執行上述的基於深度影像生成方法。 From another perspective, the present invention also proposes a computer program product. This product can be written in any programming language and/or platform. When the computer program product is loaded into the computer system and executed, the above-mentioned computer program product can be executed. Based on the depth image generation method.
在上述提出的基於深度影像生成方法中,相較於現有方法來說可以獲得更自然且計算低的合成虛擬視角影像。 In the above-mentioned depth-based image generation method, a more natural and low-computing synthetic virtual perspective image can be obtained compared to the existing methods.
雖然本發明已以實施例揭露如上,然其並非用以限定本發明,任何所屬技術領域中具有通常知識者,在不脫離本發明的精神和範圍內,當可作些許的更動與潤飾,故本發明的保護範圍當視後附的申請專利範圍所界定者為準。 Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to those defined by the attached patent application scope.
210,230:二維影像 210,230: Two-dimensional image
220:深度影像 220: Depth image
310,320,330:步驟 310, 320, 330: steps
Claims (7)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW109121387A TWI736335B (en) | 2020-06-23 | 2020-06-23 | Depth image based rendering method, electrical device and computer program product |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW109121387A TWI736335B (en) | 2020-06-23 | 2020-06-23 | Depth image based rendering method, electrical device and computer program product |
Publications (2)
Publication Number | Publication Date |
---|---|
TWI736335B true TWI736335B (en) | 2021-08-11 |
TW202201344A TW202201344A (en) | 2022-01-01 |
Family
ID=78283135
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
TW109121387A TWI736335B (en) | 2020-06-23 | 2020-06-23 | Depth image based rendering method, electrical device and computer program product |
Country Status (1)
Country | Link |
---|---|
TW (1) | TWI736335B (en) |
Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101630408A (en) * | 2009-08-14 | 2010-01-20 | 清华大学 | Depth map treatment method and device |
CN101640809A (en) * | 2009-08-17 | 2010-02-03 | 浙江大学 | Depth extraction method of merging motion information and geometric information |
CN102598674A (en) * | 2009-10-23 | 2012-07-18 | 高通股份有限公司 | Depth map generation techniques for conversion of 2d video data to 3d video data |
TWI419078B (en) * | 2011-03-25 | 2013-12-11 | Univ Chung Hua | Apparatus for generating a real-time stereoscopic image and method thereof |
TWI439961B (en) * | 2011-03-08 | 2014-06-01 | Univ Nat Chi Nan | Conversion algorithm for voids generated after converting 2D images |
CN104519348A (en) * | 2013-09-30 | 2015-04-15 | 西斯维尔科技有限公司 | Method and device for edge shape enforcement for three-dimensional video stream |
TWI493963B (en) * | 2011-11-01 | 2015-07-21 | Acer Inc | Image generating device and image adjusting method |
TWI497444B (en) * | 2013-11-27 | 2015-08-21 | Au Optronics Corp | Method and apparatus for converting 2d image to 3d image |
CN105144714A (en) * | 2013-04-09 | 2015-12-09 | 联发科技股份有限公司 | Method and apparatus of disparity vector derivation in 3d video coding |
CN105191319A (en) * | 2013-03-18 | 2015-12-23 | 高通股份有限公司 | Simplifications on disparity vector derivation and motion vector prediction in 3D video coding |
TWI527431B (en) * | 2012-04-16 | 2016-03-21 | 高通公司 | View synthesis based on asymmetric texture and depth resolutions |
TWM529333U (en) * | 2016-06-28 | 2016-09-21 | Nat Univ Tainan | Embedded three-dimensional image system |
CN107147906A (en) * | 2017-06-12 | 2017-09-08 | 中国矿业大学 | A kind of virtual perspective synthetic video quality without referring to evaluation method |
CN107578418A (en) * | 2017-09-08 | 2018-01-12 | 华中科技大学 | A kind of indoor scene profile testing method of confluent colours and depth information |
TW201828255A (en) * | 2016-12-06 | 2018-08-01 | 荷蘭商皇家飛利浦有限公司 | Apparatus and method for generating a light intensity image |
TW201944775A (en) * | 2018-04-12 | 2019-11-16 | 美商傲思丹度科技公司 | Methods for MR-DIBR disparity map merging and disparity threshold determination |
-
2020
- 2020-06-23 TW TW109121387A patent/TWI736335B/en active
Patent Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101630408A (en) * | 2009-08-14 | 2010-01-20 | 清华大学 | Depth map treatment method and device |
CN101640809A (en) * | 2009-08-17 | 2010-02-03 | 浙江大学 | Depth extraction method of merging motion information and geometric information |
CN102598674A (en) * | 2009-10-23 | 2012-07-18 | 高通股份有限公司 | Depth map generation techniques for conversion of 2d video data to 3d video data |
TWI439961B (en) * | 2011-03-08 | 2014-06-01 | Univ Nat Chi Nan | Conversion algorithm for voids generated after converting 2D images |
TWI419078B (en) * | 2011-03-25 | 2013-12-11 | Univ Chung Hua | Apparatus for generating a real-time stereoscopic image and method thereof |
TWI493963B (en) * | 2011-11-01 | 2015-07-21 | Acer Inc | Image generating device and image adjusting method |
TWI527431B (en) * | 2012-04-16 | 2016-03-21 | 高通公司 | View synthesis based on asymmetric texture and depth resolutions |
CN105191319A (en) * | 2013-03-18 | 2015-12-23 | 高通股份有限公司 | Simplifications on disparity vector derivation and motion vector prediction in 3D video coding |
CN105144714A (en) * | 2013-04-09 | 2015-12-09 | 联发科技股份有限公司 | Method and apparatus of disparity vector derivation in 3d video coding |
CN104519348A (en) * | 2013-09-30 | 2015-04-15 | 西斯维尔科技有限公司 | Method and device for edge shape enforcement for three-dimensional video stream |
TWI497444B (en) * | 2013-11-27 | 2015-08-21 | Au Optronics Corp | Method and apparatus for converting 2d image to 3d image |
TWM529333U (en) * | 2016-06-28 | 2016-09-21 | Nat Univ Tainan | Embedded three-dimensional image system |
TW201828255A (en) * | 2016-12-06 | 2018-08-01 | 荷蘭商皇家飛利浦有限公司 | Apparatus and method for generating a light intensity image |
CN107147906A (en) * | 2017-06-12 | 2017-09-08 | 中国矿业大学 | A kind of virtual perspective synthetic video quality without referring to evaluation method |
CN107578418A (en) * | 2017-09-08 | 2018-01-12 | 华中科技大学 | A kind of indoor scene profile testing method of confluent colours and depth information |
TW201944775A (en) * | 2018-04-12 | 2019-11-16 | 美商傲思丹度科技公司 | Methods for MR-DIBR disparity map merging and disparity threshold determination |
Also Published As
Publication number | Publication date |
---|---|
TW202201344A (en) | 2022-01-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7181977B2 (en) | Method and system for detecting and combining structural features in 3D reconstruction | |
US10540576B1 (en) | Panoramic camera systems | |
Li et al. | PMSC: PatchMatch-based superpixel cut for accurate stereo matching | |
JP7403528B2 (en) | Method and system for reconstructing color and depth information of a scene | |
US20170365100A1 (en) | Augmented Reality Occlusion | |
TWI550559B (en) | Image segmentation device and image processing method | |
TWI398158B (en) | Method for generating the depth of a stereo image | |
US20150097827A1 (en) | Target Region Fill Utilizing Transformations | |
Li et al. | Depth-preserving warping for stereo image retargeting | |
CN111583381B (en) | Game resource map rendering method and device and electronic equipment | |
CN107341804B (en) | Method and device for determining plane in point cloud data, and method and equipment for image superposition | |
US20160232420A1 (en) | Method and apparatus for processing signal data | |
CN111369435B (en) | Color image depth up-sampling method and system based on self-adaptive stable model | |
TWI736335B (en) | Depth image based rendering method, electrical device and computer program product | |
CN116228850A (en) | Object posture estimation method, device, electronic equipment and readable storage medium | |
Tabata et al. | Shape-Net: Room Layout Estimation from Panoramic Images Robust to Occlusion using Knowledge Distillation with 3D Shapes as Additional Inputs | |
TWI526045B (en) | Method and image processing device for adjusting stereo images | |
WO2023233575A1 (en) | Estimation device, learning device, estimation method, learning method, and program | |
Kadmin et al. | Local Stereo Matching Algorithm Using Modified Dynamic Cost Computation [J] | |
Zhang et al. | [Retracted] A 3D Face Modeling and Recognition Method Based on Binocular Stereo Vision and Depth‐Sensing Detection | |
US20230239449A1 (en) | Stereoscopic image generating device and stereoscopic image generating method | |
CN116977394A (en) | Video generation method, apparatus, device, storage medium, and computer program product | |
Lin et al. | Kinect depth image enhancement with adaptive joint multi-lateral discrete filters | |
KR20240106531A (en) | The Method and System to Construct Multi-point Real-time Metaverse Content Data Based on Selective Super-resolution | |
Zeng et al. | Deep Unrolled Weighted Graph Laplacian Regularization for Depth Completion |