200828182 九、發明說明： 【發明所屬之技術領域】 本發明係關於—種建構三維擬真關模型的方法，特別是一 種解決遮蔽域f彡像之三_纽·魏構的方法。 【先前技術】 二維擬真模型的建構過程中，常奋利用彳p & 修補美化朴砂人1 β艰處理技術來200828182 IX. Description of the invention: [Technical field to which the invention pertains] The present invention relates to a method for constructing a three-dimensional pseudo-reality model, and more particularly to a method for solving a three-dimensional configuration of a shadow domain. [Prior Art] In the construction process of two-dimensional immersive model, Chang Fen used 彳p & repairing landscaping
補吳化細獻錄，財蹟雜純贿触巾為例，常 利用影像來纽古蹟的三維_，但是取得之影像常常會受到遮 :物的影響，使得幾何模型的建構以及材f貼畴會遭遇困難。 第ΙΑ ®係為所要處理影像之遮蔽問題之示意圖，在重建三維模 於遮蔽物的影響，”不能_僅有讀之影像，岭得到遮蔽 物被當成W景魏之影像，在嚴重遮蔽的情況下，應用傳統三維 重建的流絲特_取和對紅科造成料騎誤，也因此無 法獲得良好的二維重建結果’所以在遮蔽的情況下，重建擬真的 二維貼圖模型更是一個重大挑戰。 型的過程中，會從各個不同角度來擷取欲重建主體之影像，但由For example, Wu Hua’s fine-grained records, such as the miscellaneous bribes, often use images to create three-dimensional images of the New Monuments, but the images obtained are often affected by objects: the construction of geometric models and the construction of geometric models. Will encounter difficulties. Dijon® is a schematic diagram of the problem of shadowing of the image to be processed. In reconstructing the effect of the three-dimensional model on the mask, “Can’t read the image only, the ridge gets the shadow as the image of W King Wei, in the case of severe obscuration. Under the traditional three-dimensional reconstruction of the streamline _ take and the red section caused by the riding error, and therefore can not get a good two-dimensional reconstruction results 'so in the case of shadowing, reconstruction of the immersive two-dimensional map model is a Major challenges. In the process of the type, the image of the subject to be reconstructed is taken from different angles, but by
習知技術中，針對類似三維擬真影像重建的技術方案如美國 •專利第 US6571024 號，專利名稱為 rMeth〇d and apparatus f〇r 、multl_view three d血ensional estimation」，主要技術手段在於利用 影像序列作為輸入，利用電腦視覺技術恢復攝影機拍攝方位和景 物之三維資訊，並將新物件合成至原先輸入之影像序列中。 美國專利第US6373487號，專利名稱為「Meth〇ds and apparatus for constructing a 3D model of a scene from calibrated 5 200828182 images of the scene」，主要技術手段在於利用校正過後的多部相 機’拍攝物體各角度的剪影卿〇uette)，進而使用vi腿lhuU之原 理重建物體三維資訊。 美國專利第US6643396號，專利名稱為r Acquisitl〇n 〇f 3_D s_s wrth a single hand held Camera」，主要技術手段在於利用單一相機配上 四面鏡子來取得多角度的影像即可建構出輯的場景。其中包含 自我校正技術，並可得到絕對尺寸的三維模型。In the prior art, the technical solution for the similar three-dimensional imaginary image reconstruction is US Patent No. US6571024, the patent name is rMeth〇d and apparatus f〇r, multl_view three d blood ensional estimation, and the main technical means is to use the image sequence. As input, computer vision technology is used to restore the camera's three-dimensional information about the orientation and scene, and to synthesize the new object into the original input image sequence. U.S. Patent No. 6,373,487, entitled "Meth〇ds and apparatus for constructing a 3D model of a scene from calibrated 5 200828182 images of the scene", the main technical means is to use the corrected plurality of cameras to 'shoot various angles of the object The silhouette is 〇uette), and then the three-dimensional information of the object is reconstructed using the principle of the vi leg lhuU. U.S. Patent No. 6,643,396, entitled "R Acquisitl〇n 〇f 3_D s_s wrth a single hand held Camera", the main technical means is to use a single camera with four mirrors to obtain multi-angle images to construct a scene. It includes self-correcting techniques and a three-dimensional model of absolute size.
美國專利第US5511153號，專利名稱為「Meth〇d⑽哪啦加 for three-dunensional, textured models fr〇m piural vlde〇 images d , ^ 要_手段在於侧影像序絲作三維重建。擷取影像的特徵來 十二出相躺⑽轉數’以建構出拍攝的幾何模型。並將影像 依實際物體劃分出不同區域來做貼圖。 美國專利第麵观4號，專利名稱為「祕〇d㈣挪細 〇rreconstruction of multiple Vlews wlth altenng search path and 袖社US1〇Lm〇delmg」’主要技術手段在於從多視角影像中之挑選特 往遮蔽的鋪_遞。、德傳播’並避免搜尋路徑 :般而言’主體的幾何模型會因為遮蔽產生破洞，而遮蔽物 被移叫細彳认)，但杨遮蔽物可 峡口疋在現％，而磁或不易排除的(如招牌、標 他建:在這種情況下，無論從任何⑽ _纽 乾雨也鏡。因此，树古蹟建_蝴== 6 200828182 f丈位模觀料，·不相干的鑛物總是造餘當程度的干 擾，使得數位影像或三維模型無法正確呈現標地物的形貌。干 【發明内容】 本發明提出一種解決遮蔽主體影像之三維擬真貼圖模型建 構的方法，包括下雅影像，魏分影像US Patent No. 5,511,153, the patent name is "Meth〇d (10), which is for three-dunensional, textured models fr〇m piural vlde〇images d , ^ _ means the lateral image sequence silk for three-dimensional reconstruction. Capture the characteristics of the image Twelve out the lying (10) revolutions to construct the geometric model of the shooting. The image is divided into different areas according to the actual object to make the map. US Patent No. 4, the patent name is "secret d (four) moving fine Rreconstruction of multiple Vlews wlth altenng search path and sleeves US1〇Lm〇delmg"' The main technical means is to select the special shaded from the multi-view image. "German spreads 'and avoids the search path: in general, the geometric model of the subject will cause holes due to obscuration, and the cover is moved to be fine-grained," but the Yang shield can be smashed in the present, while magnetic or It is not easy to rule out (such as signboards, marking him to build: in this case, no matter from any (10) _ New Dry Rain is also mirror. Therefore, the tree monuments are built _ butterfly == 6 200828182 f position model, irrelevant Minerals always interfere with the degree of remnant, so that the digital image or the three-dimensional model can not correctly display the topography of the landmark. [Invention] The present invention provides a method for constructing a three-dimensional immersive mapping model for shielding the image of the subject, including Xia Ya video, Wei sub-image
姐體影縣域⑼斷賴區_平面雜，·域平面屬性建構 、'確尺寸之一維成何桓型；依據平面屬性，利用多視角影像 7遮蔽區域之-闕影像；以及結合具正叙寸之三維幾何模 i與恶遮蔽之觸影像’以完成域影像之三維擬真闕模型。 區分該影像找雜區域與主體影縣域的步驟更包 利用深度/視差資訊為基礎，藉以區分深度不同之該遮蔽區域鱼該 主體影像區域；（2)姻該多視角影像間之交互關係，判斷遮蔽; 况，猎以區分出該遮蔽區域與該主體影像區域。 該遮敝區域屬性定義為平_域時，係以單應性轉投影來 (構^、正確尺寸之二維幾何模型和修補遮蔽區域之貼圖影像。 當遮蔽區域屬性定義為非平面區域時，係以影像校正來建構 正確尺寸之二維幾何模型和修補遮蔽區域之貼圖影像。 以上之關於本發日_容之說明及以下之實施方式之說明係 用以不範與轉本個之魏，並且提供本發明之專 更進一步之解釋。， 圍 【實施方式】 、本發明所提之實施示意圖係以北埔著名古蹟金廣福之實景 為幻來.細本發明所提方法之可行性，第出圖係為本發明之實施 7 200828182 例中遮蔽問題之示意圖，進行北埔古蹟數位化 時，遭遇到前述之遮蔽問題，由於金廣福左右兩邊裁種=的工作 (標不方框的部分），造成了在三維重建上的困難。兩裸樹 成:_政(角落、邊緣)不明顯外，更重要在於特二 程中’大部分的理論均無法處理這樣嚴重的遮撞情況。 從用立體技騎得狀視錢（右半邊灰_ = 造成只能得到片段破碎之視差圖。因此蔽 ^問遞 •維模型更；^= 因此’本發明提出了—侧用多視角影像解決上述遮 構方法。此方法主要分成兩部分來解決： H 尺寸之幾何麵和修補闕喊分 問題發生區域的不同，若遮蔽區域落在重建主體之平面tr 就利用建純秘本·:財娜如抑轉何(ΓΓ圓::） =狀所組成。利用共平面特性來建構數學上稱為罝鄉 ^rphy)"^F'5 . 的域點4遮蔽區域落在重建主體之非平祕域上，_用基 礎矩阵作影像校正(祕lca㈣輔助求取對應點；透過這些 得，對鱗能夠保證得到較為正確之幾何尺寸。在修補貼圖的這 们=f先在各個不同的角度拍攝未被遮擋或部分遮撞的影 求取㈣轉換_ ’並對多視㈣像作影像變形(W琴哗）’ 取後擷取相似部分的區域來修補被遮蔽的區域。在確定被遮蔽部 分的幾何正確和貼圖也完替修福您 ^ >補後.就可以使用材質貼圖的技術 建構出無遮擋的三維擬真貼圖模型。 200828182 第2A圖係為本發明所提方法之流程圖，提出解決遮蔽主體 影像之，維擬細模型建構的方法，包括下列步驟：輸入： 像JL區刀:像之遮故區域與主體影像區域(步驟卿；判斷遮蔽 主體影像之區域_紐(步驟22q);依據平峨一且正 像修補遮祕域之—嶋彡像_ );以及結合具正碟尺寸之 何模類無舰L影像，以完成主體影像之三維擬直 貼圖模型(步驟250)。 /、 在上述之區分影像之賴輯與主體影魏域的步驟中，首 先 =^分雜區域祕重紅域影像_。鮮的方法如第 圖所^可以手工的方式圈選出遮蔽區域，或者以自動對應的 =式’·先對兩幅晝面比對龍點，再計算整幅晝面的視差 lspamy) ’再根據視__性作區塊連 深度較淺)的部分切割出來，即為樹的部分。 （代表 播本1、骑疋我之平面屬性係分為平面11域與非平面區域，依 之面屬性作不同之處理，若遮蔽區域落在重建主體影像 2域上’就姻共平面的特性來建構數學上稱為單應性 =7)辩.步姻單轉_gra婦求取正碎 =應點，若賴區域落在重建主體影像之非平面區域上，則作 衫像校正(rec邱cation)來求取對應點。 區域圖料本發騎提方法巾，以單紐矩陣來建構平面 確尺寸之三維幾何模型’包括下列步驟：以輸入影像為 土衫像，在主體影像之未遮蔽區域尋找落在與遮蔽區域同一平 200828182 面上之特徵點，並在其他非基準 ^ 千〜像中π找其對應點(步驟310) ,· 針對對應點，計算影像之間的單廊 %、叫驟320)，在基準影像尹， 之特雖，_估算之單應性投 建構1 #Γ非麟衫像之職點(步驟33G) ’·根據對應點資訊來 建構二維幾何模型(步驟340)。 —現以實際處理手段說明如下以支持上述方法的可行性’當遮 敝發生在重建主體的平面區域，係 …、 ^ ，、]用，、平面的特性幫助求取所 明的早應性(h〇m〇graphy)。單應性(hom〇g响)描述了三产丘 平面的點，投影到二維影像空間中 又工曰/、 性m咖議W、純、…—、/緣之間的關係。單應 V U hn \3 u V’ =H V = h h h23 1 1 21 22 V 丄 I hi h32 λ33· 入 其中，卜，v]、0 v]為三維空間中共平面的對應點投影在第 u' y」一 /^21以+ &22v+A” 11 u + hn v+ - h 13 h21 tl + h22 v + h23 > h3i uv ^ 〇 若存在讀影像，則對r元素移項整理後可得 10 \2 200828182 u] V1 1 0 0 0 一 w 0 0 0 ul 1 - Yl丨 un 1 0 0 0 ~unun 0 0 0 un π 1 — YV 一1V- '〇 Yif -v hi 0 * = ; vnuri - V h23 0 vnvn -V_ nx9 hi _0_ nxl hi -1 J9xl 最後，至少需要四組或四组以上的對應點才能利用奇異值分Sister figure county (9) Broken area _ plane miscellaneous, · domain plane attribute construction, 'the size of one dimension is formed; according to the plane attribute, the multi-view image 7 is used to mask the area-阙 image; and the combination is positive The three-dimensional geometric model i and the occlusion touch image 'to complete the three-dimensional immersive model of the domain image. The step of distinguishing between the image-seeking region and the subject-shadow region is further based on the depth/parallax information, so as to distinguish the fishery region from the shaded region with different depths; (2) the interaction between the multi-view images and the judgment Masking; condition, hunting to distinguish the masked area from the subject image area. When the concealed area attribute is defined as a flat_domain, it is projected by a homography (a two-dimensional geometric model of the correct size and a texture image of the repaired masked area. When the masked area attribute is defined as a non-planar area, The image correction is used to construct the two-dimensional geometric model of the correct size and the texture image of the repaired mask area. The above description of the present day and the description of the following embodiments are used to describe the Wei and the Wei. Further, the invention is further explained. The following is a schematic diagram of the implementation of the present invention, which is based on the real history of the famous monument of Beipu, Jinguangfu. The feasibility of the method of the invention is as follows. The figure is a schematic diagram of the shading problem in the implementation of the invention 7 200828182. When the Beipu historical site is digitized, the above-mentioned obscuration problem is encountered, due to the work of Jin Guangfu's left and right sides cutting = part of the frame, resulting in The difficulty in three-dimensional reconstruction. Two bare trees become: _ politics (corner, edge) is not obvious, more importantly, in the second process, most of the theory can not handle such strict The occlusion situation. From the perspective of riding a stereoscopic technique (the right half of the ash _ = causes only the fragmentation of the fragment to be broken. Therefore, the mask is more important than the dimension model; ^= therefore the present invention proposes - side The multi-view image is used to solve the above-mentioned shading method. This method is mainly divided into two parts: the geometric plane of the H dimension and the repairing shouting problem are different in the region where the problem occurs. If the masked area falls on the plane of the reconstructed body, the pure secret is used. ·: Cai Na is like the suppression (ΓΓ圆::) = form. Using the coplanar characteristics to construct the mathematically known as the domain of the town ^rphy)"^F'5. On the non-flat domain of the main body, _ use the basic matrix for image correction (secret lca (four) to help find the corresponding point; through these, the scale can guarantee a more correct geometric size. In the patch map, this = f first in each Shooting at different angles without occlusion or partial occlusion (4) Conversion _ 'and multi-view (four) image image deformation (W 哗 哗)' Take the area of the similar part to repair the shaded area. Determining the geometric correctness of the shaded portion The texture is also finished for repairing you ^ > After the repair. You can use the texture mapping technology to construct an unobstructed three-dimensional immersive texture model. 200828182 2A is a flow chart of the proposed method, and proposes to solve the shielding subject Image, the method of constructing the fine model, including the following steps: Input: Like the JL area knife: the image of the masking area and the subject image area (step Qing; judge the area of the masking subject image _ New (step 22q); One is just like repairing the cover field—嶋彡 image _ ); and combining the model-free shipless L image with the size of the original disk to complete the three-dimensional straight map model of the subject image (step 250). In the above steps of distinguishing between the image and the subject image, the first step is to divide the red region image _. The fresh method, as shown in the figure, can be used to circle the masked area manually, or by automatically corresponding to the formula = first to the two sides to compare the dragon point, and then calculate the parallax of the entire surface lspamy) The part that is regarded as a shallower part of the __ sex block is cut out, which is the part of the tree. (On behalf of the broadcast 1, the plane attribute of the ride is divided into plane 11 and non-planar areas, depending on the surface attributes for different treatment, if the shadow area falls on the reconstructed subject image 2 domain, the characteristics of the common plane To construct mathematics called homography =7) arguing. Stepping a single turn _gra woman seeking a slash = should be point, if the lie area falls on the non-planar area of the reconstructed subject image, then make a shirt correction (rec Qiu cation) to find the corresponding point. The area image is issued by the method of riding the method, and the three-dimensional geometric model of the plane size is constructed by a single-matrix matrix. The method includes the following steps: the input image is a tunic image, and the unshaded area of the main image is found to be in the same area as the shadow area. Ping 200828182 feature points on the surface, and find the corresponding points in other non-reference ^ thousand ~ image π (step 310), · Calculate the single gallery % between the images for the corresponding points, call 320), in the reference image Yin, although special, _ estimated single-sponsored construction 1 #Γ非麟衫's job site (step 33G) '· Construct a two-dimensional geometric model based on the corresponding point information (step 340). - The following is the actual treatment to illustrate the feasibility of supporting the above method. 'When concealing occurs in the plane area of the reconstructed body, it is used in ..., ^, ,], and the characteristics of the plane help to find the obvious early response ( H〇m〇graphy). Homology (hom〇g) describes the point of the plane of the three hills, projected into the two-dimensional image space and the relationship between work, / m, W, pure, ... -, / edge. VU hn \3 u V' =HV = hh h23 1 1 21 22 V 丄I hi h32 λ33· into it, bu, v], 0 v] is the corresponding point of the coplanar plane in the three-dimensional space projected at the u'y "/^21 is + &22v+A" 11 u + hn v+ - h 13 h21 tl + h22 v + h23 > h3i uv ^ 〇 If there is a read image, then the r element is sorted to get 10 \ 2 200828182 u] V1 1 0 0 0 a w 0 0 0 ul 1 - Yl丨un 1 0 0 0 ~unun 0 0 0 un π 1 — YV a 1V- '〇Yif -v hi 0 * = ; vnuri - V H23 0 vnvn -V_ nx9 hi _0_ nxl hi -1 J9xl Finally, at least four or more sets of corresponding points are required to use singular value points
解（Singular Value Decomposition ; SVD )解得單廡性 (homography) 〇 利用共平面建構具正確尺寸之三維幾何模型之詳細步驟如 下： a. 在共平面但未遮檔區域擷取特徵點並作對應； b. 隨機選取η組對應點，並利用強健式方法(如或 LMedS)估异早應性(homography); c·设疋一張基準影像為三維重建基底影像，用手動或自動的 方式描繪出遮檔區域中能描述主體幾何特徵的重要特徵點 d·利用估异之單應性(homography)作投影基準影像中的重要 、 特徵點z，以求得在其他非基準影像中的對應點夂= {x;，x2_x：j，這 ， 些遮蔽區域中的對應點對為; e·結合遮蔽區域中對應點{WL和非遮蔽區域中對應點對 ，依照一般三維重建流程建立三維幾何。 第3B圖係為利用單應性(homography)建立遮蔽區域之對應 200828182 點示意^首先，由於遮蔽區域是落在一平面上，因此我們可應 用上=流程，在共平面的未遮蔽區域找尋特徵對應點，計算此平 面之早應性(h〇m〇graphy)。接著，在遮蔽區域點選代表金廣福重 要特徵之特徵點，利用單應性(h〇m〇g响)投影求得遮蔽區域的 對應點。在求得遮蔽區域的對應點之後，就可以應用—般三維重 建流程得到三維幾何。 第4A圖係為本發明所提方法中，以影像校正來建構非平面 區域^正確尺寸之三維幾何模型，包括下列步驟：以輸入影像為 基準影像’在主體影像之未遮蔽區域擷取特徵點，並在其他非基 準〜像中哥找其對應點(步驟μ。)；針對對應點，計算影像之間的 :基礎細鱗將鱗轉換成絲準影像之 T線平行(㈣);在轉正雜線上搜尋基準影像巾遮蔽區域 特欲點之對應點(倾44〇);㈣應點座標轉回基準影像之一 座標(步驟編彡叙-影像座標謂應點資二 建構三維幾何模型(步驟46〇)。 +現以貫際處理手段說明如下以支持上述方法的可行性，若 敝發生在非平通域，上狀單紐(h_卿h力便I法 於是尋求糾—種方法絲得被祕區域之對應點^要是=用 影像校正㈣f咖iGn)將影像之極線(epipQlarline)轉換成和_ 線(獅㈣平行，則對應點會座落在此校正過後的極線上 便能求取對應點。 b 如下利用影像校正建構具正確尺寸之三維幾何模型之詳細步驟 12 200828182 a·在未遮擋區域擷取特徵點並作對應，· b Ik機選取η組對應點，並利用強健式方法(如或 ^ LMedS)估算基礎矩陣； ·· C•利用基礎矩陣將極線轉換成與影像之掃瞄線平行； d·遮蔽區域之特徵點在轉正的極線上搜尋對應點； e·將對應點座_回縣鍊，對應闕為似L; f·結合遮蔽區域中對應點卜，攻和非遮蔽區域中對應點對 ，依照一般三維重建流程建立三維幾何。 、上述提到之影像校正的方法主要是利用計算出之基礎矩陣 «湘伽Qjeetive 躬遂處’其數學式為：叫，其中e為透過基礎矩陣求得之 極點(ep_e) ’ p則為我們要求之投影轉換矩陣。利用這個投影轉 換矩陣P，就可以難轉換成與触線平行，並得張校正 過後的影像。 » 、第犯圖係為利用影像校正建立遮蔽區域之對應點示意圖， 在柄敝區域找尋特徵對應點，計算影像之間的極線幾何關係。 接者，對影像作校正，使其極線魏與_線平行。在遮蔽區域 中代表金廣福重要特徵之特徵點便可在轉正上的極線上搜尋對應 ‘、、、占在求㈣敝H域的對應點之後，就可以應用—般三維重建流 程得到三維幾何模型。 1 。本毛明所提之修獅圖影像部分，仍然將遮蔽區域區分為平 面區域和非平面區域。在平面區域我們計算上述之單應性 13 200828182 (homography)，根據此轉換關係將多視角影像作投影轉換，最後 根據區域相似度填補遮蔽區域。在非平面區域，透過計算影像之 間的極賴何_ ’並對多視角影像作影像校正，最後也根據區 ^ 域相似度填補遮蔽區域。 第5A圖係為本發明所提方法中，以單應性矩陣來修補平面 區域之遮蔽區域之貼圖影像，包括下列步驟··選擇欲修補之一基 準影像(步㈣G);計算基準影像之雜肋_(步驟520);對 _ ^肖度轉之影像做投影轉_步驟53G);針對投影轉換過後之 夕張〜像作主體影像被遮蔽之區域相似影像之搜尋(步驟撕）；依 據區域相似衫像填補基準影像被遮蔽的部分(步驟。 #現以實«理手段說明如下以支持上述方法的可行性，為了 要讓重建的三維模财良好的關影像，本發明針對貼圖影像中 被^敝的部分作影像修補(image inpainting)的動作。提出解決的方 式聽各個不同的角度來拍攝被遮蔽區域的影像，然後利用上述 鲁之單應性(homography)的觀念來侧遮蔽目標區，詳細步驟如下： a·在多視角影像中選擇欲修補之基準影像；在多視角影 像中選擇前述之第1B圖當作欲修補之基準影像，而 其他輸入之多視角影像如第5B圖所示。 b·計算影像之間的單應性(h_graphy)幾何關係；針對 不同視角所拍攝的影像，求取絲準影像欲修補區域 之對應點，請參考第第5 C圖所示，並以強健式方法 估算單應性(homography)。 C.將不同角度影像做投影轉換，根據上述求出之單應性 14 200828182 (homogmphy)作投影轉換重新計算出新的座標’請參 考下列所示。 月4Singular Value Decomposition (SVD) solves homography. The detailed steps of constructing a three-dimensional geometric model with correct dimensions using coplanar are as follows: a. Extract feature points in coplanar but unobstructed regions and correspond b. Randomly select the corresponding points of the η group and use a robust method (such as or LMedS) to estimate the homography; c. Set a reference image as a 3D reconstruction base image, depicting it manually or automatically An important feature point in the occlusion area that can describe the geometric features of the subject d. Use the homography to estimate the important and feature points z in the projected reference image to obtain corresponding points in other non-reference images.夂 = {x;, x2_x: j, where the corresponding pairs of points in the masked areas are; e. Combine the corresponding points in the masked area {WL and the corresponding pairs of points in the unmasked area, and establish a three-dimensional geometry according to a general three-dimensional reconstruction process. Figure 3B is a corresponding representation of the 2828182 point used to establish the occlusion area using homography. First, since the occlusion area falls on a plane, we can apply the above = process to find features in the coplanar unmasked area. Corresponding points, calculate the earlyness of this plane (h〇m〇graphy). Next, select the feature points representing the important features of Jin Guangfu in the shadow area, and use the homography (h〇m〇g) projection to obtain the corresponding points of the masked area. After the corresponding points of the masked area are obtained, the three-dimensional reconstruction process can be applied to obtain the three-dimensional geometry. FIG. 4A is a three-dimensional geometric model for constructing a non-planar region and correct size by image correction according to the method of the present invention, comprising the steps of: taking an input image as a reference image and capturing feature points in an unmasked region of the subject image. And in other non-references ~ like the middle of the brother to find its corresponding point (step μ.); for the corresponding point, calculate the image between: the basic fine scales convert the scale into a silk image of the T line parallel ((4)); On the miscellaneous line, search for the corresponding point of the target image towel shaded area (pour 44〇); (4) Turn the coordinates back to the coordinate of one of the reference images (steps are edited - the image coordinates are said to be constructed by the two points to construct a three-dimensional geometric model (steps) 46〇). + Now the following methods are used to illustrate the feasibility of supporting the above method. If the 敝 occurs in the non-Ping Tong domain, the upper single-shaped one (h_qing ̄ force I method then seeks to correct the method) Corresponding point of the secret area ^ If you use image correction (4) f coffee iGn) to convert the epipolar Qlarline (epipQlarline) into parallel with the _ line (lion (four), the corresponding point will be located on the corrected polar line Find the corresponding point. b Use image correction as follows Detailed steps for constructing a three-dimensional geometric model of the correct size 12 200828182 a·Taking feature points in the unoccluded area and corresponding, b Ik machine selects the corresponding points of the η group, and uses the robust method (such as or ^ LMedS) to estimate the basis Matrix; ··•• Use the basic matrix to convert the polar line into parallel with the scan line of the image; d·The feature point of the shadow area searches for the corresponding point on the positive pole line; e· corresponding point阙 is like L; f· combines the corresponding points in the shadow area, the corresponding point pairs in the attack and non-shadow areas, and establishes the three-dimensional geometry according to the general three-dimensional reconstruction process. The image correction method mentioned above mainly uses the calculated basis. The mathematical formula of the matrix «Xiangjia Qjeetive 躬遂" is: called, where e is the pole obtained by the basic matrix (ep_e) 'p is the projection transformation matrix we require. With this projection transformation matrix P, it can be difficult Converted into parallel with the line of sight, and obtained the corrected image. » The first line is a schematic diagram of the corresponding point of the shadow area established by image correction, and the corresponding point is found in the handle area. Calculate the polar line relationship between images. Receiver, correct the image so that its polar line Wei is parallel to the _ line. The feature points representing the important features of Jin Guangfu in the shadow area can be searched for on the polar line on the positive After occupying the corresponding point of the (4) 敝H domain, the three-dimensional geometric model can be obtained by applying the general three-dimensional reconstruction process. 1. The lion image image part of the hair plucking still divides the occlusion area into a planar area. And non-planar areas. In the plane area, we calculate the above homography 13 200828182 (homography), according to this conversion relationship, the multi-view image is projected and converted, and finally the shadow area is filled according to the regional similarity. In the non-planar area, the image is calculated. The relationship between _ and 'image correction for multi-view images, and finally fills the masked area according to the similarity of the area. Figure 5A is a map image of the masked area of the planar area in the method of the present invention, including the following steps: selecting one of the reference images to be repaired (step (4) G); calculating the miscellaneous of the reference image Rib_(step 520); for _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ The shirt is filled with the portion of the reference image that is obscured (step. #现实实理理方法) to support the feasibility of the above method, in order to make the reconstructed three-dimensional model good image, the present invention is targeted to the texture image ^ The part of the skull is used for image inpainting. The solution is to listen to different angles to capture the image of the masked area, and then use the above concept of homography to mask the target area. The steps are as follows: a. Select the reference image to be repaired in the multi-view image; select the first picture BB as the reference image to be repaired in the multi-view image, and the other input The multi-view image is shown in Figure 5B. b. Calculate the homography (h_graphy) geometric relationship between images; for the images taken at different viewing angles, find the corresponding points of the silk image to be repaired, please refer to Figure 5 shows the homography in a robust way. C. Projection transformation of different angle images, recalculate the new one according to the above-mentioned homography 14 200828182 (homogmphy) The coordinates of 'Please refer to the following. Month 4
Wrapped Image 轉換結果如第5D圖所示，持續步驟b和e，直到所有視 角的影像投影轉換均已完成。 d·請參考第5 E圖所示’係為從多視角影像填補遮蔽區 域影像示意圖’鎌區域間的相似度填補被遮蔽影像 目標區，填觀賴之影像目標區時，首先選定被遮 蔽區的小塊區域(通常為一小塊，在轉換過的多 視角影像中作相似度的比對，其比對位置為對應到目 • 標影像之相同位置。依序進行同樣的程序，直到目標 區均修補完成。第5F _經過初步修微的結果^ 選定之小塊區域已經成功的修補完成。 第6A圖係為本發明所提方法中，以影像校正來修補非平面 ^或之遮蔽區域之關影像，包括下列步驟：選擇欲修補之一基 ==驟_，·計算基準影像與其他視角影像之極線幾何關係 :驟62〇)，_減拍攝之影—影像校正，使極線與基準影 像之掃晦線平行(步驟63〇);根據區域相似度在極線上搜尋鱼基準 影像被遮蔽影像目標區最相似之區域(步驟64〇);依據最相似之區 15 200828182 域填補基準影像被遮蔽的部分(步驟65〇);將修補後之影像轉換回 基準影像座標糸統(步驟660)。 現对際處理手段_如下以支持上述方法的可行性，若遮 ·-蔽發生在非平面區域，同樣可以·前述影像校正的方式來修補 被遮蔽區域。修補貼圖影像之步驟如下： 夕 a·在未遮擋區域擷取特徵點並作對應，· b·計算影像之間的極線幾何關係； ♦ c·將不同角度影像做影像校正； d. 根據區_似度在極、紅㈣鍾雜影像目標 之區域； e. 將修補狀影像轉細縣影縣標纽。 、、上述提到之影像校正的方法主要是利用計算出之基礎矩陣 j-投影轉換(pix)jeetive罐影像之極點轉換至無. 躬逐處’其數學式為：作=(10岸，其中e為透過基礎矩陣求得之 φ _(印lp〇le)，P則為我們要求之投影轉換矩陣。利用這個投影轉 換矩陣P ’斜以將極線轉換成與掃目旨線平行，並得出—張校正 過後的影像。 /經過上述兩種處理方式’確定所有遮蔽區域均已填補完畢 7卩可將修補過後的影像用作貼圖影像供後續處理。第6B圖則 =金廣福之遮蔽問題經過修補後之結果，遮蔽物已經完全被移 牙細節也被完整的保留下來，能夠作後續闕模型處理。 。σ具正確尺寸之二維幾何模型與無遮蔽之關影像，以完 成主體影狀三維__型：彻__e迦_卿㈣ 16 200828182 的技術，我們可以將具有正確尺 绐 寸—輪左和修補過後之貼圖 :像、、，：：成無糊題之三維擬真貼圖模型。第7Α圖錯誤， 找不到參财源。絲轉叙 處理之後，結果如第7Β _…：工過本發明的方法 回不，效果遠勝於未修補的貼圖模 =，模魏可制於統料及數位化文物展轉呈 歷史意義和文化價值。 /、有 =本她输咐_祕i，財並非用以限 ^=。林脫離本發明之精神和範_，所為之更動與_， 考^ Γ專娜護朗。11於本發_界定之保護範圍請來 考所附之申請專利範圍。 " 【圖式簡要說明】 第1A圖係為所要處理影像之遮蔽問題之示意圖； _為本發明之實_巾舰問題之示意圖； 弟1C圖係為利用立體_技術所得到片段破碎之視差圖； 第2A圖係為本發明所提方法之流程圖； 主體轉本發日骑提綠之區分該影像之歧區域與該 王體影像區域之示意圖； 第3Α圖係為本發明所提方法中，以單應性 區域具正確尺寸之三闕何模型； 構面 點 :3Β圖係為利用單應性(h〇m〇graphy)建立遮蔽區域之對應 意圖； 乐仏_林發_提方法中，以影像校正來建構非平面 -减具正確尺寸之三維幾何模型； 17 200828182 f4B圖係為利用影像校正建立遮蔽區域之對應點示意圖； 、第本發騎提方法中，以單應性矩陣來修補平面 區域之遮蔽區域之貼圖影像； 第5B圖係為不同角度擷取影像示意圖； f 5C圖係為求取對應點計算單舰(iK>mography)示意圖； ，5 D圖係為根據單應师咖卿㈣作影像變形示意圖；The Wrapped Image conversion result, as shown in Figure 5D, continues with steps b and e until the image projection conversion for all viewing angles is complete. d. Please refer to Figure 5 E for the image of the masked image from the multi-view image to fill the image of the masked area. The image is captured in the image area. When the image target area is filled in, the masked area is selected first. The small area (usually a small piece, the similarity is compared in the converted multi-view image, and the matching position is the same position corresponding to the target image. The same procedure is followed until the target The area is repaired. The 5F _ after preliminary repair results ^ The selected patch area has been successfully repaired. Figure 6A is the method proposed in the present invention, using image correction to repair the non-planar ^ or the masked area The image is closed, including the following steps: Select one of the bases to be repaired == _, · Calculate the polar line relationship between the reference image and other perspective images: Step 62 〇), _ minus the shadow of the shot - image correction, make the polar line Parallel to the broom line of the reference image (step 63〇); searching for the most similar area of the fish target image in the target area of the image according to the regional similarity (step 64〇); according to the most similar area 15 200828182 Masked reference image portion (step 65〇); Repair of the image after the image is converted back to a reference coordinate system which (step 660). The current inter-processing means is as follows to support the feasibility of the above method. If the masking occurs in a non-planar area, the masked area can be repaired by the image correction method as well. The steps for patching the image are as follows: 夕 a·Capturing feature points in the unoccluded area and making correspondings, b·calculating the polar line relationship between images; ♦ c·correcting the image at different angles; d. _like degree in the region of the extreme, red (four) clock image target; e. The patch image is transferred to the county shadow county standard. The method of image correction mentioned above is mainly to use the calculated base matrix j-projection conversion (pix) jeepive tank image to convert the pole to none. 躬where 'the mathematical formula is: for = 10 shores, e is the φ _ (print lp〇le) obtained through the basic matrix, and P is the projection transformation matrix we require. Using this projection transformation matrix P ' oblique to convert the polar line into parallel with the scan target line, and After the corrected image is taken out / After the above two treatment methods, it is determined that all the masked areas have been filled. The repaired image can be used as a texture image for subsequent processing. Figure 6B = Jin Guangfu's shadow problem As a result of the repair, the mask has been completely preserved and the details of the tooth are completely preserved, which can be used for subsequent 阙 model processing. σ has the correct size of the 2D geometric model and the unobstructed image to complete the subject image 3D __ Type: __e _ _ _ Qing (four) 16 200828182 technology, we can have the correct size - round left and repaired texture: image,,,:: into a three-dimensional immersive texture model without ambiguity. 7th If you make a mistake, you can't find the source of the money. After the silk reversal process, the result is as follows: _...: The method of the invention is not good, the effect is far better than the unpatched texture mode =, the model Wei can be made in the material and The digital cultural relics exhibition is transferred to historical significance and cultural value. /, yes = she lost _ secret i, wealth is not used to limit ^ =. Lin is out of the spirit and scope of the invention _, which is changed with _, test ^ Γ Na Hulang. 11 In the scope of protection of this issue _, please refer to the scope of the patent application attached. " [Summary of the schema] Figure 1A is a schematic diagram of the problem of shadowing of the image to be processed; Schematic diagram of the actual _ towel problem; the 1C picture is the disparity map of the fragment fragment obtained by the stereo _ technology; the 2A picture is the flow chart of the method proposed by the present invention; The schematic diagram of the image difference region and the image area of the king body; the third figure is the three geometric model with the correct size in the homogenous region in the method proposed by the invention; the facet point: the 3Β map is the use of the single Sex (h〇m〇graphy) establishes the corresponding intent of the masked area; In the music _ Linfa _ method, the three-dimensional geometric model of the correct size of the non-planar-reducing tool is constructed by image correction; 17 200828182 The f4B picture is a schematic diagram of the corresponding points for establishing the occlusion area by image correction; In the method, the texture image of the masked area of the planar area is repaired by the homography matrix; the image of FIG. 5B is a schematic diagram of the image captured by different angles; and the f 5C image is a schematic diagram of calculating the single ship (iK>mography) for obtaining the corresponding point; , 5 D is a schematic diagram of image deformation according to Shan Ying Shi Qing (4);
第E圖係為從夕視角影像填補遮蔽區域影像示意圖； 弟5F圖係為填補部分區域後之結果示意圖，· 弟6A圖係為本發明所提方法中，以影像校正來修補非平面 时或之遮蔽區域之貼圖影像； f 6B圖係為作完影像修補後的貼圖影像示意圖； =7A圖係為沒有作遮蔽補償之三維貼圖模型示意圖；以及 第7B圖係為作遮蔽補償之三維貼圖模型示意圖。 【圖式符號說明】 ^ 步驟210 步驟220 步驟230 步驟240 步驟250 輸入一影像’並區分影像之遮蔽區域與主 體影像區域 判斷遮蔽該主體影像之區域的平面屬性 依據平面屬性建構一具正確尺寸之三維 幾何模型 依據平面屬性，利用多視角影像修補遮蔽 區域之一貼圖影像 ，結合具正破尺寸之三維幾何模型與無遮 蔽之貼圖影像，以完成主體影像之三維擬 18 200828182 .真貼圖模型· 步驟310···.····…以輸入影像為基準影像，在主體影像之未 遮蔽區域尋找落在與遮蔽區域同一平面 上之特徵點，並在其他非基準影像中尋找 其對應點 步驟320……..…利用對應點，計算影像之間的單應性 步驟330...........在基準影像中，選擇遮蔽區域中能描述幾 何特徵之特徵點，利用估算之單應性投影 算出在其他非基準影像之對應點 步驟340…….....根據對應點資訊來建構三維幾何模型 步驟410·.·以輸入影像為基準影像，在主體影像之未 遮蔽區域擷取特徵點，並在其他非基準影 像中尋找其對應點 步驟420…利用對應點，計算影像之間的一基礎矩陣 步驟430...............利用基礎矩陣將極線轉換成與基準影像 之掃瞄線平行 步驟440...........在轉正的極線上搜尋基準影像中遮蔽區 域特徵點之對應點 步驟450...........將對應點座標轉回基準影像之一影像座 標 步驟460.·.··..•…根據轉回基準影像之一影像座標之對應 點資訊來建構三維幾何模型 步驟510........選擇欲修補之一基準影像 19 200828182 步驟520...........計算基準影像與其他視角影像之空間幾 何關係 步驟530….…·…對其他視角影像做投影轉換 步驟540…······..針對投影轉換過後之多張影像作主體影 像被遮蔽之部分相似影像之搜尋 步驟550..............依據部分相似影像填補基準影像被遮蔽 的部分 步驟610.........••選擇欲修補之一基準影像' 步驟620...........計算基準影像與其他視角影像之極線幾 何關係 步驟630…......對其他視角影像作影像校正，使極線與基 準影像之掃瞄線平行 步驟640···.……·根據區域相似度在極線上搜尋與基準影 像被遮蔽影像目標區最相似之區域 步驟650............依據最相似之區域填補基準影像被遮蔽 的部分 步驟660·.···.··.·.將修補後之影像轉換回基準影像座標系 統 20Figure E is a schematic diagram of filling the image of the masked area from the image of the eve view; the image of the 5F is the result of filling the partial area, and the image of the 6A is the method of the present invention, and the image correction is used to repair the non-planar or The texture image of the masked area; f 6B is the schematic image of the image after the image is repaired; the =7A is a schematic diagram of the 3D map model without shading compensation; and the 7B is the 3D map model for shading compensation schematic diagram. [Description of Schematic] ^ Step 210 Step 220 Step 230 Step 240 Step 250 Enter an image 'and distinguish the masked area of the image from the main image area to determine the plane attribute of the area that masks the subject image. Construct a correct size according to the plane attribute. According to the planar property, the 3D geometric model uses a multi-view image to repair one of the shadowed areas, and combines the 3D geometric model with the broken size and the unobscured texture image to complete the 3D simulation of the main image. 200828182. 310········... Using the input image as the reference image, searching for feature points that fall on the same plane as the masked area in the unmasked area of the subject image, and searching for corresponding points in other non-reference images. ...........Using the corresponding point to calculate the homography between the images. 330........... In the reference image, select the feature points in the masked area that describe the geometric features, using the estimation The homography projection calculates the corresponding points in other non-reference images. Steps 340...........Construct the three-dimensional geometric model based on the corresponding point information. Step 410··· taking the input image as a reference image, capturing feature points in the unmasked area of the main image, and searching for corresponding points in other non-reference images. Step 420... Using the corresponding points, calculating a basic matrix between the images Step 430...............converting the polar line into a parallel to the scan line of the reference image using the base matrix. Step 440........... Searching for corresponding points of the masked area feature points in the reference image on the line. Step 450..............Returning the corresponding point coordinates back to one of the reference image image coordinates. Step 460.····..•... Back to the corresponding point information of one of the image coordinates of the reference image to construct the three-dimensional geometric model. Step 510........ Select one of the reference images to be repaired. 19 200828182 Step 520........... Spatial geometric relationship between image and other perspective images Step 530.......·...Projection conversion step 540 for other perspective images.......................................................................................................................................................................................................... Image search step 550..............The reference image is masked according to the partial similar image Part of step 610.........••Select one of the reference images to be patched' Step 620........... Calculate the polar line relationship between the reference image and other view images Step 630... ......Image correction for other view images, so that the polar line is parallel to the scan line of the reference image. Step 640······Search for the target image of the reference image on the polar line according to the regional similarity The most similar area step 650............fills the portion of the reference image that is obscured according to the most similar area. Step 660·.········. Converts the patched image back Reference image coordinate system 20