200823799 九、發明說明： 【發明所屬之技術領域】 本發明係有關於一種影像填補方法，且特別有關於一 種可以利用多張不同角度的影像對於一特定影像進行修補 之方法。 【先前技術】200823799 IX. Description of the Invention: [Technical Field] The present invention relates to an image filling method, and more particularly to a method for repairing a specific image using a plurality of images of different angles. [Prior Art]
當拍攝景物，如建築物、古蹟景點與大型景觀或裝置 等時，標的物或場景有時會受到不相干的遮蔽物所阻檔， 而無法拍攝其完整的全貌。這些遮蔽物也許是可以自動移 除的，如車輛或行人，但有些遮蔽物可能是固定在現場， 而幾法或不易排除的’如招牌、標諸、樹木或其他建築。 在這種情況下，無論從任何角度，都無法拍攝到完整且乾 淨的標的物全貌。因此，在許多古蹟建築的數位影像保存 及二維數位模型製作等，這些不相干的遮蔽物總是造成相 當程度的干擾，使得數位影像或3D _無法正確呈現標 的物的形貌。 2001年Manuel M.01iverira提出一修補（咖㈤㈣演 算法’且2004年―u Telea提出快速行進法When photographing scenes, such as buildings, monuments, and large landscapes or installations, the subject matter or scene is sometimes blocked by irrelevant obstructions, and it is impossible to take a complete picture of it. These shelters may be automatically removed, such as vehicles or pedestrians, but some shelters may be fixed on site, and several methods may be difficult to exclude such as signs, signs, trees or other buildings. In this case, the complete and clean subject matter cannot be captured from any angle. Therefore, in the preservation of digital images and the creation of two-dimensional digital models in many monumental buildings, these irrelevant masks always cause a considerable degree of interference, so that the digital image or 3D _ cannot correctly display the appearance of the object. In 2001, Manuel M.01iverira proposed a patch (Caf (5) (4) algorithm] and in 2004, u Telea proposed the fast marching method.
Marching Method，FMM)，用以快竦 θ 、u補影像中之小區 的运失或是損壞的部分。在這些方法φ ^ _ 中’雖然可以加速小 區域之目標區的影像修補速度，但是斟 ^ ^ , 對於較大目標區域的 修補時，會產生模糊的現象。 美國專利us_⑽提出心結合影像修補與材質合 0949-Α21832TWF(N2)；P519501 10TW；yianhou 5 200823799 成（Texture Synthesis)之新的影像修補演算法。其依據影像 邊緣(Edge)的方向與強度，以及像素的信賴度來對於影像 進行修補。在此方法中，從影像中已知的影像資訊往目標 區域内擴展。若影像材質的變化有一定的規律性，則可以 得到正確性較高的影像修補結果。然而，由於仍然是以原 來影像其他位置的資訊做預測，因此，若被填補之區域的 影像雜亂，則非常容易發生錯誤。 【發明内容】 有鑑於此，本發明提供影像填補方法。 本發明實施例之影像填補方法。首先，於不同角度擷 取相應一標的物或一場景之複數影像。分別計算影像中每 一填補來源影像與影像中一特定影像間的單應性幾何關 係。依據每一填補來源影像相應之單應性幾何關係將每一 填補來源影像進行投影轉換。依據投影轉換後之填補來源 影像填補特定影像中之至少一填補目標區。 本發明上述方法可以透過程式碼方式收錄於實體媒體 中。當程式碼被機器載入且執行時，機器變成用以實行本 發明之裝置。 為使本發明之上述目的、特徵和優點能更明顯易懂， 下文特舉實施例，並配合所附圖示，詳細說明如下。 【實施方式】 第1圖顯示依據本發明實施例之影像填補方法。 0949-A21832TWF(N2)；P51950110TW；yianhou 6 200823799 如步驟S1】0 ’使用攝影機或數位相機，對-祕 的標的物或場景於不同角度拍攝多個影維空間中 分別古十曾旦 砰夕扪办像。如步驟S120， 衫像中母一填補來源影像與 :單應性幾何關係。其中，特定影像係需要進=::: ί。’且填補來源影像可關來填補特定影像中的填補目ί 第2圖顯示影像間之單應性( 係。如第2圖所示，傯今X盔二从… P W 了應"、、占關 筚物的拍摄ϋ為—、、參工間中的點，因為對建 表物的拍攝角度不同，三維空間中的對應 影像上的對庫x v 又〜到一、准 7對應,，、、占（Xl，X2, X3, X4)位置也會不同。三 共平面的對應點投影在二維影傻 工S中 應性幾hu ^ 、如像上，兩張影像間會存在單 :成何關係H。其中，單應性幾何關係H是一個如的 矩陣。早應性幾何關係Η_導過程如下。 h 首先，假設汉: l/zll h\2 '21 h22 h3l hi 13 h22> h33 uf hll hl2 hn ~u vf = h2l h22 h23 V 1 - _ 厶31厶32厶33 1 且Ud與U，，v，]為三維空 對應點座標。因此，可以得知 ^llu + \2v+^]^ 因此 V，Marching Method, FMM), used to quickly lose the loss or damage of the cells in the θ, u complement image. In these methods φ ^ _ ' can accelerate the image patching speed of the target area in the small area, but 斟 ^ ^, when repairing a larger target area, a blurring phenomenon occurs. U.S. Patent US_(10) proposes a new image patching algorithm for heart-integrated image repair and material combination 0949-Α21832TWF(N2); P519501 10TW;yianhou 5 200823799 (Texture Synthesis). It repairs the image based on the direction and intensity of the edge of the image and the reliability of the pixel. In this method, image information known from the image is expanded into the target area. If the change of the image material has a certain regularity, the image repair result with higher correctness can be obtained. However, since the information is still predicted from other locations of the original image, if the image of the filled area is cluttered, it is very easy to make an error. SUMMARY OF THE INVENTION In view of the above, the present invention provides an image filling method. The image filling method of the embodiment of the invention. First, a plurality of images of a corresponding object or a scene are captured at different angles. The homography geometric relationship between each of the filled source images and a particular image in the image is calculated separately. Each of the filled source images is projected and converted according to the homography geometric relationship corresponding to each of the filled source images. Filling the source image according to the projection conversion Fills at least one of the specific image to fill the target area. The above method of the present invention can be recorded in physical media through code. When the code is loaded and executed by the machine, the machine becomes the means for carrying out the invention. The above described objects, features and advantages of the present invention will become more apparent from the description of the appended claims. [Embodiment] FIG. 1 shows an image filling method according to an embodiment of the present invention. 0949-A21832TWF(N2);P51950110TW;yianhou 6 200823799 If step S1]0' uses a camera or a digital camera, the target object or scene of the secret is shot at different angles in different video space. Do it like this. In step S120, the mother image of the shirt image fills the source image and the homography geometric relationship. Among them, the specific image system needs to enter =::: ί. 'And fill the source image can be closed to fill the fill in the specific image. Figure 2 shows the homography between the images (system. As shown in Figure 2, now X helmet II from... PW should be ", The shooting of the shackles is -, and the point in the squad, because the angle of shooting of the building is different, the corresponding library on the corresponding image in the three-dimensional space is ~v to one, the corresponding 7 corresponds, The position of (Xl, X2, X3, X4) will be different. The corresponding point projection of the three common planes should be a few hu ^ in the two-dimensional shadow S, as in the image, there will be a single between the two images: Relationship H. Among them, the homography geometric relationship H is a matrix such as the matrix. The early geometric relationship Η _ guided process is as follows. h First, suppose the Han: l / zll h \ 2 '21 h22 h3l hi 13 h22> h33 uf Hll hl2 hn ~u vf = h2l h22 h23 V 1 - _ 厶31厶32厶33 1 and Ud and U,, v,] are three-dimensional space corresponding point coordinates. Therefore, we can know ^llu + \2v+^]^ So V,
Sf + ^2v+^3 h2lu+h22v+h^ ^f+^2v+h33 - 13-A31仙匕办32秦Λ心〇 21好〜2V+- A lw，〜办3产4331/== 〇 〇949-A21832TWF(N2);P51950110TW：yianh< 7 200823799 最後 了以得到 vn 1 〇 0 一 Ύ 〜Vi， —V1， ^ηΦη ^vnun —unvn’ -v hn hl2 h\3 0" h2l 0 h22 二 • t h23 0 t ~nx9 h3l 0 h32 1 9x1 mx\ 方程式，因此要求出單:二;且對應點可以產生二個 組的對應點。值得注音何關係矩陣Η至少需要有4 式噴是以自動化:方:進:應點的選擇可以以人工方 生方本發明實施例之單應性幾何關係之產 像與特二;源影 如步驟S121盥步驟从 ττ . ^ α 驟122，使用一特徵點尋找方法， rns角偵測（Harris Corner Detector)方法分別找出俨定 影像與填補來源影像中的特J0方法刀職出对寸疋 ]%斂點，如弟4A圖與第4B圖所 二j ’如步驟S123 ’於特定影像與填補來源影像選取 用文& ί目纟,&的疋’在此特定範圍内的特徵點將會被 影像間的對應點。如步驟Sl24,使用一特徵點搜 =法，如差值平方和（Sum〇fSq咖心脱職以，ssd) 2敎影像與填補來源影像中的特徵財分別找出落於 ^寸定範圍中之對應點，如第5A圖與第5b圖所示。由於 在計算單應性幾何_矩陣Η時，若對應點太過於集中，、 °949-Α21832TWF(N2);P51950110TW；yianhou 8 200823799 . 將會使得單應性幾何關係矩陣Η的計算產生問題。因此， 如步驟S125，將特定影像與填補來源影像之共平面上劃分 為複數個區塊，且找出分散落於每一區塊中之複數個特定 對應點，如第6Α圖與第6Β圖所示。 之後，如步驟S126，依據一最小中間平方法（Least Median of Squares，LMedS)由這些特定對應點中決定特定 影像與填補來源影像間之最佳對應點組合。在決定最佳對 應點組合時，可以隨機在劃分出之區塊中取出四點或四點 ⑩ 以上，分別計算出m組單應性幾何關係矩陣。假設有η組 對應點，且'，A分別為特定影像與填補來源影像中的對應 點，ζ = 1，2,3，·..η。在此兩影像中，由單應性幾何關係可求 出其轉換值々與&，分別為A = Ηρ，，& = Η〆。。%為殘餘值 (Residual)，表示對應點透過％矩陣，計算出來的對應點與 原對應點之間的差異。其中’ rki= J(xf -Xf)2 +{χ'Γχ\ )2 ’ 々 = l，2,3，...m，/ = 1，2,3，···η，用來表示對應點的誤差。最小 中間平方法可以將相應不同對應點組於不同單應性幾何關 ⑩ 係矩陣下之殘餘值進行排序後，取出中間的數值，再從中 找出最小的那一組。相應殘餘值最小的對應點組將被選作 為最佳對應點組合，如第7Α圖與第7Β圖所示。最後，如 步驟S127，依據最佳對應點組合中之對應點計算特定影像 與填補來源影像間之單應性幾何關係。 接下來，請再次參考第2圖。當每一填補來源影像與 特定影像間之單應性幾何關係得到時，如步驟S130，依據 相應之單應性幾何關係將每一填補來源影像進行投影轉 0949-A21832TWF(N2)；P51950110TW；yianhou 9 823799 :，如第8圖所示。如步驟_ 來源影像填補特定影 轉換後之填補 名fii千旦"多及 主夕一填補目標區。 在進灯影像修補工作時， 像中的遮蔽物件，如第9 像、雜程式移除特定影 物件移除之後，並形成木。當特定影像中之遮蔽 值传注意的是，依據特定書w弟兜圖所不。 區分為多個填補目標區二二2細真補的區域大小可以 明實施例之、>定埴、# , 、弟10圖顯不依據本發 二貝知例之决疋填補目標區之優先填補順序的方法 ^^(Lap^an Mas^) =行填補區域的輪廓，如第11圖所示。如步驟 3所 計算每—填補目標區之邊緣強度’如第 回不亚如步驟Sl53，依據邊緣強度決定填補目標區的 ^補順序三其中’邊緣強度最強的填補目標區將被最先選來進 仇補。帛13圖頒不依據本發明實施例之修補填補目標區的 方法。如步驟S141，依據填補目標區與每—填補來源影像中 相應特定影像巾填補目標區之位置的__修觀域，使用一 區塊比對方法(Block-Matching Methods)，如差值平方和、標準 化互相關法(Normalized Cross-Correlation，NCC)、標準化差值 平方和、絕對差值和(Sum of Absolute Differences，SAD)、排 行(Rank)與普查(Census)方法等分別計算每一填補來源影像中 之候選修補區域的相似度，如第14圖所示。如步驟S142，將 相似度最高的候選修補區域選作為修補區域，並如步驟S143， 取得此修補區域，並利用此修補區域填補特定影像中之此填補 目標區。 0949-A21832TWF(N2) ；P51950110TW；yianhou 10 200823799 必須提醒的是，在本發明中係計算填補目標區的週圍資訊 與其它候選修補區域的週圍資訊的差異值。差異值較小的候選 修補區域則被認定為係與填補目標區相似的。之後，便可將此 相似的候選修補區域填補填補目標區。當此填補目標區完成填 補之後，便可選擇邊緣強度次強之填補目標區，以進行填補， 直至所有需要填補的區域填補完成。 因此，本發明可以依據不同角度的多張影像相互組Sf + ^2v+^3 h2lu+h22v+h^ ^f+^2v+h33 - 13-A31 仙匕办32秦Λ心〇21好~2V+- A lw,~办3产4331/== 〇〇949- A21832TWF(N2);P51950110TW:yianh< 7 200823799 Finally, to get vn 1 〇0 a Ύ~Vi, —V1, ^ηΦη ^vnun —unvn' -v hn hl2 h\3 0" h2l 0 h22 2• t h23 0 t ~nx9 h3l 0 h32 1 9x1 mx\ equation, therefore requires the order: two; and the corresponding point can produce two groups of corresponding points. It is worthwhile to note the relationship matrix. At least 4 types of spray are required to be automated: square: advance: the choice of points can be artificially produced by the artificial image of the embodiment of the present invention. Step S121 盥 step from ττ . ^ α Step 122, using a feature point finding method, rns angle detection (Harris Corner Detector) method respectively to find the fixed image and the fill source image in the special J0 method ]% convergence point, such as the brother 4A map and the 4B map two j ' as step S123 'in the specific image and fill the source image selection text & ί 纟, & 疋 在 ' feature points in this specific range will Will be the corresponding point between the images. In step S14, a feature point search method is used, such as the sum of squared differences (Sum), and the features in the image of the filled source are respectively found in the range of Corresponding points are shown in Figures 5A and 5b. Since the corresponding points are too concentrated when calculating the homography geometry_matrix, °949-Α21832TWF(N2); P51950110TW;yianhou 8 200823799 . This will cause problems in the calculation of the homography geometry matrix Η. Therefore, in step S125, the coplanar plane of the specific image and the padding source image is divided into a plurality of blocks, and a plurality of specific corresponding points scattered in each block are found, such as FIG. 6 and FIG. Shown. Then, in step S126, the best corresponding point combination between the specific image and the padding source image is determined from the specific corresponding points according to a Least Median of Squares (LMedS) method. When determining the optimal combination of points, you can randomly take four or four points and 10 or more in the divided blocks to calculate the m-group homography geometric relationship matrix. Suppose there are η sets of corresponding points, and ', A is the corresponding point in the specific image and the filled source image, ζ = 1, 2, 3, ·.. η. In these two images, the conversion values 々 and & can be obtained from the homography geometric relationship, respectively A = Ηρ,, & = Η〆. . % is the residual value (Residual), which indicates the difference between the corresponding point calculated by the corresponding point through the % matrix and the original corresponding point. Where 'rki= J(xf -Xf)2 +{χ'Γχ\ )2 ' 々= l,2,3,...m,/ = 1,2,3,···η, used to indicate correspondence Point error. The minimum intermediate level method can sort the residual values of the corresponding corresponding point groups under different homography geometric systems, and then take out the intermediate values, and then find the smallest one. The corresponding point group with the smallest residual value will be selected as the best corresponding point combination, as shown in Figure 7 and Figure 7. Finally, in step S127, the homography geometric relationship between the specific image and the filled source image is calculated according to the corresponding point in the optimal corresponding point combination. Next, please refer to Figure 2 again. When the homography geometric relationship between each of the filled source images and the specific image is obtained, in step S130, each of the filled source images is projected to 0949-A21832TWF(N2) according to the corresponding homography geometric relationship; P51950110TW; 9 823799 : As shown in Figure 8. For example, the step _ source image fills the specific shadow after the conversion of the name fii thousand dan " more and the main eve to fill the target area. In the repair work of the light image, the shadow object in the image, such as the ninth image, the program to remove the specific image is removed, and the wood is formed. When the occlusion value in a particular image is passed, it is not according to the specific book. The size of the area that is divided into multiple filling target areas 22 and 2 can be clearly defined. The method of filling the order ^^(Lap^an Mas^) = the outline of the line fill area, as shown in Figure 11. If the edge intensity of each target area is calculated as in step 3, as in the first step, as in step S53, the order of filling the target area is determined according to the edge strength. The padding target area with the strongest edge strength will be selected first. Encourage. Figure 13 illustrates a method of repairing a target area that is not in accordance with an embodiment of the present invention. In step S141, a block-matching method, such as the sum of squares of differences, is used according to the __ repair field of the padding target area and the position of the corresponding specific image towel in each of the padding source images. Normalized Cross-Correlation (NCC), Normalized Cross-Correlation (SNC), Sum of Absolute Differences (SAD), Rank (Cenus), and Census (Census) methods are used to calculate each source of each fill. The similarity of the candidate patch areas in the image is shown in Figure 14. In step S142, the candidate patching area with the highest similarity is selected as the patching area, and in step S143, the patching area is obtained, and the patching area is filled in the specific image by using the patching area. 0949-A21832TWF(N2); P51950110TW;yianhou 10 200823799 It must be reminded that in the present invention, the difference between the surrounding information of the filling target area and the surrounding information of other candidate repaired areas is calculated. Candidate patch areas with smaller difference values are considered to be similar to the fill target area. This similar candidate patch area can then be filled to fill the target area. After the filling of the target area is completed, the target area with the second edge strength can be selected to fill the gap until all the areas to be filled are filled. Therefore, the present invention can mutually group multiple images according to different angles.
合’利用彼此間的空間幾何關係，且使用這些互補的影像 來修補目標區。 本發明之方法，或特定型態或其部份，可以以程式碼 的型態包含於實體媒體，如軟碟、光碟片、硬碟、或是任 何其他機益可讀取（如電腦可讀取）儲存媒體，苴中，告程 式碼被機器，如電腦載人且執行時，此_變成用以:鱼 本發明之I置。本發明之方法與裝置也可以以程式碼㈣ 透過一些傳送媒體，如電線或電縵、光纖、或是任何傳輸 型態進行傳送，豆中，备蘚彳饭4n u _ & 八 田^，，'、被機器’如電腦接收、載 入且執仃％ ’此機器變成用以參與本發明之裝置。 般用途處理器實作時，程式碼結合處 ^ 於應用特定邏輯電路之難裝置。 心“I員似 /然本發明已以較佳實施例揭露如上，然其並非用以 限疋本發明，任何熟悉此項技蓺 八 κ a ^ $有’在不脫離本發明之Μ 神和範圍内’當可做些許更動與潤飾， :: 範圍當視後附之申請專利範圍所界定者為準。 4 0949-Α21832TWF(N2)；P51950110TW；yianhou 200823799 【圖式簡單說明】 第1圖為一流程圖係顯示依據本發明實施例之影像填 補方法。 、 、 ^ 2圖頒示影像間之單應性對應點關係。 弟3圖為一流程圖係顯示依據本發明實施例之單應性 幾何關係之產生方法。 ^ 4A圖與第4B圖分別顯示特定影像 中之特徵點。 中之mr與第5B圖分別顯示特定影像與填補來源影像 中劃區圖塊與第6β圖分別顯示特定影像與填補來源影像 中之第最mi7:圖分別顯示特定影像與填補來源影像 轉換弟8圖為—示意圖係顯示依據單應性幾何關係之投影 第9A圖顯示一特定影像例子。 像。弟9B圖顯示將第9A圖中之遮蔽物件移除後之特定影 第10圖為一流程圖係顯示依 填補目標區之優先填補順序的方法康柄明貫施例之決定 =圖顯示特定影像中遮蔽物件的 弟12圖顯示特定影像中的邊緣。 第13圖為一流程圖係顯示依據本發明實施例之修補 〇949-A21832TWF(N2)；P5195〇11〇TW；yianh〇u 200823799 . 填補目標區的方法。 第14圖為一示意圖係顯示依據不同填補來源影像填 補特定影像中之填補目標區。 【主要元件符號說明】 S110、S120、…、S140〜步驟； S121、S122、…、S127〜步驟； S141、S142、S143〜步驟； ❿ S151、S152、S153〜步驟。The 'use' of the spatial geometry of each other and use these complementary images to repair the target area. The method of the present invention, or a specific type or part thereof, may be embodied in a physical medium such as a floppy disk, a compact disc, a hard disk, or any other machine readable (such as a computer readable computer). Take the storage medium, in the middle, when the program code is carried by the machine, such as a computer and executed, this _ becomes: for the fish of the invention. The method and apparatus of the present invention can also be transmitted by using a transmission medium such as a wire or an electric cable, an optical fiber, or any transmission type in a code (4), and a bean, 4n u _ & 八田^,, ', received by the machine, such as a computer, loaded and executed %' This machine becomes a device for participating in the present invention. When a general-purpose processor is implemented, the code combination is a difficult device for applying a specific logic circuit. The present invention has been disclosed in the preferred embodiments as above, but it is not intended to limit the invention, and any one skilled in the art can not deviate from the invention. Within the scope 'When a few changes and retouchings can be made, :: The scope is subject to the definition of the patent application scope. 4 0949-Α21832TWF(N2); P51950110TW;yianhou 200823799 [Simple diagram] Figure 1 A flow chart shows an image filling method according to an embodiment of the present invention. The picture shows the homography corresponding point relationship between images. The third figure shows a flow chart showing the homography according to an embodiment of the present invention. The geometric relationship is generated. ^ 4A and 4B respectively show the feature points in the specific image. The mr and 5B maps respectively show the specific image and the 6β map in the specific image and the filled source image respectively display the specific image. The most mi7: image in the filled source image shows the specific image and the filled source image. The image is displayed as a schematic image showing a specific image example according to the projection of the homography geometric relationship. Figure 9B. The specific image showing the removal of the obscured object in Fig. 9A is a flow chart showing the method of filling the priority of the target area. The decision of the embodiment is shown in the figure = the figure shows the brother of the object in the specific image. Figure 12 shows the edges in a particular image. Figure 13 is a flow chart showing the repair of 目标949-A21832TWF(N2); P5195〇11〇TW;yianh〇u 200823799 in accordance with an embodiment of the present invention. Figure 14 is a schematic diagram showing filling of the target area in a specific image according to different filled source images. [Main element symbol description] S110, S120, ..., S140~ steps; S121, S122, ..., S127~ steps; S141, S142, S143~step; ❿ S151, S152, S153~ steps.
0949-A21832TWF(N2)；P51950110TW.yianhou 130949-A21832TWF(N2);P51950110TW.yianhou 13