201033936 六、發明說明： .【發明所屬之技術領域】 . 本發明係關於一種立體影片（stereoscopic video)合成 方法，特別係指將一單視角影片合成為一立體影片之立體 影片合成方法。 【先前技術】 一般來說，人的雙眼幾乎是平行的，對於觀看一注視物 而言’左、右眼所看到的注視物幾乎是一樣的，但是，若 將該注視物拉近於注視者的面前，然後利用左、右眼分別 去觀察’即可發現兩眼所看到的注視物具有明顯不同的角 度與面向’因此’人們之所以可以看到立體的影像，係因 為人的大腦皮層視覺區會將該兩眼所看到注視物的視角進 行結合，換言之，一個立體影片的製作之所以可以讓觀眾 看到立體影像，便是透過特別的影像處理讓觀眾的右眼看 ❹ 到右邊的視角，左邊看到左邊的視角，再透過觀眾的大腦 將不同視角的畫面立體化。 隨著科技的進步，人們在看立體影像的時候已較少利用 配戴立體眼鏡的方式欣賞立體電影，而是透過立趙影像螢 幕裝置直接供觀眾觀看立體電影，而該立體影像螢幕裝置 的内部影像處理技術A多係利用建立深度圖（depth map ) 或追蹤攝影機拍攝的運動軌跡（camera trajectory )來合 成立體影像。 201033936 基本上’上述之建立深度圖的影像處理技術可透過許多 方式來達成，例如利用紅外線對拍攝物體進行照射以取得 • 物體的深度’其缺點在於紅外線強弱決定了誤差的程度， 若是強度太弱則拍攝出來的結果將具有許多雜訊，使得所 建立之深度圖一致性不足，導致合成後的立體影像具有過 多不連續的點，造成觀賞者視覺上的困擾與障礙。 另外，上述應用追蹤攝影機拍攝的運動轨跡的技術來合 〇 成立體影像之方法，隨著影像的精細程度和拍攝物體移動 程度增加，故需要考慮的資訊參數亦隨之增加（例如：利 用光流方程式（〇ptical F1〇w Equati〇n)來估計出移動向量 (Motion Vector))，換s之，當影片的畫面變動過於頻繁時， 會增加追蹤的困難，使得運算分析上出現較複雜且具有較 大誤差的問題。 因此，若能直接從單視角的影片來合成立體影片並直 ❿接應用於目前市面上現存具有錄影功能之立體榮幕的數位 裝置，而不需要經過複雜的三維畫面設計或使用到兩個攝 影機以上來拍攝立體影片，進而降低立體影片在製作上的 困難，方能給予新世代的立體螢幕裝置一個完整的市場空 間，也同時能夠帶領影像處理的領域向前躍進。 【發明内容】 本發月之主要目的在於提供一種立體影片（_e〇sc〇pic Wde。）合成方法，其係將單視角影片之各影像序列中的前 201033936 後段晝面（frames)對應合成為一立體影片，可直接應用 於目前市面上現存具有錄影功能之立體螢幕的數位裝置， •進而降低立體影片在製作上的困難。 為達上述之目的，本發明之立體影片合成方法，係依序 包括有以下步驟： 步称（a)’提供一來自一單視角（monoscopic video ) 影片之影像序列（sequenee )，其中，該影像序列具有複數 ❹ 個與時間相對應的畫面（frame ); 步驟（b)，計算該影像序列中前後段畫面（frames )的 基線距離（baseline distance); 步驟（c)’根據該基線距離係將一基底畫面（base franie ) 結〇複數個相關畫面’以取得一配對晝面（matching frames )’其中’該相關畫面係該基底畫面的先前畫面； 步驟（d)’適性化調整該配對畫面因配對所可能產生的 ® 位移差或角偏差； 步驟（e)’剪裁與修補該配對晝面經適性化調整後所可 能產生的像素遺失； 步驟（〇，重複步驟（c)至（e)以將所有影像序列中 的畫面完成配對畫面後’輸出一立體影片。 藉由上述之步驟’本發明之立體影片合成方法可將一般 單視角影片中影像序列中的·畫面合成為一立體影片，並可 直接應用於目前市面上現存具有錄影功能之立體螢幕的數 201033936 位裝置’而不需要經過複雜三維畫面設計或使用到兩個攝 影機以上來拍攝立體影片，故能降低立體影片在製作上的 '困難度。 【實施方式】 為使熟悉該項技藝人士瞭解本發明之目的，兹配合圖式 將本發明之較佳實施例詳細說明如下。 請參考第一圖所示，本發明之立體影片（stereoscopic 〇 video )合成方法，係依序包括有以下步驟： 步驟（a)’提供一來自一單視角（monoscopic video ) 影片之影像序列（sequence )，其中，該影像序列具有複數 個與時間相對應的畫面（frame ); 步驟（b)，§十算該影像序列中前後段畫面（frames )的 基線距離（baseline distance); 步驟（c)’根據該基線距離係將1一基底畫面（base frame ) 豳 結合複數個相關畫面，以取得一配對晝面（matching frames )，其中，該相關畫面係該基底畫面的先前晝面； 步驟（d )，適性化調整該配對晝面因配對所可能產生的 位移差或角偏差； 步驟（e)，剪裁與修補該配對畫面經適性化調整後所可 能產生的像素遺失； 步驟（f)，重複步驟（c)至（e)以將所有影像序列中 的畫面完成配對畫面後，輸出一立體影片。 201033936 -般任何-個將二維影像重建為三維影像的技術，勢必 要將該二維影像中的二維座㈣統透過搭配攝影機拍攝時 的資訊轉換成立體影像的三維座標（χ γ，ζ)系統。之前201033936 VI. Description of the invention: [Technical field to which the invention pertains] The present invention relates to a stereoscopic video synthesis method, and more particularly to a stereoscopic film synthesis method for synthesizing a single-view film into a stereoscopic film. [Prior Art] In general, the eyes of a person are almost parallel, and the gaze seen by the left and right eyes is almost the same for viewing a gaze, but if the gaze is brought closer to In front of the gaze, then use the left and right eyes to observe separately, and you can see that the gaze seen by the two eyes has distinct angles and faces. Therefore, people can see stereoscopic images because of human The cerebral cortex visual zone combines the perspectives of the gaze seen by the two eyes. In other words, the production of a stereoscopic film allows the viewer to see the stereoscopic image, which is seen by the viewer's right eye through special image processing. On the right side of the view, the left side sees the left angle of view, and then the viewer's brain stereoscopically views the different angles of view. With the advancement of technology, people have less to use stereo glasses to enjoy stereoscopic movies when viewing stereoscopic images. Instead, they use stereo cameras to directly view stereoscopic movies, and the interior of the stereoscopic video device Image processing technology A uses a depth map or a camera trajectory to synthesize a stereo image. 201033936 Basically, the image processing technology for establishing the depth map described above can be achieved in many ways, such as using infrared rays to illuminate the object to obtain the depth of the object. The disadvantage is that the intensity of the infrared light determines the degree of error, if the intensity is too weak. The result of the shooting will have a lot of noise, so that the consistency of the established depth map is insufficient, resulting in the stereoscopic image after the synthesis has too many discontinuous points, causing visual troubles and obstacles for the viewer. In addition, the above-mentioned method of tracking the motion trajectory captured by the camera to combine the method of forming a volume image, as the degree of fineness of the image and the degree of movement of the object increase, the information parameters to be considered are also increased (for example, using light) The flow equation (〇ptical F1〇w Equati〇n) is used to estimate the motion vector. In other words, when the picture of the film changes too frequently, the difficulty of tracking increases, which makes the calculation analysis more complicated. A problem with a large error. Therefore, if you can directly synthesize a stereoscopic movie from a single-view movie and directly apply it to the digital device that currently has the stereoscopic screen with video function on the market, without having to go through complicated 3D image design or use two cameras. The above three-dimensional film, in order to reduce the difficulty of the production of stereoscopic film, can give a new generation of stereoscopic screen device a complete market space, but also can lead the field of image processing forward. SUMMARY OF THE INVENTION The main purpose of this month is to provide a stereoscopic film (_e〇sc〇pic Wde.) synthesis method, which is to synthesize the first 201033936 rear-end frames in each image sequence of a single-view movie. A three-dimensional film can be directly applied to digital devices that currently have a stereoscopic screen with video functions on the market, and further reduce the difficulty in manufacturing stereoscopic movies. For the above purposes, the stereoscopic film synthesizing method of the present invention sequentially includes the following steps: Step (a) 'provides a sequence of images from a single-view video (sequenee), wherein the image The sequence has a plurality of frames corresponding to time; step (b), calculating a baseline distance of the front and back frames in the image sequence; step (c) 'based on the baseline distance A base franie encloses a plurality of related pictures 'to obtain a matching frame' where the related picture is the previous picture of the base picture; step (d) 'adaptizes the matching picture The displacement or angular deviation that may be generated by the pairing; Step (e) 'Trimming and repairing the pixel loss that may occur after the paired surface is properly adjusted; Step (〇, repeat steps (c) to (e) to After the pair of pictures in all the image sequences are completed, a stereoscopic movie is outputted. By the above steps, the stereoscopic film synthesis method of the present invention can be used for general single viewing. The image in the image sequence in the corner film is combined into a stereoscopic movie, and can be directly applied to the number of 201033936-bit devices that currently have a stereoscopic screen with video function on the market' without the need for complicated three-dimensional picture design or use of two cameras. The above-mentioned three-dimensional film can be used to reduce the difficulty of the three-dimensional film in the production. [Embodiment] In order to make the person skilled in the art understand the purpose of the present invention, the preferred embodiment of the present invention will be described in detail with reference to the drawings. As shown in the first figure, the stereoscopic video synthesis method of the present invention includes the following steps in sequence: Step (a) 'provides an image sequence from a single perspective video (monoscopic video) (sequence), wherein the image sequence has a plurality of frames corresponding to time; step (b), § ten calculates the baseline distance of the front and rear frames (frames) in the image sequence; c) 'Based on the baseline distance, a base frame is combined with a plurality of related pictures to obtain one Matching frames, wherein the related picture is the previous face of the base picture; and step (d), adapting to adjust the displacement difference or angular deviation of the paired face due to the pairing; step (e) , trimming and repairing the pixel loss that may occur after the paired picture is adaptively adjusted; step (f), repeating steps (c) to (e) to complete the paired picture after all the pictures in the image sequence, and output a stereoscopic film 201033936 - Any technology that reconstructs 2D images into 3D images, it is necessary to convert the 2D coordinates of the 2D image into the 3D coordinates of the volume image (χ γ, through the information captured by the camera). ζ) System. prior to
有學者L〇ngUet-Higgins & Prazdny提到當三維座標系統中 之Z值為正，且一觀看者的眼睛投射於一畫面時其視網 膜成像的速度包括有一角速度（angular vel〇city ) ( A，B，C ) 和一線性速度（linear vel〇city) ( V，w)，因此畫面中 的座標值（x，y)可以方程式（丨）表示 x = XiZ,y=T/Z .........〇) 而對應成像之速度可以方程式（2)及（3)表示 x=W-x-U>^Z-B+C y-B X2 +Α x y (之) y=(j^-y-V)/Z+A-C-x-3 x y+A y2 ' .........(3)〇 根據上述的技術與原理，使得本發明步驟（b)計算該 影像序列中畫面的基線距離更包括有以下步驟： 步驟（b-〇,掃描該影像序列中所有的畫面利用klt (Kanade-Lucas-Tomasi Feature Tracke〇 演算法自畫面取 出複數個特徵值及對應之座標值（Xi yi); 步驟（b·2)，將上述之角速度（A，B，C)和線性速度（u， v，w)設定為攝影機的動態參數（moti〇nparameters)，將 一靜態的攝影機模擬為一動態的攝影機，以作為該二維座 標系統轉換為一三維座標系統所需攝影機拍攝時的資訊， 因此’將上述方程式（2)乘上y，方程式（3)乘上乂後相減， 201033936 可得到以方程式（4)表示 xy~y-x = {~~A)x~(^-—B)y + C(tx2 + ν2\ " .........(4); 步驟（b-3)，比對前後段畫面的特徵值，分別取出對應 的特徵值做為複數組配對點（Xl，yi)㈣，並將該前後段 畫面的時間差（Λ〇代人方程式（4)，可得到方程式⑺表又示 如下 Ά-iyi-yi) χι) =Some scholars L〇ngUet-Higgins & Prazdny mentioned that when the Z value in the three-dimensional coordinate system is positive, and the speed of the retina imaging of a viewer's eye is projected on a picture, there is an angular velocity (Angular vel〇city) (A , B, C ) and a linear velocity (linear, vel〇city) (V, w), so the coordinate value (x, y) in the picture can be expressed by the equation (丨) x = XiZ, y = T / Z ... ......〇) and the speed of the corresponding imaging can be expressed by equations (2) and (3) x=Wx-U>^Z-B+C yB X2 +Α xy (y) y=(j^-yV /Z+ACx-3 x y+A y2 ' (3) 〇 According to the above techniques and principles, step (b) of the present invention calculates the baseline distance of the picture in the image sequence. The method includes the following steps: Step (b-〇, scan all the images in the image sequence using klt (Kanade-Lucas-Tomasi Feature Tracke algorithm to extract a plurality of feature values and corresponding coordinate values (Xi yi) from the screen; (b·2), set the above angular velocity (A, B, C) and linear velocity (u, v, w) as the camera's dynamic parameters (moti〇nparameters), and simulate a static camera as a The state of the camera, as the information required for the camera to be converted into a three-dimensional coordinate system for the two-dimensional coordinate system, so 'multiply the above equation (2) by y, equation (3) multiplied by the 乂 subtraction, 201033936 can Obtained by equation (4) xy~yx = {~~A)x~(^--B)y + C(tx2 + ν2\ ".........(4); Step (b) -3), comparing the feature values of the front and back pictures, respectively extracting the corresponding feature values as complex array pairing points (Xl, yi) (4), and the time difference of the front and back pictures (Λ〇代人(4), The equation (7) can be obtained as shown below as Ά-iyi-yi) χι) =
(~~^> Χι -(~-B).yl+C{^ +yf) .........(5)； 步驟（b-4)’將前後段晝面中的配對點對應取得一水平 位移向量（horizontal displacement vectorW 3* J 4 )和一個深 度值uepthValue)(/)’再將料畫面中的各配對點根 據該水平位移向量（毛）分成N個配對點類.別（她恤g pairs group),該配對點類別中包括有九^個配對點& y〇,其中’ N和 < 為自然數，1<= i <=从 ’ ，1<= k <=N ; 步驟（b-5)，將該水平位移向量. 、义）和深度值（％) 代入方程式（5)，可得到方程式（6)表示如下 xfk - i?2* - yf^ + C((jcf* )2 + (yf^ )2\ ..........(6) 其中，冲心)係前後 段畫面的配對點； &得如^個水平位移 步驟（b-6 )’根據上述步驟（b_5) 8 201033936 向量、深度值和配對點’將其以矩陣方式代入方程式⑺如 下(~~^> Χι -(~-B).yl+C{^ +yf) .........(5); Step (b-4)' Pairing in front and back The point correspondingly obtains a horizontal displacement vector (horizontal displacement vector W 3* J 4 ) and a depth value uepthValue) (/) ' and then divides each pairing point in the material picture into N pairing point classes according to the horizontal displacement vector (hair). Don't (g pairs group), the pairing point category includes nine^ matching points & y〇, where 'N and < is a natural number, 1<= i <= from ' , 1 <= k <=N; Step (b-5), substituting the horizontal displacement vector, the meaning, and the depth value (%) into the equation (5), the equation (6) is expressed as follows: xfk - i?2* - yf^ + C((jcf* )2 + (yf^ )2\ ..........(6) where, the heart is the pairing point of the front and back pictures; & (b-6) 'Based on the above steps (b_5) 8 201033936 vector, depth value and pairing point', it is substituted into equation (7) in matrix form as follows
Gj *hj =φ% ^ ^.........(7) 以獲得最小平方誤差解，其中， % 4k A* o4kf+(yh2 ： .： ·4 <4)2+(4>2 ❹ φ4 Δί 鲁 ’其中係以虛擬反矩陣 (pseud〇-inverse )解法計算出、以求得; 步驟（b⑺’根據步驟（b_5)和（b6)對應選取另一 組\以取得; 步驟（b_8)，將也咸硭以計算出該基底距離。 上述之步驟（b-8)計算基底距離，由於單視角影片中 攝影機拍攝的位移為準水平（_如izGnui)位移因 此，本發明之計算基底距離可直接根據上述步驟（Μ)所 定義：'及步驟(b_7)的々用以計算基底距離，在求 取1與丑1 #過程中，設定七為N個配對點類別中最小 的水平位移向量，々為N個配對點類別中最大的水平位移 9 201033936 向量，故能達到簡化計算上的複雜程度。 上述步驟（C)根據該基線距離將一基底畫面（base .frame)結合複數個相關畫面，以取得一配對晝面（matching frames )，係包括有以下步驟： 步驟（c-1 )，取出影像序列中各前後段畫面的基線距 離； 步驟（c-2 ) ’計算最佳的觀賞者之水平視差（达）； Θ 步驟（C-3)，設定一畫面為基底畫面，根據該水平視差 (化）及基線距離將複數個相關畫面，取得一配對畫面 (matching frames) ° 由於觀看者的雙眼在形成立體圖像時，其垂直視差應該 為零’因此’上述步驟（d)適性化調整該配對畫面因配對 所可能會產生位移差或角偏差，包括有以下步驟： 步驟（d-Ι )，計算該配對畫面中對應配對點（χ，乃）戌，又)的 ❹ 相差垂直（vertical disparity )距離，係由以下方程式（8) 計算 p=—2Cf* _ 歹ί) .........(8) ，其中，η為對應配對點的數量； 步驟（d-2)，係根據步驟（d-Ι)所計算之相差垂直距 離對應移動配對畫面後，利用拉凡格式演算法 (Levenberg-Marquardt algorithm)計算出待調整之比例（s) 201033936 和偏移角度（0)，以解決配對畫面在移動後可能發生之誤 差’其係由以下方程式（9)計算 scosS 一 scosB 0· 0 Y η 1 0 0 1 ι J .........(9) ’其中’ 〇丨，3；;)為配對畫面於配對結合後經適性化調整的 位置。 完成上述步驟（d)之適性化調整後有產生部分畫面的 β 影像資訊被遺漏的可能’例如影像處理業界所熟知之遮蔽 區域（occluded area )和像素空洞（pixel h〇le )，其中，該 遮蔽區域大多發生在畫面外框的周圍，因此必須透過步 驟（e)來剪裁與修補該配對畫面經適性化調整後所可能產 生的像素遺失，其包括有： 步驟（e-1)’剪裁配對畫面的外表框，去除遮蔽區域； 步驟（e2)’進行影像補償（image 處理， ® 用以解決類似像素空洞的問題。 上述之步驟（e-2)的影像補償處理係利用業界所知悉 之汁算像素平均值、亮度色差補償或影像膨脹（dilation)等 技術’因此不加以累述。 綜上所述，本發明之立體影片合成方法可將一單視角影 片中影像序列中的晝面合成為一立體影片，並可直接應用 於目則市面上現存具有錄影功能之立體螢幕的數位裝置， 而不需要經過複雜三維晝面設計或使用到兩個攝影機以上 11 201033936 來拍攝立體影片，故能降低立體影片在製作上的困難度。 【圖式簡單說明】 第一圖為本發明之立體影片合成方法步驟流程圖。 【主要元件符號說明】 (a) 步驟（a) (b) 步驟（b) (c) 步驟（c) (d) 步驟（d) (e) 步驟（e) (f) 步驟（f)Gj *hj =φ% ^ ^.........(7) to obtain the least square error solution, where, % 4k A* o4kf+(yh2 : .: ·4 <4)2+(4> 2 ❹ φ4 Δί Lu ', which is calculated by the virtual inverse matrix (pseud〇-inverse) solution; step (b(7)' selects another group according to steps (b_5) and (b6) to obtain; B_8), will also be salty to calculate the base distance. The above step (b-8) calculates the base distance, since the displacement of the camera in the single-view film is based on the displacement of the camera (_such as izGnui), the calculation of the present invention The base distance can be directly defined according to the above steps (Μ): 'and the step (b_7) is used to calculate the base distance. In the process of obtaining 1 and ugly 1 #, set the minimum level among the N pairs of points. The displacement vector, 最大 is the largest horizontal displacement of the N paired point categories, 9 201033936 vector, so the complexity of the simplified calculation can be achieved. The above step (C) combines a base picture (base .frame) according to the baseline distance. The related picture to obtain a matching frame includes the following steps: Step (c-1), taking out the baseline distance of each of the front and rear images in the image sequence; Step (c-2) 'Calculating the horizontal parallax of the best viewer (Da); Θ Step (C-3), setting a screen For the base picture, a plurality of related pictures are obtained according to the horizontal parallax (base) and the baseline distance, and a matching frame is obtained. Since the eyes of the viewer form a stereoscopic image, the vertical parallax should be zero. The above step (d) adapts the adjustment of the paired picture due to the pairing, which may cause displacement difference or angular deviation, including the following steps: Step (d-Ι), calculate the corresponding pairing point in the paired picture (χ, 戌)戌, )) The vertical disparity distance is calculated by the following equation (8): p=—2Cf* _ 歹ί) (8), where η is the corresponding pairing point The number of steps to be adjusted according to the vertical distance of the phase difference calculated by the step (d-Ι) is calculated by using the Lavenberg algorithm (Levenberg-Marquardt algorithm). ) 201033936 and offset angle (0) to The error that may occur after the movement of the pairing screen is calculated by the following equation (9). scosS a scosB 0· 0 Y η 1 0 0 1 ι J .........(9) 'where' 〇丨, 3; ;) is the position where the paired picture is adaptively adjusted after pairing. After the adaptation of the above step (d) is completed, there is a possibility that the β image information of the partial picture is missing, such as an occluded area and a pixel hoop, which are well known in the image processing industry. Most of the occlusion area occurs around the outer frame of the screen. Therefore, the pixel loss that may occur after the adaptive adjustment of the paired picture must be tailored and repaired through step (e), which includes: step (e-1)' tailoring pairing The outer frame of the screen removes the masked area; step (e2)' performs image compensation (image processing, ® to solve the problem of pixel-like holes. The image compensation processing of step (e-2) above uses the juice known to the industry. Techniques such as pixel average value, luminance chromatic aberration compensation, or image dilation are not described. Therefore, the stereoscopic film synthesis method of the present invention can synthesize the facet in the image sequence in a single-view movie into A three-dimensional film, and can be directly applied to the digital device of the existing stereo screen with video function on the market, without having to go through the complex 3D face design or use two cameras above 11 201033936 to shoot stereoscopic movies, so it can reduce the difficulty of making stereoscopic films. [Simplified illustration] The first picture is the flow chart of the method of synthesizing stereoscopic film of the present invention. [Description of main component symbols] (a) Step (a) (b) Step (b) (c) Step (c) (d) Step (d) (e) Step (e) (f) Step (f)
1212