TW201237805A - Method for depth estimation and device using the same - Google Patents

Abstract

A method for depth estimation includes the following steps. Firstly, first and second frame sets, corresponding to first and second frame time, are received. Then the motion vector data of the frame set of the second frame time with respect to that of the first frame time, are obtained. Next, stereo matching between frame data of first and second view angles of the first frame set is executed to obtain depth data of the first frame time. Then estimated depth data are estimated according to the motion vector data and the depth data of the first frame time. After that, depth data of the second frame time is obtained according to the estimated depth data, the frame data of first and second view angles of the second frame set.

Description

201237805

IW7098PA VI. Description of the Invention: [Technical Field] The present invention relates to a depth estimation method and apparatus thereof, and in particular to a depth estimation method using a dual-view image matching (Stereo Matching) technique for estimating Device. [Prior Art] In the current era of rapid technological development, stereoscopic multimedia systems have gradually gained attention in the industry. In general, in the application of stereoscopic video/video, 'Stereo Matching image processing technology' is a stereoscopic image core technology that is urgently needed in the industry. In the prior art, the dual-view image alignment technique first calculates an image depth map based on the dual-view image. As a general matter, the dual-view image matching technique has a high computational complexity = problem. Based on this, how to design a dual-view image comparison depth estimation method with low computational complexity is the direction that the industry is constantly striving for. SUMMARY OF THE INVENTION The present invention relates to a depth estimation method and apparatus thereof, and a conventional depth estimation method, and a depth estimation method and apparatus thereof according to the present invention have the advantages of low computational complexity. ... used to invent (the first aspect), propose a depth estimation method, estimate the leaf, and the binocular image (Bi_lar Vide〇) data for the depth image estimation method including the following steps. The first and second timing diagrams corresponding to the first and second time series in the first round of the binocular image, the first and second timing diagrams of each of the first and second time diagrams of the first and second time series The box data set includes a first relative to the first. Then find the first time frame of the second time frame data group and the second view frame data MateMng) ' to find the corresponding to the first and the first skin, _ order depth data. Then, according to the motion vector data, the depth data is estimated according to the second time: second: timing. Then the root angle and :: view frame data to find the second time depth data. Inventive (the second aspect), a depth data estimation input is proposed for depth estimation of input binocular image data, wherein packet depth = meta-shift: vector generation unit, dual-view image comparison unit; first time sequence &; / H * 70 receives the second timing frame corresponding to the (4) f 彡 image corresponding to the second time sequence group including the first 及 讳 八 各 各 各 各 各 各 第一 第一 第一 « « « « « « Frame data and second view frame data. Move to #一序图框MM, see the first image comparison unit according to the second view image comparison to find the corresponding to the first 7 win == data progress Data. Deep production estimate _remain _ # The first time series deep data of the inter-sequence sequence finds the first time-series moving vector data and the first time-series depth according to the second d;: degree; = view field image comparison unit root angle And the second perspective frame data, looking for:;:=: the first - is considered to have a better understanding of the above and other aspects of the present invention, the following 201237805

The preferred embodiment of the TW7098PA is described in detail with reference to the following drawings: [Embodiment] The depth estimation method of the present embodiment reduces the double by referring to the moving vector data of the input binocular video (Binocular Video) data. The amount of computation of the field image comparison (Stereo Matching). First Embodiment The depth estimation method of the present embodiment estimates the estimated depth data by referring to the motion vector data of the input binocular image data, and refers to the estimated depth data to simplify the operation of generating the corresponding depth data. Referring to Figure 1, there is shown a block diagram of a depth estimating apparatus in accordance with a first embodiment of the present invention. The depth estimating device 1 of the present embodiment is configured to perform depth estimation on the input binocular video data Vi. The input binocular image data Vi includes, for example, a plurality of dual-view frame data sets, the plurality of double-view frame data sets may correspond to respective time sequences, and each of the double-view frame data sets includes two The pen corresponds to the frame data of different perspectives. In other examples, the input binocular image data Vi may include three or more multi-view frame data. For example, the timing frame data group Vi_tl corresponding to the first time sequence t1 in the input binocular image data Vi includes the first view frame data Fl_tl and the second view frame data F2_tl; and the input binocular image data Vi corresponds to The timing frame data set Vi_t2 to the second time sequence t2 includes the first view frame data F1_t2 and the second view frame data F2_t2. The first view frame data Fl_tl and Fl_t2 are respectively the first and second 201237805

Iw /uys^A The left eye view frame data of the time sequence tl and t2; the second view frame data F2_tl and F2_t2 are, for example, the right eye view frame data of the first and second time sequences tl and t2, respectively; The first and second time sequences t1 and t2 correspond, for example, to operating time points adjacent to each other. The depth estimating device 1 includes an input unit 102, a motion vector generating unit 104, and a dual view image comparing unit 108. The input unit 102 receives the timing frame data sets Vi_tl and Vi_t2, and includes the first view frame data Fl_tl and Fl_t2 and the second view frame corresponding to the first and second time sequences t1 and t2 in the input binocular image data Vi. Information F2_tl and F2_t2. The motion vector generation unit 104 finds the first motion vector data M_12 of the timing frame data group Vi_t2 with respect to the timing frame data group Vi_tl. In an operation example, the input unit 102 and the motion vector generation unit 104 can be implemented by an image decompressor; the image decompressor performs a decompression operation on the input compressed binocular image data to restore the uncompressed original image data and Corresponding moving vector data, the input compressed binocular image data can be two image data independently of each other, and one binocular image data of one side of the left and right eye view frame data. In one example of operation, the motion vector generation unit 104 further includes, for example, an averaging filter, a Mean Fi, which can be used to perform related image processing operations on the moving vector data. The dual-view image comparison unit 106 performs dual-view image comparison based on the first view and the second view frame data F1_tl and F2_tl to find the first time-series data D_tl corresponding to the first time sequence t1. The depth estimating unit 108 is based on the motion vector data M_12 and the first time 201237805

1 W7W8PA sequence depth data D_tbu finds the corresponding depth data DE_t2 corresponding to the second time sequence t. Due to the first and second time sequence. The sequence timing of the neighbors is adjacent to each other. Generally, the input binocular image is a deep sound 1 pattern having a linear change in the first and second time sequences t1 and t2. The depth estimation unit 108 is based on the first time series. The depth data is further supplied to the dual-view image comparison unit 1 by estimating the second time-series estimation depth DE_t2 estimated by the second time-series depth data estimating unit 1 to 8 〇6: The dual-view and image matching unit 106 performs the first-view and second-view frame data F1_t2 and F2 on the basis of the estimated depth data DE_t2 corresponding to the second time sequence ^ 2 The two-view image comparison is performed: Find the second time-depth data corresponding to the second time sequence 7.2 (9). For example, the depth estimation unit 1〇8 determines the ratio of the first-view and second-view frame data/; and the F2”2 dual-view image comparison operation according to the degree data DE”2. Search for the Search Window. In an operation example, for the frame data Π 财 对应 对应 对应 对应 ( ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二 第二Accordingly, the dual-view image matching unit can correspond to the position of the center of the search window to the coordinate position (il+x, gate) in the second view frame material F2-t2. Accordingly, the depth of view data center 2 is estimated according to the second time series, and the dual-view image comparison unit 106 can effectively obtain the possible depth values between the first-view and second-view frame data and F2"2, thereby Effectively 201237805

IW7098PA η. Reducing the search for the Wei domain image comparison operation · The amount of calculation required for the comparison operation is small, and the camp 10 is referred to the second figure, and the green color is in accordance with the estimation method of the present invention. flow_. The depth step of the depth estimation of this embodiment. First, as in step (a), the input unit 1〇2 is connected to: a timing chart pivotal binocular image corresponding to the first-time sequence t1 in the following data V1 and a timing frame data level J corresponding to the second time sequence t2. , vLtl Step (6), the motion vector generates a single S 104 to find the timing and then, as in Vi_t2, relative to the timing frame data and vi_tl. Then, according to step (c), the dual-view image comparison unit view and the second view frame data F1-tl and F2_tl are subjected to the first-to-alignment to find the first corresponding to the first time sequence t1. Time: domain image D" followed by step (d), depth estimation unit 10 (10) = data M" 2 and the first - time depth data D to find = amount * 淑 mu mu - 呀 order after the depth of the leaf as steps (e) The dual-view image comparison unit 1〇6 finds and outputs the second time-series depth data D-12 according to the second time-series depth data DE_t2, the first view and the second view frame data FI and F2_t2. In the present embodiment, only the second time series estimation (4) corresponding to the second time sequence t2 is estimated by the depth estimating apparatus 1 of the present embodiment with reference to the first time series depth data D_tl corresponding to the first time sequence ti. DE_t2, and simplifies the operation of finding the operation corresponding to the second timing depth data D"2 as an example. The depth estimating apparatus i of the present embodiment is not limited thereto. 201237805

TW7098PA For example, the depth estimation device i, in the reception timing frame data set Vi t3: early 70 02, the use of the heart J motion vector generation unit financial V: ν 〇 3 t2 movement to summer - shell material M_23; The depth estimation unit (10), by using the motion vector data M-12 and M23 and corresponding to the tl and the sigh to the first, the second time sequence to the second main data and D" 2 estimated "stupid: = The second timing of the sequence t3 estimates the depth data DE t3; double = out corresponds to the operation timing between the first, the depth and the t2, but not the following: the third time sequence t3 can also be the second time sequence one In the embodiment, the depth estimation $ method and the device system thereof, the timing sequence data group of the inter-sequence and the corresponding movement data of the data group and the corresponding: the frame data are generated to correspond to the target phase. The method for estimating the depth of the ice application of the sequence and its apparatus refer to this estimation more; X: simplifying the depth corresponding to the time sequence of the target to the traditional depth estimation method, the depth of the invention is::: Fang = its device has the advantage of low computational complexity. The operation of the woody data is as follows: 201237805 xw /uy〇r/\ Please refer to FIG. 4, which is a block diagram showing the depth estimating apparatus according to the second embodiment of the present invention. The device 2 is different from the depth estimation device of the first embodiment in that the depth estimation device 2 further includes an object information estimation unit 210, an object information correction unit 212, and a control unit 214. " The object information estimation unit 210 is based on the motion vector data. M_12 finds the second time-series estimation object allocation data 0_t2 corresponding to the second timing frame data group Vi_t2. For example, the object information estimating unit 21〇 may move the vector data according to the moving vector data Μ12 The object in the input binocular image data is identified and tracked by the consistency of the neighboring pixel vector data. The object information correction unit 212 corrects the second image according to the first view and the second view frame data Fl_t2 and F2_t2. Timing estimation object allocation data 〇 to obtain second time-series object allocation data E_t2. For example, the object information correction unit 212 is a reference The accuracy of the second time-series estimation object allocation data 0_t2 is verified by the two pieces of the second time sequence t2 corresponding to the pixel data of the frame data of different viewing angles, and the second corresponding to the second time sequence VII 2 is generated. The time-series object allocation data E_t2. The control unit 214 corrects the first time-series depth data D_12' according to the second time-series object allocation data E_t2 to obtain the corrected output second-time depth data Dx_t2. For example, the control unit 214 For example, referring to the edge information related to the object in the second time-series object allocation data E_t2, the depth data corresponding to the same object is corrected to a close depth value. Due to reference to the first and third time series object allocation information and 201237805

1W7098FA is used to correct the operation of the second time series depth data D_U and D"3 according to the second time series object allocation data Ε"2 to correct the second time series depth j D-12 substantially (d) 'depth estimation device 2 can also be used for the first or third time series depth data (10) and (4) = Ding correction by referring to the first and third time series object allocation data - and E_t3. In other words, according to the first and third time series object allocation resources, t and Ε"3, the first and third timing depth data 匕" and (4) operations may be performed according to the foregoing operation according to the second time series object allocation 2 time depth data (10) Referring to Fig. 5, a flowchart of a depth estimation method according to a second embodiment of the present invention is shown in the above. The depth estimation method of the present embodiment and the depth of the first example The method is based on (4) steps, (1). Further, after step (6), first = (〇, object information estimation unit 210 based on the motion vector data 时序 timing frame data, second timing estimation of fevi-t2 The object is divided, as in step (1), the object data correcting unit 212 corrects the object distribution data 〇"2" according to the first angle of view and the 苐-view frame data F1_t2 and F2_t2 to obtain the second time-series object allocation, step (1), the control unit 214 according to the second time division; 〇2 corrects the second time series depth data W to obtain the output fourth-order depth data Dx_t2. "Lee-day command: = degree estimation method and its device reference Should the entry to a private time series data set frame timing corresponding to another time frame of the sequence 201 237 805

The motion vector data between the TW7098PA data sets and the depth data corresponding to the other time sequence are used to generate estimated depth data corresponding to the target time sequence. The depth estimation method and apparatus of the present embodiment further refer to the estimated depth data to simplify the operation of generating depth data corresponding to the target time sequence. Accordingly, the depth estimation method and apparatus thereof according to the present invention have the advantage of lower computational complexity than the conventional depth estimation method. In the above, the present invention has been disclosed in the above preferred embodiments, but it is not intended to limit the present invention. Further, the depth data and the estimated depth data referred to in the present invention refer to data having a deep nature, and are not limited to depth values, and other materials having deep nature such as disparity values are also within the scope of the present invention. A person skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, the scope of the invention is defined by the scope of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram showing a depth estimating apparatus according to a first embodiment of the present invention. Fig. 2 is a flow chart showing a depth estimation method in accordance with a first embodiment of the present invention. Fig. 3 is a block diagram showing another embodiment of the depth estimating apparatus in accordance with the first embodiment of the present invention. Fig. 4 is a block diagram showing a depth estimating apparatus in accordance with a second embodiment of the present invention.

S 12 201237805

1W7098PA Fig. 5 is a flow chart showing a depth estimation method according to a second embodiment of the present invention. [Description of Main Element Symbols] 1. Γ, 2: depth estimating means 102, 102', 202: input units 104, 104', 204: motion vector generating units 106, 106', 206: dual-view image matching unit 108 108', 208: depth estimation unit 210: object information estimation unit 212: object information correction unit 214: control unit 13

Claims (1)

201237805 . IW/WWA VII. Patent application scope: 1. A depth estimation method for depth estimation of an input binocular video (Binocular Video) data, the depth estimation method includes: (a) receiving the input binocular image a first timing frame data group corresponding to a first time sequence and a first B temple sequence frame data group corresponding to a second time sequence, wherein the first time sequence frame data group and the first The second time series frame data group respectively includes a “first-view frame data and a second view frame data; (7) finding the second time frame data group relative to the first time frame data group first Moving vector data; data data: the first-view frame of a time series frame data set, the first-view frame data for double-view image comparison (9) (10) = lng) 'to find the corresponding to the first - The time sequence - the first time depth (d) is based on the first motion vector material to find a second time series estimation depth data depth (e) according to the second time interval estimation, the first view frame of the data group Capital: first - timing frame The second timing depth data is output. The second perspective frame data, find 2. If the patent application scope method, wherein (4) (4) further includes: the depth data estimation party of the rhyme receives the input binocular image data - the third time series frame data group = the second time sequence The second time frame data group includes a 201237805 ΊΛΥ7098ΡΑ one-view frame data and a second perspective frame data. 3. If the patent application scope f 2 method, including: the wood degree asset estimation party (1) find the third time series frame data group relative to the frame data group - the second movement vector data; ... the first time series ( G丄 according to the first motion vector data, the first timing deep production second second I: timing motion vector data and the second timing depth data = second time series estimation depth data; and leaf finding data order estimation depth data The third timing chart framed the data and the second view frame data to find the third time series depth data. Finding a method, further 4 packages: the depth data estimation method according to item 3 of the patent scope is The second moving vector data is used to find an estimated object allocation data corresponding to one of the third time frame data sets; (j) correcting the third according to the first viewing frame of the third time series data group The timing estimation object is divided into Ή to serve a third time series object allocation data; and ^ the third time series ice data is corrected according to the third time series object allocation data to obtain a round of third time depth data. apply for patent The depth data estimate described in the scope of the second item includes: ° 15 I 201237805 i W /UV»FA (ΠAccording to the first moving vector sequence frame data group to estimate the object allocation data; Second time (J) according to the second time frame frame data phase data and the second view frame data repair 1 first-view frame get-the second time-series object allocation data and estimate the five object distribution data to ( k) according to the second time series depth data, to obtain an output second ^ the second time series method, further 6 packets (four) patent _ the first item of depth data estimation side (1) according to the first motion vector order One of the frame data sets estimates the object allocation resource 1 to find the corresponding time to the first time (J·) according to the first time series frame, the second perspective frame data to correct the estimated object allocation data,: angle, - Time-series object allocation data; and Becco to obtain - 00th based on the first time series of deep-production data, ... f J data to correct the first-order time Beca to serve - output first - timing depth data. Depth data estimation device for invoking video data a depth estimation, the depth estimation loading input 7G' is configured to receive a -th timing frame data group corresponding to the -th-time sequence in the input binocular image data and a second timing corresponding to a second time sequence a frame data group, wherein each of the first and second timing frame data sets includes a first view frame data and a second view 201237805 1 W/〇ys^A frame data; relative = quantity 2 units, For finding the second timing frame data group for the first-time sequence frame data group - the first - moving vector data; according to the "double 3 image ratio fine (four). Μ ·) unit, for the root - timing diagram The first view angle and the second view frame data of the frame data set are compared by the dual view image to find a first time series depth data corresponding to =; the first time sequence: the first time estimate unit 'Using to find a second time-series estimation depth data according to the first-moving vector data and the first time-series depth data; the middle: double-view image comparison unit estimates the depth according to the second timing: _ The first view of the time series frame data set and the second view Frame data to find the depth of the second timing information. Λ ^ If the scope of patent application 帛 7 item of depth data estimate - second = single; more image data in the (four) group package ^ = sequence frame frame of the first view frame data. The depth data according to item 8 of the patent application scope is estimated to be loaded with the moving vector generation unit, and the H (four) trend of the data frame of the second time series frame is obtained. And: the comparison unit further finds a third time series estimation depth data according to the first motion vector resource P depth reduction, the second motion vector data, and the first 17 201237805 TW7098PA timing depth data; and the depth estimation The unit further finds the third timing depth data according to the third timing estimation depth data, the first viewing angle of the third timing frame data group, and the second viewing angle frame data. 10. The depth data estimating apparatus according to claim 9 , further comprising: an object information estimating unit, configured to find an estimate corresponding to the third time frame data group according to the second moving vector data The object information distribution unit, the object information correction unit, correcting the third time-scheduled object allocation data according to the first view angle and the second view frame data of the third time series frame data group, to obtain a third time-series object distribution And a control unit, configured to modify the third timing depth data according to the third time-series object allocation data to obtain an output third timing depth data. 11. The depth data estimating apparatus according to claim 7, further comprising: an object information estimating unit, configured to find an estimate corresponding to the second time frame data group according to the first moving vector data And the object information correction unit, and the object information correction unit corrects the estimated object allocation data according to the first view and the second view frame data of the second time frame data group to obtain a second time series object allocation data; And a control unit, configured to correct the second timing depth data of the 201237805 J W70V8FA according to the second time-series object allocation data, and obtain an output second timing depth data. Set, i2 package, please refer to the depth data estimation device-object information estimation unit described in item 7 (4) of the patent, to find the pair of libraries to, g according to the first moving vector data; and to estimate the object distribution The one-item information correction unit is based on the first 祯& 艿 筮-, 日 &; 时序 时序 时序 图 # “ “ “ “ “ “ “ “ “ “ “ “ “ “ 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正 修正The time-series object allocation data; and the knives of the ^ 制 兀 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖 肖19