TWI485651B - Method for depth estimation and device usnig the same - Google Patents

Description

Depth data estimation method and device thereof

The present invention relates to a depth estimation method and apparatus thereof, and more particularly to a depth estimation method and apparatus for estimating using a dual-view image matching (Stereo Matching) technique.

In today's fast-changing technology era, stereoscopic multimedia systems are gradually being valued by the industry. In general, in the application of stereoscopic video/video, the dual-view image matching (Stereo Matching) image processing technology is a stereoscopic image core technology that is urgently needed in the industry. In the prior art, the dual-view image comparison technique first calculates an image depth map according to the dual-view image.

In general, the dual-view image comparison technique has a problem of high computational complexity. Based on this, how to design a dual-view image comparison depth estimation method with low computational complexity is one of the directions that the industry is constantly striving for.

The present invention relates to a depth estimation method and apparatus thereof, and the depth estimation method and apparatus thereof according to the present invention have the advantage of low computational complexity compared to the conventional depth estimation method.

According to a first aspect of the present invention, a depth estimation method is proposed for performing depth estimation on input binocular video data, and the depth estimation method comprises the following steps. First receiving the first and second timing frame data sets corresponding to the first and second time sequences in the input binocular image data, wherein each of the first and second timing frame data sets includes a first viewing angle and a second viewing angle Frame information. Then, the first moving vector data of the second timing frame data group relative to the first timing frame data group is found. Then, the two-view image matching (Stereo Matching) is performed according to the first view and the second view frame data of the first time series frame data group to find the first time-series depth data corresponding to the first time sequence. Then, the second time series estimated depth data is found according to the motion vector data and the first timing depth data. Then, according to the second timing estimation depth data, the first viewing angle of the second timing frame data group, and the second viewing angle frame data, the second timing depth data is found.

According to a second aspect of the present invention, a depth data estimating apparatus is provided for performing depth estimation on input binocular image data, including an input unit, a motion vector generating unit, a dual-view image matching unit, and a depth estimating unit. . The input unit receives a first timing frame data group corresponding to the first time sequence and a second timing frame data group corresponding to the second time sequence in the input binocular image data, wherein each of the first and second timing frames The data set includes first view frame data and second view frame data. The motion vector generation unit finds the first motion vector data of the second timing frame data group with respect to the first timing frame data group. The dual-view image comparison unit performs dual-view image comparison according to the first view and the second view frame data of the first time frame data group to find the first time depth corresponding to the first time sequence data. The depth estimating unit finds the second time series estimated depth data according to the motion vector data and the first timing depth data. The dual-view image comparison unit finds the second time-series depth data according to the second time-series estimation depth data, the first view angle of the second time-series frame data group, and the second view frame data.

In order to better understand the above and other aspects of the present invention, the preferred embodiments are described below, and in conjunction with the drawings, the detailed description is as follows:

The depth estimation method of this embodiment reduces the computational complexity of the dual-view image matching (Stereo Matching) by referring to the motion vector data of the input binocular video data.

First embodiment

The depth estimation method in this embodiment estimates the estimated depth data by referring to the motion vector data of the input binocular image data, and refers to the estimated depth data to simplify the operation of generating the corresponding depth data.

Referring to FIG. 1, a block diagram of a depth estimating apparatus according to a first embodiment of the present invention is shown. The depth estimating device 1 of the present embodiment is configured to perform depth estimation on the input binocular video data Vi. The input binocular image data Vi includes, for example, a plurality of dual-view frame data sets, the plurality of double-view frame data sets may correspond to respective time sequences, and each of the double-view frame data sets includes two The pen corresponds to the frame data of different perspectives. In other examples, the input binocular image data Vi may further include three or more multi-view frame data.

For example, the timing frame data group Vi_t1 corresponding to the first time sequence t1 in the input binocular image data Vi includes the first view frame data F1_t1 and the second view frame data F2_t1; and the input binocular image data Vi corresponds to The timing frame data set Vi_t2 to the second time sequence t2 includes a first view frame data F1_t2 and a second view frame data F2_t2. The first view frame data F1_t1 and F1_t2 are, for example, left eye view frame data in the first and second time sequences t1 and t2, respectively; the second view frame data F2_t1 and F2_t2 are, for example, first and second, respectively. The right eye view frame data in time sequence t1 and t2; the first and second time sequences t1 and t2 correspond, for example, to operating time points adjacent to each other.

The depth estimation device 1 includes an input unit 102, a motion vector generation unit 104, and a dual view image comparison unit 108.

The input unit 102 receives the timing frame data sets Vi_t1 and Vi_t2, and includes the first view frame data F1_t1 and F1_t2 and the second view frame corresponding to the first and second time sequences t1 and t2 in the input binocular image data Vi. Information F2_t1 and F2_t2. The motion vector generation unit 104 finds the first motion vector data M_12 of the timing frame data group Vi_t2 with respect to the timing frame data group Vi_t1.

In an operation example, the input unit 102 and the motion vector generation unit 104 can be implemented by an image decompressor; the image decompressor performs a decompression operation on the input compressed binocular image data to restore the uncompressed original image data and Corresponding moving vector data, the input compressed binocular image data can be two image data independently of each other, and one binocular image data of one side of the left and right eye view frame data. In one example of operation, the motion vector generation unit 104 further includes, for example, an averaging filter (Mean Filter) that can be used to perform related image processing operations on the motion vector data.

The dual-view image comparison unit 106 performs dual-view image comparison according to the first view and the second view frame data F1_t1 and F2_t1 to find the first time-series data D_t1 corresponding to the first time sequence t1.

The depth estimating unit 108 finds the second time-series estimated depth data DE_t2 corresponding to the second time sequence t2 according to the motion vector data M_12 and the first timing depth data D_t1. Since the first and second time sequences t1 and t2 are timing points adjacent to each other, in general, the input binocular image data Vi has a linearly varying depth distribution pattern in the first and second time sequences t1 and t2, and the depth estimation is performed. The unit 108 estimates the second time series estimated depth data DE_t2 according to the first time series depth data D_t1 and the motion vector data M_12.

The second timing estimation depth data DE_t2 estimated by the depth estimation unit 108 is further provided to the dual-view image comparison unit 106; the dual-view image comparison unit 106 has a second timing corresponding to the second time sequence t2. Based on the estimated depth data DE_t2, the dual view image comparison operation for the first view and the second view frame data F1_t2 and F2_t2 is performed to find the second time depth depth data D_t2 corresponding to the second time sequence t2.

For example, the depth estimation unit 108 determines, according to the second time series estimation depth data DE_t2, the search window for comparing the dual view image comparison operations of the first view and the second view frame data F1_t2 and F2_t2 ( Search Window). In an operation example, for the pixel data P(i1, j1) corresponding to the coordinates (i1, j1) in the first view frame data F1_t2, the second time series estimation depth data DE_t2 indicates that it corresponds to the depth value X. , where i1 and j1 are natural numbers. Accordingly, the dual-view image matching unit 106 can correspond to the center position of the search window to the coordinate position (i1+X, j1) in the second view frame data F2_t2.

According to this, according to the second timing estimation depth data DE_t2, the dual-view image comparison unit 106 can effectively obtain the possible depth values between the first view and the second view frame data F1_t2 and F2_t2, thereby effectively The size of the search window for the dual-view image comparison operation is reduced, and the amount of calculation required for the comparison operation is correspondingly reduced.

Referring to FIG. 2, a flow chart of a depth estimation method according to a first embodiment of the present invention is shown. The depth estimation method of this embodiment includes the following steps. First, as step (a), the input unit 102 receives the timing frame data set Vi_t1 corresponding to the first time sequence t1 and the time series frame data set Vi_t2 corresponding to the second time sequence t2 in the input binocular image data Vi. Next, as in step (b), the motion vector generation unit 104 finds the motion vector data M_12 of the timing frame data set Vi_t2 with respect to the timing frame data set Vi_t1.

Then, in step (c), the dual-view image comparison unit 106 performs dual-view image comparison according to the first view and the second view frame data F1_t1 and F2_t1 to find the first corresponding to the first time sequence t1. Timing depth data D_t1. Then, as step (d), the depth estimating unit 108 finds the second time series estimated depth data DE_t2 according to the motion vector data M_12 and the first timing depth data D_t1.

Then, in step (e), the dual view image comparison unit 106 finds and outputs the second time series depth data D_t2 according to the second time series estimation depth data DE_t2, the first view and the second view frame data F1_t2 and F2_t2.

In this embodiment, the depth estimation device 1 of the present embodiment estimates the second time-series estimated depth data corresponding to the second time sequence t2 with reference to the first time-series depth data D_t1 corresponding to the first time sequence t1. DE_t2 is used as an example to simplify the operation of finding the operation corresponding to the second timing depth data D_t2. However, the depth estimating apparatus 1 of the present embodiment is not limited thereto.

For example, the input unit 102' in the depth estimating device 1' is further configured to receive the timing frame data set Vi_t3; the motion vector generating unit 104' is further configured to generate the timing frame data set Vi_t3 relative to the timing frame data set Vi_t2 The motion vector data M_23; the depth estimating unit 108' is further configured to refer to the motion vector data M_12 and M23 and the first and second timing depth data D_t1 and D_t2 corresponding to the first and second time sequences t1 and t2 to estimate corresponding to The third time series estimation depth data DE_t3 of the third time sequence t3; the dual view image comparison unit 106' refers to the third time series estimation depth data DE_t3 to simplify the operation of finding the corresponding to the third time series depth data D_t3, such as the third The figure shows. In this embodiment, the third time sequence t3 is an operation sequence between the first and second time sequences t1 and t2, but not limited thereto. For example, the third time sequence t3 may also be The timing of the operation after the second time sequence t2.

The depth estimation method and apparatus of the embodiment refer to the motion vector data corresponding to the time series frame data group corresponding to the target time sequence and the frame data group corresponding to another time sequence, and the depth data corresponding to the other time sequence. To generate estimated depth data corresponding to the time sequence of this target. The depth estimation method and apparatus of the present embodiment further refer to the estimated depth data to simplify the operation of generating depth data corresponding to the target time sequence. Accordingly, the depth estimation method and apparatus related to the present invention have the advantage of low computational complexity compared to the conventional depth estimation method.

Second embodiment

The depth estimation method in this embodiment corrects the operation of generating the corresponding depth data by referring to the object segmentation information of the input binocular image data.

Referring to FIG. 4, a block diagram of a depth estimating apparatus according to a second embodiment of the present invention is shown. The depth estimating device 2 of the present embodiment is different from the depth estimating device of the first embodiment in that the depth estimating device 2 further includes an object information estimating unit 210, an object information correcting unit 212, and a control unit 214.

The object information estimating unit 210 finds the second time-series estimated object allocation data O_t2 corresponding to the second timing frame data set Vi_t2 based on the motion vector data M_12. For example, the object information estimating unit 210 can identify and track the objects in the input binocular image data Vi according to the consistency of each pen pixel moving vector data in the moving vector data M_12 and the surrounding pixel moving vector data.

The object information correcting unit 212 corrects the second time-series estimated object allocation data O_t2 according to the first viewing angle and the second viewing angle frame data F1_t2 and F2_t2 to obtain the second time-series object allocation data E_t2. For example, the object information correction unit 212 verifies the accuracy of the second time-scheduled object allocation data O_t2 by referring to the pixel data corresponding to the frame data of the different viewing angles corresponding to the second time sequence t2. Corresponding to the second time series object allocation data E_t2 of the second time sequence t2.

The control unit 214 corrects the second time-series depth data D_t2 according to the second time-series object allocation data E_t2 to obtain the corrected output second-time depth data Dx_t2. For example, the control unit 214 corrects the depth data corresponding to the same object to a close depth value, for example, by referring to the edge information related to the object in the second time-series object allocation data E_t2.

The operation of correcting the first and third timing depth data D_t1 and D_t3 by referring to the first and third time-series object allocation data E_t1 and E_t3 and the operation of correcting the second timing depth data D_t2 according to the second time-series object allocation data E_t2 are substantially Similarly, the depth estimating device 2 can perform the correcting operation for the first or third time-series depth data D_t1 and D_t3 with reference to the first and third time-series object allocation data E_t1 and E_t3 via substantially the same operation. In other words, the operations of modifying the first and third timing depth data D_t1 and D_t3 according to the first and third time-series object allocation data E_t1 and E_t3 may be based on the foregoing operation of correcting the second time-series depth data D_t2 according to the second time-series object allocation data E_t2. Obtained, and will not be repeated here.

Referring to FIG. 5, a flow chart of a depth estimation method according to a second embodiment of the present invention is shown. The depth estimation method of the present embodiment is different from the depth estimation method of the first embodiment in that it further includes steps (i)-(k) after the step (b). Further, after step (b), step (i) is first performed, and the object information estimating unit 210 finds the second time-series estimated object allocation data O_t2 corresponding to the second timing frame data set Vi_t2 according to the motion vector data M_12.

Then, in step (j), the object data correcting unit 212 corrects the second time-series estimated object allocation data O_t2 according to the first viewing angle and the second viewing angle frame data F1_t2 and F2_t2 to obtain the second time-series object allocation data E_t2. Then, as step (k), the control unit 214 corrects the second timing depth data D_t2 according to the second time-series object allocation data E_t2 to obtain the output second timing depth data Dx_t2.

The depth estimation method and apparatus of the embodiment refer to the motion vector data corresponding to the time series frame data group corresponding to the target time sequence and the frame data group corresponding to another time sequence, and the depth data corresponding to the other time sequence. To generate estimated depth data corresponding to the time sequence of this target. The depth estimation method and apparatus of the present embodiment further refer to the estimated depth data to simplify the operation of generating depth data corresponding to the target time sequence. Accordingly, the depth estimation method and apparatus related to the present invention have the advantage of low computational complexity compared to the conventional depth estimation method.

In conclusion, the present invention has been disclosed in the above preferred embodiments, and is not intended to limit the present invention. In addition, the depth data and the estimated depth data referred to in the present invention refer to data having a deep nature, and are not limited to depth values, such as disparity values and other materials having deep nature, and are also within the scope of the present invention. A person skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, the scope of the invention is defined by the scope of the appended claims.

1, 1', 2. . . Depth estimating device

102, 102', 202. . . Input unit

104, 104', 204. . . Moving vector generation unit

106, 106', 206. . . Dual-view image comparison unit

108, 108', 208. . . Depth estimation unit

210. . . Object information estimation unit

212. . . Object information correction unit

214. . . control unit

Fig. 1 is a block diagram showing a depth estimating apparatus according to a first embodiment of the present invention.

2 is a flow chart showing a depth estimation method according to a first embodiment of the present invention.

Fig. 3 is a block diagram showing another embodiment of the depth estimating apparatus according to the first embodiment of the present invention.

Fig. 4 is a block diagram showing a depth estimating apparatus according to a second embodiment of the present invention.

FIG. 5 is a flow chart showing a depth estimation method according to a second embodiment of the present invention.

(a)-(e). . . Steps

Claims (12)

A depth estimation method for performing depth estimation on an input binocular video (Binocular Video) data, the depth estimation method comprising: (a) receiving a first one of the input binocular image data corresponding to a first time sequence a timing frame data group and a second timing frame data group corresponding to a second time sequence, wherein the first timing frame data group and the second timing frame data group each include a first perspective frame Data and a second view frame data; (b) finding a first movement vector data of the second time series frame data group relative to the first time frame data group; (c) according to the first time The first view frame data of the sequence frame data group and the second view frame data are subjected to two-view image matching (Stereo Matching) to find a first time series depth corresponding to the first time sequence. Data; (d) finding a second time-series estimation depth data according to the first motion vector data and the first timing depth data; and (e) estimating the depth data according to the second timing, the second timing frame data The first view frame data of the group and The second view frame data is used to find the second time series depth data. The method for estimating a depth data according to claim 1, wherein the step (a) further comprises: receiving a third timing frame data group corresponding to a third time sequence in the input binocular image data, the The three timing frame data group includes a first perspective frame data and a second perspective frame data. The method for estimating the depth data according to item 2 of the patent application scope further includes: (f) finding a second movement vector data of the third time series frame data group relative to the second time frame data group; g) finding a third timing estimation depth data according to the first motion vector data, the first timing depth data, the second motion vector data, and the second timing depth data; and (h) according to the third timing The depth data, the first view frame data of the third time series frame data group, and the second view frame data are estimated to find the third time series depth data. The method for estimating a depth data according to item 3 of the patent application scope further includes: (i) finding an estimated object allocation data corresponding to one of the third time series frame data groups according to the second motion vector data; (j) And correcting the estimated object allocation data according to the first view frame data of the third time series frame data group and the second view frame data to obtain a third time series object allocation data; and (k) according to the third The time series object allocation data corrects the third time series depth data to obtain an output third time series depth data. For example, the method for estimating depth data as described in item 1 of the patent application scope includes: (i) finding an estimated object allocation data corresponding to one of the second timing frame data sets according to the first motion vector data; (j) the first viewing frame data according to the second timing frame data group and The second view frame data is corrected to obtain the second time-series object allocation data; and (k) correcting the second time-series depth data according to the second time-series object allocation data to obtain an output second Timing depth data. The method for estimating a depth data according to the first aspect of the patent application, further comprising: (i) finding, according to the first motion vector data, an estimated object allocation data corresponding to one of the first timing frame data sets; And correcting the estimated object allocation data according to the first viewing angle and the second viewing angle frame data of the first timing frame data group to obtain a first time series object allocation data; and (k) according to the first The time-series object allocation data corrects the first timing depth data to obtain an output first timing depth data. A depth data estimating device for performing depth estimation on an input binocular video (Binocular Video) data, the depth estimating device comprising: an input unit configured to receive the input binocular image data corresponding to a first time sequence a first timing frame data group and a second timing frame data group corresponding to a second time sequence, wherein each of the first and second timing frame data sets includes a first perspective frame data And a second perspective a frame vector data, coupled to the input unit, for finding a first motion vector data of the second timing frame data group relative to the first time frame data group; The viewfinder unit is coupled to the input unit for performing dual-view image comparison according to the first view and the second view frame data of the first time frame data group a first time depth data corresponding to the first time sequence; and a depth estimation unit coupled to the motion vector generation unit and the dual view image comparison unit for Locating a second time-series estimation depth data by using the motion vector data and the first timing depth data; wherein the dual-view image comparison unit estimates the depth data according to the second timing, and the second timing frame data group The first view angle and the second view frame data are used to find the second time series depth data. The depth data estimating device of claim 7, wherein the input unit further receives a third timing frame data group corresponding to a third time sequence in the input binocular image data, the third timing chart The frame data group includes a first view frame data and a second view frame data. The depth data estimating device of claim 8, wherein: the motion vector generating unit is further configured to find a second movement of the third timing frame data group with respect to one of the second timing frame data groups. Vector data The dual-view image comparison unit further finds a third time-series estimated depth data according to the first motion vector data, the first timing depth data, the second motion vector data, and the second timing depth data; The depth estimation unit further finds the third time series depth data according to the third time series estimation depth data, the first view angle of the third time series frame data group, and the second view frame data. The depth data estimating device of claim 9, further comprising: an object information estimating unit coupled to the motion vector generating unit, configured to find the corresponding to the third timing according to the second motion vector data One of the frame data sets estimates the object allocation data; an object information correction unit is coupled to the object information estimating unit, and corrects the first viewing angle and the second viewing frame data according to the third time series frame data group Estimating the object allocation data to obtain a third time-series object allocation data; and a control unit coupled to the object information correction unit for correcting the third time-series depth data according to the third time-series object allocation data to obtain a The third timing depth data is output. The depth data estimating device of claim 7, further comprising: an object information estimating unit coupled to the motion vector generating unit, configured to find the corresponding to the second timing according to the first motion vector data One of the frame data sets estimates the item allocation information; and An object information correction unit is coupled to the object information estimating unit, and corrects the estimated object allocation data according to the first viewing angle and the second viewing angle frame data of the second timing frame data group to obtain a second timing The object allocation data; and a control unit coupled to the object information correction unit for correcting the second time series depth data according to the second time series object allocation data to obtain an output second time series depth data. The depth data estimating device of claim 7, further comprising: an object information estimating unit coupled to the motion vector generating unit, configured to find the corresponding to the first time according to the first motion vector data One of the sequence frame data sets estimates the object allocation data; and an object information correction unit is coupled to the object information estimating unit, according to the first view and the second view frame of the first time series frame data set Correcting the estimated object allocation data to obtain a first time-series object allocation data; and a control unit coupled to the object information correction unit for correcting the first timing according to the first time-series object allocation data Depth data to get an output first timing depth data.