CN112488915B - Depth reconstruction method and device based on superpixel relation analysis - Google Patents
Depth reconstruction method and device based on superpixel relation analysis Download PDFInfo
- Publication number
- CN112488915B CN112488915B CN201910864317.2A CN201910864317A CN112488915B CN 112488915 B CN112488915 B CN 112488915B CN 201910864317 A CN201910864317 A CN 201910864317A CN 112488915 B CN112488915 B CN 112488915B
- Authority
- CN
- China
- Prior art keywords
- super
- pixel
- superpixel
- adjacent
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/579—Depth or shape recovery from multiple images from motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/61—Scene description
Abstract
The present disclosure relates to depth reconstruction based on superpixel relationship analysisA method and apparatus, the method comprising: determining a feature point matching result M between adjacent image frames in the image frame set and a super-pixel segmentation result S of a target image frame; according to the feature point matching result M and the super-pixel segmentation result S, determining a homography matrix H corresponding to each super-pixel and a motion relation R between adjacent super-pixelse(ii) a According to the motion relation R between adjacent superpixelseDetermining the spatial relationship R between adjacent superpixelss(ii) a According to the spatial relationship R between adjacent superpixelssDetermining a plane parameter theta of each super pixel; and according to the homography matrix H corresponding to each super pixel and the plane parameter theta of each super pixel, performing depth reconstruction on the dynamic scene corresponding to the image frame set, thereby effectively realizing the depth reconstruction on the complex motion scene.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to a depth reconstruction method and apparatus based on superpixel relationship analysis.
Background
Monocular camera-based depth reconstruction methods have been an important and extremely challenging piece of research, which has important applications in many fields, such as automatic driving, object recognition, motion recognition, etc. However, for many complex Motion scenes in practical application, such as rigid moving vehicles, pedestrians moving in a deformation manner, and the like, when a conventional Motion StructuRe From Motion (SFM) method is used for performing dynamic scene depth solution, the relative scale of each unit StructuRe cannot be directly determined, so that the conventional SFM method cannot perform depth reconstruction on the complex Motion scenes.
Disclosure of Invention
In view of this, the present disclosure provides a depth reconstruction method and apparatus based on superpixel relationship analysis, which effectively implement depth reconstruction of a complex motion scene.
According to a first aspect of the present disclosure, there is provided a depth reconstruction method based on superpixel relationship analysis, including: determining a feature point matching result M between adjacent image frames in an image frame set and a super-pixel segmentation result S of a target image frame, wherein the image frame set comprises at least two image frames, the target image frame is one of the image frame set, and the super-pixel segmentation result S comprises a plurality of super-pixels; determining the motion relation between the homography matrix H corresponding to each super pixel and the adjacent super pixels according to the feature point matching result M and the super pixel segmentation result SRe(ii) a According to the motion relation R between the adjacent super pixelseDetermining a spatial relationship R between said neighboring superpixelss(ii) a According to the spatial relation R between said adjacent superpixelssDetermining a plane parameter theta of each super pixel; and performing depth reconstruction on the dynamic scene corresponding to the image frame set according to the homography matrix H corresponding to each super pixel and the plane parameter theta of each super pixel.
In a possible implementation manner, the motion relationship R between the adjacent superpixels and the homography matrix H corresponding to each superpixel is determined according to the feature point matching result M and the superpixel segmentation result SeThe method comprises the following steps: for any superpixel i, by applying a first energy function E (H, R)e) Optimizing to obtain a homography matrix H corresponding to the superpixel iiAnd a motion relation R between said super-pixel i and an adjacent super-pixel je(i, j); wherein the first energy function E (H, R)e) Comprises the following steps: a homography matrix H corresponding to the super pixeliRelated first optimization term Edata(Hi) Homography matrix H corresponding to said super-pixeliThe motion relation R between the super pixel i and the adjacent super pixele(i, j) and homography matrix H corresponding to the adjacent pixel jjAssociated second optimization term Epair(Hi，Hj，Re(i, j)), and a motion relationship R between said super-pixel i and an adjacent super-pixele(i, j) related constant term Eo(Re(i，j))。
In a possible implementation manner, the homography matrix H corresponding to the adjacent pixel jjIs the optimized homography matrix.
In a possible implementation, said motion relation R between said adjacent superpixels is used as a function of said motion relation ReDetermining a spatial relationship R between said neighboring superpixelssThe method comprises the following steps: relating the motion R between said adjacent superpixelseIs determined as a spatial relationship R between said neighboring superpixelssThe relationship type includes: coplanar, Hinge and Crack.
In one possible implementation, said determining is based on a spatial relationship R between said adjacent superpixelssDetermining a plane parameter θ for each superpixel, comprising: determining a reference superpixel set S among the plurality of superpixels based on a preset algorithmtWherein the reference super pixel set StCorresponding to a background part in the target image frame; the reference super pixel set StDetermining the scale factor s of each reference superpixel as 1; according to the reference super pixel set StAnd the spatial relationship R between said adjacent superpixelssThe plane parameter θ for each superpixel is determined.
In one possible implementation, the set S of super-pixels is based on the referencetAnd the spatial relationship R between said adjacent superpixelssDetermining a plane parameter θ for each superpixel, comprising: for any superpixel i, optimizing a second energy function E (theta, s) to obtain a plane parameter theta of the superpixel iiAnd a scale factor si(ii) a Wherein the second energy function E (θ, s) includes: with the plane parameter theta of the super-pixel iiAnd a scale factor siRelated third optimization term Efit(θi，si) With the plane parameter θ of the superpixel iiAnd a plane parameter theta of an adjacent super pixel j corresponding to the super pixel ijRelated fourth optimization term ERel(θi，θj) And a scale factor s associated with said super-pixel iiAssociated constant term Eocc(si)。
In a possible implementation, the motion relation R between said super-pixel i and all the adjacent super-pixels j is determinedsWhen the relationship types of (i, j) are all Crack, the second energy function E (θ, s) further includes: with the plane parameter theta of the super-pixel iiPlane parameter theta of super pixel kkRelated fifth optimization term Epair(θi，θk) Wherein the super pixel k is a super pixel set SrA set S of super-pixelsrCan reach the reference superpixel set S based on the target pathtAnd the relation type of the superpixel relation on the target path is Coplannar or Hinge.
In a possible implementation, the plane parameter θ of said neighboring super-pixel jjAnd a plane parameter theta of the superpixel kkThe optimized plane parameters.
In a possible implementation manner, the performing depth reconstruction on the dynamic scene corresponding to the image frame set according to the homography matrix H corresponding to each super pixel and the plane parameter θ of each super pixel includes: decomposing the homography matrix H corresponding to each super pixel to obtain a camera rotation parameter R corresponding to each super pixel0And a camera translation parameter t0(ii) a According to the camera rotation parameter R corresponding to each super pixel0And a camera translation parameter t0And constructing a dense depth map of the dynamic scene corresponding to the image frame set according to the plane parameter theta of each super pixel.
According to a second aspect of the present disclosure, there is provided a depth reconstruction apparatus based on superpixel relationship analysis, including: the image processing device comprises a first determining module, a second determining module and a third determining module, wherein the first determining module is used for determining a feature point matching result M between adjacent image frames in an image frame set and a super pixel segmentation result S of a target image frame, the image frame set comprises at least two image frames, the target image frame is one of the image frame set, and the super pixel segmentation result S comprises a plurality of super pixels; a second determining module, configured to determine a homography matrix H corresponding to each super pixel and a motion relation R between adjacent super pixels according to the feature point matching result M and the super pixel segmentation result Se(ii) a A third determination module for determining a motion relation R between said adjacent superpixelseDetermining a spatial relationship R between said neighboring superpixelss(ii) a A fourth determination module for determining a spatial relationship R between said adjacent superpixelssDetermining a plane parameter theta of each super pixel; a reconstruction module for reconstructing said image from said homography H corresponding to each superpixel and said image of each superpixelAnd the plane parameter theta is used for carrying out depth reconstruction on the dynamic scene corresponding to the image frame set.
Determining a feature point matching result M between adjacent image frames in an image frame set comprising at least two image frames and a super-pixel segmentation result S of a target image frame, wherein the target image frame is one frame in the image frame set, the super-pixel segmentation result S comprises a plurality of super-pixels, and determining a homography matrix H corresponding to each super-pixel and a motion relation R between the adjacent super-pixels according to the feature point matching result M and the super-pixel segmentation result SeAccording to the motion relation R between adjacent superpixelseDetermining the spatial relationship R between adjacent superpixelssAccording to the spatial relationship R between adjacent superpixelssAnd determining a plane parameter theta of each super pixel, and performing depth reconstruction on the dynamic scene corresponding to the image frame set according to the homography matrix H corresponding to each super pixel and the plane parameter theta of each super pixel. Therefore, the spatial relationship of the super pixels is determined through the motion relationship of the super pixels, so that the plane parameter of each super pixel can be determined, and the deep reconstruction of a complex motion scene is effectively realized.
Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.
FIG. 1 is a schematic flow chart of a depth reconstruction method based on superpixel relationship analysis according to an embodiment of the present disclosure;
FIG. 2 shows a schematic diagram of a depth reconstruction system based on superpixel relationship analysis according to an embodiment of the present disclosure;
FIG. 3 shows a schematic diagram of the results of depth reconstruction using two consecutive image frames according to an embodiment of the present disclosure;
fig. 4 shows a schematic structural diagram of a depth reconstruction apparatus based on superpixel relationship analysis according to an embodiment of the present disclosure.
Detailed Description
Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.
Fig. 1 shows a schematic flowchart of a depth reconstruction method based on superpixel relationship analysis according to an embodiment of the present disclosure. As shown in fig. 1, the method may include:
step S11, determining a feature point matching result M between adjacent image frames in the image frame set and a super-pixel segmentation result S of the target image frame, where the image frame set includes at least two image frames, the target image frame is one frame in the image frame set, and the super-pixel segmentation result S includes a plurality of super-pixels.
Step S12, according to the feature point matching result M and the super-pixel segmentation result S, determining the homography matrix H corresponding to each super-pixel and the motion relation R between the adjacent super-pixelse。
Step S13, according to the motion relation R between the adjacent super pixelseDetermining the spatial relationship R between adjacent superpixelss。
Step S14, according to the spatial relation R between the adjacent super-pixelssThe plane parameter θ for each superpixel is determined.
And step S15, performing depth reconstruction on the dynamic scene corresponding to the image frame set according to the homography matrix H corresponding to each super pixel and the plane parameter theta of each super pixel.
The spatial relationship of the super-pixels is determined through the motion relationship between the super-pixels, so that the plane parameter of each super-pixel can be determined, and the depth reconstruction of a complex motion scene is effectively realized.
Fig. 2 shows a schematic diagram of a depth reconstruction system based on superpixel relationship analysis according to an embodiment of the present disclosure. As shown in fig. 2, the depth reconstruction system includes five modules: the device comprises a preprocessing module, a relation analysis module, a motion selection module, a reconstruction module and an optimization module.
Inputting at least two image frames included in the image frame set into a preprocessing module, preprocessing the at least two image frames by the preprocessing module, and outputting a feature point matching result M between adjacent image frames and a super-pixel segmentation result S of a target image frame.
In one example, feature point matching results between adjacent image frames are determined by a CPM algorithm.
It should be noted that, the method for determining the feature point matching result M between adjacent image frames may also use other algorithms besides the CPM algorithm described above, and this disclosure is not limited in this respect.
In an example, a superpixel segmentation result S of the target image frame is determined by the SLIC algorithm.
It should be noted that, the method for determining the super-pixel segmentation result S of the target image frame may also use other algorithms besides the SLIC algorithm described above, and this disclosure is not limited in this regard.
For the relationship between adjacent superpixels, two relationships are defined: kinematic relationships and spatial relationships. The motion relationship is related to the homography matrix H and can be obtained by point matching, and the spatial relationship is determined by the plane parameter theta. For the kinematic and spatial relationships, there are 3 relationship types each, defined as: coplanar, Hinge and Crack. Table 1 shows a specific definition of the kinematic relationships. Table 2 shows a specific definition of the spatial relationship.
TABLE 1
TABLE 2
Wherein Hi is the super pixel SiCorresponding homography matrix, thetaiIs a super pixel SiP is the canonical form of the pixel point P, Bi,jIs a neighboring super-pixel SiAnd SjA set of pixels on a common boundary.
Still taking the above fig. 2 as an example, the feature point matching result M between the adjacent image frames output by the preprocessing module and the super-pixel segmentation result S of the target image frame are input into the relationship analysis module, and the relationship analysis module determines the homography matrix H corresponding to each super-pixel and the motion relationship R between the adjacent super-pixels according to the feature point matching result M between the adjacent image frames determined by the preprocessing module and the super-pixel segmentation result S of the target image framee。
In one possible implementation, according to the feature point matching result M and the super-pixel segmentation result S, determining the homography matrix H corresponding to each super-pixel and the motion relationship R between adjacent super-pixelseThe method comprises the following steps: for any superpixel i, by applying a first energy function E (H, R)e) Optimizing to obtain a homography matrix H corresponding to the super pixel iiAnd the motion relation R between a superpixel i and an adjacent superpixel je(i, j); wherein the first energy function E (H, R)e) Comprises the following steps: homography matrix H corresponding to super pixeliRelated first optimization term Edata(Hi) Homography matrix H corresponding to a super pixeliThe motion relation R between a super-pixel i and an adjacent super-pixele(i, j) homography matrix H corresponding to adjacent pixel jjAssociated second optimization term Epair(Hi，Hj，Re(i, j)), and a motion relationship R with the super-pixel i and the neighboring super-pixele(i, j) related constant term Eo(Re(i，j))。
In one possible implementation, the homography matrix H corresponding to the adjacent pixel jjIs the optimized homography matrix.
For example, the first energy function E (H, R)e) Is expressed as follows:
determining a homography matrix H corresponding to the super pixel i by optimizing the first energy functioniAnd the motion relation R between a superpixel i and an adjacent superpixel je(i, j) E is the set of adjacent superpixel pairs corresponding to superpixel i, and λs1And λs2Are the weight coefficients.
At a first energy function E (H, R)e) In (1), a homography matrix H corresponding to the super pixeliRelated first optimization term Edata(Hi) The specific representation form for matching the feature point matching result M is as follows:
wherein, | ZiI is a normalization parameter, Su(Si) Is the support matching of the super-pixel i, taugIs the threshold parameter, ωcThe specific representation form of (A) is as follows:
ωc(Si,pl)＝exp(-ωs(Si,Sj)/γ),pl∈Sj。
wherein, ω issγ is a constant parameter for the geodesic distance between a superpixel i and the center of a superpixel j.
At a first energy function E (H, R)e) In (1), a homography matrix H corresponding to the super pixeliThe motion relation R between a super-pixel i and an adjacent super-pixele(i, j) homography matrix H corresponding to adjacent pixel jjAssociated second optimization term Epair(Hi，Hj，Re(i, j)) are represented in the following specific forms:
at a first energy function E (H, R)e) And the motion relation R between the super pixel i and the adjacent super pixele(i, j) related constant term Eo(Re(i, j)) is represented in the following specific form:
the first energy function E (H, R) is set to be equal to or higher than a predetermined threshold when a plurality of image frames are included in the image frame sete) Further comprising: with the motion relation R between the super-pixel i and the adjacent super-pixele(i, j) related prior term Emul(Re(i, j)), at which time the first energy function E (H, R)e) Is expressed in the following specific form:
wherein the prior term Emul(Re(i, j)) is added according to the tracking superpixel relationship, and is expressed in the following form:
wherein R isp(i, j) is a priori the tracked motion relationships,
indicates how many frames the relationship holds, τmIs a threshold parameter.
By optimizing the first energy function E (H, R)e) Determining that a super-pixel i corresponds toHomography matrix HiAnd the motion relation R between a superpixel i and an adjacent superpixel je(i, j) includes:
first, a homography matrix H is randomly initializedi. Wherein, the homography matrix H can be initialized through the characteristic point matching result MiThe homography matrix H may also be initialized by other methodsiThis disclosure is not particularly limited thereto.
Secondly, according to the optimized homography matrix H corresponding to the adjacent pixelsjDetermining a second optimization term Epair(Hi，Hj，Re(i, j)), a support vector that considers all point matches involved as local is optimized.
And finally, using a new support vector set to correspond to the homography matrix H by adopting a method of randomly generating model parameters according to a fast propagation methodiAnd (6) optimizing.
Wherein, the optimization process can be ended by setting the optimization times, or the homography matrix H can be usediThe optimization process is ended when convergence is reached, and the optimization process may also be ended in other ways, which is not specifically limited by the present disclosure.
Optimizing the first energy function E (H, R) by the abovee) The homography matrix H corresponding to each super pixel and the motion relation R between the adjacent super pixels can be obtainede。
In a possible implementation, the motion relation R between adjacent superpixels is used as a basiseDetermining the spatial relationship R between adjacent superpixelssThe method comprises the following steps: relating the motion between adjacent super-pixels to ReIs determined as a spatial relationship R between adjacent superpixelssThe relationship type includes: coplanar, Hinge and Crack.
In practical application, the motion relation and the control relation between the adjacent super pixels are in one-to-one correspondence, so that the motion relation R between the adjacent super pixels is obtainedeThen, the motion relation R between the adjacent super pixelseIs determined as a spatial relationship R between adjacent superpixelssThe type of relationship (c). I.e. phase inversionMotion relation R between adjacent super pixelseWhen the relationship type of (1) is Coplanar, the spatial relationship R thereofsThe type of relationship of (a) is also Coplanar; when the motion relation R between adjacent superpixelseWhen the relationship type of (1) is Hinge, the spatial relationship RsThe relationship type of (1) is also Hinge; when the motion relation R between adjacent super-pixelseWhen the relationship type of (1) is Crack, the spatial relationship R issIs also Crack.
Still taking the example of FIG. 2 above, the relationship analysis module outputs the homography H corresponding to each super-pixel and the spatial relationship R between adjacent super-pixelssTo the motion selection module.
In one possible implementation, the spatial relationship R between adjacent superpixels is based onsDetermining a plane parameter θ for each superpixel, comprising: determining a reference superpixel set S among a plurality of superpixels based on a preset algorithmtWherein the reference super pixel set StA background portion in the corresponding target image frame; reference super pixel set StDetermining the scale factor s of each reference superpixel as 1; from a reference set of superpixels StAnd the spatial relationship R between adjacent superpixelssThe plane parameter θ for each superpixel is determined.
Still taking the example of fig. 2 above, the motion selection module determines a reference super-pixel set S among the plurality of super-pixels based on a predetermined algorithmtAnd decomposing the homography matrix Hi corresponding to each super pixel to obtain the inverse depth d corresponding to each super pixeliAnd the normal vector n of the planeiAnd further outputs a reference super-pixel set StThe spatial relationship R between adjacent superpixelssAnd an inverse depth d for each super-pixeliAnd the normal vector n of the planeiTo the reconstruction module such that the reconstruction module determines a planar parameter θ for each superpixel.
In one possible implementation, the set S of superpixels is based on a referencetAnd the spatial relationship R between adjacent superpixelssDetermining a plane parameter θ for each superpixel, comprising: for any superpixel i, by optimizing the second energy function E (θ, s),obtaining the plane parameter theta of the superpixel iiAnd a scale factor si(ii) a Wherein, the second energy function E (θ, s) includes: with the plane parameter theta of the super-pixel iiAnd a scale factor siRelated third optimization term Efit(θi，si) And the plane parameter theta of the super pixel iiPlane parameter theta of adjacent super pixel j corresponding to super pixel ijRelated fourth optimization term ERel(θi，θj) And scale factor s with super pixel iiAssociated constant term Eocc(si)。
In one possible implementation, the motion relationship R between a superpixel i and all neighboring superpixels jsWhen the relationship types of (i, j) are all Crack, the second energy function E (θ, s) further includes: with the plane parameter theta of the super-pixel iiPlane parameter theta of super pixel kkAssociated fifth optimization term Epari (θ)i，θk) Wherein the super-pixel k is a super-pixel set SrOther super-pixels, super-pixel sets SrCan reach the reference superpixel set S based on the target pathtAnd the relation type of the superpixel relation on the target path is Coplanar or Hinge.
In a possible implementation, the plane parameter θ of the neighboring superpixel jjAnd the plane parameter theta of the superpixel kkThe optimized plane parameters.
For example, the second energy function E (θ, s) is expressed as follows:
wherein λ isr1、λr2And λr3Are the weight coefficients.
In the second energy function E (theta, s), the plane parameter theta of the super-pixel iiPlane parameter theta of adjacent super pixel j corresponding to super pixel ijRelated fourth optimization term ERel(θi，θj) In the concrete form of representationThe following were used:
in the second energy function E (θ, s), the scale factor s associated with the super-pixel iiAssociated constant term Eocc(si) The specific representation form of (A) is as follows:
Eocc(si)＝δ(si≠1)。
wherein the content of the first and second substances,
in the second energy function E (theta, s), the plane parameter theta of the super-pixel iiAnd a scale factor siRelated third optimization term Efit(θi，si) Mainly used for maintaining local characteristics, the specific representation form is as follows:
wherein, taurIs a threshold parameter.
Definition of SrRepresenting all the underlying super-pixel sets StThe super pixel set that can be achieved. For one not in StWe define that it can reach StIf and only if we can find a target path from the superpixel to StAnd the relation type of the superpixel relation on the target path is Coplanar or Hinge, and at this time, the following is defined:
Ec＝{(i,k)|(i,k)∈E,Si∈Sr,Sk∈S\Sr}。
in the second energy function E (theta, s), the plane parameter theta of the super-pixel iiPlane parameter theta of super pixel kkRelated fifth optimization term Epair(θi，θk) The specific expression form of (A) is as follows:
according to the fifth optimization term Epair(θi，θk) It can be ensured that the plane parameters of all superpixels apply to the second energy function E (θ, s) described above.
Determining a planar parameter θ of the superpixel i by optimizing said second energy function E (θ, s)iAnd a scale factor siThe method comprises the following steps:
first, a plane parameter θ is randomly initializedi。
Secondly, according to the optimized homography matrix theta corresponding to the adjacent pixelsjAnd thetakDetermining a fourth optimization term ERel(θi，θj) And a fifth optimization term Epari(θi，θk)。
Finally, according to a fast propagation method, a method for randomly generating model parameters is adopted to carry out plane parameter thetaiAnd (6) optimizing.
Wherein, the optimization process can be ended by setting the optimization times, or the plane parameter theta can be setiThe optimization process is ended when convergence is reached, and the optimization process may also be ended in other ways, which is not specifically limited by the present disclosure.
In one possible implementation manner, performing depth reconstruction on a dynamic scene corresponding to an image frame set according to a homography matrix H corresponding to each super pixel and a plane parameter θ of each super pixel includes: decomposing the homography matrix H corresponding to each super pixel to obtain the camera rotation parameter R corresponding to each super pixel0And a camera translation parameter t0(ii) a According to the camera rotation parameter R corresponding to each super pixel0And a camera translation parameter t0And the plane parameter theta of each super pixel is used for constructing a dense depth map of the dynamic scene corresponding to the image frame set.
Still taking the example of fig. 2 above, the motion selection module decomposes the homography matrix Hi corresponding to each super-pixel and obtains the camera rotation corresponding to each super-pixelParameter R0And a camera translation parameter t0And output to a reconstruction module, which is finally based on the camera rotation parameter R corresponding to each superpixel0And a camera translation parameter t0And a plane parameter theta of each super pixel, constructing a dense depth map D of the dynamic scene corresponding to the image frame set, and further optimizing by an optimization module to obtain an optimized dense depth map DR. Fig. 3 shows a schematic diagram of the result of depth reconstruction using two consecutive image frames according to an embodiment of the present disclosure.
Determining a feature point matching result M between adjacent image frames in an image frame set comprising at least two image frames and a super-pixel segmentation result S of a target image frame, wherein the target image frame is one frame in the image frame set, the super-pixel segmentation result S comprises a plurality of super-pixels, and determining a homography matrix H corresponding to each super-pixel and a motion relation R between the adjacent super-pixels according to the feature point matching result M and the super-pixel segmentation result SeAccording to the motion relation R between adjacent superpixelseDetermining the spatial relationship R between adjacent superpixelssAccording to the spatial relationship R between adjacent superpixelssAnd determining a plane parameter theta of each super pixel, and performing depth reconstruction on the dynamic scene corresponding to the image frame set according to the homography matrix H corresponding to each super pixel and the plane parameter theta of each super pixel. Therefore, the spatial relationship of the super pixels is determined through the motion relationship of the super pixels, so that the plane parameter of each super pixel can be determined, and the deep reconstruction of a complex motion scene is effectively realized.
Fig. 4 shows a schematic structural diagram of a depth reconstruction apparatus based on superpixel relationship analysis according to an embodiment of the present disclosure. As shown in fig. 4, the apparatus 40 includes:
a first determining module 41, configured to determine a feature point matching result M between adjacent image frames in an image frame set and a super-pixel segmentation result S of a target image frame, where the image frame set includes at least two image frames, the target image frame is one frame in the image frame set, and the super-pixel segmentation result S includes multiple super-pixels;
a second determining module 42, configured to determine a homography H corresponding to each super pixel and a motion relation R between adjacent super pixels according to the feature point matching result M and the super pixel segmentation result Se；
A third determination module 43 for determining the motion relation R between adjacent superpixelseDetermining the spatial relationship R between adjacent superpixelss；
A fourth determination module 44 for determining the spatial relationship R between adjacent superpixelssDetermining a plane parameter theta of each super pixel;
and the reconstruction module 45 is configured to perform depth reconstruction on the dynamic scene corresponding to the image frame set according to the homography matrix H corresponding to each super pixel and the plane parameter θ of each super pixel.
In one possible implementation, the second determining module 42 is specifically configured to:
for any superpixel i, by applying a first energy function E (H, R)e) Optimizing to obtain a homography matrix H corresponding to the super pixel iiAnd the motion relation R between a superpixel i and an adjacent superpixel je(i，j)；
Wherein the first energy function E (H, R)e) Comprises the following steps: homography H corresponding to a super-pixel iiRelated first optimization term Edata(Hi) Homography matrix H corresponding to a super-pixel iiThe motion relation R between a super-pixel i and an adjacent super-pixel je(i, j) homography matrix H corresponding to adjacent pixel jjAssociated second optimization term Epair(Hi，Hj，Re(i, j)), and a motion relationship R with the super-pixel i and the neighboring super-pixel je(i, j) related constant term Eo(Re(i，j))。
In one possible implementation, the homography matrix H corresponding to the adjacent pixel jjIs the optimized homography matrix.
In a possible implementation manner, the third determining module 43 is specifically configured to:
relating the motion between adjacent superpixels to ReDetermination of the type of relationshipAs a spatial relationship R between adjacent superpixelssThe relationship type includes: coplanar, Hinge and Crack.
In one possible implementation, the fourth determining module 44 includes:
a first determination submodule for determining a set S of reference superpixels among the plurality of superpixels on the basis of a preset algorithmtWherein the reference super pixel set StA background portion in the corresponding target image frame;
a second determining submodule for determining the reference superpixel set StDetermining the scale factor s of each reference superpixel as 1;
a third determining submodule for determining a reference set S of superpixels fromtAnd the spatial relationship R between adjacent superpixelssThe plane parameter θ for each superpixel is determined.
In a possible implementation manner, the third determining submodule is specifically configured to:
for any superpixel i, the plane parameter theta of the superpixel i is obtained by optimizing the second energy function E (theta, s)iAnd a scale factor si；
Wherein, the second energy function E (θ, s) includes: with the plane parameter theta of the super pixel iiAnd scale factor siRelated third optimization term Efit(θi，si) And the plane parameter theta of the super pixel iiPlane parameter theta of adjacent super pixel j corresponding to super pixel ijRelated fourth optimization term ERel(θi，θj) And scale factor s with super pixel iiAssociated constant term Eocc(si)。
In one possible implementation, the motion relationship R between a superpixel i and all neighboring superpixels jsWhen the relationship types of (i, j) are all Crack, the second energy function E (θ, s) further includes: with the plane parameter theta of the super-pixel iiPlane parameter theta of super pixel kkAssociated fifth optimization term Epair(θi，θk) Wherein the super-pixel k is a super-pixel set SrOther super-pixels, super-pixel sets SrCan reach the reference super-pixel set S based on the target pathtAnd the relation type of the superpixel relation on the target path is Coplanar or Hinge.
In a possible implementation, the plane parameter θ of the neighboring superpixel jjAnd the plane parameter theta of the superpixel kkThe optimized plane parameters.
In one possible implementation, the reconstruction module 45 is specifically configured to:
decomposing the homography matrix H corresponding to each super pixel to obtain the camera rotation parameter R corresponding to each super pixel0And a camera translation parameter t0；
According to the camera rotation parameter R corresponding to each super pixel0And a camera translation parameter t0And the plane parameter theta of each super pixel is used for constructing a dense depth map of the dynamic scene corresponding to the image frame set.
The apparatus 40 provided in the present disclosure can implement each step in the method embodiment shown in fig. 1 and/or fig. 2, and implement the same technical effect, and for avoiding repetition, details are not described here again.
Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein were chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (9)
1. A depth reconstruction method based on superpixel relationship analysis is characterized by comprising the following steps:
determining a feature point matching result M between adjacent image frames in an image frame set and a super-pixel segmentation result S of a target image frame, wherein the image frame set comprises at least two image frames, the target image frame is one of the image frame set, and the super-pixel segmentation result S comprises a plurality of super-pixels;
determining a homography matrix H corresponding to each super pixel and a motion relation R between adjacent super pixels according to the feature point matching result M and the super pixel segmentation result Se；
According to the motion relation R between the adjacent super pixelseDetermining a spatial relationship R between said neighboring superpixelss；
According to the spatial relation R between said adjacent superpixelssDetermining a plane parameter theta of each super pixel;
according to the homography matrix H corresponding to each super pixel and the plane parameter theta of each super pixel, carrying out depth reconstruction on the dynamic scene corresponding to the image frame set;
determining a homography matrix H corresponding to each super pixel and a motion relation R between adjacent super pixels according to the feature point matching result M and the super pixel segmentation result SeThe method comprises the following steps:
for any superpixel i, by applying a first energy function E (H, R)e) Optimizing to obtain a homography matrix H corresponding to the superpixel iiAnd a motion relation R between said super-pixel i and an adjacent super-pixel je(i，j)；
Wherein the first energy function E (H, R)e) Comprises the following steps: a homography H corresponding to said super-pixel iiRelated first optimization term Edata(Hi) Homography H corresponding to said super-pixel iiThe motion relation R between the super pixel i and the adjacent super pixel je(i, j) homography H corresponding to said adjacent superpixel jjAssociated second optimization term Epair(Hi，Hj，Re(i, j)), and a motion relationship R between said super-pixel i and an adjacent super-pixel je(i, j) related constant term Eo(Re(i，j))。
2. The method of claim 1, wherein the homography matrix H for the neighboring superpixel j corresponds tojIs the optimized homography matrix.
3. The method according to claim 1, wherein said determining is based on a motion relationship R between said neighboring superpixelseDetermining a spatial relationship R between said neighboring superpixelssThe method comprises the following steps:
relating the motion R between said adjacent superpixelseIs determined as a spatial relationship R between said neighboring superpixelssThe relationship type includes: coplanar, Hinge and Crack.
4. A method according to claim 3, wherein said correlation is based on the spatial relationship R between said adjacent superpixelssDetermining a plane parameter θ for each superpixel, comprising:
determining a reference superpixel set S among the plurality of superpixels based on a preset algorithmtWherein the reference super pixel set StCorresponding to a background part in the target image frame;
the reference super pixel set StDetermining the scale factor s of each reference superpixel as 1;
according to the reference super pixel set StAnd the spatial relationship R between said adjacent superpixelssThe plane parameter θ for each superpixel is determined.
5. The method of claim 4, wherein the set S of superpixels is based on the referencetAnd the spatial relationship R between said adjacent superpixelssDetermining a plane parameter θ for each superpixel, comprising:
for any superpixel i, optimizing a second energy function E (theta, s) to obtain a plane parameter theta of the superpixel iiAnd a scale factor si；
Wherein the second energyThe quantity function E (θ, s) includes: with the plane parameter theta of the super-pixel iiAnd a scale factor siRelated third optimization term Efit(θi，si) With the plane parameter θ of the superpixel iiAnd the plane parameter theta of the adjacent super pixel j corresponding to the super pixel ijRelated fourth optimization term ERel(θi，θj) And a scale factor s associated with said super-pixel iiAssociated constant term Eocc(si)。
6. Method according to claim 5, characterized in that the motion relation R between said superpixel i and all neighboring superpixels j is determined assWhen the relationship types of (i, j) are all Crack, the second energy function E (θ, s) further includes: with the plane parameter theta of the super-pixel iiPlane parameter theta of super pixel kkRelated fifth optimization term Epair(θi，θk) Wherein the super pixel k is a super pixel set SrA set S of super-pixelsrCan reach the reference superpixel set S based on the target pathtAnd the relation type of the superpixel relation on the target path is Coplannar or Hinge.
7. The method of claim 6, wherein the plane parameter θ of the neighboring superpixel jjAnd a plane parameter θ of the superpixel kkThe optimized plane parameters.
8. The method according to claim 1, wherein the depth reconstruction of the dynamic scene corresponding to the image frame set according to the homography H corresponding to each super pixel and the plane parameter θ of each super pixel comprises:
decomposing the homography matrix H corresponding to each super pixel to obtain a camera rotation parameter R corresponding to each super pixel0And a camera translation parameter t0；
According to the camera rotation parameter R corresponding to each super pixel0And a camera translation parameter t0And constructing a dense depth map of the dynamic scene corresponding to the image frame set according to the plane parameter theta of each super pixel.
9. A depth reconstruction device based on superpixel relationship analysis, comprising:
the image processing device comprises a first determining module, a second determining module and a third determining module, wherein the first determining module is used for determining a feature point matching result M between adjacent image frames in an image frame set and a superpixel segmentation result S of a target image frame, the image frame set comprises at least two image frames, the target image frame is one frame in the image frame set, and the superpixel segmentation result S comprises a plurality of superpixels;
a second determining module, configured to determine a homography matrix H corresponding to each super pixel and a motion relation R between adjacent super pixels according to the feature point matching result M and the super pixel segmentation result Se；
A third determination module for determining a motion relation R between said adjacent superpixelseDetermining a spatial relationship R between said neighboring superpixelss；
A fourth determination module for determining a spatial relationship R between said adjacent superpixelssDetermining a plane parameter theta of each super pixel;
the reconstruction module is used for carrying out depth reconstruction on the dynamic scene corresponding to the image frame set according to the homography matrix H corresponding to each super pixel and the plane parameter theta of each super pixel;
the second determining module is specifically configured to:
for any superpixel i, by applying a first energy function E (H, R)e) Optimizing to obtain a homography matrix H corresponding to the superpixel iiAnd a motion relation R between said super-pixel i and an adjacent super-pixel je(i，j)；
Wherein the first energy function E (H, R)e) Comprises the following steps: a homography H corresponding to said super-pixel iiIs related to the firstOptimization term Edata(Hi) Homography H corresponding to said super-pixel iiThe motion relation R between the super pixel i and the adjacent super pixel je(i, j) homography H corresponding to said adjacent superpixel jjAssociated second optimization term Epair(Hi，Hj，Re(i, j)), and a motion relationship R between said super-pixel i and an adjacent super-pixel je(i, j) related constant term Eo(Re(i，j))。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910864317.2A CN112488915B (en) | 2019-09-12 | 2019-09-12 | Depth reconstruction method and device based on superpixel relation analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910864317.2A CN112488915B (en) | 2019-09-12 | 2019-09-12 | Depth reconstruction method and device based on superpixel relation analysis |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112488915A CN112488915A (en) | 2021-03-12 |
| CN112488915B true CN112488915B (en) | 2022-06-21 |
Family
ID=74920772
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910864317.2A Active CN112488915B (en) | 2019-09-12 | 2019-09-12 | Depth reconstruction method and device based on superpixel relation analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112488915B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114419073B (en) * | 2022-03-09 | 2022-08-12 | 荣耀终端有限公司 | Motion blur generation method and device and terminal equipment |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8432434B2 (en) * | 2011-07-08 | 2013-04-30 | Mitsubishi Electric Research Laboratories, Inc. | Camera and method for focus based depth reconstruction of dynamic scenes |
| CN106709948A (en) * | 2016-12-21 | 2017-05-24 | 浙江大学 | Quick binocular stereo matching method based on superpixel segmentation |
| CN108038905B (en) * | 2017-12-25 | 2018-12-07 | 北京航空航天大学 | A kind of Object reconstruction method based on super-pixel |
| CN108416840B (en) * | 2018-03-14 | 2020-02-18 | 大连理工大学 | Three-dimensional scene dense reconstruction method based on monocular camera |
-
2019
- 2019-09-12 CN CN201910864317.2A patent/CN112488915B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112488915A (en) | 2021-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Le Pendu et al. | Light field inpainting propagation via low rank matrix completion | |
| US10547871B2 (en) | Edge-aware spatio-temporal filtering and optical flow estimation in real time | |
| Sun et al. | A quantitative analysis of current practices in optical flow estimation and the principles behind them | |
| US10991150B2 (en) | View generation from a single image using fully convolutional neural networks | |
| US9153032B2 (en) | Conversion method and apparatus with depth map generation | |
| Hu et al. | Single-image real-time rain removal based on depth-guided non-local features | |
| CN107767339B (en) | Binocular stereo image splicing method | |
| GB2520613A (en) | Target region fill utilizing transformations | |
| CN111027438B (en) | Human body posture migration method, mobile terminal and computer storage medium | |
| CN107025660B (en) | Method and device for determining image parallax of binocular dynamic vision sensor | |
| WO2011060579A1 (en) | Method for generating depth maps from monocular images and systems using the same | |
| CN111988593B (en) | Three-dimensional image color correction method and system based on depth residual optimization | |
| Yan et al. | Stereoscopic image stitching based on a hybrid warping model | |
| KR20160058058A (en) | Image processing method and apparatus | |
| CN112233212A (en) | Portrait editing and composition | |
| KR102637901B1 (en) | A method of providing a dolly zoom effect by an electronic device and the electronic device utilized in the method | |
| CN107958446A (en) | Information processing equipment and information processing method | |
| Purohit et al. | Depth-guided dense dynamic filtering network for bokeh effect rendering | |
| CN112488915B (en) | Depth reconstruction method and device based on superpixel relation analysis | |
| CN113949808A (en) | Video generation method and device, readable medium and electronic equipment | |
| US9325962B2 (en) | Method and system for creating dynamic floating window for stereoscopic contents | |
| Ali et al. | Robust regularization for single image dehazing | |
| CN111669563B (en) | Stereo image visual comfort enhancement method based on reinforcement learning | |
| US11533464B2 (en) | Method for synthesizing intermediate view of light field, system for synthesizing intermediate view of light field, and method for compressing light field | |
| Wang et al. | Adaptive video stabilization based on feature point detection and full-reference stability assessment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20210312 Assignee: ZHONGQU (BEIJING) TECHNOLOGY CO.,LTD. Assignor: TSINGHUA University Contract record no.: X2022110000074 Denomination of invention: Depth Reconstruction Method and Device Based on Hyperpixel Relation Analysis Granted publication date: 20220621 License type: Common License Record date: 20221103 |
|
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20210312 Assignee: Beijing Xinqing Tech Co.,Ltd. Assignor: TSINGHUA University Contract record no.: X2022110000077 Denomination of invention: Depth Reconstruction Method and Device Based on Hyperpixel Relation Analysis Granted publication date: 20220621 License type: Common License Record date: 20221108 |