CN112561780A - City scene grid model optimization method with additional multi-sight feature constraint - Google Patents

City scene grid model optimization method with additional multi-sight feature constraint Download PDF

Info

Publication number
CN112561780A
CN112561780A CN202011398537.XA CN202011398537A CN112561780A CN 112561780 A CN112561780 A CN 112561780A CN 202011398537 A CN202011398537 A CN 202011398537A CN 112561780 A CN112561780 A CN 112561780A
Authority
CN
China
Prior art keywords
dimensional
model
vertex
straight line
line segments
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011398537.XA
Other languages
Chinese (zh)
Other versions
CN112561780B (en
Inventor
谢洪
闫利
陈长军
俞思帆
黄健铭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan University WHU
Original Assignee
Wuhan University WHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan University WHU filed Critical Wuhan University WHU
Priority to CN202011398537.XA priority Critical patent/CN112561780B/en
Publication of CN112561780A publication Critical patent/CN112561780A/en
Application granted granted Critical
Publication of CN112561780B publication Critical patent/CN112561780B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G06T5/70—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G06T17/205—Re-meshing
    • G06T3/08—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection

Abstract

The invention provides an urban scene grid model optimization method with additional multi-sight characteristic constraints. The method comprises the steps of extracting straight-line segments of each frame of multi-view image in a multi-view image sequence to obtain a plurality of two-dimensional straight-line segments, calculating a binary descriptor, and matching the two-dimensional straight-line segments between two continuous frames of multi-view images by using the binary descriptor to obtain a plurality of groups of two-dimensional straight-line segments with the same name; selecting the longest two-dimensional straight line segment in each group of two-dimensional straight line segments with the same name to construct a longest two-dimensional straight line segment set, back-projecting all pixels of all the line segments in the set onto the three-dimensional mesh model through camera parameters, and combining all vertex sets of triangular surface patches where all back-projection points are located into a plurality of edge vertex sets of the three-dimensional mesh model; and after the edge vertex set is obtained, constructing an energy function related to the three-dimensional mesh model vertex, and gradually optimizing the three-dimensional mesh model vertex position by a gradient descent method. The invention has the maximum linear information quantity and can restore the accuracy and the aesthetic degree of the edge of the model to the maximum extent.

Description

City scene grid model optimization method with additional multi-sight feature constraint
Technical Field
The invention belongs to the field of remote sensing mapping, and particularly relates to an urban scene grid model optimization method with additional multi-sight feature constraints.
Background
Three-dimensional reconstruction based on a multi-view three-dimensional reconstruction technology is one of important ways for obtaining an object triangular mesh model, and the main processes of the three-dimensional reconstruction include dense matching point cloud generation, point cloud network construction model generation and model texture mapping. However, due to the limitation of the dense matching algorithm, the final generated point cloud inevitably contains a certain number of noise points, especially in the edge region of the reconstructed object. The noise is difficult to remove through a point cloud filtering mode and the like, so that wrong fluctuation exists in the reconstructed three-dimensional model of the urban real scene, and particularly, the phenomenon of distortion or excessive smoothness exists in the edge area of the building. A large number of regular buildings exist in the urban scene, and the three-dimensional model needs to be subjected to straightening constraint, so that the precision and the aesthetic degree of the urban scene three-dimensional model are improved.
The traditional grid model optimization method adjusts the coordinates of the vertexes on the initial grid model based on the maximum projection consistency of the vertexes of the grid on the image, and due to the characteristic of the measure of the image consistency, the optimized model has a smooth phenomenon and cannot restore the straightness of areas such as the edge of a building. On the basis of the traditional optimization method, some works add straight line constraint, wherein most methods generate three-dimensional line segments, and the three-dimensional line segments are used for constraining the movement of vertexes on the edge of the mesh model so as to enable the vertexes to be regularly arranged into a straight line. The three-dimensional line segments are derived from two-dimensional multi-sight features, and lack a certain amount of information compared with the original two-dimensional straight line features.
The traditional mesh model optimization method takes the image consistency measure between the projection points of the three-dimensional model vertexes on the multi-view image as an energy function, and optimizes the vertexes of the initial three-dimensional model by minimizing the energy function. Due to the limitations of the image gray level constraint term and the regularization term, particularly fine and sharp features cannot be recovered in the model refinement process. The prior art of adding line constraint on the basis of the traditional method is mostly based on three-dimensional line feature constraint, the three-dimensional line feature is derived from two-dimensional line features, and a part of information content is lost. Therefore, the existing model optimization method has the following disadvantages:
the traditional optimization method does not consider the sharp characteristics of model edges and does not meet the optimization requirements of urban scene models with a large number of straight building edges;
some prior arts introduce line feature constraints, but have the problems of missing line feature information amount and large calculation amount.
Therefore, the invention discloses an urban scene grid model optimization method with additional multi-sight feature constraints, which introduces multi-sight two-dimensional line feature construction straight line constraints with the largest information amount on the basis of optimizing the overall grid model vertex position based on image consistency, and restores the straight and sharp characteristics of the building edge area while enriching the model details on the grid edge area vertex.
Disclosure of Invention
The method optimizes the vertex coordinates of the mesh model, so that each vertex has the maximum consistency on the multi-view image. Meanwhile, multi-sight characteristic constraint is added on the vertexes of the edge area, so that the vertexes are arranged into a straight line. To realize the above functions, the technical problems to be solved by the present invention are: the edge of the three-dimensional mesh model is susceptible to noise and has the phenomenon of over-smooth distortion, which is caused by the limitation of a dense matching algorithm, but the optimization method based on the image gray level consistency cannot solve the problem.
The invention provides an urban scene grid model optimization method with additional multi-sight characteristic constraints, which comprises the following steps:
step 1: extracting a plurality of two-dimensional straight line segments from each frame of multi-view image in the multi-view image sequence through a straight line segment extraction algorithm; constructing a binary descriptor of each two-dimensional line segment, and matching the two-dimensional line segments between two continuous frames of multi-view images by using the binary descriptors of the two-dimensional line segments to obtain a plurality of groups of two-dimensional line segments with the same name;
step 2: selecting two-dimensional straight line segments with the longest length in each group of two-dimensional straight line segments with the same name to construct a longest two-dimensional straight line segment set, back-projecting each pixel on each line segment in the longest two-dimensional straight line segment set onto the three-dimensional grid model through respective camera pose parameters, wherein all vertex sets of triangular surface pieces where all back-projection points are located on the three-dimensional grid model are a plurality of edge vertex sets of the three-dimensional grid model;
and step 3: after the edge vertex set is obtained in the step 2, an energy function related to the three-dimensional mesh model vertex is constructed, the energy is gradually reduced through a gradient descent method, and the position of the three-dimensional mesh model vertex is gradually optimized until the energy is minimum, and the three-dimensional mesh model vertex is optimized to the optimal position;
preferably, the multi-view image sequence in step 1 is:
I＝{I1,I2,...,IK}
wherein K is the number of images in the multi-view image sequence, IkFor the kth frame of multiview images in a multiview image sequence, Ik(x, y) is the pixel of the x-th row and y-th column of the K-th frame multiview image in the multiview image sequence, K is the [1, K ]]， x∈[1,R]，y∈[1,C]R is the number of rows of the multi-view image, and C is the number of columns of the multi-view image;
the two-dimensional straight-line segments extracted from each frame of image in the step 1 are as follows:
Figure BDA0002811643570000031
k∈[1,K]
linek,j＝{Ik(x1,y1),Ik(x2,y2),...,Ik(xN,yN)}
wherein, linekA plurality of two-dimensional straight line segments, lines, extracted from the k-th frame of the multi-view imagek,jIs the jth two-dimensional straight-line segment corresponding to the kth frame multi-view image, j belongs to [1, L ]k],LkLine is the number of two-dimensional line segments extracted from the k-th multi-view imagek,jIs composed of multiple pixels I on the k-th framek(xn,yn) Composition of wherein xn∈[1,R]And yn∈[1,C]Respectively representing the row number and the column number of the pixel, and N is a two-dimensional straight line segmentk,jThe total number of the pixels, and the coordinates of the N pixels on the k-th frame multi-view image are connected;
step 1, the binary descriptor of each two-dimensional straight-line segment is:
Figure BDA0002811643570000032
k∈[1,K]
wherein, LBDk,jA binary descriptor of a jth two-dimensional straight-line segment corresponding to the kth frame of multi-view image;
step 1, matching two-dimensional line segments between two continuous frames of multi-view images by using a binary descriptor of the two-dimensional line segments is as follows:
finding the corresponding relation of the straight line descriptors on the continuous two frames of multi-view images:
{LBDk,LBDk+1},k∈[1,K-1]，LBDkLBD in (1)k,m、LBDk+1Medium LBDk+1,nIf the Euclidean distance between two descriptor vectors is closest, the matching degree is highest, namely two lines are successfully matched, and then the line is matchedk+1,nAddition to linek,mThe same name straight line group LsIs marked as
Figure BDA0002811643570000033
The matching strategy can obtain a plurality of groups of two-dimensional straight line segments with the same name as:
Figure BDA0002811643570000034
Figure BDA0002811643570000035
wherein S represents the same-name straight line group number, S represents the number of all same-name two-dimensional straight line group, and LsRepresents the s-th group of homonymous straight lines,
Figure BDA0002811643570000036
representing two-dimensional straight line segments on a multi-view image of a k-th framek,jWherein j is [1, L ]k]Within the interval;
preferably, the longest two-dimensional straight line segment set constructed in step 2 is:
Figure BDA0002811643570000037
wherein the content of the first and second substances,
Figure BDA0002811643570000038
represents the longest straight line segment in the s-th group of homonymous straight lines from the k-thsAnd (5) frame image.
Step 2, the three-dimensional grid model is as follows:
M＝{V1,V2,..,VP,F1,F2,..,FQ}
wherein, VpOne vertex, F, representing a three-dimensional mesh modelqA triangular patch representing a three-dimensional mesh model, P ∈ [1, P [ ]]，q∈[1,Q]P represents the number of vertexes of the three-dimensional mesh model, and Q represents the number of triangular patches of the three-dimensional mesh model. The data stored in the three-dimensional mesh model is the position coordinates of each vertex and the index relation between the patches and the vertices, namely which three vertices each triangle patch is composed of.
2, positioning a plurality of three-dimensional model edge vertex sets on the three-dimensional grid model by the longest two-dimensional straight line segment set as follows:
Figure BDA0002811643570000041
Vs Edenotes the group sSet of edge vertices corresponding to the first-name-line segment, S ∈ [1, S ∈]The superscript E is marked as the edge vertex;
each group of homonymous straight lines L is selectedsTwo-dimensional straight line segment with longest middle length
Figure BDA0002811643570000042
Set all pixels on the line Ik(xn,yn)},n∈[1,N]Back-projection to initial model M ═ { V ═ V1,V2,..,VP,F1,F2,..,FQOn F, the set of all triangular patches intersected with the projection point is Fs ESet of vertices V corresponding to set of patchess EDefined as a group of homonymous lines LsA corresponding edge region;
preferably, the energy function in step 3 is:
E(V)＝Ephoto(V)+Eline(VE)+Esmooth(V)
wherein V represents the set of all the vertexes V ═ V of the three-dimensional mesh model1,V2,..,VP}，VERepresenting the set of the three-dimensional mesh model edge vertexes positioned in the step 2,
Figure BDA0002811643570000043
Ephotoand ElineComposing data items in energy functions, EsmoothIs a smoothing term. EphotoRepresenting the image gray consistency constraint energy, and having the function of constraining the maximum gray consistency of the grid vertex on the multi-view image; elineConstraining energy for the multi-sight characteristic, wherein the function is to constrain the edge vertexes of the three-dimensional grid model to be regularly arranged into a straight line; esmoothThe energy is constrained for the smoothing term, which acts to smooth the model noise through the topological relationship of the model itself. The overall energy function has the effect of
Step 3, image gray level consistency constraint energy EphotoComprises the following steps:
Figure BDA0002811643570000044
in the formula: image IijIs the ith frame multiview image IiThe multi-view image I of the jth frame is re-projected to the surface of the three-dimensional grid modeljObtaining the product; h (I)i,Iij)(vi) Representing an image IiAnd IijIn between the pixel vi1-ZNCC index of (a) wherein viFor the vertex V in the ith frameiZNCC represents a normalized cross correlation coefficient; the pair is a set of related image pairs, and is a set constructed by selecting all related images j satisfying three conditions of overlapping degree, scale difference and angle difference with a reference image for each frame of multi-view image i.
Step 3, the multiple sight features are restricted ElineComprises the following steps:
Figure BDA0002811643570000051
in the formula:
Figure BDA0002811643570000052
is the edge vertex V of the three-dimensional mesh models EIn the ith frame of multi-view image IiThe pixel of the projected point on the image,
Figure BDA0002811643570000053
representing the s-th group of homonymous straight lines LsTwo-dimensional straight-line segments on the i-th frame multiview,
Figure BDA0002811643570000054
representing two-dimensional pixels
Figure BDA0002811643570000055
To a straight line
Figure BDA0002811643570000056
The sum of the distances of (a).
Step 3 the smoothing term EsmoothComprises the following steps:
Figure BDA0002811643570000057
in the formula k1And k2Representing the principal curvature of the model surface at the current point V. For discrete mesh data, the gradient of the energy can be approximated by the laplacian of the vertices of the three-dimensional mesh model.
After the energy function is constructed and the gradients of all energy items are obtained, the E (V) can be gradually minimized by a gradient descent method, and the optimization of the grid vertex is realized.
The method is improved on the basis of the traditional optimization method, the vertex at the edge of the grid model is positioned through linear pixel back projection, the multi-view two-dimensional line characteristic with the largest information amount is introduced as linear constraint, and the movement of the vertex at the edge is driven, so that the straightness at the edge of the grid model is restored while the global details of the model are enriched.
The method considers the problem that the edges of the grid model are distorted and smooth due to the influence of noise, and introduces linear characteristic constraint;
compared with the three-dimensional line feature, the two-dimensional multi-sight feature has the largest straight line information amount, and the accuracy and the attractiveness of the edge of the model can be restored to the greatest extent.
Drawings
FIG. 1: the method of the invention is a flow chart.
FIG. 2: and matching straight lines with the same name.
FIG. 3: the edge vertices are located.
FIG. 4: multiple line-of-sight feature constraints.
FIG. 5: grid laplacian.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is further described in detail below with reference to specific examples. It should be understood that the specific examples described herein are intended to be illustrative only and are not intended to be limiting.
The following describes a specific embodiment of the present invention with reference to fig. 1 to 5 as an optimization method of an urban scene grid model with additional multi-view feature constraints, which includes the following steps:
step 1: extracting a plurality of two-dimensional straight line segments from each frame of multi-view image in the multi-view image sequence through a straight line segment extraction algorithm; constructing a binary descriptor of each two-dimensional line segment, and matching the two-dimensional line segments between two continuous frames of multi-view images by using the binary descriptors of the two-dimensional line segments to obtain a plurality of groups of two-dimensional line segments with the same name;
step 1, the multi-view image sequence is as follows:
I＝{I1,I2,...,IK}
where K is 100, I is the number of images in the multiview image sequencekFor the kth frame of multiview pictures, I, of the multiview picture sequencek(x, y) is the pixel of the x-th row and y-th column of the K-th frame multiview image in the multiview image sequence, K is the [1, K ]]， x∈[1,R]，y∈[1,C]R1200 is the number of rows of the multiview image, and C800 is the number of columns of the multiview image;
the two-dimensional straight-line segments extracted from each frame of image in the step 1 are as follows:
Figure BDA0002811643570000061
k∈[1,K]
linek,j＝{Ik(x1,y1),Ik(x2,y2),...,Ik(xN,yN)}
wherein, linekA plurality of two-dimensional straight line segments, lines, extracted from the k-th frame of the multi-view imagek,jIs the jth two-dimensional straight-line segment corresponding to the kth frame multi-view image, j belongs to [1, L ]k],LkLine is the number of two-dimensional line segments extracted from the k-th multi-view imagek,jIs composed of multiple pixels I on the k-th framek(xn,yn) Composition of wherein xn∈[1,R]And yn∈[1,C]Respectively representing the row number and the column number of the pixel, and N is a two-dimensional straight line segmentk,jTotal number of pixels onAnd the coordinates of the N pixels on the k-th frame of multi-view image are connected;
step 1, the binary descriptor of each two-dimensional straight-line segment is:
Figure BDA0002811643570000071
k∈[1,K]
wherein, LBDk,jA binary descriptor of a jth two-dimensional straight-line segment corresponding to the kth frame of multi-view image;
step 1, matching two-dimensional line segments between two continuous frames of multi-view images by using a binary descriptor of the two-dimensional line segments is as follows:
finding the corresponding relation of the straight line descriptors on the continuous two frames of multi-view images:
{LBDk,LBDk+1},k∈[1,K-1]，LBDkLBD in (1)k,m、LBDk+1Medium LBDk+1,nIf the Euclidean distance between two descriptor vectors is closest, the matching degree is highest, namely two lines are successfully matched, and then the line is matchedk+1,nAddition to linek,mThe same name straight line group LsIs marked as
Figure BDA0002811643570000072
The matching strategy can obtain a plurality of groups of two-dimensional straight line segments with the same name as:
Figure BDA0002811643570000073
Figure BDA0002811643570000074
wherein S represents the same-name straight line group number, S represents the number of all same-name two-dimensional straight line group, and LsRepresents the s-th group of homonymous straight lines,
Figure BDA0002811643570000075
representing two-dimensional straight line segments on a multi-view image of a k-th framek,jWherein j is [1, L ]k]In the interval, the matching process of two groups of homonymous straight lines is illustrated in fig. 2.
Step 2: selecting two-dimensional straight line segments with the longest length in each group of two-dimensional straight line segments with the same name to construct a longest two-dimensional straight line segment set, back-projecting each pixel on each line segment in the longest two-dimensional straight line segment set onto the three-dimensional mesh model through respective camera pose parameters, wherein all vertex sets of triangular surface sheets where all back-projection points are located on the three-dimensional mesh model are a plurality of edge vertex sets of the three-dimensional mesh model, and as shown in fig. 3, two edge vertex sets of the three-dimensional mesh model are positioned by two longest two-dimensional straight line segments;
the longest two-dimensional line segment set constructed in the step 2 is as follows:
Figure BDA0002811643570000076
wherein the content of the first and second substances,
Figure BDA0002811643570000077
represents the longest straight line segment in the s-th group of homonymous straight lines from the k-thsAnd (5) frame image.
Step 2, the three-dimensional grid model is as follows:
M＝{V1,V2,..,VP,F1,F2,..,FQ}
wherein, VpOne vertex, F, representing a three-dimensional mesh modelqA triangular patch representing a three-dimensional mesh model, P ∈ [1, P [ ]]，q∈[1,Q]P10000 denotes the number of vertices of the three-dimensional mesh model, and Q10000 denotes the number of triangle patches of the three-dimensional mesh model. The data stored in the three-dimensional mesh model is the position coordinate of each vertex and the index relation between the patches and the vertices, namely which three vertices each triangle patch is composed of.
2, positioning a plurality of three-dimensional model edge vertex sets on the three-dimensional grid model by the longest two-dimensional straight line segment set as follows:
Figure BDA0002811643570000081
Vs Erepresenting the edge vertex set corresponding to the S-th group of straight-line segments with the same name, S is the [1, S ∈]The superscript E is marked as the edge vertex;
each group of homonymous straight lines L is selectedsTwo-dimensional straight line segment with longest middle length
Figure BDA0002811643570000082
Set all pixels on the line Ik(xn,yn)},n∈[1,N]Back-projection to initial model M ═ { V ═ V1,V2,..,VP,F1,F2,..,FQOn F, the set of all triangular patches intersected with the projection point is Fs ESet of vertices V corresponding to set of patchess EDefined as a group of homonymous lines LsA corresponding edge vertex;
and step 3: after the edge vertex set is obtained in the step 2, an energy function related to the three-dimensional mesh model vertices is constructed, the energy is gradually reduced through a gradient descent method, and the positions of the three-dimensional mesh model vertices are gradually optimized until the energy is minimum, and the three-dimensional mesh model vertices are all optimized to the optimal positions:
the energy function in step 3 is:
E(V)＝Ephoto(V)+Eline(VE)+Esmooth(V)
wherein V represents the set of all the vertexes V ═ V of the three-dimensional mesh model1,V2,..,VP}，VERepresenting the set of the three-dimensional mesh model edge vertexes positioned in the step 2,
Figure BDA0002811643570000083
Ephotoand ElineComposing data items in energy functions, EsmoothIs a smoothing term. EphotoRepresenting the gray consistency constraint energy of the image, which is used for constraining the gray consistency of the grid vertex on the multi-view image to be the mostLarge; elineConstraining energy for the multi-sight characteristic, wherein the function is to constrain the edge vertexes of the three-dimensional grid model to be regularly arranged into a straight line; esmoothThe energy is constrained for the smoothing term, which acts to smooth the model noise through the topological relationship of the model itself. The overall energy function has the effect of
Step 3, image gray level consistency constraint energy EphotoComprises the following steps:
Figure BDA0002811643570000084
in the formula: image IijIs the ith frame multiview image IiThe multi-view image I of the jth frame is re-projected to the surface of the three-dimensional grid modeljObtaining the product; h (I)i,Iij)(vi) Representing an image IiAnd IijIn between the pixel vi1-ZNCC index of (a) wherein viFor the vertex V in the ith frameiZNCC represents a normalized cross correlation coefficient; the pair is a set of related image pairs, and is a set constructed by selecting all related images j satisfying three conditions of overlapping degree, scale difference and angle difference with a reference image for each frame of multi-view image i.
Step 3, the multiple sight features are restricted ElineComprises the following steps:
Figure BDA0002811643570000091
in the formula:
Figure BDA0002811643570000092
is the edge vertex V of the three-dimensional mesh models EIn the ith frame of multi-view image IiThe pixel of the projected point on the image,
Figure BDA0002811643570000093
representing the s-th group of homonymous straight lines LsTwo-dimensional straight-line segments on the i-th frame multiview,
Figure BDA0002811643570000094
representing two-dimensional pixels
Figure BDA0002811643570000095
To a straight line
Figure BDA0002811643570000096
The sum of the distances of (a). As shown in fig. 4, projection points on the corresponding multi-view sequence images with edge vertices are distributed around the corresponding two-dimensional straight line segments in a scattered manner, and the multi-view feature constraint energy term is to reduce deviation values of the projection distances from the two-dimensional straight line segments, so that the edge vertices are arranged in order.
Step 3 the smoothing term EsmoothComprises the following steps:
Figure BDA0002811643570000097
in the formula k1And k2Representing the principal curvature of the model surface at the current point V. For discrete grid data, the above energy gradient can be approximated with the laplacian operator. As shown in fig. 5, the laplacian segment of the vertex of the three-dimensional mesh model is the directed distance from the vertex to the average value of the vertex of the neighborhood of the point ring, and the convex noise vertex can be effectively flattened.
After the energy function is constructed and the gradients of all energy items are obtained, the E (V) can be gradually minimized by a gradient descent method, and the optimization of the grid vertex is realized.
It should be understood that parts of the specification not set forth in detail are well within the prior art.
It should be understood that the above description of the preferred embodiments is specific and not intended to limit the scope of the invention, and that similar data compression and reconstruction may be made by those skilled in the art without departing from the scope of the invention as defined in the appended claims.

Claims (4)

1. A city scene grid model optimization method with additional multi-sight feature constraints is characterized by comprising the following steps:
step 1: extracting a plurality of two-dimensional straight line segments from each frame of multi-view image in the multi-view image sequence through a straight line segment extraction algorithm; constructing a binary descriptor of each two-dimensional line segment, and matching the two-dimensional line segments between two continuous frames of multi-view images by using the binary descriptors of the two-dimensional line segments to obtain a plurality of groups of two-dimensional line segments with the same name;
step 2: selecting two-dimensional straight line segments with the longest length in each group of two-dimensional straight line segments with the same name to construct a longest two-dimensional straight line segment set, back-projecting each pixel on each line segment in the longest two-dimensional straight line segment set onto the three-dimensional mesh model through respective camera pose parameters, wherein all vertex sets of triangular patches where all back-projection points are located on the three-dimensional mesh model are a plurality of edge vertex sets of the three-dimensional mesh model;
and step 3: and (3) after the edge vertex set is obtained in the step (2), constructing an energy function related to the vertices of the three-dimensional mesh model, gradually reducing the energy through a gradient descent method, and gradually optimizing the positions of the vertices of the three-dimensional mesh model until the energy is minimum, wherein the vertices of the three-dimensional mesh model are optimized to the optimal position.
2. The urban scene grid model optimization method with additional multi-view feature constraints according to claim 1, wherein:
step 1, the multi-view image sequence is as follows:
I＝{I1,I2,...,IK}
wherein K is the number of images in the multi-view image sequence, IkFor the kth frame of multiview pictures, I, of the multiview picture sequencek(x, y) is the pixel of the x-th row and y-th column of the K-th frame multiview image in the multiview image sequence, K is the [1, K ]]，x∈[1,R]，y∈[1,C]R is the number of rows of the multi-view image, and C is the number of columns of the multi-view image;
the two-dimensional straight-line segments extracted from each frame of image in the step 1 are as follows:
Figure FDA0002811643560000011
linek,j＝{Ik(x1,y1),Ik(x2,y2),...,Ik(xN,yN)}
wherein, linekA plurality of two-dimensional straight line segments, lines, extracted from the k-th frame of the multi-view imagek,jIs the jth two-dimensional straight-line segment corresponding to the kth frame multi-view image, j belongs to [1, L ]k],LkLine is the number of two-dimensional line segments extracted from the k-th multi-view imagek,jIs composed of multiple pixels I on the k-th framek(xn,yn) Composition of (a) wherein xn∈[1,R]And yn∈[1,C]Respectively representing the row number and the column number of the pixel, and N is a two-dimensional straight line segmentk,jThe total number of the pixels, and the coordinates of the N pixels on the k-th frame multi-view image are connected;
step 1, the binary descriptor of each two-dimensional straight-line segment is:
Figure FDA0002811643560000021
wherein, LBDk,jA binary descriptor of a jth two-dimensional straight-line segment corresponding to the kth frame of multi-view image;
step 1, matching the two-dimensional line segments between two continuous frames of multi-view images by using the binary descriptors of the two-dimensional line segments is as follows:
finding the corresponding relation of the straight line descriptors on the continuous two frames of multi-view images:
{LBDk,LBDk+1},k∈[1,K-1]，LBDkLBD in (1)k,m、LBDk+1Medium LBDk+1,nIf the Euclidean distance between two descriptor vectors is closest, the matching degree is highest, namely two lines are successfully matched, and then the line is matchedk+1,nAddition to linek,mThe same name of the same straightLine group LsIs marked as
Figure FDA0002811643560000022
The matching strategy can obtain a plurality of groups of two-dimensional straight line segments with the same name as:
Figure FDA0002811643560000023
Figure FDA0002811643560000024
wherein S represents the same-name straight line group number, S represents the number of all same-name two-dimensional straight line group, and LsRepresents the s-th group of homonymous straight lines,
Figure FDA0002811643560000025
representing two-dimensional straight line segments on a multi-view image of a k-th framek,jWherein j is [1, L ]k]Within the interval.
3. The urban scene grid model optimization method with additional multi-view feature constraints according to claim 1, wherein:
the longest two-dimensional line segment set constructed in the step 2 is as follows:
Figure FDA0002811643560000026
wherein the content of the first and second substances,
Figure FDA0002811643560000027
represents the longest straight line segment in the s-th group of homonymous straight lines from the k-thsFrame images;
step 2, the three-dimensional grid model is as follows:
M＝{V1,V2,..,VP,F1,F2,..,FQ}
wherein, VpOne vertex, F, representing a three-dimensional mesh modelqA triangular patch representing a three-dimensional mesh model, P ∈ [1, P [ ]]，q∈[1,Q]P represents the number of vertexes of the three-dimensional mesh model, and Q represents the number of triangular patches of the three-dimensional mesh model; the data stored in the three-dimensional mesh model is the position coordinate of each vertex and the index relation between the patches and the vertices, namely each triangular patch consists of three corresponding vertices;
step 2, the set of the edge vertexes of the plurality of three-dimensional models on the three-dimensional grid model is positioned by the longest two-dimensional straight line segment set, and the set is as follows:
Figure FDA0002811643560000031
Figure FDA0002811643560000032
representing the edge vertex set corresponding to the S-th group of straight-line segments with the same name, S is the [1, S ∈]The superscript E is marked as the edge vertex;
each group of homonymous straight lines L is selectedsTwo-dimensional straight line segment with longest middle length
Figure FDA0002811643560000033
Set all pixels on the line Ik(xn,yn)},n∈[1,N]Back-projection to initial model M ═ { V ═ V1,V2,..,VP,F1,F2,..,FQOn the surface, the set of all triangular patches intersected with the projection point is
Figure FDA0002811643560000034
Vertex set corresponding to patch set
Figure FDA0002811643560000035
Defined as a group of homonymous lines LsCorresponding edge area.
4. The urban scene grid model optimization method with additional multi-view feature constraints according to claim 1, wherein:
the energy function in step 3 is:
E(V)＝Ephoto(V)+Eline(VE)+Esmooth(V)
wherein V represents the set of all the vertexes V ═ V of the three-dimensional mesh model1,V2,..,VP}，VERepresenting the set of the three-dimensional mesh model edge vertexes positioned in the step 2,
Figure FDA0002811643560000036
Ephotoand ElineData items in the composition energy function, EsmoothIs a smoothing term; ephotoRepresenting the image gray consistency constraint energy, and having the function of constraining the maximum gray consistency of the grid vertex on the multi-view image; elineConstraining energy for the multi-sight characteristic, wherein the function is to constrain the edge vertexes of the three-dimensional grid model to be regularly arranged into a straight line; esmoothEnergy is constrained for the smoothing term, and the function is to smooth model noise through the self topological relation of the model;
the overall energy function has the effect of:
step 3, image gray level consistency constraint energy EphotoComprises the following steps:
Figure FDA0002811643560000037
in the formula: image IijIs the ith frame multiview image IiThe multi-view image I of the jth frame is re-projected to the surface of the three-dimensional grid modeljObtaining the product; h (I)i,Iij)(vi) Representing an image IiAnd IijIn between the pixel vi1-ZNCC index of (a) wherein viFor the vertex V in the ith frameiZNCC represents a normalized cross-correlation coefficient; the pair is a set of related image pairs, which is selected to satisfy the overlapping degree with the reference image,Constructing an obtained set of all related images j of the three conditions of the scale difference and the angle difference;
step 3, the multiple sight features are restricted ElineComprises the following steps:
Figure FDA0002811643560000041
in the formula:
Figure FDA0002811643560000042
is the edge vertex of the three-dimensional mesh model
Figure FDA0002811643560000043
In the ith frame of multi-view image IiThe pixel of the projected point on the image,
Figure FDA0002811643560000044
representing the s-th group of homonymous straight lines LsTwo-dimensional straight-line segments on the i-th frame multiview,
Figure FDA0002811643560000045
representing two-dimensional pixels
Figure FDA0002811643560000046
To a straight line
Figure FDA0002811643560000047
The sum of the distances of (a);
step 3 the smoothing term EsmoothComprises the following steps:
Figure FDA0002811643560000048
in the formula k1And k2Representing the principal curvature of the model surface at the current point V; for discrete grid data, the gradient of the energy can be approximately represented by a Laplacian operator of the vertex of the three-dimensional grid model;
after the energy function is constructed and the gradients of all energy items are obtained, the E (V) can be gradually minimized by a gradient descent method, and the optimization of the grid vertex is realized.
CN202011398537.XA 2020-12-02 2020-12-02 City scene grid model optimization method with additional multi-sight feature constraint Active CN112561780B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011398537.XA CN112561780B (en) 2020-12-02 2020-12-02 City scene grid model optimization method with additional multi-sight feature constraint

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011398537.XA CN112561780B (en) 2020-12-02 2020-12-02 City scene grid model optimization method with additional multi-sight feature constraint

Publications (2)

Publication Number Publication Date
CN112561780A true CN112561780A (en) 2021-03-26
CN112561780B CN112561780B (en) 2022-04-15

Family

ID=75047714

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011398537.XA Active CN112561780B (en) 2020-12-02 2020-12-02 City scene grid model optimization method with additional multi-sight feature constraint

Country Status (1)

Country Link
CN (1) CN112561780B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130127847A1 (en) * 2010-08-25 2013-05-23 Hailin Jin System and Method for Interactive Image-based Modeling of Curved Surfaces Using Single-view and Multi-view Feature Curves
CN105205808A (en) * 2015-08-20 2015-12-30 武汉大学 Multi-vision image dense coupling fusion method and system based on multiple characteristics and multiple constraints
US20160086338A1 (en) * 2013-05-22 2016-03-24 National University Corporation Kobe University Line-of-sight measurement device, line-of-sight measurement method and line-of-sight measurement program
US20170085733A1 (en) * 2014-05-12 2017-03-23 Dacuda Ag Method and apparatus for scanning and printing a 3d object

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130127847A1 (en) * 2010-08-25 2013-05-23 Hailin Jin System and Method for Interactive Image-based Modeling of Curved Surfaces Using Single-view and Multi-view Feature Curves
US20160086338A1 (en) * 2013-05-22 2016-03-24 National University Corporation Kobe University Line-of-sight measurement device, line-of-sight measurement method and line-of-sight measurement program
US20170085733A1 (en) * 2014-05-12 2017-03-23 Dacuda Ag Method and apparatus for scanning and printing a 3d object
CN105205808A (en) * 2015-08-20 2015-12-30 武汉大学 Multi-vision image dense coupling fusion method and system based on multiple characteristics and multiple constraints

Also Published As

Publication number Publication date
CN112561780B (en) 2022-04-15

Similar Documents

Publication Publication Date Title
CN109872397B (en) Three-dimensional reconstruction method of airplane parts based on multi-view stereo vision
CN110458957B (en) Image three-dimensional model construction method and device based on neural network
Li et al. Feature-preserving 3D mesh simplification for urban buildings
Xiao et al. Image-based façade modeling
CN101916454B (en) Method for reconstructing high-resolution human face based on grid deformation and continuous optimization
US20060244757A1 (en) Methods and systems for image modification
Vicente et al. Balloon shapes: Reconstructing and deforming objects with volume from images
CN101303772A (en) Method for modeling non-linear three-dimensional human face based on single sheet image
CN109147025B (en) RGBD three-dimensional reconstruction-oriented texture generation method
CN109410133B (en) Face texture repairing method based on 3DMM
CN109671039B (en) Image vectorization method based on layering characteristics
CN112862736B (en) Real-time three-dimensional reconstruction and optimization method based on points
CN113160420A (en) Three-dimensional point cloud reconstruction method and device, electronic equipment and storage medium
CN112784621A (en) Image display method and apparatus
CN106127743B (en) The method and system of automatic Reconstruction bidimensional image and threedimensional model accurate relative location
CN115393548A (en) Grid texture simplifying algorithm suitable for three-dimensional reconstruction
CN113781621A (en) Three-dimensional reconstruction processing method, device, equipment and storage medium
CN116977596A (en) Three-dimensional modeling system and method based on multi-view images
CN116363290A (en) Texture map generation method for large-scale scene three-dimensional reconstruction
CN112561780B (en) City scene grid model optimization method with additional multi-sight feature constraint
Sangeetha et al. A novel exemplar based Image Inpainting algorithm for natural scene image completion with improved patch prioritizing
CN110147809A (en) Image processing method and device, storage medium and vision facilities
Cushen et al. Markerless real-time garment retexturing from monocular 3d reconstruction
CN115482268A (en) High-precision three-dimensional shape measurement method and system based on speckle matching network
CN115082640A (en) Single image-based 3D face model texture reconstruction method and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant