CN115063485B - Three-dimensional reconstruction method, device and computer-readable storage medium - Google Patents
Three-dimensional reconstruction method, device and computer-readable storage medium Download PDFInfo
- Publication number
- CN115063485B CN115063485B CN202210999929.4A CN202210999929A CN115063485B CN 115063485 B CN115063485 B CN 115063485B CN 202210999929 A CN202210999929 A CN 202210999929A CN 115063485 B CN115063485 B CN 115063485B
- Authority
- CN
- China
- Prior art keywords
- point
- line
- projection error
- dimensional
- feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000004364 calculation method Methods 0.000 claims description 38
- 238000004891 communication Methods 0.000 claims description 13
- 238000005457 optimization Methods 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 6
- 230000004927 fusion Effects 0.000 claims description 4
- 230000003044 adaptive effect Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
- Apparatus For Radiation Diagnosis (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the application relates to the technical field of three-dimensional reconstruction, and discloses a three-dimensional reconstruction method, a three-dimensional reconstruction device and a computer readable storage medium, wherein the three-dimensional reconstruction method comprises the following steps: extracting point features, line features and face features of each of a plurality of images, the plurality of images representing multiple views of a target object; calculating a point projection error of the point feature, a line projection error of the line feature, and a plane projection error of the plane feature; generating a three-dimensional model of the target object from the point projection errors, the line projection errors, and the surface projection errors. And redundant points, lines or planes are combined through the constraint of the points, the lines and the planes, so that the data of the three-dimensional model is more concise and accurate, the operation efficiency is improved, and the fineness of the three-dimensional model is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of three-dimensional reconstruction, in particular to a three-dimensional reconstruction method, a three-dimensional reconstruction device and a computer-readable storage medium.
Background
At present, with the progress of science and technology, a plurality of problems are solved in more and more fields through three-dimensional reconstruction, the three-dimensional reconstruction includes traditional modeling and live-action three-dimensional reconstruction, the traditional modeling is that modeling personnel model a target object or a scene according to the knowledge of the target object or the scene and combining drawings or self experience, the model fineness obtained in the mode is difficult to guarantee and is often difficult to apply under the condition of needing fine contrast with the real object or the scene, the live-action three-dimensional reconstruction reversely finds out a three-dimensional model according to a single image or an image sequence, the model fineness is higher, the generation speed is high, and some defects of the traditional modeling are overcome.
The existing live-action three-dimensional reconstruction is usually established according to one characteristic of an image, the error is large, the model fineness is low, and the efficiency and the effect are poor when the three-dimensional model needs to be subjected to texture processing and the like in the follow-up process.
Disclosure of Invention
In view of the foregoing problems, embodiments of the present invention provide a three-dimensional reconstruction method, which is used to solve the problems in the prior art.
According to an aspect of an embodiment of the present invention, there is provided a three-dimensional reconstruction method, including: extracting point features, line features and face features of each of a plurality of images, the plurality of images representing multiple views of a target object; calculating a point projection error of the point feature, a line projection error of the line feature, and a plane projection error of the plane feature; and generating a three-dimensional model of the target object according to the point projection error, the line projection error and the surface projection error.
The point projection error, the line projection error and the surface projection error are calculated by combining the point feature, the line feature and the surface feature of the target object, and the three-dimensional model of the target object is obtained by combining the point feature, the line feature and the surface feature, and redundant points, lines or surfaces are combined by combining the two constraints, so that the data of the three-dimensional model is simpler and more accurate, the operation efficiency is improved, the three-dimensional model is more precise and contains more image data, more expansion modes are provided in the subsequent application of the three-dimensional model, the precision of the three-dimensional model is improved, and the subsequent processing efficiency is optimized.
In an optional manner, the calculating a point projection error of the point feature, a line projection error of the line feature, and a plane projection error of the plane feature further includes:
let the point projection error beThe calculation formula is as follows:wherein j represents the number of the point features,in order to be a feature of the point,for the point features on the imageThe corresponding three-dimensional point is displayed on the screen,is an internal parameter of the image and is,andas an external parameter of the image(ii) a Let the line projection error beThe calculation formula is as follows:wherein k represents the number of said line features,andrespectively being characteristic of said lineThe start point and the end point of (c),is a characteristic of the lineCorresponding three-dimensional line segmentProjection on the image, the calculation formula of which is:whereinto representThe co-factor matrix of (a) is,the internal parameters of the image are expressed in the form of:wherein, in the process,is the number of the image in question,andis the focal length of a pixel of the image,andthe pixel point coordinates of the image are obtained; let the surface projection error beThe calculation formula is as follows:
wherein j represents the number of the point features, k represents the number of the line features, m represents the number of the face features,for the purpose of the point feature,is composed ofThe corresponding three-dimensional point is displayed on the screen,in order to be a feature of the face,in order to be a feature of the line,is composed ofA corresponding three-dimensional line segment is formed,is composed ofThe normal vector of (a) is,andare respectively two-dimensional planesCorresponding three-dimensional planeAnd the distance of the origin of coordinates to the straight line.
The point projection error, the line projection error and the surface projection error of the image are calculated according to the known parameters and formulas, so that the point feature, the line feature and the surface feature on the image can obtain an accurate three-dimensional point, an accurate three-dimensional line segment and an accurate three-dimensional plane according to the point projection error, the line projection error and the surface projection error.
In an alternative manner, the generating a three-dimensional model from the point projection error, the line projection error, and the plane projection error further includes:
obtaining a camera pose and a three-dimensional point cloud according to the point projection error, the line projection error and the surface projection error; generating a patch according to the camera pose and the three-dimensional point cloud; and generating the three-dimensional model according to the patch.
The surface patch is generated by combining the point characteristic, the line characteristic, the surface characteristic, the point projection error, the line projection error and the surface projection error, so that the surface patch is restrained by the point, the line and the surface, the edge of the surface patch is clearer, and the three-dimensional model generated according to the surface patch is more precise and accurate.
In an optional manner, the calculating a point projection error of the point feature, a line projection error of the line feature, and a plane projection error of the plane feature further includes:
calculating a total projection error from the point projection error, the line projection error, and the surface projection errorWhereinis the weight of the error of the point projection,is the weight of the line projection error,is the weight of the surface projection error,the error of the projection of the point is represented,which is representative of the line projection error,representing the surface projection error(ii) a Generating a three-dimensional point cloud and/or a patch according to the total projection error constraint; and generating the three-dimensional model according to the three-dimensional point cloud and/or the patch.
The weights are added into the total projection errors obtained through calculation according to the point projection errors, the line projection errors and the surface projection errors, so that the weights can be adjusted according to the differences of the points, the lines and the surfaces according to the total projection errors, error items of the points, the lines and the surfaces are distinguished, the weights can be adjusted according to actual conditions when three-dimensional point clouds and surface patches are generated according to the total projection errors, and a three-dimensional model generated according to the three-dimensional point clouds and/or the surface patches is more precise and accurate.
In an alternative manner, the generating a three-dimensional model from the point projection error, the line projection error, and the plane projection error further includes:
performing disparity fusion of left and right views by combining a block dense reconstruction algorithm with weights set according to the point features, the line features or the plane features, wherein the block dense reconstruction algorithm formula comprises:whereinis an adaptive weight set according to the difference of the point feature, the line feature and the face feature,in order to self-define the parameters,a weight set according to the point feature, the line feature, or the face feature.
Setting weights according to the point features, the line features and the face features by introducing the point features, the line features and the face features as constraints in the block dense reconstruction algorithmTo generate three dimensionsAnd the point cloud can be ensured to be under the constraints of point characteristics, line characteristics and surface characteristics.
In an alternative manner, the generating a three-dimensional model from the point projection error, the line projection error, and the plane projection error further includes:
extracting semantic features of each image; generating a semantic texture patch according to the point projection error, the line projection error, the surface projection error and the semantic features; and generating the three-dimensional model according to the semantic texture patches.
Through extracting the semantic features of each image, generating semantic texture patches by combining point projection errors, line projection errors and plane projection errors, and generating a three-dimensional model according to the semantic texture patches, different parts in the three-dimensional model can be colored, textured and the like more conveniently and rapidly, and the service efficiency of the three-dimensional model is improved.
In an optional manner, after the generating the three-dimensional model according to the semantic texture patch, the method further includes:
and performing multi-detail level optimization on the three-dimensional model.
When the three-dimensional model generated by the semantic texture surface patch is optimized in multiple detail levels, the constraint of the midpoint, the line and the surface of the semantic texture surface patch can be combined to carry out the optimization in multiple detail levels, so that the efficiency of the optimization in multiple detail levels is higher.
According to another aspect of the embodiments of the present invention, there is provided a three-dimensional reconstruction apparatus including:
the extraction module is used for extracting the point feature, the line feature and the surface feature of each image in a plurality of images, and the images are shot from different angles;
a calculation module for calculating a point projection error of the point feature, a line projection error of the line feature, and a plane projection error of the plane feature;
and the generating module is used for generating a three-dimensional model according to the point projection error, the line projection error and the surface projection error.
According to another aspect of the embodiments of the present invention, there is provided a three-dimensional reconstruction apparatus including:
the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation of the three-dimensional reconstruction method.
According to another aspect of the embodiments of the present invention, there is provided a computer-readable storage medium, in which at least one executable instruction is stored, and the executable instruction causes a three-dimensional reconstruction device to perform operations corresponding to the method.
According to the three-dimensional reconstruction method, the three-dimensional reconstruction device and the computer-readable storage medium, the point, line and surface features are comprehensively extracted to carry out three-dimensional reconstruction together, so that the generated three-dimensional model is provided with constraints of points, lines and surfaces, redundant points on the same line segment or redundant line segments on the same plane are convenient to combine or remove, the number of miscellaneous lines of the generated three-dimensional model is reduced, and the fineness of the generated three-dimensional model can be improved.
The foregoing description is only an overview of the technical solutions of the embodiments of the present invention, and the embodiments of the present invention can be implemented according to the content of the description in order to make the technical means of the embodiments of the present invention more clearly understood, and the detailed description of the present invention is provided below in order to make the foregoing and other objects, features, and advantages of the embodiments of the present invention more clearly understandable.
Drawings
The drawings are only for purposes of illustrating embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 shows a schematic flow chart of a three-dimensional reconstruction method provided in an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a three-dimensional reconstruction apparatus provided in an embodiment of the present invention;
fig. 3 shows a schematic structural diagram of a three-dimensional reconstruction apparatus provided in an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein.
Aiming at the problem of low model fineness of the existing three-dimensional reconstruction technology, the inventor notices that the existing real-scene three-dimensional reconstruction technology used in the fields of building, cultural heritage digitization, mapping and the like is often extracted according to one feature of points, lines or planes of an image, so that three-dimensional reconstruction is realized, the fineness of the generated three-dimensional model is limited, when the three-dimensional model is generated by only constraining any one of the point features, the line features or the plane features, because only single constraint exists in the three-dimensional model, the three-dimensional model cannot be combined with each other to combine or eliminate redundant points, lines or planes, the generated three-dimensional model is easy to have more miscellaneous lines, redundant line segments in the same plane are difficult to remove or combine with the plane, the generated three-dimensional model is relatively complicated, and the three-dimensional model is not beneficial to further processing of the subsequent three-dimensional model, for example, the occupation of resources is relatively large and the consumed time is relatively long when the subsequent texture adding and other operations are performed on the model. Therefore, it is important to develop a three-dimensional reconstruction method capable of improving the accuracy of the three-dimensional model.
In order to solve the above problems, the inventors of the present application have studied and designed a three-dimensional reconstruction method, which performs three-dimensional reconstruction by comprehensively extracting features of points, lines, and planes, so that the generated three-dimensional model is constrained by the points, the lines, and the planes, and is convenient for merging or removing redundant points on the same line segment or redundant line segments on the same plane, so that the generated three-dimensional model has fewer miscellaneous lines, and the fineness of the generated three-dimensional model can be improved.
Fig. 1 shows a flowchart of a three-dimensional reconstruction method provided by an embodiment of the present invention, and as shown in fig. 1, the method includes the following steps:
step 110: point features, line features and face features of each of a plurality of images are extracted, the plurality of images representing multiple views of a target object.
In this step, the target object refers to an object targeted by three-dimensional reconstruction, such as an article, a person, or a scene, and a plurality of the above objects may also jointly constitute the target object, for example, a person and an article in one scene are jointly used as the target object.
In this step, the image is a photograph including the target object, and may also be a video frame extracted from a video including the target object, and the source and the form of the image may be various, and only the three-dimensional reconstruction method provided in the present application needs to be able to acquire a certain amount of initial two-dimensional information of the target object from the image, which is not particularly limited in the present application.
The method for extracting the point feature, the line feature and the plane feature of each image in the plurality of images can be an LBP (Local Binary Patterns) algorithm, an HOG (Histogram of Oriented Gradient) feature extraction algorithm, a SIFT (Scale-invariant feature transform) operator and the like, the point feature, the line feature and the plane feature in each image are accurately extracted, only a proper feature extraction mode needs to be selected according to an actual application scene and actual needs, and the method is not particularly limited.
In order to obtain enough two-dimensional information of the target object to generate the three-dimensional model, multiple views of the target object, i.e. multiple different angles of the target object, should be included in the multiple images, for example, if the images are photographs of the target object obtained by shooting, the photographs of the multiple target objects should include images obtained by shooting from the multiple different angles of the target object by the camera.
By extracting point features, line features and surface features in each image from a plurality of images containing a plurality of angles of the target object, enough target object information can be acquired as basic data, and the subsequent calculation for generating the three-dimensional model is facilitated.
Step 120: point projection errors of the point features, line projection errors of the line features, and plane projection errors of the plane features are calculated.
In this step, point projection errors of the point features, line projection errors of the line features, and plane projection errors of the plane features are calculated, respectively, from the point features, the line features, and the plane features extracted in step 110.
For example, when the plurality of images in step 110 are captured by a camera, image parameters required for calculating the point projection error, the line projection error and the surface projection error may be given by the camera, or may be directly obtained according to image attributes, and it is understood that the required parameters of the same image do not change due to the change of the obtaining means, so that different parameter obtaining modes may be adopted according to actual situations in order to obtain known parameters of the image necessary for calculating the projection error, which is not particularly limited in the embodiment of the present application.
The method for calculating the point projection error, the line projection error and the surface projection error can be carried out by a BA (bubble Adjustment) algorithm, and aims to obtain the error between the theoretical projection position and the actual position of the point feature, the line feature and the surface feature on the three-dimensional plane, thereby obtaining the accurate positions of the three-dimensional point, the three-dimensional line segment and the three-dimensional plane of the target object and being convenient for generating an accurate three-dimensional model according to the accurate positions subsequently.
By calculating the point projection error of the point characteristic, the line projection error of the line characteristic and the plane projection error of the plane characteristic, the error of the two-dimensional characteristic point in three-dimensional projection is known, so that the error correction is conveniently carried out when the two-dimensional characteristic point is converted into three-dimensional in the follow-up process, the three-dimensional reconstruction of the two-dimensional characteristic point in the image can be ensured, and the three-dimensional model of the target object is obtained.
Step 130: and generating a three-dimensional model of the target object according to the point projection error, the line projection error and the surface projection error.
In this step, a three-dimensional point corresponding to a point feature of the target object is calculated by the point projection error, a three-dimensional line segment corresponding to a line feature of the target object is calculated by the line projection error, and a three-dimensional plane corresponding to a surface feature of the target object is calculated by the surface projection error, thereby generating a three-dimensional model of the target object.
The three-dimensional model may be a mesh grid structure combined with attributes of the target object, for example, the three-dimensional model is a mesh grid structure combined with semantic generation, and may also be combined with other attributes according to actual needs, so as to facilitate further application and expansion of the three-dimensional model for different application scenarios, which is not particularly limited in the embodiments of the present application.
When the three-dimensional model is constrained by any two or more of the point feature, the line feature and the surface feature, the redundant useless points, lines or surfaces can be found by combining other features, so that the miscellaneous lines in the generated three-dimensional model are reduced, and the fineness is improved. For example, in the embodiment of the present application, a point projection error, a line projection error, and a plane projection error are obtained through a point feature, a line feature, and a plane feature, and then, an accurate three-dimensional point, a three-dimensional line segment, and a three-dimensional plane are calculated.
In this case, the combination of the point feature, the line feature, and the face feature may be performed by a BA (Bundle Adjustment) algorithm.
By generating the three-dimensional model of the target object according to the point projection error, the first projection error and the surface projection error, the three-dimensional reconstruction method provided by the embodiment of the application can generate a more detailed three-dimensional model by combining the characteristics of points, lines and surfaces at the same time.
As can be seen from the combination of the above steps 110, 120 and 130, according to the three-dimensional reconstruction method provided by the present application, the point feature, the line feature and the plane feature of each of the plurality of images are extracted, and the plurality of images include a plurality of angles of the target object; calculating point projection errors of the point features, line projection errors of the line features and surface projection errors of the surface features; and generating a three-dimensional model of the target object according to the point projection error, the line projection error and the surface projection error. By utilizing the scheme of the embodiment, the point projection error, the line projection error and the plane projection error are calculated by combining the point feature, the line feature and the plane feature of the target object, the three-dimensional model of the target object is obtained by combining the point projection error, the line projection error and the plane projection error, redundant points, lines or planes in the three-dimensional model can be combined by combining two constraints, so that the data of the three-dimensional model is simpler and more accurate, the operation efficiency is improved, the three-dimensional model is more precise and contains more image data, more expansion modes are provided in the subsequent application of the three-dimensional model, and the subsequent processing efficiency of the three-dimensional model is improved.
In one embodiment of the present invention, calculating a point projection error of a point feature, a line projection error of a line feature, and a plane projection error of a plane feature further comprises:
step a01: set the point projection error asThe calculation formula is as follows:where j represents the number of point features,in order to be a point feature,as a feature of a point on the imageThe corresponding three-dimensional point is displayed on the screen,is an internal parameter of the image and is,and withAs an external parameter of the image(ii) a Set the line projection error toThe calculation formula is as follows:where k represents the number of line features,andare respectively line featuresThe start point and the end point of (c),is a line featureCorresponding three-dimensional line segmentProjection on the image, the calculation formula is:whereinto representThe co-factor matrix of (a) is,is an intrinsic parameter of the image, which is expressed in the form of:whereinis the number of the image or images,andis the focal length of a pixel of the image,andthe coordinates of pixel points of the image are obtained; set the surface projection error asThe calculation formula is as follows:
where j represents the number of point features, k represents the number of line features, m represents the number of surface features,in order to be a point feature, the method comprises the following steps of,is composed ofThe corresponding three-dimensional point is displayed on the screen,in order to be a surface feature,is characterized by the characteristics of a line,is composed ofA corresponding three-dimensional line segment, wherein,is composed ofThe normal vector of (a) is,andare respectively two-dimensional planesCorresponding three-dimensional planeAnd the distance of the origin of coordinates to the straight line.
For example, knowing the number j of point features in an image, knowing the feature pointsKnowing the characteristic pointsCorresponding three-dimensional pointKnowing the intrinsic parameters of the imageThe expression isWherein isThe number of the image or images is/are,andas an imageThe focal length of the pixel of (a),andas an imageAnd knowing the extrinsic parameters of the imageThen can be according to the formulaCalculating the point projection error of the point feature。
For example, the number k of line features in the image is known, andandare respectively the line characteristicsThe start point coordinates and the end point coordinates of (2) are knownIs a characteristic of the lineCorresponding three-dimensional line segmentProjection on the image, the calculation formula of which is:wherein, in the process,representing intrinsic parameters of an imageThe co-factor matrix of (a), wherein,is the number of the image in question,andis the focal length of a pixel of the image,andthe coordinates of the pixel points of the image can be calculated according to a formulaCalculating the line projection error of the line feature。
For example, given the number of point features in an image is j, the number of line features is k, the number of surface features is m,in order to be a point feature,is composed ofThe corresponding three-dimensional point is displayed on the screen,in order to be a surface characteristic,in order to be a line feature,is composed ofA corresponding three-dimensional line segment, wherein,is composed ofThe normal vector of (a) is,andare respectively two-dimensional planesCorresponding three-dimensional planeAnd the distance of the origin of coordinates to the straight line,may be according to a formulaAnd calculating to obtain the surface projection error of the surface feature.
The point projection error, the line projection error and the surface projection error of the image are calculated according to the known parameters and formulas, so that the point feature, the line feature and the surface feature on the image can obtain an accurate three-dimensional point, an accurate three-dimensional line segment and an accurate three-dimensional plane according to the point projection error, the line projection error and the surface projection error.
In an embodiment of the present invention, generating the three-dimensional model according to the point projection error, the line projection error, and the surface projection error further includes:
step b01: obtaining a camera pose and a three-dimensional point cloud according to the point projection error, the line projection error and the surface projection error; generating a surface patch according to the camera pose and the three-dimensional point cloud; and generating a three-dimensional model according to the patches.
Wherein, the camera pose refers to the external parameters of the image, and the image is setExternal parameters ofExpressed by the formula:where the rotation uses a rotation vector, it can be transformed into a rotation matrix by the rodregs equation.
The three-dimensional point cloud is generated by combining point features, line features and surface features with point projection errors, line projection errors and surface projection errors.
The patch is merged according to the point feature, the line feature and the surface feature, combined with the point projection error, the line projection error and the surface projection error, for example, when a patch is located in a three-dimensional plane obtained by the surface projection error, the patch is merged with the three-dimensional plane, for example, when one edge of a patch intersects with a three-dimensional line segment obtained by the image line projection error, the patch takes the three-dimensional line segment as a new edge to form a new patch, that is, the generation of the patch is constrained by the point feature, the line feature and the surface feature, the point projection error, the line projection error and the surface projection error.
A patch is generated by combining point characteristics, line characteristics, surface characteristics, point projection errors, line projection errors and surface projection errors, so that the patch is constrained by the points, the lines and the surface, the edge of the patch is clearer, and a three-dimensional model generated according to the patch is finer.
In one embodiment of the present invention, calculating a point projection error of a point feature, a line projection error of a line feature, and a plane projection error of a plane feature further comprises:
step c01: calculating a total projection error based on the point projection error, the line projection error and the surface projection errorWhereinis the weight of the error of the point projection,is the weight of the line projection error,is the weight of the surface projection error and,the error of the projection of the point is represented,which is indicative of the line projection error,representing the surface projection error; generating a three-dimensional point cloud and/or a patch according to the total projection error constraint; and generating a three-dimensional model according to the three-dimensional point cloud and/or the patch.
The weights are added into the total projection errors obtained through calculation according to the point projection errors, the line projection errors and the surface projection errors, so that the weights can be adjusted according to the differences of the points, the lines and the surfaces according to the total projection errors, error items of the points, the lines and the surfaces are distinguished, the weights can be adjusted according to actual conditions when three-dimensional point clouds and surface patches are generated according to the total projection errors, and a three-dimensional model generated according to the three-dimensional point clouds and/or the surface patches is finer.
In an embodiment of the present invention, generating the three-dimensional model according to the point projection error, the line projection error, and the surface projection error further includes:
step d01: and performing parallax fusion of left and right views by combining a block dense reconstruction algorithm with weights set according to point features, line features or plane features, wherein the block dense reconstruction algorithm formula comprises:wherein, in the process,is self-adaptive weight set according to the difference of point characteristics, line characteristics and surface characteristics,in order to self-define the parameters,weights set according to point features, line features, or surface features.
The block dense reconstruction algorithm is an image optimization algorithm, patch refers to a range of 3 × 3 or 5 × 5 with a certain pixel as a center, and the average error minimization is performed by using the patches of the pixel points in the left view and the right view in the conventional block dense reconstruction algorithm.
Setting weights according to point features, line features and face features by introducing the point features, the line features and the face features as constraints in a block dense reconstruction algorithmTherefore, the point feature, the line feature and the surface feature can be restrained when the three-dimensional point cloud is generated.
In an embodiment of the present invention, generating the three-dimensional model according to the point projection error, the line projection error, and the surface projection error further includes:
step d02: extracting semantic features of each image; generating a semantic texture surface patch according to the point projection error, the line projection error, the surface projection error and the semantic features; and generating a three-dimensional model according to the semantic texture surface patch.
The semantic features refer to semantic information of an image, the semantic information is divided into a visual layer, an object layer and a concept layer, the visual layer is a commonly understood bottom layer and contains textures, shapes and the like, the object layer is a middle layer and generally contains attribute features, such as the state of a certain object at a certain moment, and the concept layer is a high layer and is the object expressed by the image and closest to human understanding. In this step, the semantic features may be a visual layer, an object layer, or a concept layer, and different semantic features are extracted according to actual needs, which is not particularly limited in this embodiment of the present application.
Through extracting the semantic features of each image, generating a semantic texture surface patch by combining point projection errors, line projection errors and plane projection errors, and generating a three-dimensional model according to the semantic texture surface patch, different parts in the three-dimensional model can be colored, textured and the like more conveniently and rapidly, and the service efficiency of the three-dimensional model is improved.
In an embodiment of the present invention, after generating the three-dimensional model according to the semantic texture patch, the method further includes:
step d03: and performing multi-detail level optimization on the three-dimensional model.
In this step, the multi-level-of-Detail optimization refers to optimization according to the multi-level-of-Detail (Levels of Detail) of the model, which was proposed by Clark in 1976, and it is considered that when an object covers a small area of the screen, the object can be used to describe a thicker model, and a geometric level model for a visible surface judgment algorithm is provided, so as to rapidly draw a complex scene.
When the three-dimensional model generated by the semantic texture surface patch is subjected to multi-detail-level optimization, the multi-detail-level optimization can be performed by combining constraints of points, lines and surfaces of the semantic texture surface patch, so that the multi-detail-level optimization efficiency is higher.
Fig. 2 shows a functional block diagram of a three-dimensional reconstruction apparatus 200 according to an embodiment of the present invention. As shown in fig. 2, the apparatus includes: an extraction module 210, a calculation module 220, and a generation module 230.
An extraction module 210 for extracting a point feature, a line feature and a face feature of each of a plurality of images, the plurality of images representing multiple views of a target object;
a calculating module 220, configured to calculate a point projection error of the point feature, a line projection error of the line feature, and a surface projection error of the surface feature;
a generating module 230, configured to generate a three-dimensional model according to the point projection error, the line projection error, and the surface projection error;
in some embodiments, the calculation module 220 further comprises:
a first calculation unit for setting the projection error asThe calculation formula is as follows:where j represents the number of point features,in order to be a point feature,as a point feature on the imageThe corresponding three-dimensional point is displayed on the screen,is an internal parameter of the image and is,and withAs an external parameter of the image;
A second calculation unit for setting the line projection error toThe calculation formula is as follows:where k represents the number of line features,andare respectively line featuresThe start point and the end point of (c),is a line featureCorresponding three-dimensional line segmentProjection on the image, the calculation formula is:wherein, in the process,to representThe co-factor matrix of (a) is,is an intrinsic parameter of the image, which is expressed by the following form:whereinis the number of the image or images,andis the focal length of a pixel of the image,andthe pixel point coordinates of the image are obtained;
a third calculation unit for setting the surface projection error asThe calculation formula is as follows:where j represents the number of point features, k represents the number of line features, m represents the number of surface features,in order to be a point feature,is composed ofThe corresponding three-dimensional point is displayed on the screen,in order to be a surface feature,is characterized by the characteristics of a line,is composed ofA corresponding three-dimensional line segment is formed,is composed ofThe normal vector of (a) is,andare respectively two-dimensional planesCorresponding three-dimensional planeAnd the distance of the origin of coordinates to the straight line.
In some embodiments, the calculation module 220 further comprises:
a fourth calculation unit for calculating a total projection error based on the point projection error, the line projection error and the surface projection errorWhereinis the weight of the point projection error,is the weight of the line projection error,is the weight of the surface projection error,the error of the projection of the point is represented,which is indicative of the line projection error,representing a surface projection error;
the first generating unit is used for generating a three-dimensional point cloud and/or a patch according to the total projection error constraint;
and the second generation unit is used for generating a three-dimensional model according to the three-dimensional point cloud and/or the patch.
In some embodiments, the generation module 230 further comprises:
the third generation unit is used for obtaining a camera pose and a three-dimensional point cloud according to the point projection error, the line projection error and the surface projection error;
a fourth generating unit, configured to generate a patch according to the camera pose and the three-dimensional point cloud;
and the fifth generating unit is used for generating the three-dimensional model according to the patch.
In some embodiments, the generation module 230 further comprises:
a sixth generating unit, configured to perform disparity fusion of left and right views through a block dense reconstruction algorithm and a weight combination set according to the point feature, the line feature, or the plane feature, where the block dense reconstruction algorithm formula includes:whereinis self-adaptive weight set according to the difference of point characteristics, line characteristics and surface characteristics,in order to self-define the parameters,weights set according to point features, line features, or surface features.
In some embodiments, the generation module 230 further comprises:
a seventh generating unit, configured to extract semantic features of each image;
an eighth generating unit, configured to generate a semantic texture patch according to the point projection error, the line projection error, the surface projection error, and the semantic feature;
and the ninth generating unit is used for generating the three-dimensional model according to the semantic texture surface patch.
In some embodiments, the three-dimensional reconstruction device 200 further comprises:
and the fifth calculation unit is used for performing multi-detail-level optimization on the three-dimensional model.
Fig. 3 is a schematic structural diagram of a three-dimensional reconstruction apparatus according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the three-dimensional reconstruction apparatus.
As shown in fig. 3, the three-dimensional reconstruction apparatus may include: a processor 302, a memory 306, a communication interface 304, and a communication bus 308.
The processor 302, memory 306, and communication interface 304 communicate with each other via a communication bus 308.
The memory 306 is configured to store at least one executable instruction 310, and the executable instruction 310 causes the processor 302 to perform the relevant steps in the three-dimensional reconstruction method embodiment as described above.
The embodiment of the present invention further provides a computer-readable storage medium, where at least one executable instruction is stored in the storage medium, and when the executable instruction runs on a three-dimensional reconstruction device, the three-dimensional reconstruction device may execute the three-dimensional reconstruction method in any method embodiment described above.
The algorithms or displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system is apparent from the description above. In addition, embodiments of the present invention are not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the invention and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: rather, the invention as claimed requires more features than are expressly recited in each claim.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names. The steps in the above embodiments should not be construed as limiting the order of execution unless specified otherwise.
Claims (9)
1. A method of three-dimensional reconstruction, comprising:
extracting point features, line features and face features of each of a plurality of images, the plurality of images representing multiple views of a target object;
calculating a point projection error of the point feature, a line projection error of the line feature, and a plane projection error of the plane feature;
generating a three-dimensional model of the target object according to the point projection error, the line projection error and the surface projection error;
the calculating a point projection error of the point feature, a line projection error of the line feature, and a plane projection error of the plane feature further comprises: let the point projection error beThe calculation formula is as follows:
wherein j represents the number of the point features,for the purpose of the point feature,for the point features on the imageThe corresponding three-dimensional point is displayed on the screen,is an internal parameter of the image and is,andas an external parameter of the image;
wherein k represents the number of said line features,andrespectively being characteristic of said lineThe start point and the end point of (c),is a characteristic of the lineCorresponding three-dimensional line segmentProjection on the image, the calculation formula of which is:
wherein,representThe co-factor matrix of (a) is,the internal parameters of the image are expressed in the form of:
wherein,is the number of the image in question,andis the focal length of a pixel of the image,andthe coordinates of the pixel points of the image are obtained;
wherein j represents the number of the point features, k represents the number of the line features, m represents the number of the face features,for the purpose of the point feature,is composed ofThe corresponding three-dimensional point is displayed on the screen,in order to be a feature of the face,in order to be a feature of the line,is composed ofA corresponding three-dimensional line segment, wherein,is composed ofThe normal vector of (a) is calculated,andare respectively two-dimensional planesCorresponding three-dimensional planeAnd the distance of the origin of coordinates to the straight line.
2. The three-dimensional reconstruction method of claim 1 wherein said generating a three-dimensional model from said point projection errors, said line projection errors and said plane projection errors further comprises:
obtaining a camera pose and a three-dimensional point cloud according to the point projection error, the line projection error and the surface projection error;
generating a patch according to the camera pose and the three-dimensional point cloud;
and generating the three-dimensional model according to the patch.
3. The three-dimensional reconstruction method according to any one of claims 1-2, wherein said calculating a point projection error of said point feature, a line projection error of said line feature and a plane projection error of said plane feature further comprises:
calculating a total projection error from the point projection error, the line projection error, and the surface projection errorWhereinis the weight of the point projection error,is the weight of the line projection error,is the weight of the surface projection error and,is representative of the error of the projection of the point,which is representative of the line projection error,representing the surface projection error;
generating a three-dimensional point cloud and/or a patch according to the total projection error constraint;
and generating the three-dimensional model according to the three-dimensional point cloud and/or the patch.
4. The three-dimensional reconstruction method of claim 1 wherein said generating a three-dimensional model from said point projection errors, said line projection errors and said plane projection errors further comprises:
performing disparity fusion of left and right views by combining a block dense reconstruction algorithm with weights set according to the point features, the line features or the plane features, wherein the block dense reconstruction algorithm formula comprises:
5. The three-dimensional reconstruction method of claim 1, wherein said generating a three-dimensional model from said point projection error, said line projection error, and said plane projection error, further comprises:
extracting semantic features of each image;
generating a semantic texture surface patch according to the point projection error, the line projection error, the surface projection error and the semantic features;
and generating the three-dimensional model according to the semantic texture surface patch.
6. The method of claim 5, wherein after generating the three-dimensional model from the semantic texture patches, the method further comprises:
and performing multi-detail level optimization on the three-dimensional model.
7. A three-dimensional reconstruction apparatus, comprising:
the extraction module is used for extracting point features, line features and surface features of each image in a plurality of images, and the plurality of images are shot from different angles;
a calculation module for calculating a point projection error of the point feature, a line projection error of the line feature, and a plane projection error of the plane feature;
a generating module for generating a three-dimensional model according to the point projection error, the line projection error and the surface projection error;
the calculation module further comprises:
a first calculation unit for setting the projection error asThe calculation formula is as follows:where j represents the number of point features,in order to be a point feature, the method comprises the following steps of,as a point feature on the imageThe corresponding three-dimensional point is displayed on the screen,is an internal parameter of the image and is,andas an external parameter of the image;
A second calculation unit for setting the line projection error toThe calculation formula is as follows:where k represents the number of line features,andare respectively line characteristicsThe start point and the end point of (c),is a line featureCorresponding three-dimensional line segmentProjection on the image, the calculation formula is:whereinrepresentThe co-factor matrix of (a) is,is an intrinsic parameter of the image, which is expressed by the following form:whereinis the number of the image or images,andis the focal length of a pixel of the image,andthe pixel point coordinates of the image are obtained;
a third calculation unit for setting the surface projection error asThe calculation formula is as follows:wherein j represents the number of point features, k represents the number of line features, m represents the number of surface features,in order to be a point feature, the method comprises the following steps of,is composed ofThe corresponding three-dimensional point is displayed on the screen,in order to be a surface feature,in order to be a line feature,is composed ofA corresponding three-dimensional line segment is formed,is composed ofThe normal vector of (a) is calculated,andare respectively two-dimensional planesCorresponding three-dimensional planeAnd the distance of the origin of coordinates to the straight line.
8. A three-dimensional reconstruction apparatus, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface are communicated with each other through the communication bus;
the memory is configured to store at least one executable instruction that causes the processor to perform the operations of the three-dimensional reconstruction method of any one of claims 1-6.
9. A computer-readable storage medium having stored therein at least one executable instruction which, when run on a three-dimensional reconstruction device, causes the three-dimensional reconstruction device to perform the operations of the three-dimensional reconstruction method of any one of claims 1-6.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210999929.4A CN115063485B (en) | 2022-08-19 | 2022-08-19 | Three-dimensional reconstruction method, device and computer-readable storage medium |
PCT/CN2023/113315 WO2024037562A1 (en) | 2022-08-19 | 2023-08-16 | Three-dimensional reconstruction method and apparatus, and computer-readable storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210999929.4A CN115063485B (en) | 2022-08-19 | 2022-08-19 | Three-dimensional reconstruction method, device and computer-readable storage medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN115063485A CN115063485A (en) | 2022-09-16 |
CN115063485B true CN115063485B (en) | 2022-11-29 |
Family
ID=83208595
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210999929.4A Active CN115063485B (en) | 2022-08-19 | 2022-08-19 | Three-dimensional reconstruction method, device and computer-readable storage medium |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN115063485B (en) |
WO (1) | WO2024037562A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115063485B (en) * | 2022-08-19 | 2022-11-29 | 深圳市其域创新科技有限公司 | Three-dimensional reconstruction method, device and computer-readable storage medium |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015154601A1 (en) * | 2014-04-08 | 2015-10-15 | 中山大学 | Non-feature extraction-based dense sfm three-dimensional reconstruction method |
CN110021065A (en) * | 2019-03-07 | 2019-07-16 | 杨晓春 | A kind of indoor environment method for reconstructing based on monocular camera |
WO2021120175A1 (en) * | 2019-12-20 | 2021-06-24 | 驭势科技(南京)有限公司 | Three-dimensional reconstruction method, apparatus and system, and storage medium |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111784842B (en) * | 2020-06-29 | 2024-04-12 | 北京百度网讯科技有限公司 | Three-dimensional reconstruction method, device, equipment and readable storage medium |
CN112102411B (en) * | 2020-11-02 | 2021-02-12 | 中国人民解放军国防科技大学 | Visual positioning method and device based on semantic error image |
CN113313832B (en) * | 2021-05-26 | 2023-07-04 | Oppo广东移动通信有限公司 | Semantic generation method and device of three-dimensional model, storage medium and electronic equipment |
CN114241050B (en) * | 2021-12-20 | 2024-05-07 | 东南大学 | Camera pose optimization method based on Manhattan world assumption and factor graph |
CN114708293A (en) * | 2022-03-22 | 2022-07-05 | 广东工业大学 | Robot motion estimation method based on deep learning point-line feature and IMU tight coupling |
CN115063485B (en) * | 2022-08-19 | 2022-11-29 | 深圳市其域创新科技有限公司 | Three-dimensional reconstruction method, device and computer-readable storage medium |
-
2022
- 2022-08-19 CN CN202210999929.4A patent/CN115063485B/en active Active
-
2023
- 2023-08-16 WO PCT/CN2023/113315 patent/WO2024037562A1/en unknown
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015154601A1 (en) * | 2014-04-08 | 2015-10-15 | 中山大学 | Non-feature extraction-based dense sfm three-dimensional reconstruction method |
CN110021065A (en) * | 2019-03-07 | 2019-07-16 | 杨晓春 | A kind of indoor environment method for reconstructing based on monocular camera |
WO2021120175A1 (en) * | 2019-12-20 | 2021-06-24 | 驭势科技(南京)有限公司 | Three-dimensional reconstruction method, apparatus and system, and storage medium |
Non-Patent Citations (1)
Title |
---|
基于递推自适应权重的快速稠密立体匹配;杨奎 等;《北京航空航天大学学报》;20130731;第39卷(第7期);第963-967页 * |
Also Published As
Publication number | Publication date |
---|---|
WO2024037562A1 (en) | 2024-02-22 |
CN115063485A (en) | 2022-09-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108961390B (en) | Real-time three-dimensional reconstruction method based on depth map | |
WO2020206903A1 (en) | Image matching method and device, and computer readable storage medium | |
US20190108646A1 (en) | Image processing method, apparatus, and storage medium | |
CN110223383A (en) | A kind of plant three-dimensional reconstruction method and system based on depth map repairing | |
KR20180054487A (en) | Method and device for processing dvs events | |
US9047706B1 (en) | Aligning digital 3D models using synthetic images | |
CN112651881B (en) | Image synthesizing method, apparatus, device, storage medium, and program product | |
CN109685879B (en) | Method, device, equipment and storage medium for determining multi-view image texture distribution | |
US10169891B2 (en) | Producing three-dimensional representation based on images of a person | |
CN115375857B (en) | Three-dimensional scene reconstruction method, device, equipment and storage medium | |
WO2023078052A1 (en) | Three-dimensional object detection method and apparatus, and computer-readable storage medium | |
CN115063485B (en) | Three-dimensional reconstruction method, device and computer-readable storage medium | |
CN115512055A (en) | Method and device for performing indoor structure three-dimensional reconstruction based on two-dimensional video and computer equipment | |
CN109766896A (en) | A kind of method for measuring similarity, device, equipment and storage medium | |
CN112465984A (en) | Monocular camera sequence image three-dimensional reconstruction method based on double-layer filtering | |
CN117557740B (en) | Three-dimensional model segmentation level switching method and device, electronic equipment and storage medium | |
CN116363290A (en) | Texture map generation method for large-scale scene three-dimensional reconstruction | |
CN107610216B (en) | Particle swarm optimization-based multi-view three-dimensional point cloud generation method and applied camera | |
CN117726747A (en) | Three-dimensional reconstruction method, device, storage medium and equipment for complementing weak texture scene | |
CN112633293A (en) | Three-dimensional sparse point cloud reconstruction image set classification method based on image segmentation | |
CN116092035A (en) | Lane line detection method, lane line detection device, computer equipment and storage medium | |
CN113077504B (en) | Large scene depth map generation method based on multi-granularity feature matching | |
JP2020004219A (en) | Apparatus, method, and program for generating three-dimensional shape data | |
WO2019080257A1 (en) | Electronic device, vehicle accident scene panoramic image display method and storage medium | |
Yang et al. | A high-realistic texture mapping algorithm based on image sequences |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |