CN112991524A - Three-dimensional reconstruction method, electronic device and storage medium - Google Patents

Three-dimensional reconstruction method, electronic device and storage medium Download PDF

Info

Publication number
CN112991524A
CN112991524A CN202110426718.7A CN202110426718A CN112991524A CN 112991524 A CN112991524 A CN 112991524A CN 202110426718 A CN202110426718 A CN 202110426718A CN 112991524 A CN112991524 A CN 112991524A
Authority
CN
China
Prior art keywords
point cloud
cloud data
dimensional
target object
initial
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110426718.7A
Other languages
Chinese (zh)
Other versions
CN112991524B (en
Inventor
户磊
蔡麟
李廷照
张举勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hefei Dilusense Technology Co Ltd
Original Assignee
Beijing Dilusense Technology Co Ltd
Hefei Dilusense Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Dilusense Technology Co Ltd, Hefei Dilusense Technology Co Ltd filed Critical Beijing Dilusense Technology Co Ltd
Priority to CN202110426718.7A priority Critical patent/CN112991524B/en
Publication of CN112991524A publication Critical patent/CN112991524A/en
Application granted granted Critical
Publication of CN112991524B publication Critical patent/CN112991524B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Abstract

The embodiment of the invention relates to the field of image processing, and discloses a three-dimensional reconstruction method, electronic equipment and a storage medium. The three-dimensional reconstruction method comprises the following steps: dividing the point cloud data of the target object in the current frame into point cloud data in a designated area and point cloud data in a non-designated area; setting different deformation weights for point cloud data in a designated area and point cloud data in a non-designated area; fitting the three-dimensional splicing model of the previous frame to the point cloud data of the current frame according to the deformation weight; generating a three-dimensional splicing model of the current frame according to the fitted data; and judging whether the point cloud data of the next frame exists, if so, returning to execute the step of dividing the point cloud data of the target object in the current frame into a designated area and a non-designated area, and if not, taking the three-dimensional splicing model of the current frame as the target three-dimensional model of the target object. By adopting the embodiment, the problems of high cost, low precision and low speed of three-dimensional reconstruction are solved.

Description

Three-dimensional reconstruction method, electronic device and storage medium
Technical Field
The embodiment of the invention relates to the field of image processing, in particular to a three-dimensional reconstruction method, electronic equipment and a storage medium.
Background
Three-dimensional reconstruction refers to the establishment of a mathematical model suitable for computer representation and processing of a three-dimensional object, is the basis for processing, operating and analyzing the properties of the three-dimensional object in a computer environment, and is also a key technology for establishing virtual reality expressing an objective world in a computer. The current three-dimensional reconstruction method mainly comprises the following steps: method 1 is to scan an object with a high precision three-dimensional scanner. The method 2 is that a color image of a plurality of visual angles of a target object is obtained based on a multi-visual angle color camera, and three-dimensional information of a human body is estimated through a multi-visual angle algorithm. The method 3 is to collect color images and depth information of a target object by fixing a plurality of RGB-D cameras, splice point cloud data according to the position relation among the pre-marked cameras, and carry out three-dimensional reconstruction.
However, the high-precision three-dimensional scanner in the method 1 is expensive and not easy to popularize; in the method 2, although the price of the color camera is low, the accuracy of the three-dimensional information generated based on the color image is low, so that the accuracy of a three-dimensional model obtained by three-dimensional reconstruction is low; the preparation work in the method 3 is various, and the speed of three-dimensional reconstruction is slow.
Disclosure of Invention
An object of an embodiment of the present invention is to provide a method, an electronic device, and a storage medium for three-dimensional reconstruction, so as to solve the problems of high cost, low precision, and low speed of three-dimensional reconstruction.
In order to solve the above technical problem, an embodiment of the present invention provides a method for three-dimensional reconstruction, including: dividing the point cloud data of the target object in the current frame into point cloud data in a designated area and point cloud data in a non-designated area; setting different deformation weights for the point cloud data in the designated area and the point cloud data in the non-designated area; fitting the three-dimensional splicing model of the previous frame to the point cloud data of the current frame according to the deformation weight; generating a three-dimensional splicing model of the current frame according to the fitted data; judging whether the point cloud data of the next frame exists, if so, returning to execute the step of dividing the point cloud data of the target object in the current frame into a designated area and a non-designated area, and if not, taking the three-dimensional splicing model of the current frame as the target three-dimensional model of the target object.
An embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of three-dimensional reconstruction described above.
Embodiments of the present invention also provide a computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, implements the above-mentioned method for three-dimensional reconstruction.
According to the three-dimensional reconstruction method in the embodiment of the application, in the process of image acquisition of the target object, the degree of change of different parts of the target object is different, the point cloud data of the current frame of the target object is divided into the point cloud data in the designated area and the point cloud data in the non-designated area, and different deformation weights are set for the point cloud data in the designated area and the point cloud data in the non-designated area, so that the point cloud data in the designated area and the point cloud data in the non-designated area are closer to the real condition of the target object according to the different deformation weights, the fitting accuracy and the accuracy of a three-dimensional splicing model are improved, and the accuracy of a target three-.
In addition, the step of dividing the point cloud data of the target object in the current frame into point cloud data in a designated area and point cloud data in a non-designated area comprises the following steps: acquiring a designated part in the initial three-dimensional model template, wherein the initial three-dimensional model template is generated according to the depth information of the first frame of the target object and a preset three-dimensional template generation model; and searching point cloud data in a designated area corresponding to the designated part in the point cloud data according to the designated part in the initial three-dimensional model template. Generating an initial three-dimensional model template through the depth information of the first frame, and fitting and fusing based on the initial three-dimensional model template to reduce the calculated amount of three-dimensional reconstruction; the appointed part in the initial three-dimensional model template is obtained, the point cloud data of the appointed area corresponding to the appointed part is searched from the point cloud data of the current frame through the appointed part in the initial three-dimensional model template, and the point cloud data of the appointed area of the current frame is obtained without performing example segmentation on the color image corresponding to the current frame, so that the time cost is reduced.
In addition, acquiring the designated part in the initial three-dimensional model template comprises: acquiring a first frame color image of the target object; performing example segmentation on the first frame color image, and determining a specified part of the target object in the first frame color image; and projecting the initial three-dimensional model template to the segmented first frame color image according to the designated part of the target object in the first frame color image, and determining the designated part in the initial three-dimensional model template. And projecting the initial three-dimensional model template to the two-dimensional image segmented by the example, so that the designated position in the initial three-dimensional model template can be quickly determined.
In addition, before the dividing the point cloud data of the target object in the current frame into the point cloud data in the designated area and the point cloud data in the non-designated area, the method further comprises: and acquiring an image of the rotating target object by adopting a single color-depth camera to obtain at least two frames of image data of the target object. And only a single color-depth camera is adopted to acquire the image of the target object, so that the cost of the equipment is reduced.
In addition, according to the designated part in the initial three-dimensional model template, point cloud data in a designated area corresponding to the designated part is searched in the point cloud data, and the method comprises the following steps: searching points with a distance smaller than a preset threshold value from the designated position in the initial three-dimensional model template as point cloud data in the designated area; and taking other point cloud data outside the specified area as the point cloud data in the non-specified area.
In addition, fitting the three-dimensional stitching model of the previous frame to the point cloud data of the current frame according to the deformation weight comprises the following steps: carrying out rigid transformation on the three-dimensional splicing model of the previous frame to generate a new three-dimensional splicing model; and determining a non-rigid transformation deformation field according to the deformation weight, and deforming the new three-dimensional splicing model according to the non-rigid deformation field to obtain fitted data.
In addition, before acquiring the designated location in the initial three-dimensional model template, the method further comprises: and carrying out rigid transformation and non-rigid transformation on the initial three-dimensional model template so as to optimize the initial three-dimensional model template. The accuracy of the initial three-dimensional model template is improved through rigid transformation and non-rigid transformation.
In addition, if the target object is a human body, the designated portion is an arm.
Drawings
One or more embodiments are illustrated by way of example in the accompanying drawings, which correspond to the figures in which like reference numerals refer to similar elements and which are not to scale unless otherwise specified.
Fig. 1 is a flowchart of a three-dimensional reconstruction method according to a first embodiment of the present invention;
fig. 2 is a flowchart of a three-dimensional reconstruction method according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of an example segmented image according to a second embodiment of the present invention;
fig. 4 is a schematic flow chart of a three-dimensional reconstruction method according to a second embodiment of the present invention;
fig. 5 is a flowchart of a three-dimensional reconstruction method according to a third embodiment of the present invention;
fig. 6 is a flowchart of a three-dimensional reconstruction method according to a fourth embodiment of the present invention;
fig. 7 is a block diagram of an electronic device according to a fifth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments.
The following embodiments are divided for convenience of description, and should not constitute any limitation to the specific implementation manner of the present invention, and the embodiments may be mutually incorporated and referred to without contradiction.
A first embodiment of the invention relates to a method of three-dimensional reconstruction. The flow is shown in figure 1:
step 101: and dividing the point cloud data of the target object in the current frame into point cloud data in a specified area and point cloud data in a non-specified area.
Step 102: different deformation weights are set for the point cloud data in the designated area and the point cloud data in the non-designated area.
Step 103: and fitting the three-dimensional splicing model of the previous frame to the point cloud data of the current frame according to the deformation weight.
Step 104: and generating a three-dimensional splicing model of the current frame according to the fitted data.
Step 105: and judging whether the point cloud data of the next frame exists, if so, returning to the step of dividing the point cloud data of the target object in the current frame into a designated area and a non-designated area, and if not, executing the step 106.
Step 106: and taking the three-dimensional splicing model of the current frame as a target three-dimensional model of a target object. Generating a three-dimensional splicing model of the current frame according to the fitted data
According to the three-dimensional reconstruction method in the embodiment of the application, in the process of image acquisition of the target object, the degree of change of different parts of the target object is different, the point cloud data of the current frame of the target object is divided into the point cloud data in the designated area and the point cloud data in the non-designated area, and different deformation weights are set for the point cloud data in the designated area and the point cloud data in the non-designated area, so that the point cloud data in the designated area and the point cloud data in the non-designated area are closer to the real condition of the target object according to the different deformation weights, the fitting accuracy and the accuracy of a three-dimensional splicing model are improved, and the accuracy of a target three-.
A second embodiment of the invention relates to a method of three-dimensional reconstruction. The second embodiment is a detailed description of the first embodiment, and the flow thereof is shown in fig. 2.
Step 201: the method comprises the steps of obtaining a designated position in an initial three-dimensional model template, wherein the initial three-dimensional model template is generated according to a first frame of image data of a target object and a preset three-dimensional template generation model, and the image data comprises a color image of the target object and corresponding depth information.
Specifically, the target object may be an object, such as a box, or the like, or may be an animal, a human body, a plant, or the like. The target object is introduced for a human body in this example. The initial three-dimensional model template is generated according to a first frame of image data of the target object and a preset three-dimensional template generation model. The preset three-dimensional template generation model can be obtained by adopting a deep learning network after training a training set, and the three-dimensional template generation model is used for generating an initial three-dimensional model of the target object. The deep learning network may be a CNN network or the like, in which the first frame image data of a plurality of training samples is included in the training set. The image data includes: color images and corresponding depth information.
And inputting the first frame image data of the target object into the three-dimensional template generation model to obtain an initial three-dimensional model of the target object. Since the initial three-dimensional model of the target object only adopts the first frame image data, the generated initial three-dimensional model is inaccurate, and at this time, the initial three-dimensional model can be regarded as a skeleton model of the target object, and the target three-dimensional model of the target object is obtained based on the skeleton model, so that the speed of generating the target three-dimensional model can be shortened.
In one example, before performing the three-dimensional reconstruction, image data of the target object at a plurality of viewing angles needs to be acquired. For example, the image data of the target object may be acquired by a plurality of RGB-D cameras, or the image data of the target object under a plurality of viewing angles may be acquired by photographing a fixed target object by rotating a single RGB-D camera.
In another example, a single color-depth camera is used to image capture a rotating target object, resulting in at least two frames of image data of the target object.
Specifically, a rotating device may be provided for the target object, for example: a turntable. The rotating device can rotate 360 degrees. A color-depth camera (i.e., an RGB-D camera) is disposed beside the rotating device to photograph a target object located on the rotating device. Usually, image data of one rotation of the target object can be acquired, and the acquisition frequency can be set according to actual needs, for example: the target object rotates one circle, and 150 frames of image data are acquired. It is understood that the number of frames collected may be less than 150 frames, or greater than 150 frames.
It is worth mentioning that the image data of the target object is acquired only through a single RGB-D camera, so that the complexity of acquisition equipment is reduced, the equipment cost is also reduced, and meanwhile, the positions of the rotating device and the camera are simple and quick, so that the high-precision three-dimensional reconstruction of the target object can be completed only by acquiring the image data of the target object without depending on large-scale acquisition equipment. Continuously acquiring 360 degrees of image data of the target object, such as one rotation of 150 frames of data, can improve the accuracy of the reconstruction result.
Acquiring image data of a target object from an RGB-D camera also includes internal parameters of the camera. The depth information is converted into point cloud data by using the acquired camera parameters, in this example, the depth information is embodied in a depth map, and the calculation formula of converting the pixel point (u, v) from the depth map coordinate to the world coordinate is as follows, noting that the depth Z at one pixel coordinate (u, v) on the depth map D is D (u, v):
Figure BDA0003029796950000051
Figure BDA0003029796950000052
wz-z formula (3);
wherein fx and fy are focal lengths of the depth camera in x and y directions respectively, and cx and cy are horizontal and vertical offsets of an image origin relative to an optical center imaging point respectively. And converting the depth map into point cloud data, and constructing an adjacency relation between the point clouds by using the depth map to obtain a triangular surface patch of the point cloud.
By processing the depth information, the point cloud data of the target object can be obtained for subsequent fusion.
In one example, after acquiring the first frame color image of the target object, acquiring a first frame color image of the target object; carrying out example segmentation on the first frame color image, and determining a specified part of a target object in the first frame color image; and according to the designated part of the target object in the first frame of color image, projecting the initial three-dimensional model template to the segmented first frame of color image, and determining the designated part in the initial three-dimensional model template.
The first frame color image is subjected to example segmentation to obtain a segmented color image, and the segmented color image is shown in fig. 3 (colors are not shown in fig. 3). The initial three-dimensional model template is projected on a first frame of color image by projection, the color image is divided into an area where a specified part is located and an area where a non-specified part is located due to example division of the first frame of color image, and the specified part in the initial three-dimensional model template can be determined according to the division result.
The specified portion may be a portion in the target object where the probability of change is large, for example: an arm of a human body. In this example, the target object is a human body, and the designated portion of the target object is an arm.
It is worth mentioning that in this example, the color image is subjected to example segmentation, and the points in the arm in the initial three-dimensional model template of the target object can be obtained through projection. Because the proportion of the point cloud of the arm relative to the whole body is smaller, the arm is easier to change in the process of acquiring data. By segmenting the arm, the point cloud data of the appointed part and the point cloud data of the non-appointed part are processed differently, and the reconstruction precision can be improved.
Step 202: and searching point cloud data in a designated area corresponding to the designated part in the point cloud data of the current frame according to the designated part in the initial three-dimensional model template.
Specifically, the point cloud belonging to the designated part in the point cloud data of the current frame can be searched through the designated part in the initial three-dimensional model template. The point cloud data corresponding to the designated position in the initial three-dimensional model template in the point cloud can be obtained by searching in a nearest neighbor mode.
It should be noted that, by using the designated location in the initial three-dimensional model template, the point cloud data in the designated area corresponding to the designated location in the current frame can be segmented, and the point cloud data of the designated location obtained by performing instance segmentation again through the color image is not needed, so that the time overhead is reduced.
Step 201 and step 202 are detailed descriptions of step 101 in the first embodiment.
Step 203: different deformation weights are set for the point cloud data in the designated area and the point cloud data in the non-designated area.
The designated location in this example is typically a location that is subject to movement during data acquisition, such as: the arm part of the human body. Based on this, the deformation weight of the point cloud data in the specified area may be set to be larger than the deformation weight of the point cloud data in the non-specified area, for example: the deformation weight of the point cloud data in the designated area may be set to be 5 times of the deformation weight of the point cloud data in the non-designated area, the deformation weight of the point cloud data in the designated area may be set to be e1, and the deformation weight of the point cloud data in the non-designated area may be set to be e0, where e1 is 5e 0.
Step 204: and fitting the three-dimensional splicing model of the previous frame to the point cloud data of the current frame according to the deformation weight.
In one example, rigid transformation is carried out on the three-dimensional splicing model of the previous frame to generate a new three-dimensional splicing model; and solving the deformation field of the non-rigid transformation according to the deformation weight, and deforming the new three-dimensional splicing model according to the non-rigid deformation field to obtain fitted data.
Specifically, the three-dimensional splicing model of the ith frame is marked as Fi-1In this case, F is first transformed by the rigid transformation of the ith framei-1Point cloud data P transformed to i +1 th frameiThe positions are relatively close to each other, and the three-dimensional splicing model F is splicedi-1To PiPerforming rigid fitting, passing the deformation field of rigid transformation through Fi-1And searching, wherein in the example, 0 equivalent surface is extracted from the point cloud of the first frame to be used as the three-dimensional splicing model of the first frame.
The objective function of the rigid fit is shown in equation (4):
Figure BDA0003029796950000071
where scale represents the scaling factor, rot represents the rotation matrix, trans represents the translation variable, piAnd q isiSearching the point cloud data P of the current frame in a nearest neighbor modeiAnd a three-dimensional mosaic model Fi-1The corresponding point in (1). The values of the scaling coefficient scale, the rotation matrix rot and the translation variable trans can be obtained through the energy in the gauss-newton minimization formula (4). Substituting the solved scaling coefficient scale, the rotation matrix rot and the translation variable trans into a formula (5), and realizing F according to the formula (5)i-1Fitting to Point cloud data PiThe method comprises the following steps:
pi scale (rot qi + trans) formula (5);
the corresponding points in the point cloud data can be rectified by a rigid fit.
Making a three-dimensional splicing model after rigid fittingFor a new three-dimensional mosaic model, non-rigid fitting is carried out on the new three-dimensional mosaic model according to the deformation weight, namely, a graph node G on the three-dimensional mosaic model after rigid fitting is utilizedsThe new three-dimensional mosaic model is added to the point cloud data PiPerforming a non-rigid fit, wherein Gs＝{G0,G1,...,Gn-1And the graph nodes are points sampled on the new three-dimensional splicing model.
The objective function of the non-rigid fit is shown in equation (6):
E(M)＝Edata+Esmoothformula (6);
where M represents a non-rigid deformation field, and M ═ R0,T0,R1,T1,...,Rn-1,Tn-1}，Rn-1A rotation matrix, T, representing the (n-1) th graph node of the ith framen-1A translation matrix representing the (n-1) th graph node representing the ith frame; edataRepresenting data energies, which are similar to rigid transformations, used to constrain corresponding points in the point cloud data to be as close to energy as possible. Since the deformation weight of the point cloud data in the designated area and the deformation weight of the point cloud data in the non-designated area are included in this example, formula (6) is sorted in this example to obtain formula (7):
E(M)＝(E'data×e1+E″data×e0)+Esmoothformula (7);
wherein, E'dataRepresenting the data energy within the specified area, e1 representing the distortion weight of the point cloud data within the specified area; e ″)dataRepresenting the data energy within the unspecified area, and e0 representing the deformation weight of the point cloud data within the unspecified area.
In this example, the non-rigid deformation field M ═ { R ═ R0,T0,R1,T1,...,Rn-1,Tn-1Point q applied to the rigidly fitted three-dimensional mosaic modeliThe deformed point is obtained as shown in equation (8):
Figure BDA0003029796950000081
where wi (j) represents the non-rigid weight of the jth graph node for the ith point.
Figure BDA0003029796950000082
Incidentally, E'dataAnd E ″)dataIs calculated in a manner similar to equation (9).
EsmoothThe energy used to constrain the non-rigid deformation field to be as smooth as possible is shown in equation (10):
Figure BDA0003029796950000083
where neighbor (i) represents a set of subscripts of adjacent graph nodes of the ith graph node.
Similarly, the three-dimensional splicing model F of the previous frame is obtained by nearest neighbor searchingi-1Point cloud data P of current frameiCorresponding relation between them, then solving the non-rigid deformation field M by Gauss-Newton iterationiThen according to a non-rigid deformation field MiSplicing the three-dimensional model Fi-1Point cloud data P to current frameiA non-rigid fit is performed.
Step 205: and generating a three-dimensional splicing model of the current frame according to the fitted data.
Performing TSDF fusion on the fitted data, namely fusing the TSDF of the previous i framesi-1. Using fused TSDFi-1Obtaining a three-dimensional splicing model F of the (i + 1) th frame by marking-cubei。
Step 206: and judging whether the point cloud data of the next frame exists, if so, returning to the step 201, and if not, executing the step 207.
Step 207: and taking the three-dimensional splicing model of the current frame as a target three-dimensional model of a target object.
Fig. 4 shows a schematic diagram of an effect of three-dimensional reconstruction, an initial three-dimensional model template can be generated by combining a color image and point cloud data, a three-dimensional mosaic model is arranged above a double-headed arrow in the template, point cloud data of each frame is arranged below the double-headed arrow in the template, a final column of target three-dimensional models are obtained after fusion, and the final column of target three-dimensional models show the effect of the target three-dimensional models in three view angles.
By the three-dimensional reconstruction method in this example, a target three-dimensional model of a target object can be accurately and quickly generated.
The steps of the above methods are divided for clarity, and the implementation may be combined into one step or split some steps, and the steps are divided into multiple steps, so long as the same logical relationship is included, which are all within the protection scope of the present patent; it is within the scope of the patent to add insignificant modifications to the algorithms or processes or to introduce insignificant design changes to the core design without changing the algorithms or processes.
The third embodiment of the present invention relates to a method for three-dimensional reconstruction, and the third embodiment is a further improvement of the second embodiment, and the main improvement lies in that: in this embodiment, the point cloud data in the designated area is searched according to the distance from the designated position in the initial three-dimensional model template. The flow is shown in fig. 5.
Step 301: the method comprises the steps of obtaining a designated position in an initial three-dimensional model template, wherein the initial three-dimensional model template is generated according to a first frame of image data of a target object and a preset three-dimensional template generation model, and the image data comprises a color image of the target object and corresponding depth information.
Step 302: and searching points with a distance smaller than a preset threshold value from the designated part in the initial three-dimensional model template as point cloud data in the designated area.
Specifically, in order to increase the searching speed, the distance between the point cloud data of the current frame and the designated part in the initial three-dimensional model template may be acquired, and a point whose distance from the designated part is smaller than a preset threshold may be used as a point in the designated area.
Step 303: and taking other point cloud data outside the specified area as point cloud data in the non-specified area.
And taking the point cloud data outside the specified area in the point cloud data of the current frame as the point cloud data in the non-specified area.
Step 304: different deformation weights are set for the point cloud data in the designated area and the point cloud data in the non-designated area.
Step 305: and fitting the three-dimensional splicing model of the previous frame to the point cloud data of the current frame according to the deformation weight.
Step 306: and generating a three-dimensional splicing model of the current frame according to the fitted data.
Step 307: and judging whether the point cloud data of the next frame exists, if so, returning to the step 301, otherwise, executing the step 308.
Step 308: and taking the three-dimensional splicing model of the current frame as a target three-dimensional model of a target object.
Steps 304 to 308 are substantially the same as steps 203 to 207 in the second embodiment, and will not be described herein again.
In the example, if each frame acquires and divides the arm for acquiring the point cloud through the color image, the time overhead is relatively long, but in the example, the arm for acquiring the point cloud is acquired only through the color image of the first frame, the arm of the initial three-dimensional model template is acquired, the point cloud of the arm in the subsequent frame is indirectly acquired through the arm of the initial three-dimensional model template, the time overhead is reduced, and the speed of three-dimensional reconstruction is improved; the point cloud data corresponding to the arm part is found by relaxation, and meanwhile, the deformation weight of the point cloud data of the arm is increased, so that the accuracy of three-dimensional reconstruction can be improved.
The fourth embodiment of the present invention relates to a method for three-dimensional reconstruction, and the fourth embodiment is a further improvement of the above embodiment, and the main improvement lies in: before obtaining the designated part in the initial three-dimensional model template, the initial three-dimensional model template is optimized, and the flow is shown in fig. 6.
Step 401: and carrying out rigid transformation and non-rigid transformation on the initial three-dimensional model template so as to optimize the initial three-dimensional model template.
The procedure of the rigid fitting and the non-rigid fitting is similar to that in the second embodiment.
Specifically, before rigid fitting, an initial scaling ratio can be obtained through the difference of the initial three-dimensional model template and the point cloud data P of the first frame in the y direction, and then the initial three-dimensional model template and the point cloud data P of the first frame are used0Coordinates of the center point translate the initial three-dimensional model template to P0Is detected.
Obtaining the corresponding point p of the point cloud and the initial three-dimensional model template through nearest neighbor searchingiAnd q isiAnd obtaining a variable scaling coefficient scale, a rotation matrix rot and a translation variable trans through the energy in the Gaussian and Newton minimization formula (4), and correcting the point corresponding to the specified part in the point cloud data according to the formula (5).
The initial three-dimensional model template after rigid fitting can be subjected to non-rigid fitting, and the graph node G of the initial three-dimensional model template can be utilizedsAnd carrying out non-rigid registration on the initial three-dimensional model template to the point cloud data of the first frame. Similarly, the corresponding relation between the initial three-dimensional model template and the point cloud data of the first frame is obtained through nearest neighbor searching, and then the non-rigid deformation field M is solved through Gaussian and Newton iteration. And then applying the M to the initial three-dimensional model template to obtain a new initial three-dimensional model template. Through rigid transformation and non-rigid transformation, the initial three-dimensional model template is optimized, and the accuracy of the initial three-dimensional model template is improved.
It should be noted that the TSDF of the depth map of the first frame is calculated0And obtaining a three-dimensional splicing model F of the first frame through Marching-cube0。
Step 402: and acquiring a designated part in the initial three-dimensional model template.
Step 403: and searching point cloud data in a designated area corresponding to the designated part in the point cloud data of the current frame according to the designated part in the initial three-dimensional model template.
Step 404: different deformation weights are set for the point cloud data in the designated area and the point cloud data in the non-designated area.
Step 405: and fitting the three-dimensional splicing model of the previous frame to the point cloud data of the current frame according to the deformation weight.
Step 406: and generating a three-dimensional splicing model of the current frame according to the fitted data.
Step 407: and judging whether the point cloud data of the next frame exists, if so, returning to the step 402, otherwise, executing the step 408.
Step 408: and taking the three-dimensional splicing model of the current frame as a target three-dimensional model of a target object.
It should be understood that this embodiment is a system example corresponding to the first embodiment, and may be implemented in cooperation with the first embodiment. The related technical details mentioned in the first embodiment are still valid in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related-art details mentioned in the present embodiment can also be applied to the first embodiment.
It should be noted that each module referred to in this embodiment is a logical module, and in practical applications, one logical unit may be one physical unit, may be a part of one physical unit, and may be implemented by a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, elements that are not so closely related to solving the technical problems proposed by the present invention are not introduced in the present embodiment, but this does not indicate that other elements are not present in the present embodiment.
A fifth embodiment of the present invention relates to an electronic device, which includes: at least one processor 501; and a memory 502 communicatively coupled to the at least one processor 501; wherein the memory stores instructions executable by the at least one processor 501, the instructions being executable by the at least one processor to enable the at least one processor 501 to perform the method of three-dimensional reconstruction described above.
The memory 502 and the processor 501 are connected by a bus, which may include any number of interconnected buses and bridges that link one or more of the various circuits of the processor 501 and the memory 502. The bus may also link various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor is transmitted over a wireless medium via an antenna, which further receives the data and transmits the data to the processor.
A sixth embodiment of the present invention relates to a computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the method for three-dimensional reconstruction described above.
The processor is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And the memory may be used to store data used by the processor in performing operations.
Those skilled in the art can understand that all or part of the steps in the method of the foregoing embodiments may be implemented by a program to instruct related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples for carrying out the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.

Claims (10)

1. A method of three-dimensional reconstruction, comprising:
dividing the point cloud data of the target object in the current frame into point cloud data in a designated area and point cloud data in a non-designated area;
setting different deformation weights for the point cloud data in the designated area and the point cloud data in the non-designated area;
fitting the three-dimensional splicing model of the previous frame to the point cloud data of the current frame according to the deformation weight;
generating a three-dimensional splicing model of the current frame according to the fitted data;
judging whether the point cloud data of the next frame exists, if so, returning to execute the step of dividing the point cloud data of the target object in the current frame into a designated area and a non-designated area, and if not, taking the three-dimensional splicing model of the current frame as the target three-dimensional model of the target object.
2. The method of claim 1, wherein the dividing the point cloud data of the target object in the current frame into point cloud data in a designated area and point cloud data in a non-designated area comprises:
acquiring a designated part in an initial three-dimensional model template, wherein the initial three-dimensional model template is generated according to image data of a first frame of the target object and a preset three-dimensional template generation model, and the image data comprises a color image and corresponding depth information of the target object;
and searching point cloud data in a designated area corresponding to the designated part in the point cloud data of the current frame according to the designated part in the initial three-dimensional model template.
3. The method of claim 2, wherein the obtaining of the specified location in the initial three-dimensional model template comprises:
acquiring a first frame color image of the target object;
performing example segmentation on the first frame color image, and determining a specified part of the target object in the first frame color image;
and projecting the initial three-dimensional model template to the segmented first frame color image according to the designated part of the target object in the first frame color image, and determining the designated part in the initial three-dimensional model template.
4. The method of three-dimensional reconstruction according to any one of claims 1 to 3, wherein before the dividing target object's point cloud data in the current frame is point cloud data in a designated area and point cloud data in a non-designated area, the method further comprises:
and acquiring an image of the rotating target object by adopting a single color-depth camera to obtain at least two frames of image data of the target object.
5. The method of claim 2 or 3, wherein the searching point cloud data in a designated area corresponding to a designated location in the point cloud data according to the designated location in the initial three-dimensional model template comprises:
searching points with a distance smaller than a preset threshold value from the designated position in the initial three-dimensional model template as point cloud data in the designated area;
and taking other point cloud data outside the specified area as the point cloud data in the non-specified area.
6. The method of any one of claims 1 to 3, wherein the fitting the three-dimensional stitching model of the previous frame to the point cloud data of the current frame according to the deformation weight comprises:
carrying out rigid transformation on the three-dimensional splicing model of the previous frame to generate a new three-dimensional splicing model;
and determining a non-rigid transformation deformation field according to the deformation weight, and deforming the new three-dimensional splicing model according to the non-rigid deformation field to obtain fitted data.
7. The method of three-dimensional reconstruction according to claim 2 or 3, wherein prior to said obtaining a specified location in said initial three-dimensional model template, said method further comprises:
and carrying out rigid transformation and non-rigid transformation on the initial three-dimensional model template so as to optimize the initial three-dimensional model template.
8. The method of three-dimensional reconstruction according to claim 2 or 3, wherein the target object is a human body, and the designated portion is an arm.
9. An electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of three-dimensional reconstruction according to any one of claims 1-8.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of three-dimensional reconstruction of one of claims 1 to 8.
CN202110426718.7A 2021-04-20 2021-04-20 Three-dimensional reconstruction method, electronic device and storage medium Active CN112991524B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110426718.7A CN112991524B (en) 2021-04-20 2021-04-20 Three-dimensional reconstruction method, electronic device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110426718.7A CN112991524B (en) 2021-04-20 2021-04-20 Three-dimensional reconstruction method, electronic device and storage medium

Publications (2)

Publication Number Publication Date
CN112991524A true CN112991524A (en) 2021-06-18
CN112991524B CN112991524B (en) 2022-03-25

Family

ID=76341285

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110426718.7A Active CN112991524B (en) 2021-04-20 2021-04-20 Three-dimensional reconstruction method, electronic device and storage medium

Country Status (1)

Country Link
CN (1) CN112991524B (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150036918A1 (en) * 2011-07-11 2015-02-05 Kabushiki Kaisha Toshiba Image processing method and system
CN105184861A (en) * 2015-10-20 2015-12-23 南京大学 Automatic human three-dimensional (3D) reconstruction system
CN105913489A (en) * 2016-04-19 2016-08-31 东北大学 Indoor three-dimensional scene reconstruction method employing plane characteristics
CN108182704A (en) * 2016-12-08 2018-06-19 广州映博智能科技有限公司 Localization method based on Shape context feature
CN108711185A (en) * 2018-05-15 2018-10-26 清华大学 Joint rigid moves and the three-dimensional rebuilding method and device of non-rigid shape deformations
CN108805979A (en) * 2018-02-05 2018-11-13 清华-伯克利深圳学院筹备办公室 A kind of dynamic model three-dimensional rebuilding method, device, equipment and storage medium
CN108876894A (en) * 2018-02-01 2018-11-23 北京旷视科技有限公司 Three-dimensional face model and three-dimensional headform's generation method and generating means
CN110717494A (en) * 2019-07-16 2020-01-21 北京航空航天大学青岛研究院 Android mobile terminal indoor scene three-dimensional reconstruction and semantic segmentation method
CN112132971A (en) * 2020-09-08 2020-12-25 合肥的卢深视科技有限公司 Three-dimensional human body modeling method, device, electronic equipment and storage medium

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150036918A1 (en) * 2011-07-11 2015-02-05 Kabushiki Kaisha Toshiba Image processing method and system
CN105184861A (en) * 2015-10-20 2015-12-23 南京大学 Automatic human three-dimensional (3D) reconstruction system
CN105913489A (en) * 2016-04-19 2016-08-31 东北大学 Indoor three-dimensional scene reconstruction method employing plane characteristics
CN108182704A (en) * 2016-12-08 2018-06-19 广州映博智能科技有限公司 Localization method based on Shape context feature
CN108876894A (en) * 2018-02-01 2018-11-23 北京旷视科技有限公司 Three-dimensional face model and three-dimensional headform's generation method and generating means
CN108805979A (en) * 2018-02-05 2018-11-13 清华-伯克利深圳学院筹备办公室 A kind of dynamic model three-dimensional rebuilding method, device, equipment and storage medium
CN108711185A (en) * 2018-05-15 2018-10-26 清华大学 Joint rigid moves and the three-dimensional rebuilding method and device of non-rigid shape deformations
CN110717494A (en) * 2019-07-16 2020-01-21 北京航空航天大学青岛研究院 Android mobile terminal indoor scene three-dimensional reconstruction and semantic segmentation method
CN112132971A (en) * 2020-09-08 2020-12-25 合肥的卢深视科技有限公司 Three-dimensional human body modeling method, device, electronic equipment and storage medium

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
SEONGHWA CHOI.ETC: ""Point Cloud Deformation for Single Image 3d Reconstruction"", 《IEEE》 *
周文: ""基于RGB-D相机的三维人体重建方法研究"", 《中国优秀博硕士学位论文全文数据库(硕士) 信息科技辑》 *
陈光: ""基于多态统计的三维人体重建及其在虚拟试衣中的应用"", 《中国优秀博硕士学位论文全文数据库(博士) 信息科技辑》 *

Also Published As

Publication number Publication date
CN112991524B (en) 2022-03-25

Similar Documents

Publication Publication Date Title
US11954870B2 (en) Dynamic scene three-dimensional reconstruction method, apparatus and system, server, and medium
CN108648240B (en) Non-overlapping view field camera attitude calibration method based on point cloud feature map registration
CN106780619B (en) Human body size measuring method based on Kinect depth camera
CN104484648B (en) Robot variable visual angle obstacle detection method based on outline identification
CN109993793B (en) Visual positioning method and device
CN111340797A (en) Laser radar and binocular camera data fusion detection method and system
KR102450931B1 (en) Image registration method and associated model training method, apparatus, apparatus
CN112132908B (en) Camera external parameter calibration method and device based on intelligent detection technology
CN110060329B (en) Mobile terminal human body model reconstruction method based on color depth video stream data
CN112949478A (en) Target detection method based on holder camera
CN110717936B (en) Image stitching method based on camera attitude estimation
CN111612731B (en) Measuring method, device, system and medium based on binocular microscopic vision
CN111998862B (en) BNN-based dense binocular SLAM method
CN111060006A (en) Viewpoint planning method based on three-dimensional model
CN111383204A (en) Video image fusion method, fusion device, panoramic monitoring system and storage medium
CN114219855A (en) Point cloud normal vector estimation method and device, computer equipment and storage medium
CN115222884A (en) Space object analysis and modeling optimization method based on artificial intelligence
CN102202159A (en) Digital splicing method for unmanned aerial photographic photos
CN114612352A (en) Multi-focus image fusion method, storage medium and computer
CN112991524B (en) Three-dimensional reconstruction method, electronic device and storage medium
CN113112532B (en) Real-time registration method for multi-TOF camera system
CN114399547B (en) Monocular SLAM robust initialization method based on multiframe
CN114332187B (en) Monocular target ranging method and device
Fotia et al. Applying 3D and photogrammetric scanning systems to the case of cultural heritage
Gasz et al. The Registration of Digital Images for the Truss Towers Diagnostics

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20230324

Address after: 230091 room 611-217, R & D center building, China (Hefei) international intelligent voice Industrial Park, 3333 Xiyou Road, high tech Zone, Hefei, Anhui Province

Patentee after: Hefei lushenshi Technology Co.,Ltd.

Address before: 100083 room 3032, North B, bungalow, building 2, A5 Xueyuan Road, Haidian District, Beijing

Patentee before: BEIJING DILUSENSE TECHNOLOGY CO.,LTD.

Patentee before: Hefei lushenshi Technology Co.,Ltd.

TR01 Transfer of patent right